References & Citations
Statistics > Machine Learning
Title: Assessing and tuning brain decoders: cross-validation, caveats, and guidelines
(Submitted on 16 Jun 2016 (v1), last revised 7 Nov 2016 (this version, v2))
Abstract: Decoding, ie prediction from brain images or signals, calls for empirical evaluation of its predictive power. Such evaluation is achieved via cross-validation, a method also used to tune decoders' hyper-parameters. This paper is a review on cross-validation procedures for decoding in neuroimaging. It includes a didactic overview of the relevant theoretical considerations. Practical aspects are highlighted with an extensive empirical study of the common decoders in within-and across-subject predictions, on multiple datasets --anatomical and functional MRI and MEG-- and simulations. Theory and experiments outline that the popular " leave-one-out " strategy leads to unstable and biased estimates, and a repeated random splits method should be preferred. Experiments outline the large error bars of cross-validation in neuroimaging settings: typical confidence intervals of 10%. Nested cross-validation can tune decoders' parameters while avoiding circularity bias. However we find that it can be more favorable to use sane defaults, in particular for non-sparse decoders.
Submission history
From: Gael Varoquaux [view email][v1] Thu, 16 Jun 2016 14:29:28 GMT (785kb,D)
[v2] Mon, 7 Nov 2016 15:40:46 GMT (692kb,D)
Link back to: arXiv, form interface, contact.