UnfoldDecode for discriminability analysis on single participant level?

[riikkamononen]

Hi,

We have used Unfold.jl to recover overlapping MEG responses to simultaneously presented visual
stimuli, and it has worked great. Now we would like to compute discriminabilities between the
estimated responses. As we have been very happy with Unfold so far, we’re hoping we could use
UnfoldDecode for the decoding of our data. However, we have understood that it is not currently
possible to run the decoding analysis on a single participant level.

So our question is, is it possible to use UnfoldDecode for decoding and statistical testing of the
estimated responses on individual level, or could that kind of functionality be implemented? I guess
part of the question is how the computational folds are defined and whether one can freely
choose the folding, right?

I’ve also got a side-question: When estimating the visual evoked responses with Unfold.jl we have
used a model with no intercept with good results. I remember that in the Unfold-workshop it was
told that one practically always wants to include the intercept term in the model. In more detail our
stimulus consists of several small regions that are shown in parallel with uncorrelated timing
sequences. The model we used was of the form y ~ region1 + region2 + … + regionN. Does it make
sense to you to have a model without intercept in this case, or should we maybe add the intercept
term?

And thanks for the great toolbox!
Best, Riikka

[behinger]

Decoding

As I understand, you want to figure out how well you can decode your results.

Stimulation orthogonal

If your stimulation is orthogonal (i.e. no correlations between predictors & no biased overlap), you wouldn't need any specific overlap correction. I.e. you'd want to use standard epoching & some LDA/SVM on each time-point, or the whole epoch. Is that correct? If yes, there is no tutorial yet, but it should be very easy to write for me. In terms of decoders, cross-validation scheme etc. available, I would entirely rely on MLJ.jl, the machine learning backend of julia.

Overlap-corrected decoding

But given you are using unfold already, likely you have biased overlap. In that case, you should be able to use our overlap-corrected MLJ implementation. It is not super polished right now, but I'm happy to to provide support there as well

Always include the intercept?

Indeed, your case sounds like one, where it doesnt make sense to include one. I think it really starts making sense when interactions come into play.

[riikkamononen]

Thanks for the quick reply!

Yes, I think the overlap-corrected MLJ-implementation is the way to go. So does that mean we can basically follow the quickstart example on the UnfoldDecode github page? (except maybe changing LDA to some MLJ-supported model and ofc modifying other details too)

The quickstart example:

LDA = @load LDA pkg=MultivariateStats
des = Dict("fixation" => (@formula(0~1+condition+continuous),firbasis((-0.1,1.),100)));
uf_lda = fit(UnfoldDecodingModel,des,evt,dat,LDA(),"fixation"=>:condition)

Defining custom folds

In addition, we would like to use specific folds to create the train and test sets. In UnfoldDecode.jl/src/fit.jl, in the function Unfold.fit, the folding seems to be done using MLJBase.train_test_pairs(). So I guess defining custom folds should be possible by doing some modifications there, if needed, right?

[behinger]

yes - I hope it works because I noticed the doc's seem to have failed silently.

To your question: yes, ideally we'd have a simple keyword-argument to simply add either latencies or an eventcolumn+type to define what to use as potential custom folds. Then we could do a (not tested code, just from top of my head)

 lat_to_splits(folds) = [f:f1 for f,f1 in zip(folds[1:end-1],folds[2:end])]
if isa(folds,Array)
  # folds =  [100,250,300,3500]
  train_test = lat_to_splits(folds)
elseif isa(folds, Pair)
  # folds = "eventcolumn" => "eventvalue"
 train_test = subset(tbl, first(folds)  => .==(last(folds)).latency |>lat_to_splits
elseif isa(folds, Integer)
   folds = 5
    train_test = MLJBase.train_test_pairs(CV(; nfolds = nfolds), 1:size(tbl, 1))
else
error("folds needs to be an array of latencies, a pair (eventcolumn=>eventvalue) or an integer")
end

in the code I also marked in a comment that boundaries between folds are currently not automatically checked / excluded (e.g. you could have leakage of one overlapping event to the other fold which I think is not ideal, but maybe also doesnt matter? I didnt check simulations for this). This should be fixed at some point, but I simply didnt come around to do it. I should document it properly, but also havent done so...

If you would want to implement that, I'd be very open for PRs.

[riikkamononen]

Great, thanks! I'll take a look into customizing the folds, and possibly could also try to implement that properly. At least as soon as I get the UnfoldDecode fit-function to work when called from Python...

And about the leakage of overlapping events, in our case we have a small break between the intended folds, which should at least reduce the possible problem. So in this case probably doesn't matter, but in general I don't know.