UnfoldDecode - decoding several targets using the same encoding model?

[riikkamononen]

Hi again,

In the UnfoldDecode fit-function one target can be decoded at a time. However, in our case we have up to 60 stimuli (kind of experimental conditions), all encoded with the same Unfold-model and with the same basisfunction, and we want to decode each one of them, i.e. have up to 60 targets.

So would it make sense to decode several targets in one UnfoldDecode.fit call, i.e. use the same Unfold-fit to decode all the intended targets instead of fitting a new encoding model for each target? So, after fitting the encoding model to the train and test sets in each fold all intended targets would be decoded for that fold using the same fitted models, before moving to the next fold. In that case the target argument should be changed to some kind of dict or array of pairs, and the returned UnfoldDecodingModel should be something like an array of vectors of DecodingFits..? Or is it indeed better to fit the encoding model separately for each target?

Sorry about the messy explanation, I hope you get what I mean...

And btw, is the target classified against all other categories (one vs. all) or in a pairwise manner (one vs. one)?

Also, in UnfoldDecode.fit the target variable is defined as Pair, which causes some problems when calling the function from Python. Is there a reason why the target should be defined as Pair instead of Tuple? The model seems to run either way.

Best, Riikka

[behinger]

Hi!
I unfortunately cannot spend much time right now on UnfoldDecode, as I have my tenurecase coming up + interviews for PhD-Students (who might just work on UnfoldDecode ;-)) coming up. It is very cool suggestions you are doing!

1. multi-target

This might be helpful in your situation. Do you have 60 event types, or a condition with 60 levels? I'm not super sure how common it is to have multiple targets. I'm somewhat hesitant as this will add a bit more complexity to a codebase that is not well tested yet. But maybe the changes necessary are not that large

2. one vs. all / one vs. one: I think this totally depends on your MLJModel (see comment below as well). LDA() by default is multiclass; I'd have to check the documentation in detail

3. pairs: I like pairs because it is clear that it can only be two values :). Feel free to open a PR with something like

Unfold.fit(UnfoldDecodingModel, design, tbl, dat::AbstractMatrix, model::MLJ.Model, target::Any;kwargs...) = fit(UnfoldDecodingModel,design,tbl,dat,model,Pair(target);kwargs...)

(didnt test it)

---

coincidentially I think I found a severe training-leakage bug:

        # XXX remove the boundary data to ensure no leakage
        uf_train = Unfold.fit(
            UnfoldLinearModelContinuousTime,
            design,
            tbl,
            dat;
            eventcolumn = eventcolumn,
            unfold_fit_options...,
        )

tbl should be tbltrain I think.

---

I also think this is not correct:

        y_train = coerce(tbltrain[ix_train, target[2]], OrderedFactor)

it should better be MultiClass, not OrderedFactor. For 2-classes (my test cases) it doesnt matter, but in your case it might - or LDA ignores it. But other models might not.

[riikkamononen]

Hi!

1. Multi-target

We have 60 event columns in our events matrix, so I guess that’s 60 event types. And yeah, it might be that this kind of functionality is not very useful to most users. So I’ll see if I can implement the functionality myself locally and maybe that’s enough. I’m not an expert on Julia, so the result might not be the most elegant ever…

2. One vs. all / one vs. one: Ok, I’ll check the LDA documentation!

3. Pairs: I’ll test that, thanks! Gotta check how the PRs work too

I’ll also check the bugs.

Thanks for your help so far, and good luck with the tenurecase! Looking forward to the possible new features by the (hopefully several) new PhD-students working on UnfoldDecode :-D