Repository navigation
fcma: random argument in preprocessing.prepare_fcma_data is not clear #363
Description
I ran the following code:
permuted_raw_data, _, permuted_labels = prepare_fcma_data(images, epoch_list, mask, random=2) print(permuted_labels)
which gives me this output:
[0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1]
However, I get the exact same output, when I run the same code again or when I set random=0, random=1, random=2, or when I leave out the random argument altoghether. From what I understand, this should not happen, as the lables should be shuffled? Do I get the documentation wrong or might there be a bug?
Also, I the end, my fcma results give me the same classification performance, regardless of what I set the random argument to.
Activity
yidawang commentedon Apr 26, 2018
Setting the
randomargument to 1 or 2 shuffles the data not the labels. http://brainiak.org/docs/brainiak.fcma.html#brainiak.fcma.preprocessing.RandomTypeWhat is your dataset? What is the classification results you got from
random=0?mihaic commentedon Apr 27, 2018
@yidawang, what do you think about changing the docstring to say "images" instead of (or in addition to) "data", to indicate as clearly as possible what is shuffled?
yidawang commentedon Apr 27, 2018
@mihaic sounds good, I can do it later today.
fcma: Clarify that image data is shuffled (#365)
mihaic commentedon Apr 30, 2018
@moopet17, hopefully PR #365 makes it clear that the images are what gets shuffled.
If you are still worried about the classification performance not changing, please provided some data to reproduce what you are seeing.
moopet17 commentedon May 1, 2018
@yidawang: Thank you.
@mihaic: Yes, that makes it clear. It would be great if the documentation would also be clarified here: http://brainiak.org/docs/brainiak.fcma.html?highlight=fcma%20preprocessing%20prepare_fcma_data#brainiak.fcma.preprocessing.prepare_fcma_data This is the first place to look if one tries to understand what a BrainIAK function does.
mihaic commentedon May 1, 2018
At this moment, we only have HTML documentation for BrainIAK releases. I have just created issue #367 to add a version of the HTML docs corresponding to the master branch of BrainIAK.
How about the classification performance? Is it unexpected? If not, please close this issue.
moopet17 commentedon May 1, 2018
Ok the real issue seems to be that TO ME, the documentation suggested that
random=1equalsrandom=RandomType.REPRODUCIBLEand so on. Turns out that is not the case. Setting the random argument to an integer does not do anything and always defaults toRandomType.NORANDOM. So it seems to me that this beahavior should be fixed or the documentation should be clearer on that.mihaic commentedon May 7, 2018
I agree, the documentation can be interpreted as allowing
random=1. We can improve it.Otherwise, can you please confirm that the classification performance changes as expected when using
UNREPRODUCIBLE?[-]`random` argument in `fcma.preprocessing.prepare_fcma_data` function is broken[/-][+]fcma: `random` argument in `preprocessing.prepare_fcma_data` is not clear[/+]