Repository navigation
SRM could be faster and more memory efficient #416
Description
Current implementation of SRM could be enhanced when the number of subjects and sessions are big.
Our experiments show that our implementation is more memory-efficient and faster while having similar accuracy (measured with test R2 reconstruction error) than ProbSRM .
I have a PR ready for this. If you think it is interesting please let me know.
Best,
Hugo RICHARD
Metadata
Metadata
Assignees
Labels
No labels
Activity
mihaic commentedon Apr 4, 2019
Absolutely! Please submit a PR.
qihongl commentedon May 25, 2019
Thanks @hugorichard ! This is a really exciting addition!
Do you have some simulation results showing it is qualitatively similar to the old SRM and faster when the data is big enough? I would love to take a look. It seems that your example only shows the results on fastSRM. Besides qualitatively verifying the code, I think these results will be useful to guide users' decision.
I'm not sure I'm the best person to review the technical stuff, such as the detail of EM updates. Can we invite some reviewers who actually work on SRM?
hugorichard commentedon May 27, 2019
Thanks @qihongl for your feedback !
Yes you are right the example does not show that FastSRM is a better choice than the old SRM. It just shows that the implementation is running and how to use the API. I tried to use the same example as in the old SRM but I cannot get my hands on the masker that was used to mask the movie watching data which prevents me from using an interesting atlas to perform the reduction. The mask given in brainiak/examples/funcalign/download-data.sh ("data/MNI152_T1_3mm_brain_mask.nii") does not match the shape of the data given in the same file ("data/movie_data.mat").
Let me share with you some results on a real dataset (Forrest data using the full brain, 19 subjects and 7 sessions). The data may be too big for a small example but it shows why FastSRM should be used instead of the old SRM.
memory_usage_forrest_all.pdf
fit_time_forrest_all.pdf
perfs_forrest_basc_n_iter_10.pdf
If you think that it would be better to have the code reproducing this experiment included in brainiak I will provide it. Forrest data are freely available at https://openneuro.org/datasets/ds000113/versions/1.0.0/download but they are quite large.
The experiment is described briefly below:
All sessions but one are used to compute spatial maps for every subjects. Then spatial maps and data from the left-out session of all subjects but one are used to compute the shared response in the left-out session. At last, the shared response during the left-out session and the spatial maps of the test subject are used to predict the data of the left-out subject during the left-out session. The performance of the model is measured by comparing the prediction and true data using the coefficient of determination (mean r2 score).
qihongl commentedon May 27, 2019
Wow, in terms of cross-validated r^2, it seems that fast SRM performed much better (fast SRM >> SRM ~= det SRM). It seems that somehow fast SRM can "find" more useful latent dimensions? Just out of curiosity, do you have some intuition about why is this the case? Do you have a paper about your method? I think your result is impressive enough as a paper :) though I think a more common practice is to hold out either a session or a subject entirely?
Providing a runnable example using a real/large dataset does sound complicated. @snastase might have some
.niidata suited for testing your code? He has been working on a command line interface of ISC: https://github.com/snastase/isc-tutorial , which seems to handle data of a similar form - a list of files that eventually get converted to a list of 2d arrays before running the analysis. Alternatively, using a toy simulated data is probably fine. I just thought explicitly showing which step in fast SRM corresponds to which step in the old SRM can be useful for many users who are familiar with the old SRM.@hugorichard @mihaic @snastase And is it possible to "sync" fast SRM and ISC s.t. they take input files formatted in the same way? Maybe we will need a brief brainiak meeting to talk about what format makes the most sense. Though I think this might save time in the future.
@hejiaz @snastase do you have time to review this PR? I'm not sure if my expertise is enough to help with this.
hugorichard commentedon May 28, 2019
Thanks for your answer!
My intuition is that the projection onto the atlas reduces the noise which allows the algorithm to find a better solution.
I am writing a paper with Jonathan Pillow about this.
Thanks for your comments about the held-out session and subject. I use the "co-smoothing" set up essentially to gain time. I do not think it would change the result to hold out one subject entirely because the shared response do not depends heavily on the presence of one subject when a lot of subjects are used for training. But i will check this and let you know.
Thanks a lot for this link to the isc tutorial. I think the data made available on in the section "real fMRI data" is exactly what I am looking for to create a simple and meaningful example. I'll update my example based on this.
The API of ISC and FastSRM are indeed almost the same. The only difference is that ISC takes as input a list of matrices n_timeframes x n_voxels and FastSRM a list of paths pointing to matrices n_timeframes x n_voxels. I just want to point out that having the option to pass a list of path is necessary for FastSRM because when we use big datasets it might not be possible to hold them into RAM.
qihongl commentedon May 28, 2019
@hugorichard I see! Thanks for the comments! They are highly informative! And I'm looking forward to reading your paper!
hugorichard commentedon Jun 3, 2019
@qihongl I just updated my example with the data you provided. We only get a small increase in R2 compared to the old SRM but still a good speed-up and lower memory usage.
qihongl commentedon Jun 3, 2019
@hugorichard Cool I think that's an awesome tutorial. I can see some old SRM users would find that very intuitive and eager to try FastSRM. At least I am :)
@mihaic I think this PR is great. Though I don't think I'm the best person to review the code since we have so many people working/worked on SRM development. Can we get some help? @hejiaz @TuKo @cameronphchen Or someone who's currently working with Peter Ramadge?
hugorichard commentedon Jul 10, 2019
@qihongl @mihaic Any news ? If there is anything I can do don't hesitate to contact me.
mihaic commentedon Jul 10, 2019
Sorry for the silence, @hugorichard. We are having difficulty getting help. We'll contact @hejiaz and @TuKo again.