Requirements
These tutorials require
- A local installation or Python
- Logging in to Flywheel using the CLI
Getting data from Flywheel
Flywheel stores data and its accompanying metadata in a proprietary system in the cloud. This data can be downloaded manually using the web interface or through the SDK. The process of getting data from Flywheel involves two steps
- Finding the data’s container
- Finding the file object and downloading it or a part of it
Accessing data in Python
Suppose we ran FMRIPREP and would like to download some of its outputs. In this example FMRIPREP was run and attached to a session container (it could also be attached to a subject or project). The first step is to find the session container:
>>> import flywheel
>>> fw = flywheel.Client()
>>> session_id = "5cd31cbc08a9960041650843"
>>> analysis = fw.get_container_analyses(
... session_id, filter='label="fmriprep 06/03/2019 10:58:53"')[0]
>>> print(analysis.label)
fmriprep 06/03/2019 10:58:53
In this case we know that there is only one analysis with this name so we access it from the list that is returned by indexing at [0]. The analysis variable represents the analysis container.
Listing file attachments
Now we can see which files are attached to this analysis data container:
>>> print("\n".join([file_obj.name for file_obj in analysis.files]))
sub-X_5cf535cb36da2300443b2fb9.html.zip
fmriprep_work_sub-X_5cf535cb36da2300443b2fb9.zip
fmriprep_sub-X_5cf535cb36da2300443b2fb9.zip
These are the names of each file object attached to the container. These names should be meaningful to us.
Downloading file attachments
The most straightforward approach to getting data from Flywheel is to directly download the zip file. Then you can unzip it and access its contents on your local machine. To do that,
>>> analysis.download_file("fmriprep_sub-X_5cf535cb36da2300443b2fb9.zip",
... "fmriprep_sub-X_5cf535cb36da2300443b2fb9.zip")
and the file fmriprep_sub-X_5cf535cb36da2300443b2fb9.zip will exist in your current working directory.
Extracting files from zip archive attachments
It is very common for these attachments to be zip files. Often we don’t need all the data in a zip file. Instead we just want to get one or more files from the zip directly. First, we can get information about the contents of the zip
>>> zip_info = analysis.get_file_zip_info(
... 'fmriprep_sub-X_5cf535cb36da2300443b2fb9.zip')
>>> print('\n'.join([member.path for member in zip_info.members]))
5cf535cb36da2300443b2fb9/
5cf535cb36da2300443b2fb9/fmriprep/
5cf535cb36da2300443b2fb9/fmriprep/logs/
5cf535cb36da2300443b2fb9/fmriprep/logs/CITATION.tex
5cf535cb36da2300443b2fb9/fmriprep/logs/CITATION.html
5cf535cb36da2300443b2fb9/fmriprep/logs/CITATION.md
5cf535cb36da2300443b2fb9/fmriprep/sub-X/
5cf535cb36da2300443b2fb9/fmriprep/sub-X/figures/
5cf535cb36da2300443b2fb9/fmriprep/sub-X/figures/sub-X_task-mixedgamblestask_run-02_carpetplot.svg
5cf535cb36da2300443b2fb9/fmriprep/sub-X/figures/sub-X_task-mixedgamblestask_run-01_carpetplot.svg
5cf535cb36da2300443b2fb9/fmriprep/sub-X/figures/sub-X_task-mixedgamblestask_run-01_rois.svg
5cf535cb36da2300443b2fb9/fmriprep/sub-X/figures/sub-X_task-mixedgamblestask_run-01_flirtbbr.svg
5cf535cb36da2300443b2fb9/fmriprep/sub-X/figures/sub-X_t1_2_mni.svg
5cf535cb36da2300443b2fb9/fmriprep/sub-X/figures/sub-X_task-mixedgamblestask_run-02_flirtbbr.svg
5cf535cb36da2300443b2fb9/fmriprep/sub-X/figures/sub-X_task-mixedgamblestask_run-02_rois.svg
5cf535cb36da2300443b2fb9/fmriprep/sub-X/figures/sub-X_seg_brainmask.svg
5cf535cb36da2300443b2fb9/fmriprep/sub-X/func/
5cf535cb36da2300443b2fb9/fmriprep/sub-X/func/sub-X_task-mixedgamblestask_run-01_space-MNI152NLin2009cAsym_boldref.nii.gz
5cf535cb36da2300443b2fb9/fmriprep/sub-X/func/sub-X_task-mixedgamblestask_run-02_space-MNI152NLin2009cAsym_desc-brain_mask.nii.gz
5cf535cb36da2300443b2fb9/fmriprep/sub-X/func/sub-X_task-mixedgamblestask_run-01_desc-confounds_regressors.tsv
5cf535cb36da2300443b2fb9/fmriprep/sub-X/func/sub-X_task-mixedgamblestask_run-02_space-MNI152NLin2009cAsym_desc-preproc_bold.nii.gz
5cf535cb36da2300443b2fb9/fmriprep/sub-X/func/sub-X_task-mixedgamblestask_run-01_space-MNI152NLin2009cAsym_desc-preproc_bold.nii.gz
5cf535cb36da2300443b2fb9/fmriprep/sub-X/func/sub-X_task-mixedgamblestask_run-01_space-MNI152NLin2009cAsym_desc-brain_mask.nii.gz
5cf535cb36da2300443b2fb9/fmriprep/sub-X/func/sub-X_task-mixedgamblestask_run-02_space-MNI152NLin2009cAsym_boldref.nii.gz
5cf535cb36da2300443b2fb9/fmriprep/sub-X/func/sub-X_task-mixedgamblestask_run-02_desc-confounds_regressors.tsv
5cf535cb36da2300443b2fb9/fmriprep/sub-X/anat/
5cf535cb36da2300443b2fb9/fmriprep/sub-X/anat/sub-X_from-MNI152NLin2009cAsym_to-T1w_mode-image_xfm.h5
5cf535cb36da2300443b2fb9/fmriprep/sub-X/anat/sub-X_space-MNI152NLin2009cAsym_desc-preproc_T1w.nii.gz
5cf535cb36da2300443b2fb9/fmriprep/sub-X/anat/sub-X_desc-brain_mask.nii.gz
5cf535cb36da2300443b2fb9/fmriprep/sub-X/anat/sub-X_desc-preproc_T1w.nii.gz
5cf535cb36da2300443b2fb9/fmriprep/sub-X/anat/sub-X_label-GM_probseg.nii.gz
5cf535cb36da2300443b2fb9/fmriprep/sub-X/anat/sub-X_space-MNI152NLin2009cAsym_dseg.nii.gz
5cf535cb36da2300443b2fb9/fmriprep/sub-X/anat/sub-X_space-MNI152NLin2009cAsym_label-WM_probseg.nii.gz
5cf535cb36da2300443b2fb9/fmriprep/sub-X/anat/sub-X_label-CSF_probseg.nii.gz
5cf535cb36da2300443b2fb9/fmriprep/sub-X/anat/sub-X_space-MNI152NLin2009cAsym_desc-brain_mask.nii.gz
5cf535cb36da2300443b2fb9/fmriprep/sub-X/anat/sub-X_label-WM_probseg.nii.gz
5cf535cb36da2300443b2fb9/fmriprep/sub-X/anat/sub-X_from-orig_to-T1w_mode-image_xfm.txt
5cf535cb36da2300443b2fb9/fmriprep/sub-X/anat/sub-X_dseg.nii.gz
5cf535cb36da2300443b2fb9/fmriprep/sub-X/anat/sub-X_from-T1w_to-MNI152NLin2009cAsym_mode-image_xfm.h5
5cf535cb36da2300443b2fb9/fmriprep/sub-X/anat/sub-X_space-MNI152NLin2009cAsym_label-GM_probseg.nii.gz
5cf535cb36da2300443b2fb9/fmriprep/sub-X/anat/sub-X_space-MNI152NLin2009cAsym_label-CSF_probseg.nii.gz
5cf535cb36da2300443b2fb9/fmriprep/dataset_description.json
5cf535cb36da2300443b2fb9/fmriprep/sub-X.html
That’s all the fmriprep output! Suppose we only want to get the confound regressors out of this zip. we can easily do this with a few lines:
>>> from os import listdir, path as op
>>> confounds_file = [member for member in zip_info.members if
... member.path.endswith('confounds_regressors.tsv')][0]
>>> confounds_file_name = op.split(confounds_file.path)[1]
>>> print(confounds_file_name)
>>> analysis.download_file_zip_member('fmriprep_sub-X_5cf535cb36da2300443b2fb9.zip',
... confounds_file.path,
... confounds_file_name)
>>> listdir(".") # Verify that it was downloaded
[ 'sub-X_task-mixedgamblestask_run-01_desc-confounds_regressors.tsv' ]
Since BIDS requires that subject and session are in each filename, we can iterate over all the analyses and download these files to the current working directory. Alternatively, we can read the data directly into memory:
>>> mem_tsv = analysis.read_file_zip_member('fmriprep_sub-X_5cf535cb36da2300443b2fb9.zip',
... confounds_file.path)
>>> import pandas as pd
>>> from io import BytesIO
>>> confounds_df = pd.read_csv(BytesIO(mem_tsv), delimiter='\t')
>>> confounds_df.head()
csf white_matter global_signal std_dvars dvars framewise_displacement ... trans_x trans_y trans_z rot_x rot_y rot_z
0 583.33704 0.0 485.55243 NaN NaN NaN ... 0.000000 0.000000 0.000000 -0.000170 -0.000000 0.000000
1 579.08600 0.0 483.35240 1.179028 18.223396 0.072119 ... 0.000000 0.020775 0.046312 -0.000271 -0.000000 0.000000
2 579.21936 0.0 481.93520 1.102987 17.048092 0.034897 ... 0.000000 0.000000 0.045737 0.000000 -0.000000 0.000000
3 574.88620 0.0 481.27450 1.024027 15.827655 0.033394 ... 0.010164 -0.010357 0.042451 0.000000 -0.000000 0.000192
4 575.22850 0.0 481.20953 0.997114 15.411689 0.032764 ... 0.016909 -0.008537 0.034754 0.000000 -0.000223 0.000085
[5 rows x 25 columns]
A full example
Suppose we want to extract the QC tsv from all of the QSIPREP gears attached to the PNC project. First we get a reference to the project container.
>>> pnc = fw.projects.find_first('label=PNC_CS_810336')
>>> analyses = fw.get_analyses('projects', pnc.id, 'sessions')
>>> qsiprep_results = [ana for ana in analyses if
... ana.label.startswith("qsiprep_0.3.16_0.8.0RC3")]
We can write a temporary function that gets the non-html zip file and extracts the qc file.
>>> import os.path as op
>>> def download_qc_files(analysis, output_dir):
... # Safely get a list of files
... analysis_files = analysis.files or []
... # Search for a non-html zip file
... zip_files = [f_obj for f_obj in analysis_files if
... f_obj.name.endswith('.zip') and
... not f_obj.name.endswith('.html.zip')]
...
... # if it succeeded, there will be exactly one results zip
... if not len(zip_files) == 1:
... return False
... zip_obj = zip_files[0]
...
... # List the contents of the zip archive
... zip_members = analysis.get_file_zip_info(zip_obj.name)['members']
... # find the confounds file and the qc file
... confounds_files = [f_obj for f_obj in zip_members if 'confounds.tsv' in f_obj.path]
... qc_files = [f_obj for f_obj in zip_members if 'ImageQC' in f_obj.path]
...
... # We need both files
... if not confounds_files and qc_files:
... return False
... confounds_files = [f_obj for f_obj in zip_members if 'confounds.tsv' in f_obj.path]
... qc_files = [f_obj for f_obj in zip_members if 'ImageQC' in f_obj.path]
... # We need both files
... if not confounds_files and qc_files:
... return False
... # Download the confounds file
... confounds_file = confounds_files[0].path
... downloaded_confounds_file = output_dir + "/" + op.split(confounds_file)[1]
... if not op.exists(downloaded_confounds_file):
... analysis.download_file_zip_member(zip_obj.name,
... confounds_file,
... downloaded_confounds_file)
... # Download the qc file
... qc_file = qc_files[0].path
... downloaded_qc_file = output_dir + "/" + op.split(qc_file)[1]
... if not op.exists(downloaded_qc_file):
... analysis.download_file_zip_member(zip_obj.name,
... qc_file,
... downloaded_qc_file)
... return True
...
Then we can loop over the analysis objects and download the files we want:
>>> import os
>>> failed_analyses = []
>>> out_dir = "qsiprep_tsvs"
>>> os.makedirs(out_dir, exist_ok=True)
>>> for result in tqdm(qsiprep_results):
... if not load_qc_txt(result, out_dir):
... failed_analyses.append(result)
And all the files we wanted will be in qsiprep_tsvs/!
Downloading BIDS
Using the CLI
We don’t recommend using this approach, but it is possible to use the Flywheel CLI to download BIDS directly without using Python. See their documentation for usage.