Skip to content
Merged
Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
66 commits
Select commit Hold shift + click to select a range
50da63f
compcor skeleton
Sep 1, 2016
9e0d327
incremental changes, baby test
Sep 2, 2016
9f7886c
add doctest
Sep 2, 2016
55aec65
add doctest
Sep 2, 2016
b45b7b1
Merge branch 'compcor1594' of http://github.com/shoshber/nipype into …
Sep 2, 2016
f454cbe
fix imports
Sep 3, 2016
ca930d3
better comments, minor re-arranging
Sep 3, 2016
414f11a
set up test prereq's
Sep 4, 2016
bca197e
complete _run_interface implementation
Sep 4, 2016
124d346
implement _list_outputs
Sep 4, 2016
ef50858
pretty ok test
Sep 4, 2016
5e8b997
compcore -> compcor
Sep 6, 2016
6127924
Merge http://github.com/nipy/nipype into compcor1594
Sep 6, 2016
ac8dbfe
TCompCor skeleton, tests
Sep 7, 2016
a5402f7
make python2 happy
Sep 7, 2016
8f5eabc
factor out tSTD calculation
Sep 7, 2016
d87ac71
tCompCor first pass
Sep 7, 2016
757bd0e
alias ACompCor => CompCor
Sep 7, 2016
a54120f
prevent name collision in tests
Sep 7, 2016
c1b0a87
better unique file names
Sep 7, 2016
9a88a11
refactor toy maker function
Sep 7, 2016
61da367
testing skeleton for tsnr
Sep 8, 2016
89b530a
factor out deleting temp files
Sep 8, 2016
87108d9
pre-refactor tests
Sep 8, 2016
33d2f30
move tsnr to own file
Sep 8, 2016
1511bea
naive refactor
Sep 8, 2016
b66db5f
make compcor tests more sensitive to changes, simplify regress_poly()
Sep 8, 2016
4b9dbe4
fix test oops
Sep 8, 2016
bd9113a
m rm unnecessary/confusing test msgs
Sep 8, 2016
e78b0ae
make regress_poly take any kind of input data
oesteban Sep 8, 2016
5bf2d54
improve code readability
Sep 9, 2016
fd45c05
fixed for real this time?
Sep 9, 2016
62bb3bd
change absolute imports to explicit relative imports
Sep 9, 2016
da1e59a
add use_regress_poly Boolean to compcor
Sep 9, 2016
6f76129
Merge branch 'fix-shoshber-compcor1594' of https://github.com/oesteba…
Sep 9, 2016
045bfdc
Merge branch 'oesteban-fix-shoshber-compcor1594' into compcor1594
Sep 9, 2016
ee726a1
undo accidental changes in merge
Sep 9, 2016
0a5bc7a
move tsnr back into misc
Sep 9, 2016
61c96ad
remove rogue file
Sep 9, 2016
4fd169a
fix lingregress (swapped z/y! )
Sep 9, 2016
dfdd6b0
more intelligent testing
Sep 9, 2016
1995a01
use regress_poly from TSNR in compcor
Sep 9, 2016
13838bd
add mask.nii back
Sep 9, 2016
027d47e
smarter tests
Sep 9, 2016
5464902
rsfmri workflow test skeleton
Sep 9, 2016
f8cea61
set up mocks
Sep 10, 2016
1c655f8
mock only what isn't being tested
Sep 10, 2016
08bccfa
run compcor tests in tempfile
Sep 10, 2016
4401bb8
set up 2 out of 3 inputs for compcor node
Sep 10, 2016
29c8b74
Merge http://github.com/nipy/nipype into compcor1594
Sep 10, 2016
402b6b1
add __init__.py file
Sep 11, 2016
c18a780
mocked-up rsfmri workflow runs; use tempfiles
Sep 11, 2016
083c16f
completed rsfmri wf test
Sep 11, 2016
be215f5
refactor rsfmri fsl workflow
Sep 11, 2016
5f39bfc
make python 3 happy
Sep 11, 2016
6869a5f
use mkdtemp for test_tsnr
Sep 11, 2016
c9aa6d6
enable extra_regressors (needed by rsfmri_surface_vol...)
Sep 12, 2016
9aa2300
make percentile threshold a parameter for tCompCor
Sep 12, 2016
276fb57
use CompCor interface in example
Sep 12, 2016
ea60782
forgot to regress pre-computing std
Sep 12, 2016
6a1497d
use relative paths in doctests
Sep 12, 2016
e245073
update documentation
Sep 12, 2016
6a88b3a
robuster tests
Sep 12, 2016
5347881
fix dumb error, comments
Sep 12, 2016
cb2faa2
clarify documentation
Sep 13, 2016
bb45785
remove stuff
Sep 13, 2016
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
81 changes: 81 additions & 0 deletions nipype/algorithms/compcor.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
from ..interfaces.base import (BaseInterfaceInputSpec, TraitedSpec,
BaseInterface, traits, File)
import nibabel as nb
import numpy as np
from scipy import linalg, stats
import os

from nipype.pipeline.engine import Workflow

class CompCoreInputSpec(BaseInterfaceInputSpec):
realigned_file = File(exists=True, mandatory=True, desc='already realigned brain image (4D)')
mask_file = File(exists=True, mandatory=True, desc='mask file that determines ROI (3D)')
num_components = traits.Int(6, usedefault=True) # 6 for BOLD, 4 for ASL
# additional_regressors??

class CompCoreOutputSpec(TraitedSpec):
components_file = File(desc='text file containing the noise components', exists=True)

class CompCore(BaseInterface):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shoshber - checking if this is a pun intended class name :) we should use the name from the paper just for searchability purposes. also see how doi's are now included to be able to generate a citation list for an interface.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not the best place to mention this (I can make a separate issue) but is there a keyword search functionality in nipype? In other words it would be great if interfaces had keywords (like "motion correction", "brain masking"...it would be important to choose them carefully!) and then there was some way to search through them to find an interface that does what you want. Seems relevant here as it is best to have the name match what's in the paper, but then that name isn't actually all that descriptive about what the interface does!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes it's a pun :) I'll change it.

re doi's, is this what you're referring to? #1464

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup! It's just a question of adding references_ field such as here:

references_ = [{'entry': BibTeX("@book{FrackowiakFristonFrithDolanMazziotta1997,"

On Tue, Sep 6, 2016 at 2:15 PM, Shoshana Berleant [email protected]
wrote:

In nipype/algorithms/compcor.py
#1599 (comment):

+import numpy as np
+from scipy import linalg, stats
+import os
+
+from nipype.pipeline.engine import Workflow
+
+class CompCoreInputSpec(BaseInterfaceInputSpec):

  • realigned_file = File(exists=True, mandatory=True, desc='already realigned brain image (4D)')
  • mask_file = File(exists=True, mandatory=True, desc='mask file that determines ROI (3D)')
  • num_components = traits.Int(6, usedefault=True) # 6 for BOLD, 4 for ASL
  • additional_regressors??

+class CompCoreOutputSpec(TraitedSpec):

  • components_file = File(desc='text file containing the noise components', exists=True)

+class CompCore(BaseInterface):

Yes it's a pun :) I'll change it.

re doi's, is this what you're referring to? #1464
#1464


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/nipy/nipype/pull/1599/files/ef50858d3b3016018f0078af6b7da29412803c00#r77720814,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAOkp7UyOpbrANACeVwGabDKi06T-Wveks5qndgEgaJpZM4JyQmp
.

'''
Interface with core CompCor computation, used in aCompCor and tCompCor
Example
-------
>>> ccinterface = CompCore()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to add this on top of the file: https://github.com/nipy/nipype/blob/master/nipype/interfaces/afni/preprocess.py#L5 for the doctests to run independently of CWD

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

couldn't get that to work :(

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see the file header I mention in my previous comment it your code. Did you try adding it? Also notice that the after adding the header the input files should be treated as if they were in the CWD: https://github.com/nipy/nipype/blob/master/nipype/interfaces/afni/preprocess.py#L45

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, I tried that earlier. It didn't work, I tried for ~10 minutes to get it to work, then decided it wasn't really worth it for now. Should I consider it worth it for now?

>>> ccinterface.inputs.realigned_file = 'nipype/testing/data/functional.nii'
>>> ccinterface.inputs.mask_file = 'nipype/testing/data/mask.nii'
>>> ccinterface.inputs.num_components = 1
'''
input_spec = CompCoreInputSpec
output_spec = CompCoreOutputSpec

def _run_interface(self, runtime):
imgseries = nb.load(self.inputs.realigned_file).get_data()
mask = nb.load(self.inputs.mask_file).get_data()
voxel_timecourses = imgseries[mask > 0]
# Zero-out any bad values
voxel_timecourses[np.isnan(np.sum(voxel_timecourses, axis=1)), :] = 0

# from paper:
# "Voxel time series from the noise ROI (either anatomical or tSTD) were
# placed in a matrix M of size Nxm, with time along the row dimension
# and voxels along the column dimension."
# voxel_timecourses.shape == [nvoxels, time]
M = voxel_timecourses.T
numvols = M.shape[0]
numvoxels = M.shape[1]

# "The constant and linear trends of the columns in the matrix M were removed ..."
timesteps = range(numvols)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you may want to use legendre polynomials for this as shown here:

https://github.com/nipy/nipype/blob/master/nipype/algorithms/misc.py#L310

also this should become a boolean flag + order being another parameter that's set to some default value but user changeable.

Copy link
Contributor Author

@berleant berleant Sep 8, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, @chrisfilo suggested this earlier--I'm working on refactoring TSNR to share the code.

I'll add the boolean flag.

for voxel in range(numvoxels):
m, b, _, _, _ = stats.linregress(M[:, voxel], timesteps)
M[:, voxel] = M[:, voxel] - [m*t + b for t in timesteps]

# "... prior to column-wise variance normalization."
stdM = np.std(M, axis=0)
# set bad values to division identity
stdM[stdM == 0] = 1.
stdM[np.isnan(stdM)] = 1.
stdM[np.isinf(stdM)] = 1.
M = M / stdM

# "The covariance matrix C = MMT was constructed and decomposed into its
# principal components using a singular value decomposition."
u, _, _ = linalg.svd(M, full_matrices=False)
components = u[:, :self.inputs.num_components]
components_file = os.path.join(os.getcwd(), "components_file.txt")
np.savetxt(components_file, components, fmt="%.10f")
return runtime

def _list_outputs(self):
outputs = self._outputs().get()
outputs['components_file'] = os.path.abspath("components_file.txt")
return outputs

class aCompCor(Workflow):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this be BaseInterface?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Like @satra 's comment on the issue (#1594), I am proposing to write an interface containing the core CompCor logic. tCompCor and aCompCor would be workflows that contain nodes for creating the masks, a node for the core CompCor logic, and possibly (?) nodes for applying the extracted noise components. What do you think?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see... Workflows go here:
https://github.com/nipy/nipype/tree/master/nipype/workflows and follow a
slightly different pattern (they are functions returning a workflow object
rather than sublasses of Workflow).

I personally would just create two Interfaces: aCompCor(BaseInterface) that
takes a mask, calculates PCA and return the components, and
tCompCor(aCompCor) that takes a percentile argument n, creates a mask that
includes only the n% most variable voxels and reuses the code from its
parent aCompCor to return components. This is as flexible as the workflows,
but easier to integrate with pipelines as well as implement. Let me know
what do you think.

On Fri, Sep 2, 2016 at 10:16 PM, Shoshana Berleant <[email protected]

wrote:

In nipype/algorithms/compcor.py
#1599 (comment):

-class aCompCor(Node):
+class aCompCor(Workflow):

Like @satra https://github.com/satra 's comment on the issue (#1594
#1594), I am proposing to write an
interface containing the core CompCor logic. tCompCor and aCompCor would be
workflows that contain nodes for creating the masks, a node for the core
CompCor logic, and possibly (?) nodes for applying the extracted noise
components. What do you think?


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/nipy/nipype/pull/1599/files/50da63f9a8c74e60ef43b48349d9e5e19f5ad719..9e0d32796cec4ef237e2ccb2b7b74554f01c6ad8#r77431634,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAOkp4CKN3GLOC9k9ZZtcY92UazGtooFks5qmQK7gaJpZM4JyQmp
.

pass

class tCompCor(Workflow):
pass
60 changes: 60 additions & 0 deletions nipype/algorithms/tests/test_compcor.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# emacs: -*- mode: python; py-indent-offset: 4; indent-tabs-mode: nil -*-
# vi: set ft=python sts=4 ts=4 sw=4 et:
import nipype
from nipype.testing import assert_equal, assert_true, assert_false, skipif
from nipype.algorithms.compcor import CompCore

import nibabel as nb
import numpy as np
import os

functionalnii = 'func.nii'
masknii = 'mask.nii'

def test_compcore():
# setup
noise = np.fromfunction(fake_noise_fun, fake_data.shape)
realigned_file = make_toy(fake_data + noise, functionalnii)

mask = np.ones(fake_data.shape[:3])
mask[0,0,0] = 0
mask[0,0,1] = 0
mask_file = make_toy(mask, masknii)

# run
ccinterface = CompCore(realigned_file=realigned_file, mask_file=mask_file)
ccresult = ccinterface.run()

# asserts
components = ccinterface._list_outputs()['components_file']
assert_true(os.path.exists(components))
assert_true(os.path.getsize(components) > 0)
assert_equal(ccinterface.inputs.num_components, 6)

# apply components_file to realigned_file, is it better? the same as before adding noise?

# remove temporary nifti files
os.remove(functionalnii)
os.remove(masknii)
os.remove(components)

def make_toy(ndarray, filename):
toy = nb.Nifti1Image(ndarray, np.eye(4))
nb.nifti1.save(toy, filename)
return filename

fake_data = np.array([[[[8, 5, 3, 8, 0],
[6, 7, 4, 7, 1]],

[[7, 9, 1, 6, 5],
[0, 7, 4, 7, 7]]],


[[[2, 4, 5, 7, 0],
[1, 7, 0, 5, 4]],

[[7, 3, 9, 0, 4],
[9, 4, 1, 5, 0]]]])

def fake_noise_fun(i, j, l, m):
return m*i + l - j