ENH: Support + and += operators for Tractogram #495

MarcCote · 2016-11-09T18:31:07Z

This PR adds the functionality of concatenating two Tractogram objects using either tractogram += other_tractogram or tractogram = tractogram1 + tractogram2.

This will definitively interest @jchoude, @arnaudbore, @Garyfallidis, @FrancoisRheaultUS and many others.

coveralls · 2016-11-09T18:43:51Z

Coverage increased (+0.01%) to 95.959% when pulling bd3f921 on MarcCote:enh_tractogram_operators into ec4567f on nipy:master.

codecov-io · 2016-11-09T18:50:35Z

Current coverage is 94.02% (diff: 94.70%)

Merging #495 into master will increase coverage by <.01%

@@             master       #495   diff @@
==========================================
  Files           166        166          
  Lines         21832      21992   +160   
  Methods           0          0          
  Messages          0          0          
  Branches       2325       2343    +18   
==========================================
+ Hits          20527      20679   +152   
- Misses          875        878     +3   
- Partials        430        435     +5

Powered by Codecov. Last update 6104bd1...8f8fd5a

matthew-brett

Sorry to be slow to review - see comments.

matthew-brett · 2016-11-26T18:16:01Z

nibabel/streamlines/tractogram.py

+        other : :class:`PerArrayDict` object
+            Its data will be appended to the data of this dictionary.
+
+        Notes


Maybe add the method returns None.

matthew-brett · 2016-11-26T18:16:55Z

nibabel/streamlines/tractogram.py

+
+        Notes
+        -----
+        The entries in both dictionaries must match.


More specifically, the keys in each dictionary must be the same.

matthew-brett · 2016-11-26T18:18:11Z

nibabel/streamlines/tractogram.py

+        other : :class:`PerArraySequenceDict` object
+            Its data will be appended to the data of this dictionary.
+
+        Notes


Returns None, keys must match.

matthew-brett · 2016-11-26T18:21:23Z

nibabel/streamlines/tractogram.py

@@ -136,6 +162,32 @@ def __setitem__(self, key, value):

        self.store[key] = value

+    def extend(self, other):


Can't you just inherit this method? I guess you'd have make the docstrings and message a bit more generic, but it seems a shame to duplicate the code.

I had to duplicate it (almost every line except one) because one method is using self[key] = np.concatenate([self[key], other[key]]) and the other self[key].extend(other[key]). If you think of an alternative, I'm all ears.

matthew-brett · 2016-11-26T18:22:20Z

nibabel/streamlines/tractogram.py

+        other : :class:`Tractogram` object
+            Its data will be appended to the data of this tractogram.
+
+        Notes


Returns None, keys much match.

matthew-brett · 2016-11-26T18:32:19Z

nibabel/streamlines/tests/test_tractogram.py

+        t = DATA['tractogram'].copy()
+
+        # Double the tractogram.
+        new_t = t + t


Could use:

def extender(a, b): a.extend(b) return a import operator for op, in_place in ((operator.add, False, (operator.iadd, True), (extender, True)): first_arg = copy(t) new_t = op(first_arg, t) assert_equal(new_t is first_arg, in_place) # etc

matthew-brett · 2016-11-26T18:34:30Z

nibabel/streamlines/tests/test_tractogram.py

+    def test_extend(self):
+        total_nb_rows = DATA['tractogram'].streamlines.total_nb_rows
+        sdict = PerArraySequenceDict(total_nb_rows, DATA['data_per_point'])
+        sdict2 = PerArraySequenceDict(total_nb_rows, DATA['data_per_point'])


Would be nice to check for situation where data was not the same.

matthew-brett · 2016-11-26T18:35:06Z

nibabel/streamlines/tests/test_tractogram.py

+            assert_arrays_equal(sdict[k][len(DATA['tractogram']):], v)
+
+        # Test incompatible PerArrayDicts.
+        assert_raises(ValueError, sdict.extend, PerArraySequenceDict())


Check for situation where there are extra keys in one or other? Other than empty case here?

matthew-brett · 2016-11-26T18:38:30Z

nibabel/streamlines/tests/test_tractogram.py

@@ -233,6 +248,20 @@ def test_getitem(self):
            assert_arrays_equal(sdict[-1][k], v[-1])
            assert_arrays_equal(sdict[[0, -1]][k], v[[0, -1]])

+    def test_extend(self):
+        total_nb_rows = DATA['tractogram'].streamlines.total_nb_rows
+        sdict = PerArraySequenceDict(total_nb_rows, DATA['data_per_point'])


Avoid duplication by inheriting this test class from TestPerArrayDict, and this in TestPerArrayDict?

tested_cls = PerArraySequenceDict def test_extend(self): sdict = self.tested_cls(total_nb_rows, DATA['data_per_point']) # etc

Or a mixin with just this method.

I'm not sure it will work as you intend or maybe I don't get your point. There is a couple of differences between testing a PerArrayDict and a PerArraySequenceDict. For instance, the constructor of the first class takes the number of streamlines whereas the second takes the total number of points in a ArraySequence. Also, one checks data_per_streamline where the other data_per_point.

I got it to work :)

I see the complications now. What do you mean by "I got it to work" ?

It is related to my previous comment. I meant I succeeded in reducing code duplication.

matthew-brett · 2016-11-26T18:40:01Z

nibabel/streamlines/tests/test_tractogram.py

@@ -181,6 +181,21 @@ def test_getitem(self):
            assert_arrays_equal(sdict[-1][k], v[-1])
            assert_arrays_equal(sdict[[0, -1]][k], v[[0, -1]])

+    def test_extend(self):


Test for + and +=? See below for general suggestion.

By choice, PerArrayDict and PerArraySequenceDict don't support + and +=. Only Tractogram objects have it.

Sorry for my slow understanding, but why no + and +=? Just because native dict objects don't support these? In which case, why extend which doesn't exist for dict either?

Didn't see the need for them att the time. I don't think these dicts are going to be used extensively but I can add them if you want.

No, it's fine, just trying to work out how you were thinking of these.

MarcCote · 2016-11-29T03:27:12Z

@matthew-brett thanks for the feedback. I addressed most of your comments except those related to code duplication (see my replies above).

coveralls · 2016-11-29T03:37:45Z

Coverage increased (+0.06%) to 96.006% when pulling aec94cf on MarcCote:enh_tractogram_operators into ec4567f on nipy:master.

coveralls · 2016-12-22T19:53:24Z

Coverage increased (+0.06%) to 96.002% when pulling 7d5cefb on MarcCote:enh_tractogram_operators into ec4567f on nipy:master.

coveralls · 2016-12-23T05:45:51Z

Coverage increased (+0.06%) to 96.004% when pulling 040e2a0 on MarcCote:enh_tractogram_operators into ec4567f on nipy:master.

coveralls · 2017-01-06T16:10:13Z

Coverage increased (+0.01%) to 96.004% when pulling 5d98758 on MarcCote:enh_tractogram_operators into 6104bd1 on nipy:master.

MarcCote · 2017-01-07T22:00:12Z

@matthew-brett this PR is ready for a second round of reviews.

coveralls · 2017-01-07T22:14:34Z

Coverage increased (+0.01%) to 96.004% when pulling fc51e76 on MarcCote:enh_tractogram_operators into 6104bd1 on nipy:master.

matthew-brett · 2017-01-08T00:54:20Z

Thanks for your patience, I should be able to get to this on Monday.

matthew-brett

Some small comments

matthew-brett · 2017-01-09T14:11:31Z

nibabel/streamlines/tests/test_tractogram.py

-                                        'mean_torsion': mean_torsion_func,
-                                        'mean_colors': mean_colors_func}
+    DATA['data_per_point_func'] = {
+        'colors': lambda: (e for e in DATA['colors']),


Indentation?

I found the code cleaner when I break the line this way compared to in the middle of the generator comprehension.

Sure - I was wondering if the indentation you got here was PEP8 compatible - fine if so.

It is passing the flake8 test of one of the Travis bots :).

matthew-brett · 2017-01-09T14:11:43Z

nibabel/streamlines/tests/test_tractogram.py

+        'colors': lambda: (e for e in DATA['colors']),
+        'fa': lambda: (e for e in DATA['fa'])}
+    DATA['data_per_streamline_func'] = {
+        'mean_curvature': lambda: (e for e in DATA['mean_curvature']),


Indentation?

I found the code cleaner when I break the line this way compared to in the middle of the generator comprehension.

matthew-brett · 2017-01-09T14:13:39Z

nibabel/streamlines/tests/test_tractogram.py

@@ -181,6 +181,21 @@ def test_getitem(self):
            assert_arrays_equal(sdict[-1][k], v[-1])
            assert_arrays_equal(sdict[[0, -1]][k], v[[0, -1]])

+    def test_extend(self):


Sorry for my slow understanding, but why no + and +=? Just because native dict objects don't support these? In which case, why extend which doesn't exist for dict either?

matthew-brett · 2017-01-09T14:16:48Z

nibabel/streamlines/tests/test_tractogram.py

+            assert_arrays_equal(sdict[k][len(DATA['tractogram']):],
+                                new_data[k])
+
+        # Extending with an empty PerArrayDicts should change nothing.


Should be PerArrayDict (no 's' at end)?

matthew-brett · 2017-01-09T14:20:19Z

nibabel/streamlines/tests/test_tractogram.py

+        sdict2 = PerArrayDict(len(DATA['tractogram']), new_data)
+        assert_raises(ValueError, sdict.extend, sdict2)
+
+        # Other dict has the right number of entries but wrong shape.


I think the check is, that the keys must be the same. Is that what you mean by "shape" here?

By entries, I mean keys. By shape, I mean the shape (except for the first dimension) of the ndarray or ArraySequence that will be appended to the the value at dict[k] where k is one of the entry. This is because we know these dict are dictionaries of ndarray or ArraySequence.

Hum - but isn't the error in fact coming from the fact that mean_color != other, rather than the shape difference? Maybe you need two tests here, one for the keys and one for the shapes, where the shapes test has an entry with the same name, but a different shape.

Yes,you are right. I'll make another test.

matthew-brett · 2017-01-09T14:59:23Z

nibabel/streamlines/tests/test_tractogram.py

@@ -233,6 +248,20 @@ def test_getitem(self):
            assert_arrays_equal(sdict[-1][k], v[-1])
            assert_arrays_equal(sdict[[0, -1]][k], v[[0, -1]])

+    def test_extend(self):
+        total_nb_rows = DATA['tractogram'].streamlines.total_nb_rows
+        sdict = PerArraySequenceDict(total_nb_rows, DATA['data_per_point'])


I see the complications now. What do you mean by "I got it to work" ?

matthew-brett · 2017-01-09T14:59:55Z

nibabel/streamlines/tests/test_tractogram.py

+        total_nb_rows = DATA['tractogram'].streamlines.total_nb_rows
+        sdict = PerArraySequenceDict(total_nb_rows, DATA['data_per_point'])
+
+        # Test compatible PerArrayDicts.


PerArraySequenceDicts

matthew-brett · 2017-01-09T15:00:16Z

nibabel/streamlines/tests/test_tractogram.py

+            assert_arrays_equal(sdict[k][len(DATA['tractogram']):],
+                                new_data[k])
+
+        # Extending with an empty PerArrayDicts should change nothing.


PerArraySequenceDict

matthew-brett · 2017-01-09T15:00:38Z

nibabel/streamlines/tests/test_tractogram.py

+        sdict2 = PerArraySequenceDict(np.sum(list_nb_points), new_data)
+        assert_raises(ValueError, sdict.extend, sdict2)
+
+        # Other dict has the right number of entries but wrong shape.


Wrong keys? (As above).

matthew-brett · 2017-01-09T15:01:12Z

nibabel/streamlines/tests/test_tractogram.py

+
+        for op, in_place in ((operator.add, False), (operator.iadd, True),
+                             (extender, True)):
+            first_arg = t.copy()


Needs deepcopy?

Are you asking if we need a deepcopy or you are suggesting me to use deepcopy?

Yes, asking if you need deepcopy.

No, because .copy() is doing a deepcopy (https://github.com/nipy/nibabel/blob/master/nibabel/streamlines/tractogram.py#L346).

matthew-brett

Response to replies.

matthew-brett · 2017-01-17T00:15:09Z

nibabel/streamlines/tests/test_tractogram.py

-                                        'mean_torsion': mean_torsion_func,
-                                        'mean_colors': mean_colors_func}
+    DATA['data_per_point_func'] = {
+        'colors': lambda: (e for e in DATA['colors']),


Sure - I was wondering if the indentation you got here was PEP8 compatible - fine if so.

matthew-brett · 2017-01-17T00:15:43Z

nibabel/streamlines/tests/test_tractogram.py

@@ -181,6 +181,21 @@ def test_getitem(self):
            assert_arrays_equal(sdict[-1][k], v[-1])
            assert_arrays_equal(sdict[[0, -1]][k], v[[0, -1]])

+    def test_extend(self):


No, it's fine, just trying to work out how you were thinking of these.

matthew-brett · 2017-01-17T00:17:49Z

nibabel/streamlines/tests/test_tractogram.py

+        sdict2 = PerArrayDict(len(DATA['tractogram']), new_data)
+        assert_raises(ValueError, sdict.extend, sdict2)
+
+        # Other dict has the right number of entries but wrong shape.


Hum - but isn't the error in fact coming from the fact that mean_color != other, rather than the shape difference? Maybe you need two tests here, one for the keys and one for the shapes, where the shapes test has an entry with the same name, but a different shape.

coveralls · 2017-01-19T00:23:42Z

Coverage increased (+0.02%) to 96.008% when pulling 8f8fd5a on MarcCote:enh_tractogram_operators into 6104bd1 on nipy:master.

MarcCote · 2017-01-19T22:42:00Z

@matthew-brett should be ready to be merged if you don't have any additional comments.

matthew-brett · 2017-01-19T22:42:44Z

Thanks for the edits - and sorry about the wait.

MarcCote · 2017-01-19T22:53:02Z

No worry. Thanks for the review.

matthew-brett reviewed Nov 26, 2016

View reviewed changes

MarcCote added 7 commits January 6, 2017 10:57

Add Tractogram concatenation

cc6b31b

Add docstrings

c69a5ba

Support __add__. Remove unnecessary syntatic sugar.

c99f970

Addressed @matthew-brett's comments

eaadbb5

RF: Add function to easily make fake streamline for testing purposes.

09f4ed6

PEP8

d3410ee

Supports extending empty tractograms

5d98758

MarcCote force-pushed the enh_tractogram_operators branch from 040e2a0 to 5d98758 Compare January 6, 2017 15:57

PEP8

fc51e76

matthew-brett reviewed Jan 9, 2017

View reviewed changes

matthew-brett reviewed Jan 17, 2017

View reviewed changes

arnaudbore mentioned this pull request Jan 18, 2017

Tractography converter UNFmontreal/toad#43

Closed

Addressed @matthew-brett's comments

8f8fd5a

matthew-brett merged commit e7f23f6 into nipy:master Jan 19, 2017

MarcCote deleted the enh_tractogram_operators branch January 19, 2017 22:51

		@@ -136,6 +162,32 @@ def __setitem__(self, key, value):

		self.store[key] = value

		def extend(self, other):

ENH: Support + and += operators for Tractogram #495

ENH: Support + and += operators for Tractogram #495

Uh oh!

Conversation

MarcCote commented Nov 9, 2016

Uh oh!

coveralls commented Nov 9, 2016

Uh oh!

codecov-io commented Nov 9, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Current coverage is 94.02% (diff: 94.70%)

Uh oh!

matthew-brett left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MarcCote Nov 29, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MarcCote commented Nov 29, 2016

Uh oh!

coveralls commented Nov 29, 2016

Uh oh!

coveralls commented Dec 22, 2016

Uh oh!

coveralls commented Dec 23, 2016

Uh oh!

coveralls commented Jan 6, 2017

Uh oh!

MarcCote commented Jan 7, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coveralls commented Jan 7, 2017

Uh oh!

matthew-brett commented Jan 8, 2017

Uh oh!

matthew-brett left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

codecov-io commented Nov 9, 2016 •

edited

Loading

MarcCote Nov 29, 2016 •

edited

Loading

MarcCote commented Jan 7, 2017 •

edited

Loading