Test failure on s390x: test_nib_tck2trk

[sanjayankur31]

While updating the Fedora package to 2.4.1, we received build failures on s390x.

Hardware information:

CPU info:
Architecture:        s390x
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Big Endian
CPU(s):              4
On-line CPU(s) list: 0-3
Thread(s) per core:  1
Core(s) per socket:  1
Socket(s) per book:  1
Book(s) per drawer:  1
Drawer(s):           4
NUMA node(s):        1
Vendor ID:           IBM/S390
Machine type:        2964
CPU dynamic MHz:     5000
CPU static MHz:      5000
BogoMIPS:            3033.00
Hypervisor:          KVM/Linux
Hypervisor vendor:   KVM
Virtualization type: full
Dispatching mode:    horizontal
L1d cache:           128K
L1i cache:           96K
L2d cache:           2048K
L2i cache:           2048K
L3 cache:            65536K
L4 cache:            491520K
NUMA node0 CPU(s):   0-3
Flags:               esan3 zarch stfle msa ldisp eimm dfp edat etf3eh highgprs te vx sie


Memory:
              total        used        free      shared  buff/cache   available
Mem:       10306200      259056     8939776         116     1107368     9935280
Swap:       2097148        2568     2094580


Storage:
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda3       103G  3.2G   95G   4% /

Only one test fails:

BUILDSTDERR: ======================================================================
BUILDSTDERR: ERROR: nibabel.tests.test_scripts.test_nib_tck2trk
BUILDSTDERR: ----------------------------------------------------------------------
BUILDSTDERR: Traceback (most recent call last):
BUILDSTDERR:   File "/usr/lib/python3.7/site-packages/nose/case.py", line 197, in runTest
BUILDSTDERR:     self.test(*self.arg)
BUILDSTDERR:   File "/builddir/build/BUILD/nibabel-2.4.1/nibabel/tests/test_scripts.py", line 488, in test_nib_tck2trk
BUILDSTDERR:     trk = nib.streamlines.load(standard_trk)
BUILDSTDERR:   File "/builddir/build/BUILD/nibabel-2.4.1/nibabel/streamlines/__init__.py", line 96, in load
BUILDSTDERR:     return tractogram_file.load(fileobj, lazy_load=lazy_load)
BUILDSTDERR:   File "/builddir/build/BUILD/nibabel-2.4.1/nibabel/streamlines/trk.py", line 372, in load
BUILDSTDERR:     arr_seqs = create_arraysequences_from_generator(trk_reader, n=3)
BUILDSTDERR:   File "/builddir/build/BUILD/nibabel-2.4.1/nibabel/streamlines/array_sequence.py", line 387, in create_arraysequences_from_generator
BUILDSTDERR:     for data in gen:
BUILDSTDERR:   File "/builddir/build/BUILD/nibabel-2.4.1/nibabel/streamlines/trk.py", line 683, in _read
BUILDSTDERR:     buffer=f.read(nb_pts * pts_and_scalars_size))
BUILDSTDERR: TypeError: buffer is too small for requested array
BUILDSTDERR: ----------------------------------------------------------------------

The complete build log is here:
build.txt

The build completes fine on x86_64, i686, ppc64le, and armv7hl

[effigies]

Thanks for the report. Not having an s390x system to work on, do you think you could drop into pdb and produce a listing of the variables in scope?

e.g.,

$ nosetests --pdb nibabel/tests/test_scripts.py
(pdb) locals()

[sanjayankur31]

I'll do it over the weekend. I'll have to figure out how we Fedora folks get access to s390x test machines too .

[effigies]

Hi @sanjayankur31 any news? Looking to release in the next week or so, and would love to try to get this resolved before then.

[sanjayankur31]

I've got access to a machine now, but I haven't had a chance to look into it. Sorry. Hopefully this week.

[effigies]

Okay. If not, we'll get to it when we can. Bugs are always there to be fixed.

[yarikoptic]

confirming that the bug is still there with a doctest to follow up

======================================================================
ERROR: nibabel.tests.test_scripts.test_nib_tck2trk
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/yoh/nibabel/nibabel/tests/test_scripts.py", line 488, in test_nib_tck2trk
    trk = nib.streamlines.load(standard_trk)
  File "/home/yoh/nibabel/nibabel/streamlines/__init__.py", line 96, in load
    return tractogram_file.load(fileobj, lazy_load=lazy_load)
  File "/home/yoh/nibabel/nibabel/streamlines/trk.py", line 372, in load
    arr_seqs = create_arraysequences_from_generator(trk_reader, n=3)
  File "/home/yoh/nibabel/nibabel/streamlines/array_sequence.py", line 387, in create_arraysequences_from_generator
    for data in gen:
  File "/home/yoh/nibabel/nibabel/streamlines/trk.py", line 683, in _read
    buffer=f.read(nb_pts * pts_and_scalars_size))
TypeError: buffer is too small for requested array

======================================================================
FAIL: Doctest: nibabel.brikhead.AFNIHeader.__init__
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/doctest.py", line 2224, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for nibabel.brikhead.AFNIHeader.__init__
  File "/home/yoh/nibabel/nibabel/brikhead.py", line 285, in __init__

----------------------------------------------------------------------
File "/home/yoh/nibabel/nibabel/brikhead.py", line 298, in nibabel.brikhead.AFNIHeader.__init__
Failed example:
    header.get_data_dtype()
Expected:
    dtype('int16')
Got:
    dtype('<i2')


----------------------------------------------------------------------
Ran 7835 tests in 90.279s

FAILED (SKIP=62, errors=1, failures=1)

here is some variables

(sid_s390x-dchroot)yoh@zelenka:~/nibabel$ python -m nose -s -v --pdb --pdb-failures /home/yoh/nibabel/nibabel/tests/test_scripts.py:test_nib_tck2trk
nibabel.tests.test_scripts.test_nib_tck2trk ... > /home/yoh/nibabel/nibabel/streamlines/trk.py(683)_read()
-> buffer=f.read(nb_pts * pts_and_scalars_size))
(Pdb) p nb_pts
50331648
(Pdb) p pts_and_scalars_size
12
(Pdb) p f
<nibabel.openers.Opener object at 0x3ffb6749f08>

PS if interested -- there is a software s390 emulator -- https://hercules-390.github.io/html/ although I have not tried it for ages. Also I see qemu-system-misc package in debian claiming to provide emulators for all kinds of architectures, including s390. would be cool to try, might do if get bored ;-)

[yarikoptic]

hm, I am a bit confused. backtrace points to line 683 but listing to 701

(Pdb) bt
  /usr/lib/python2.7/unittest/case.py(329)run()
-> testMethod()
  /usr/lib/python2.7/dist-packages/nose/case.py(197)runTest()
-> self.test(*self.arg)
  /home/yoh/nibabel/nibabel/tests/test_scripts.py(488)test_nib_tck2trk()
-> trk = nib.streamlines.load(standard_trk)
  /home/yoh/nibabel/nibabel/streamlines/__init__.py(96)load()
-> return tractogram_file.load(fileobj, lazy_load=lazy_load)
  /home/yoh/nibabel/nibabel/streamlines/trk.py(372)load()
-> arr_seqs = create_arraysequences_from_generator(trk_reader, n=3)
  /home/yoh/nibabel/nibabel/streamlines/array_sequence.py(387)create_arraysequences_from_generator()
-> for data in gen:
> /home/yoh/nibabel/nibabel/streamlines/trk.py(683)_read()
-> buffer=f.read(nb_pts * pts_and_scalars_size))

but listing lists 701 as the current one

(Pdb) l
696     
697                 # In case the 'count' field was not provided.
698                 header[Field.NB_STREAMLINES] = count
699     
700                 # Set the file position where it was (in case it was already open).
701  ->             f.seek(start_position, os.SEEK_CUR)

why?

as for variables around 683 -- seems to be kosher or what am I missing?

(Pdb) p (nb_pts, nb_pts_and_scalars)
(50331648, 3)
(Pdb) p f4_dtype
dtype('float32')
(Pdb) p nb_pts * pts_and_scalars_size
603979776
(Pdb) 603979776 / (50331648*3.)
4.0

[yarikoptic]

ah -- file is too small so either nb_pts is off or file writing (which generated that file) is buggy?

(Pdb) p len(f.read(nb_pts * pts_and_scalars_size))
4796
(Pdb) p nb_pts * pts_and_scalars_size
603979776

(Pdb) p f.name
'/tmp/tmpPB1cFj/standard.trk'
(Pdb) 
[2]+  Stopped                 python -m nose -s -v --pdb --pdb-failures /home/yoh/nibabel/nibabel/tests/test_scripts.py:test_nib_tck2trk
(sid_s390x-dchroot)yoh@zelenka:~/nibabel$ ls -ld /tmp/tmpPB1cFj/standard.trk
-rw-r--r-- 1 yoh yoh 5800 Jul 29 14:49 /tmp/tmpPB1cFj/standard.trk

[effigies]

@yarikoptic Thanks, it looks like an endianness problem. The nb_pts should be 3, and

>>> np.int32(3).newbyteorder()
50331648

So the question is whether that part of the file got written big endian or is was little endian but is being read as big endian. Are you able to copy that file over and read it on a little-endian machine?

[effigies]

@MarcCote I'm going to rope you in on this one. It looks like there's an issue either with reading or writing TRK files on big-endian systems. I don't have one to play with at the moment, so I can't give you a lot more details, but it may be easier for you to check through the potential break points for logic errors.

[yarikoptic]

FWIW here is the file http://onerussian.com/tmp/standard.trk

[effigies]

Yup, failing on mine. So it seems nb_pts is written in the native byte order.

[effigies]

It was the other way around. Part of the header was written big-endian, so the endianness for the entire file was detected as big-endian. When encountering the number of points, which were separately encoded little-endian, the ordering was reversed.

This was the header after being byte swapped to big-endian:

{
    'magic_number': b'TRACK',
    'dimensions': array([1024, 1280, 1792], dtype=int16),
    'voxel_sizes': array([4.6006e-41, 2.3049e-41, 8.9683e-44], dtype=float32),
    'origin': array([0., 0., 0.], dtype=float32),
    'nb_scalars_per_point': 0,
    'scalar_name': array([b'', b'', b'', b'', b'', b'', b'', b'', b'', b''], dtype='|S20'),
    'nb_properties_per_streamline': 0,
    'property_name': array([b'', b'', b'', b'', b'', b'', b'', b'', b'', b''], dtype='|S20'),
    'voxel_to_rasmm': array([[4.6006e-41, 0.0000e+00, 0.0000e+00, 0.0000e+00],
       [0.0000e+00, 2.3049e-41, 0.0000e+00, 0.0000e+00],
       [0.0000e+00, 0.0000e+00, 8.9683e-44, 0.0000e+00],
       [0.0000e+00, 0.0000e+00, 0.0000e+00, 4.6006e-41]], dtype=float32),
    'reserved': b'',
    'voxel_order': b'RAS',
    'pad2': b'',
    'image_orientation_patient': array([0., 0., 0., 0., 0., 0.], dtype=float32),
    'pad1': b'',
    'invert_x': b'',
    'invert_y': b'',
    'invert_z': b'',
    'swap_xy': b'',
    'swap_yz': b'',
    'swap_zx': b'',
    'nb_streamlines': 2013265920,
    'version': 2,
    'hdr_size': 1000,
    'endianness': '>',
    '_offset_data': 1000
}

I've created #782, which should resolve this.

@yarikoptic Doctest bug included.

[MarcCote]

Sorry, I'm just seeing this issue (and your fix) now. Good catch :D.