Skip to content

BUG: work round int overflow in size calculation #325

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jul 9, 2015
Merged

BUG: work round int overflow in size calculation #325

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions nibabel/tests/test_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1227,3 +1227,21 @@ def assert_rt(data,
slope = slope,
post_clips = post_clips,
nan_fill = nan_fill)


def test_array_from_file_overflow():
# Test for int overflow in size calculation in array_from_file
shape = (1500,) * 6
class NoStringIO: # Null file-like for forcing error
def seek(self, n_bytes):
pass
def read(self, n_bytes):
return b''
try:
array_from_file(shape, np.int8, NoStringIO())
except IOError as err:
message = str(err)
assert_equal(message,
'Expected {0} bytes, got {1} bytes from {2}\n'
' - could the file be damaged?'.format(
11390625000000000000, 0, 'object'))
5 changes: 4 additions & 1 deletion nibabel/volumeutils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@
import gzip
import bz2
from os.path import exists, splitext
from operator import mul
from functools import reduce

import numpy as np

Expand Down Expand Up @@ -504,7 +506,8 @@ def array_from_file(shape, in_dtype, infile, offset=0, order='F', mmap=True):
pass
if len(shape) == 0:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could also specify the dtype in np.prod, rather than introduce operator.mul:

n_bytes = int(np.prod(shape, dtype=np.int64) * in_dtype.itemsize)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The advantage of using operator.mul is that is uses Python integers internally, and so completely prevents overflow for any possible value of the shape. In fact the test tests overflow for values above the int64 range, although of course you cannot make an array that large.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough. Might change the comment to reflect that it's not just avoiding int32 overflows, but that's really nit-picky.

return np.array([])
n_bytes = int(np.prod(shape) * in_dtype.itemsize)
# Use reduce and mul to work around numpy integer overflow
n_bytes = reduce(mul, shape) * in_dtype.itemsize
if n_bytes == 0:
return np.array([])
# Read data from file
Expand Down