UnicodeDecodeError backtrace during scan

[dengste]

Using scancode 2.0.1. I get an abort with a UnicodeDecodeError backtrace. I'm guessing this is due to same illegal encoding in some file, but I cannot figure out which file exactly is responsible. The backtrace does not tell me much because I cannot see the original backtrace that triggered this but only the one from the multiprocessing pool:

Traceback (most recent call last):
  File "/home/engsterd/scancode-toolkit-2.0.1/bin/scancode", line 11, in <module>
    load_entry_point('scancode-toolkit', 'console_scripts', 'scancode')()
  File "/home/engsterd/scancode-toolkit-2.0.1/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/engsterd/scancode-toolkit-2.0.1/src/scancode/utils.py", line 76, in main
    standalone_mode=standalone_mode, **extra)
  File "/home/engsterd/scancode-toolkit-2.0.1/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/engsterd/scancode-toolkit-2.0.1/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/engsterd/scancode-toolkit-2.0.1/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/engsterd/scancode-toolkit-2.0.1/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/engsterd/scancode-toolkit-2.0.1/src/scancode/cli.py", line 402, in scancode
    ignore=user_ignore)
  File "/home/engsterd/scancode-toolkit-2.0.1/src/scancode/cli.py", line 515, in scan
    result = scanned.next()
  File "/home/engsterd/scancode-toolkit-2.0.1/local/lib/python2.7/site-packages/click/_termui_impl.py", line 259, in next
    rv = next(self.iter)
  File "/home/engsterd/scancode-toolkit-2.0.1/src/scancode/pool.py", line 46, in wrap
    return func(self, timeout=timeout or 1e10)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 668, in next
    raise value
UnicodeDecodeError: 'ascii' codec can't decode byte 0xb1 in position 28: ordinal not in range(128)

[pombredanne]

@dengste
Couple questions to help track this bug:

• Can you paste the exact scancode args and options you have used?
• Is this while scanning public or private code? If public can you send a link?
• Was the code being scanned extracted from an archive? if so which tool did you use to extract this?
• Can you rerun your scan adding these options --verbose --diag --processes 1 : these will provide a different style for progress (one line per file), printout more diagnostic information and run on a single process. Eventually this should provide more details.
• What is your locale (run locale)?
• Can you and paste the output of this:
  python -c "import sys;print('filesystem encoding:', repr(sys.getfilesystemencoding()), 'default encoding:', repr(sys.getdefaultencoding()))"

[dengste]

I simply ran scancode with

./scancode <directory> test.json

Unfortunately, it's private data and I cannot share it. It's a git repo, so the files were not extracted from an archive. Running with the suggested options simply says

[....]
Scanned: path1/path2/somefile
Traceback (most recent call last):

and it follows the exact same backtrace as above. Funny thing is: If I rerun scancode on the subdirectory path1/path2, it works...

Locale output:

LANG=en_US.UTF-8
LANGUAGE=en_US:en
LC_CTYPE=en_US.UTF-8
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Python output:
('filesystem encoding:', "'UTF-8'", 'default encoding:', "'ascii'")

Is there some way to modify the code so that I can see the original backtrace that triggers this? Then I can debug this further myself.

[dengste]

I found the offending file: it has a plus-minus sign in iso-encoding in it. On my shell (zsh) I can create such a file with

 touch foo$'\261'bar

Scanning that file will create the above backtrace. The file exists this way because it was checked in with an old MSysGit version for Windows, which still used codepages instead of UTF-8.

[pombredanne]

@dengste Thanks. Let me add a test and track this pesky bug

[pombredanne]

@dengste I pushed some tests (failing) in a branch that show the problem thanks to your research.
Now there are various ways to resolve this:

1. always return a proper error for a path that is not unicode decodable: Do not scan but the scan will have a clear error message for this path.
2. if a path is not unicode decodable, do not error and fall back to bytes. Then in the scan outputs (that demand unicode encoded as UTF-8) return %-encoded paths for these pesky paths.

None of these solutions is entirely satisfying: 1. skips a file to scan but is quite visible (though I even wonder if we might not need to also %-encode this somehow) while 2. does the work quietly but is a tad more complex and the paths reported in the outputs may not be easy to recreate and relate to an actual path.

Note: we used to have %-encoding and dropped it in favor of unicode/utf-8.
@sschuberth Ping: I recall you reported some tickets in this area too.

[pombredanne]

FWIW the error exists also in the std Python and is not so much a Python error per se.
Dealing with unknown encodings (aka. bytes) is always a potential issue when you want unicode out.

Python 2.7.13 (default, Jun 16 2017, 16:03:08) 
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> list(os.walk(u'non_unicode'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/pombreda/.pyenv/versions/2.7.13/lib/python2.7/os.py", line 286, in walk
    if isdir(join(top, name)):
  File "/home/pombreda/.pyenv/versions/2.7.13/lib/python2.7/posixpath.py", line 73, in join
    path += '/' + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xb1 in position 4: ordinal not in range(128)
>>> list(os.walk('non_unicode'))
[('non_unicode', [], ['foo\xb1bar'])]
>>> 

[pombredanne]

Actually this is no longer a bug on Python 3....

Python 3.4.3 (default, Nov 17 2016, 01:08:31) 
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> list(os.walk('non_unicode'))
[('non_unicode', [], ['foo\udcb1bar'])]
>>> list(os.walk(b'non_unicode'))
[(b'non_unicode', [], [b'foo\xb1bar'])]
>>> 

[sschuberth]

I indeed did see the same issue not too long ago in one of our scans, but I forgot to file a ticket about it:

  File "/usr/lib/python2.7/multiprocessing/pool.py", line 668, in next
    raise value
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 6: ordinal not in range(128)

[pombredanne]

@sschuberth thanks!

So since this is eventually a bug in Python that is fixed in Py3, here is what I suggest:

Do not let a scan fail (ever) but do not attempt to scan the file either since most everything is expecting Unicode throughout inside in preparation to the Py3 move. Instead return an error for this path as a percent encoded path and also add its bytes representation in the error message.

And when Python3 support is there with #295 there will be no error messages anymore on these type of files (hopefully... as things get complicated at times with the expectations of third-party libraries and natives wrt to unicode, paths and file system encodings).

[sschuberth]

Some more info from my side: I saw this happening on our CI when scanning https://github.com/expressjs/serve-static/tree/v1.12.3, however, locally the scan completed just fine. locale output is the same on both machines, Python commonly is 2.7.12 and sys.getfilesystemencoding() reports UTF-8, git ls-tree consistently shows the problematic path as "test/fixtures/snow \342\230\203/.gitkeep". So what could make a difference?

[pombredanne]

@sschuberth Thanks! are you sure you use the same scancode version on each sides?