Add fnmatch.filterfalse function

[stevendaprano]

BPO	30413
Nosy	@brettcannon, @rhettinger, @vstinner, @stevendaprano, @serhiy-storchaka, @wm75
Dependencies	bpo-30415: Improve fnmatch testing
Files	filterfalse.diff filterfalse.alternate_patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2017-05-20.13:19:02.837>
labels = ['3.7', 'type-feature', 'library']
title = 'Add fnmatch.filterfalse function'
updated_at = <Date 2019-08-21.16:32:30.493>
user = 'https://github.com/stevendaprano'

bugs.python.org fields:

activity = <Date 2019-08-21.16:32:30.493>
actor = 'gvanrossum'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2017-05-20.13:19:02.837>
creator = 'steven.daprano'
dependencies = ['30415']
files = ['46879', '46880']
hgrepos = []
issue_num = 30413
keywords = ['patch']
message_count = 17.0
messages = ['294029', '294031', '294036', '294038', '294039', '294041', '294078', '294110', '294111', '294132', '294133', '294510', '350056', '350062', '350066', '350090', '350093']
nosy_count = 7.0
nosy_names = ['brett.cannon', 'rhettinger', 'vstinner', 'steven.daprano', 'serhiy.storchaka', 'wolma', 'Roee Nizan']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue30413'
versions = ['Python 3.7']

Linked PRs

• gh-74598: add fnmatch.filterfalse for excluding names #121185

[stevendaprano]

There has been some discussion on Python-Ideas about adding fnmatch.filter_false to complement filter, for when you want to ignore matching files rather than non-matching ones.

https://mail.python.org/pipermail/python-ideas/2017-May/045694.html

I don't believe there have been any strong objections, and work-arounds have a considerable performance penalty (according to the OP, I have not confirmed that).

Here's a patch which refactors filter and adds filter_false.

Sorry I haven't got my head around the brave new world on Github, so I'm going old school here with a diff :-)

[serhiy-storchaka]

In general LGTM. I left few minor comments on Rietveld. Needed the versionadded directive, Misc/NEWS and What's New entries.

But shouldn't the new function have name "filterfalse" for parallel with itertools.filterfalse? And this would better match the style of the fnmatch module, fnmatchcase doesn't contain an underscore.

Tests for fnmatch.filter are too poor. I think we should add more tests (this is other issue).

[wm75]

Hi,
seems I had the same thoughts as you, Steven. I had started working on a patch independently yesterday, but after making my changes to fnmatch itself, I found I had too many other things to do to address unittests and doc changes to turn this into a real patch - so thank you for spending time on all of this.
I just downloaded your patch and merged it with mine because I think my version of fnmatch.py may be simpler and slightly faster (though like you I didn't run performance tests). Feel free to do whatever you like with this alternate version - it's not that there is much new in it to take credit for :)

Another thing I noted: fnmatch takes a rather unusual approach to determine whether normcase is NOP or not. It imports posixpath only to see if it is the same as os.path. That looks pretty weird and wasteful to me (especially if you are running this on Windows and are effectively importing posixpath just for fun then). I think it would be better to just check os.name instead (like pathlib does for example). I moved the corresponding check to the top of the module to make this easier to address, should that be of interest. I'm also using the result of the check in fnmatch.fnmatch because I don't see any reason why you wouldn't want to benefit from it there as well.

[serhiy-storchaka]

Your patch makes filter() returning normalized names. This may be not what is expected.

[wm75]

Does it? I thought it does so only if normalize_case is True.
Did I miss something?

[serhiy-storchaka]

Yes, it does so only if normalize_case is True. Currently fnmatch.filter() returns non-normalized names.

[stevendaprano]

I'm happy for you to change the name to filterfalse.

[wm75]

@serhiy: my bad! I just hadn't realized this behavior of the original.
With this requirement I cannot see any simpler solution than Steven's.

Some other questions though to everyone involved:

1. what do you think about "os.path is posixpath" vs just checking os.name == 'posix' as I suggested earlier?

2. speaking of missing functionality in filter:
   What do you think of a keyword argument like 'case' to both filter and filterfalse that, when True, would make these functions behave equivalently to
   [n for n in names if fnmatchcase(n, pattern)]
   The default would be False, of course. I know this would be inconsistent in terms of argument vs separate functions, but it is easy to explain and learn and without it Windows users of filter/filterfalse would really suffer from the much poorer performance due to the slow normcase call (even slower now with the new fspath protocol) even if they pass in normalized names already.

3. not directly related to this issue, but I came across it in this context:

isn't normcase in both posixpath and ntpath doing isinstance(str, bytes) checks that are redundant with os.fspath? Is this oversight or am I missing something again?

[serhiy-storchaka]

1. what do you think about "os.path is posixpath" vs just checking os.name == 'posix' as I suggested earlier?

Currently os.path is posixpath if and only if os.name == 'posix'. But this can be changed in future. I think it is safer to the current check. The posixpath module doesn't contain classes, enums or namedtuples, it's import is fast.

And it would be safer to keep the check in the function rather than move it at module level.

What do you think of a keyword argument like 'case' to both filter and filterfalse that, when True, would make these functions behave equivalently to
[n for n in names if fnmatchcase(n, pattern)]

This is a different issue. fnmatch.filter() was added for speeding up glob.glob() (see bpo-409973). I don't know whether there is a good use case for filtercase().

isn't normcase in both posixpath and ntpath doing isinstance(str, bytes) checks that are redundant with os.fspath?

Good catch! It seems to me that they are redundant. Please open a new issue for this.

[wm75]

Good catch! It seems to me that they are redundant. Please open a new issue for this.

done: bpo-30427

[wm75]

Yet another thing I just realized (sorry for being so annoying):

With os.normcase calling os.fspath in 3.6+ it is not really a NOP anymore even on posix. As a consequence, you can now do some weird things with fnmatch: in all cases, and only in these, where the module *does* call normcase you can pass in Path-like objects as the names.
This works with fnmatch.fnmatch on all platforms, never for fnmatchcase, and platform-dependently (on Windows, but not on posix platforms) for filter/filterfalse.
It's mostly that last case that worries me because it makes it easy to accidentally write code that is not platform-independent.

Also note that you can also pass a Path-like object as pat everywhere except in fnmatchcase.