pathlib.glob('**') returns only directories

[jitterman]

BPO	26115
Nosy	@pitrou

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-01-14.22:26:27.607>
created_at = <Date 2016-01-14.21:20:32.100>
labels = ['type-bug', 'invalid']
title = "pathlib.glob('**') returns only directories"
updated_at = <Date 2016-02-04.07:25:04.510>
user = 'https://bugs.python.org/jitterman'

bugs.python.org fields:

activity = <Date 2016-02-04.07:25:04.510>
actor = 'SilentGhost'
assignee = 'none'
closed = True
closed_date = <Date 2016-01-14.22:26:27.607>
closer = 'SilentGhost'
components = []
creation = <Date 2016-01-14.21:20:32.100>
creator = 'jitterman'
dependencies = []
files = []
hgrepos = []
issue_num = 26115
keywords = []
message_count = 3.0
messages = ['258223', '258237', '259537']
nosy_count = 3.0
nosy_names = ['pitrou', 'SilentGhost', 'jitterman']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue26115'
versions = ['Python 3.5']

Linked PRs

• GH-70303: Emit FutureWarning when pathlib glob pattern ends with ** #105413
• GH-70303: Make pathlib.Path.glob('**') return both files and directories #114684

[jitterman]

The title says it all.

The shell version of '*' and '**' return both directories and files.
Path('.').glob('*') returns both directories and files, but Path('.').glob('**') returns only directories. That seems wrong to me.

[SilentGhost]

It is, however, exactly what documentation says it should do:

The “**” pattern means “this directory and all subdirectories, recursively”.

[jitterman]

It may be what the documentation says it will do, but is not what it should do. I believe that because:

1. Currently ** in pathlib matches only directories, but **.py matches files. That seems inconsistent.
2. In bash, and csh, ** matches files and directories. To get the same in pathlib one must use **/*, which is inconsistent with what we have used for many decades.
3. With the traditional meaning of **, it is easy to constrain the match to directories by adding slash to the end of the glob (just use **/).
4. There is considerable value in supporting the traditional meaning of glob strings. Globbing is a very powerful feature, and it is often offered to the end user in shell-like situations. For example, sftp offers globbing. When offering globbing to the end users it is best to be consistent the globbing they are already familiar with.
5. There is no significant advantage to the difference between pathlib globbing and traditional globbing.

Globbing in pathlib is different from traditional globbing in another important way. pathlib does not distinguish between hidden files and directories and normal files and directories. There may be isolated cases where that is preferred, but generally that is not true. Indeed, the primary characteristic of being hidden is that it should not be included in globbing. One marks a file or directory to be hidden specifically to mean 'do not include this one when selecting groups of files or directories'. Once the glob string has been expanded, it is possible to filter out the hidden files and directories, but it very difficult to do if there are several levels of directories because when weeding out the matches that should not be be included you have to look for hidden items at all levels of the path.

Globbing has been available and largely unchanged for almost 50 years. I encourage you to strongly consider making pathlib globbing more consistent with what we have all grown up with.

[barneygale]

Re-opening - this is a valid issue. As the previous poster notes, pathlib's ** is incompatible with every other glob implementation out there, including Python's own glob.glob():

If recursive is true, the pattern “**” will match any files and zero or more directories, subdirectories and symbolic links to directories. If the pattern is followed by an os.sep or os.altsep then files will not match.

It's easy enough to solve - just delete these lines:

cpython/Lib/pathlib.py

Lines 1046 to 1048 in 24af451

	if pattern_parts[-1] == '**':
	# GH-70303: '**' only matches directories. Add trailing slash.
	pattern_parts.append('')

However we'll need to go through a deprecation period first. nope!

[terryjreedy]

The Path.glob says that patterns are the same as for fnmatch, except that ** is always recursive. The fnmatch doc says nothing about **. The glob dob says glob is essentially scandir+fnmatch, and that the behavior of ** depends on options selected.

If recursive is true, the pattern ** will match any files and zero or more directories, subdirectories and symbolic links to directories. If the pattern is followed by an os.sep or os.altsep then files will not match.

If include_hidden is true, ** pattern will match hidden directories.)

To me, the current Path.glob ** is a behavior that should be fixed.