[easy doc] Document os.rename() behavior on Windows when src and dst are on different filesystems

[stephan]

BPO	28356
Nosy	@pfmoore, @tjguk, @zware, @eryksun, @zooba, @rhyn0
PRs	bpo-28356: Document os.rename() behavior on Windows for differing filesystems #27376
Files	os_rename_windows.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2016-10-04.13:42:32.460>
labels = ['easy', '3.8', '3.9', '3.10', 'type-feature', 'OS-windows', 'docs']
title = '[easy doc] Document os.rename() behavior on Windows when src and dst are on different filesystems'
updated_at = <Date 2021-08-02.13:03:11.787>
user = 'https://bugs.python.org/stephan'

bugs.python.org fields:

activity = <Date 2021-08-02.13:03:11.787>
actor = 'vstinner'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation', 'Windows']
creation = <Date 2016-10-04.13:42:32.460>
creator = 'stephan'
dependencies = []
files = ['50180']
hgrepos = []
issue_num = 28356
keywords = ['patch', 'easy', 'newcomer friendly']
message_count = 13.0
messages = ['278035', '278036', '278038', '278040', '278041', '278048', '278113', '387667', '388283', '388295', '388297', '388322', '398201']
nosy_count = 9.0
nosy_names = ['paul.moore', 'tim.golden', 'docs@python', 'zach.ware', 'eryksun', 'steve.dower', 'stephan', 'scienidlex', 'rhyn0']
pr_nums = ['27376']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue28356'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[stephan]

Hi,

I am just migrating my code from python 2.7.12 to 3.5.2
on Windows and stumbled on the following difference:

In python 2.7.12:
os.rename(filepath1, filepath2)
works even if
filepath1 and filepath2 are on different drives
(or one on a local drive, the other on a network share).

In python 3.5.2 I get for the same operation:
"OSError: [WinError 17] The system cannot move the file to a different disk drive"

My question:

• is this a bug?
• if not, where is this difference mentioned in the docs?
  I did find nothing, but I think it should be mentioned,
  otherwise I assume it's a bug.

[vstinner]

On Python 3 on Windows, os.rename() is implemented as MoveFileExW() with flags=0.

The doc says: "When moving a file, the destination can be on a different file system or volume. If the destination is on another drive, you must set the MOVEFILE_COPY_ALLOWED flag in dwFlags."

https://msdn.microsoft.com/en-us/library/windows/desktop/aa365240%28v=vs.85%29.aspx?f=255&MSPPError=-2147217396

I guess that the portable fix is to try rename() or fall back on copy(src, dst) + delete(src).

--

On Python 2 on Windows, os.rename() is implemented as MoveFileW(). It seems like this function behaves as MoveFileEx() called with MOVEFILE_COPY_ALLOWED:

"A new file may be on a different file system or drive."
https://msdn.microsoft.com/en-us/library/windows/desktop/aa365239(v=vs.85).aspx

--

Should we add a flag to os.rename() to allow copy, to have a portable API?

[vstinner]

Ah, on Linux rename() also fails if src and dst are on two different filesystems: OSError: [Errno 18] Invalid cross-device link.

By the way, the Python document doesn't say anything about operation on two different filesystems :-/
https://docs.python.org/dev/library/os.html#os.rename

[eryksun]

3.3 added os.replace, which on Windows entailed a switch from calling MoveFile to MoveFileEx in order to specify the MOVEFILE_REPLACE_EXISTING flag. However, not passing the MOVEFILE_COPY_ALLOWED broke compatibility with os.rename on Windows for versions prior to 3.3. I don't know whether or not this was discussed as an intentional breaking change in order to align the behavior with POSIX rename(). The change seems reasonable to me, plus at this point I don't think much can be done other than to add a note to the docs that the behavior changed in 3.3.

[stephan]

Hi,

I tryed os.replace() as replacement for os.rename() too,
but as you said it does not work if the files are on different drives.

For now I switched to shutil.move() but I suppose its not
as performant/optimal as an "move" or "rename" directly supported
by the OS.

[eryksun]

In scanning over bpo-8828, I see no discussion of the consequences of not using MOVEFILE_COPY_ALLOWED in Antoine's patch, so it appears that this behavior change was unintentional.

For now I switched to shutil.move() but I suppose its not
as performant/optimal as an "move" or "rename" directly
supported by the OS.

Correct, the copy employed by shutil.move for a cross-volume move is not as optimized as what MoveFile uses (basically CopyFile), but unless you're moving a file that's very large it shouldn't matter, and even then without testing I don't know if it's a significant difference relative to the throughput of the disk(s) involved.

[vstinner]

In scanning over bpo-8828, I see no discussion of the consequences of not using MOVEFILE_COPY_ALLOWED in Antoine's patch, so it appears that this behavior change was unintentional.

Depending on your point of view, it *can* be seen as an enhancement.

See the documentation of MOVEFILE_COPY_ALLOWED: "If the file is
successfully copied to a different volume and the original file is
unable to be deleted, the function succeeds leaving the source file
intact." This behaviour can be seen as a source of bug. It can be
acceptable if if is expected.

Since shutil.move() already implements the copy+delete fallback, I
suggest to *not* change os.rename() but *document* the behaviour
change:

• os.rename() can fail if source and destination are on two different
  file systems
• Use shutil.move() to support move to a different directory

https://docs.python.org/dev/library/shutil.html#shutil.move already
explains well the behaviour when two different filesystems are used:
"... os.rename() is used. Otherwise, src is copied to dst using
copy_function and then remove ..."

I also suggest to mention in shutil.move() doc that the source can be
left undeleted if delete fails for some reason when copy+delete is
used.

[eryksun]

The documentation of os.rename() should mention that on Windows the "operation will fail if src and dst are on different filesystems". For POSIX, instead of "will" it says "may", but the failure is a certainty in Windows since MOVEFILE_COPY_ALLOWED isn't used.

[vstinner]

This issue is now a matter of *documenting* the Windows behavior. Is there any volunteer to propose a PR?

[scienidlex]

i have problem with python version 3.9.1

[eryksun]

T-VEy, this is a documentation-only issue for versions of Python that are under active development for bug fixes and enhancements -- 3.8, 3.9, and 3.10.

[scienidlex]

Eryk, can you please go on details on how are they using different strategies to fix bugs but still it's not enough for the time being?Because I know nothing about python and I recognized it in a minute of opening the files.I don't even have it installed on my pc.I don't even have python, not even version 2.7.

On Mon, Mar 8, 2021 at 23:48, Eryk Sun<[email protected]> wrote:
Eryk Sun <[email protected]> added the comment:

T-VEy, this is a documentation-only issue for versions of Python that are under active development for bug fixes and enhancements -- 3.8, 3.9, and 3.10.

----------
versions: +Python 3.10, Python 3.8

---

Python tracker <[email protected]>
<https://bugs.python.org/issue28356\>

---

[rhyn0]

Hi all,

This is my first issue so feedback is welcome.

Following @vstinner 's suggestions:

• os.rename() can fail if source and destination are on two different
  file systems
• Use shutil.move() to support move to a different directory

And from @eryksun :

... on Windows the "operation will fail if src and dst are on different filesystems".

Just 2 short lines:
2292,6 +2292,8 @@ features:
will fail with an :exc:`OSError` subclass in a number of cases:

On Windows, if \*dst* exists a :exc:\`FileExistsError\` is always raised.

+ The operation will fail if *src* and *dst* are on different filesystems. Use
+ :func:`shutil.move` to support moves to a different filesystem.

If nothing to change, I will make a PR soon.

[furkanonder]

@eryksun The problem seems to be solved. I think we can close the problem.