dep is not working when GOPATH on Windows ReFS partition

[amsokol]

What version of Go (go version) and dep (git describe --tags) are you using?

go-1.8.3
v0.0.2

What dep command did you run?

dep init

What did you expect to see?

result os init command

What did you see instead?

any dep command never returns

The reason of problem:

I have Windows 10.
Microsoft is moving from NTFS to new ReFS file system.
ReFS is filesystem for storage.
My GOPATH is on ReFS storage.
The reason of problem is ReFS does not support hard link any more (https://docs.microsoft.com/en-us/windows-server/storage/refs/refs-overview).

That is why code from lockfile.go:
// return value intentionally ignored, as ignoring it is part of the algorithm

_ = os.Link(tmplock.Name(), name)

Is not working. os.Link returns "The request is not supported.".

That is the reason why code in source_manager.go:

err = lockfile.TryLock()
for err != nil {
	if _, ok := err.(interface {
		Temporary() bool
	}); ok {
		time.Sleep(time.Second * 1)
	} else {
		return nil, CouldNotCreateLockError{
			Path: glpath,
			Err:  fmt.Errorf("unable to lock %s: %s", glpath, err.Error()),
		}
	}
	err = lockfile.TryLock()
}

never returns from for

[ibrasho]

@matjam Any thoughts on this?

[sdboyer]

@nightlyone might have an idea, as he's the original author of the library we're relying on for this. any guidance? 😄 🙏

[matjam]

I've had some luck with alternative locking schemes. I will take a look at fixing the original package.

We might have to detect the OS and use native locking semantics. Or some other scheme. I am not sure why it actually needs to do a hardlink.

The workaround, I presume, is to not put your GOPATH on ReFS.

[nightlyone]

@sdboyer thanks for the heads up!

@matjam the algorithm indeed requires hard links to work correctly. Without the specific atomicity guarantees of setting a hard link and rename a file in a directory the whole locking beaks. I could be more friendly to the users by documenting these constraints better and being more specific about which errors I ignore.

Filed nightlyone/lockfile#20 to track this.

[nightlyone]

Please note that my lockfile library by design (project scope) also does not work in NFS scenarios and other file systems mounted on multiple hosts at the same time.

The design space of the lockfile library is to manage concurrent invocations of unrelated machine local processes manipulating data on locally to that.

If you need to serialize accesses to a potentially shared file system, such as the GOPATH, there needs to be a more advanced locking schema.

[matjam]

ReFS breaks all existing lock file semantics then, in this case. If Microsoft migrate your C: drive to ReFS I suspect lots of stuff will break. Developers have been relying on being able to use linking for lockfiles for a long time.

We can implement a workaround in dep by using the system tempdir. But it's only a workaround.

[matjam]

Looking at statements from Microsoft there is no current plans to make ReFS supported for the OS drive. So, if we move the process lock file to the system lock file area then that should be sufficient. It will need to be a "per user" lockfile.

[matjam]

@sdboyer Sam, feel free to assign this one to me, I'll get a fix for it you agree with the proposed fix.

[sdboyer]

@matjam unfortunately i can't (thanks github), but making the note here is enough - thanks!

[nightlyone]

Looking at statements from Microsoft there is no current plans to make ReFS supported for the OS drive. So, if we move the process lock file to the system lock file area then that should be sufficient. It will need to be a "per user" lockfile.

I believe you even need to provide a lockfile per GOPATH since having multiple independent GOPATHs (often one per project) is quite common and supported by the current Go tooling.

[matjam]

Yup! @sdboyer pointed that out in the slack.

actually, not too hard, really. But yeah, it would need to be per GOPATH rather than per user.

[matjam]

@amsokol can you do a "go test" of github.com/theckman/go-flock and let us know if that works for you on your filesystem?

[amsokol]

Dear @matjam ,

I have vacation right now and have no access to my PC.
I try to test next week.
I'm sorry.

[matjam]

@amsokol There is a PR created that should address this. Please test and confirm.

[matjam]

I tested this on Windows 10 with an ReFS partition. The go-flock approach appears to work.

however there are other problems in the CI build.

You might want to check out my branch and use that, if you're still using ReFS.