testing/fstest: test that ReadDir on file returns error, or returns 0 entries if file is empty

[bcmills]

The Plan 9 builders are consistently failing on the new check added to testing/fstest.TestFS in CL 296391

--- FAIL: TestDirFS (0.07s)
    os_test.go:2720: TestFS found errors:
        a: ReadDir of non-dir file should return an error
        b: ReadDir of non-dir file should return an error
        dir/x: ReadDir of non-dir file should return an error
FAIL
FAIL	os	9.258s

2021-03-12T02:56:04-e8b8278/plan9-arm
2021-03-12T02:30:33-e87c4bb/plan9-arm
2021-03-12T01:47:01-a607408/plan9-arm
2021-03-12T00:52:00-71a6c13/plan9-arm
2021-03-11T21:43:04-7fc638d/plan9-arm
2021-03-11T20:46:21-b3896fc/plan9-arm
2021-03-11T20:01:00-4dd9c7c/plan9-arm
2021-03-11T19:11:46-43d5f21/plan9-arm
2021-03-11T19:04:42-b0733ba/plan9-386-0intro
2021-03-11T19:04:42-b0733ba/plan9-arm
2021-03-11T19:00:06-64d323f/plan9-arm
2021-03-11T18:50:12-ae9cd12/plan9-arm
2021-03-11T18:25:50-86d6678/plan9-386-0intro
2021-03-11T18:25:50-86d6678/plan9-arm
2021-03-11T18:25:41-1853411/plan9-arm

It's not obvious to me whether this is a defect in the os implementation of ReadDir on Plan 9, or an overly-optimistic assertion in testing/fstest.TestFS.

CC @josharian @rsc @ianlancetaylor @fhs @millerresearch @0intro

[bcmills]

Marking as release-blocker for Go 1.17 until we determine whether this is a defect in testing/fstest (which would affect all platforms), or in ReadDir on plan9 (which would not necessarily need to block the release because plan9 is not a first-class port).

[bcmills]

(And if this is a defect on plan9, the test still ought to be fixed or bypassed somehow so that the builders can provide useful test coverage for everything else.)

[gopherbot]

Change https://golang.org/cl/301190 mentions this issue: Revert "testing/fstest: test that ReadDirFile on a non-dir fails"

[fhs]

It seems to be a defect in the os implementation of ReadDir on Plan 9. ReadDir succeeds on empty files:

cpu% cd src/os
cpu% /tmp/readdir testdata/hello
2021/03/12 12:37:56 readdir testdata/hello: stat buffer too short
cpu% /tmp/readdir testdata/dirfs/a
cpu% /tmp/readdir testdata/dirfs/b
cpu% /tmp/readdir testdata/dirfs/dir/x
cpu% /tmp/readdir /lib/words
2021/03/12 12:39:39 readdir /lib/words: malformed stat buffer

[millerresearch]

On Plan 9, the same read syscall is used to read both directories and regular files. There's a higher level dirread library function to read raw directory entries and parse them into Dir structures, but there's no prohibition against using that function to read from a regular file which happens to contain zero or more syntactically correct directory entries. Go's ReadDir is semantically very close to Plan 9's dirread, but go doc for the fs.ReadDirFile interface says:

... ReadDir should return an error for non-directories.

Does should in Go documentation have the same meaning as in Internet RFCs, or does it actually mean must? And if so, should (or must) the same restriction apply to os.ReadDir and os.File.ReadDir and os.Readdir and os.File.Readdir and os.File.Readdirnames? (Any others I've missed?)

[fhs]

For reference, APE's opendir does a stat at the beginning to check if it's a directory. In 9front, they changed it by removing the stat call and instead added a slash to the end of filename. Open syscall returns an error if there is a slash after a regular filename:

cpu% syscall open /lib/words 0
syscall: return: 3
cpu% syscall open /lib/words/ 0
syscall: return: -1 error: '/lib/words/' not a directory

Too bad we're dealing with a file that's already open, so we can't use this trick to avoid the stat syscall. So, depending on the meaning of "should", we may need to add an extra stat syscall in readdir, possibly only when we read nothing from the file to check whether it's an empty regular file or empty directory.

[josharian]

We had a similar conversation about FreeBSD and the cost of an extra stat at https://groups.google.com/g/golang-dev/c/rh8jwxyG1PQ.

I personally am tempted to pay the cost of the stat in each case in order to have stronger fs.FS invariants.

This and the FreeBSD case seem like a policy decision, and they should probably be decided together.

I'm going to re-open this as a place to make that decision.

cc @rsc who wrote the original "should" in the docs

[millerresearch]

In practice, applications doing a ReadDir will often have previously done a stat to check whether the file is a directory. File.Stat could cache the file type, to save the extra stat syscall in case it's followed by ReadDir. If the application uses os.Stat instead or doesn't check, ReadDir can do a stat syscall and cache the file type for subsequent calls.