errors: add ErrUnsupported

[ianlancetaylor]

UPDATE: This proposal has shifted from the original description to the one described in comments below.

• errors: add ErrUnsupported #41198 (comment)
• errors: add ErrUnsupported #41198 (comment)
• errors: add ErrUnsupported #41198 (comment)
• errors: add ErrUnsupported #41198 (comment)

Go has developed a pattern in which certain interfaces permit their implementing types to provide optional methods. Those optional methods are used if available, and otherwise a generic mechanism is used.

For example:

• io.WriteString checks whether an io.Writer has a WriteString method, and either calls it or calls Write.
• io.Copy checks the source for a WriterTo method, and then checks the destination for a ReaderFrom method.
• net/http.(*timeoutWriter).Push checks for a Push method, and returns ErrNotSupported if not found.

The io/fs proposal (#41190) proposes various other optional methods, such as ReadFile, where there is again a generic implementation if the method is not defined.

The use of WriterTo and ReaderFrom by io.Copy is awkward, because in some cases whether the implementation is available or not is only known at run time. For example, this happens for os.(*File).ReadFrom, which uses the copy_file_range system call which is only available in certain cases (see the error handling in https://golang.org/src/internal/poll/copy_file_range_linux.go). When os.(*File).ReadFrom is called, but copy_file_range is not available, the ReadFrom method falls back to a generic form of io.Copy. This loses the buffer used by io.CopyBuffer, leading to release notes like https://golang.org/doc/go1.15#os, leading in turn to awkward code and, for people who don't read the release notes, occasional surprising performance loss.

The use of optional methods in the io/fs proposal seems likely to lead to awkwardness with fs middleware, which must provide optional methods to support higher performance, but must then fall back to generic implementations with the underlying fs does not provide the method.

For any given method, it is of course possible to add a result parameter indicating whether the method is supported. However, this doesn't help existing methods. And in any case there is already a result parameter we can use: the error result.

I propose that we add a new value errors.ErrUnimplemented whose purpose is for an optional method to indicate that although the method exists at compile time, it turns out to not be available at run time. This will provide a standard well-understood mechanism for optional methods to indicate that they are not available. Callers will explicitly check for the error and, if found, fall back to the generic syntax.

In normal use this error will not be returned to the program. That will only happen if the program calls one of these methods directly, which is not the common case.

I propose that the implementation be simply the equivalent of

var ErrUnimplemented = errors.New("unimplemented operation")

Adding this error is a simple change. The only goal is to provide a common agreed upon way for methods to indicate whether they are not available at run time.

Changing ReadFrom and WriteTo and similar methods to return ErrUnimplemented in some cases will be a deeper change, as that could cause some programs that currently work to fail unexpectedly. I think the overall effect on the ecosystem would be beneficial, in avoiding problems like the one with io.CopyBuffer, but I can see reasonable counter arguments.

[darkfeline]

This seems to overlap with the existing way of testing whether a method/interface is supported:

foo, ok := bar.(Foo) // Does bar support Foo?

Will we end up in situations where callers need to do two checks for whether a method is supported (the type assertion and an ErrUnimplemented check)? I know we can forbid that by convention, but it seems like a really easy trap to fall into.

[ianlancetaylor]

This new error is intended to be used with type assertions. Type assertions test availablity at compile time, and the error tests availablity at run time. Something like io.copyBuffer would change to look like this:

	if wt, ok := src.(WriterTo); ok {
		written, err := wt.WriteTo(dst)
		if err != errors.ErrUnimplemented {
			return written, err
		}
		// WriteTo existed but was not actually implemented, carry on as though it did not exist.
	}

In other words, yes, for these cases you are expected to write two tests.

[darkfeline]

I see, this is about the implementation determining whether the method can be provided at runtime time, rather than the caller determining whether the "dynamic type" of a value implements the method at runtime. I see the value for this for existing code, but new code can use an interface type with a different implementation determined at runtime.

type Foo interface {}
type Bar interface {}
func NewFoo() Foo {
  if runtimeCheck() {
    return specialFoo{} // supports .(Bar)
  } else {
    return basicFoo{}
  }
}

Does this proposal include recommendations for the wider community on whether to use ErrUnimplemented or the above for new code?

[josharian]

Can we spell it without the Err prefix, since it is in the errors package?

[tooolbox]

The use of optional methods in the io/fs proposal seems likely to lead to awkwardness with fs middleware, which must provide optional methods to support higher performance, but must then fall back to generic implementations with the underlying fs does not provide the method.

Would it suffice for io/fs to define an ErrNotImplemented sentinel? It's not clear to me that it's a positive change to broadly sanction the concept of "methods that don't actually work at runtime".

[narqo]

(A minor suggestion) I feel Unimplemented is hard to read. Can I suggest

errros.NotImplemented

[mvdan]

The use of optional methods in the io/fs proposal seems likely to lead to awkwardness with fs middleware, which must provide optional methods to support higher performance, but must then fall back to generic implementations with the underlying fs does not provide the method.

Has any research gone into whether this can be solved at compile time, not needing to add extra run time checks which Go programs would need to add? I know it's a hard problem to solve, but I assume it's worth a try before falling back to run time checks.

[rsc]

An early version of the FS interface draft had such an error, which is maybe part of what inspired this proposal.
I removed it from the FS interface draft before publishing, because in almost all cases I could think of, it was better for a method to either (1) report a definitive failure, or (2) fall back to code that does implement the behavior.

The fact that io.Copy checks for two optional methods complicates a lot and may well have been a mistake. I would rather not make general conclusions about the awkwardness of io.Copy.

Let's look instead at fs.ReadFile, which you raised as an example. Suppose I have an fs wrapper type that adds a prefix to all the underlying calls:

type Subdir struct {
    prefix string
    fsys fs.FS
}

If that type wants to make the ReadFile method on fs available, it can already do:

func (s *Subdir) ReadFile(file string) ([]byte, error) {
    file = path.Join(s.prefix, file)
    if fsys, ok := s.fsys.(fs.ReadFileFS); ok {
        return fsys.ReadFile(file)
    }
    return fs.ReadFile(fsys, file)
}

With this proposal, the code could instead do:

func (s *Subdir) ReadFile(file string) ([]byte, error) {
    file = path.Join(s.prefix, file)
    if fsys, ok := s.fsys.(fs.ReadFileFS); ok {
        return fsys.ReadFile(file)
    }
    return errors.ErrUnimplemented
}

The last line is the only one that changed.

Compared to the first version, being able to write the second version is only slightly less work for the wrapper author, but far more work for the call sites. Now every time a method like this gets called, the caller must check for ErrUnimplemented and do something else to retry the operation a different way. That is, ErrUnimplemented is a new kind of error for Go programs, a "it didn't quite fail, you have to do more work!" error.

And it's not the case that you only have to worry about this if you've tested for an optional interface right before the call. Suppose you have code that takes a value of the concrete type *Subdir as an argument. You can see from godoc etc that there's a ReadFile method, 100% guaranteed. But now every time you call ReadFile you have to check for ErrUnimplemented.

The pattern of "handle the call one way or another" seems much better for more code than the pattern of "refuse to handle the call". It preserves the property that when an error happens, it's a real failure and not something that needs retrying.

In that sense, ErrUnimplemented is a bit like EINTR. I'm wary of introducing that as a new pattern.

[rsc]

On a much more minor point, given that package errors today exports four symbols, none of which has type error, I don't believe that "this is an error" is implied by errors.. (For example, errors.Unwrap is not an error about unwrapping.)

If we are at some point to add one or more standard error values to package errors, the names should probably continue the convention of using an Err prefix. That will be avoid readers needing to memory which symbols in package errors are and are not errors.

[rsc]

For the record, although it's mostly unrelated to this discussion, io.CopyBuffer was a mistake and should not have been added. It introduced new API for a performance optimization that could have been achieved without the new API.

The goal was to reuse a buffer across multiple Copy operations, as in:

buf := make([]byte, size)
for _, op := range ops {
    io.CopyBuffer(op.dst, op.src, buf)
}

But this could instead be done using:

w := bufio.NewWriterSize(nil, size)
for _, op := range ops {
    w.Reset(op.dst)
    io.Copy(w, op.src)
    w.Flush()
}

There was no need to add CopyBuffer to get a buffer reused across Copy operations. Nothing to be done about it now, but given that the entire API was a mistake I am not too worried about the leakiness of the abstraction.

Credit to @bcmills for helping me understand this.

[rhysh]

The result of an optional interface check is always the same for a particular value. Code can branch based off of that and know that the value won't suddenly implement or un-implement the optional method. (Consider an http.Handler that uses http.Flusher several times per response.)

What are the rules for methods that return ErrUnimplemented? If a method call on a value returns it, does that method need to return it on every subsequent call? If a method call on a value doesn't return it (maybe does a successful operation, maybe returns a different error), is the method allowed to return it on a future call?

If there were a way to construct types with methods at runtime (possibly by trimming the method set of an existing type), with static answers for "does it support this optional feature", would that address the need?

[tooolbox]

Has any research gone into whether this can be solved at compile time, not needing to add extra run time checks which Go programs would need to add? I know it's a hard problem to solve, but I assume it's worth a try before falling back to run time checks.

This is where my mind goes on this subject. It seems like the conclusion has been made that this isn't possible, but I think the space is worth exploring.

If there were a way to construct types with methods at runtime (possibly by trimming the method set of an existing type), with static answers for "does it support this optional feature", would that address the need?

I suppose if you had interface A with method set X, and wanted to wrap it with B with method set Y (superset of X) you could re-wrap it with A afterwards to ensure that the final result had only the method set of X. Then at compile time you're assured to not call methods of B which are not actually supported by the underlying A.