Write::write_all erroring when encountering Ok(0) interacts poorly with the contract of Write::write

[marshallpierce]

The documentation of write (emphasis added) specifies that returning Ok(0) is a valid thing to do, albeit with a possible implementation-specific "this writer is dead" meaning:

A return value of 0 typically means that the underlying object is no longer able to accept bytes and will likely not be able to in the future as well, or that the buffer provided is empty.

Despite the above, Write::write_all's current implementation returns an error when an underlying write returns Ok(0), effectively making this a forbidden return value for implementations of Write. However, in cases where a Write is transforming or buffering data, it can be useful to return Ok(0) to avoid violating this other, more firmly worded, requirement of write:

A call to write represents at most one attempt to write to any wrapped object.

I came across this issue while working on a streaming base64 encoder Write for the base64 crate, and subsequently found other cases where these features have clashed (hat tip to jebrosen on IRC).

In the case of base64, suppose a user constructs a base64 writer with an internal buffer that delegates to, say, stdout. The user calls write with a 6-byte input, which is base64 encoded into 8 bytes in the internal buffer, and subsequently written to the delegate (stdout). The delegate writes 5 bytes, and the remaining 3 bytes are kept in the buffer. On the next call to write from the user, I want to consume no input (and therefore return Ok(0)), but rather simply try again to write the remaining 3 bytes to the delegate. (Even with a different implementation that does consume as much input as can be encoded and still fit within the buffer, eventually that buffer will fill, and we're back to returning Ok(0).)

The same issue affects flate2:

// miniz isn't guaranteed to actually write any of the buffer provided,
// it may be in a flushing mode where it's just giving us data before
// we're actually giving it any data. We don't want to spuriously return
// `Ok(0)` when possible as it will cause calls to write_all() to fail.
// As a result we execute this in a loop to ensure that we try our
// darndest to write the data.

That violates the "at most one write" part of write, but it could be avoided if returning Ok(0) wasn't going to break write_all.

BufWriter in the std lib has a similar problem:

impl<W: Write> Write for BufWriter<W> {
    fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
        if self.buf.len() + buf.len() > self.buf.capacity() {
            self.flush_buf()?;
        }
        if buf.len() >= self.buf.capacity() {
            self.panicked = true;
            let r = self.inner.as_mut().unwrap().write(buf);
            self.panicked = false;
            r
        } else {
            Write::write(&mut self.buf, buf)
        }
    }
...

If BufWriter::write is called when its buffer is nonempty with an input that won't fit in the buffer, the second if leads to two calls to the delegate writer for one BufWriter::write call, which violates the contract of write. Similarly, returning Ok(0) there would solve the issue: the caller could then retry with the same buffer, which would be passed to the delegate writer untouched.

No doubt there are complexities I don't see yet, but given that write_all's documentation doesn't say anything about erroring on Ok(0), is it feasible to simply continue looping instead of erroring on Ok(0)? If the concept of "writer exhaustion" the "typically..." clause in write's docs is referring to needs to be represented, could that not be an ErrorKind?

[SimonSapin]

@rust-lang/libs, any thoughts?

[sfackler]

A call to write represents at most one attempt to write to any wrapped object.

This requirement seems at best annoying and at worst unenforceable. For example, I have no idea if OpenSSL's SSL_write behaves this way not. I could in theory add some kind of buffering layer between OpenSSL and the downstream Write implementation but that really does not seem worth it. I think I'd probably advocate for removing that language in favor of something that's a bit softer.

[marshallpierce]

AFAICT, the at-most-one requirement is necessary for this other part to be true in practice, though:

If an error is returned then no bytes in the buffer were written to this writer.

And that's a pretty useful property.

[sfackler]

That is also not a thing that you can guarantee when e.g. wrapping third party libraries.

[marshallpierce]

I agree; I just think it's a good thing to strive for. But either way, I don't think we have to cross that bridge if we change the (undocumented) behavior that write_all bails on Ok(0).

[sfackler]

That will break the use of write_all with all writers which return Ok(0).

[alexcrichton]

IIRC handling Ok(0) was intended to handle some weird cases of pipes on some OS somewhere where a write to a closed pipe/socket didn't return an error but rather returned Ok(0). The "at most one write" language I think should be toned down to a general guideline, as it's basically just saying "please don't make write equivalent to write_all". This is already the case in flate2 where write only goes until something succeeds. It's a special case because of a wonky C library API and just the requirements of the protocol it's implementing.

[marshallpierce]

I'm not sure I follow -- what would the middle ground be between "one write" and "like write_all", and how would that obviate the need to return Ok(0) when some intermediate buffer can't be written fast enough yet it's not an error?

[alexcrichton]

To me the middle ground is "if you can, make sure write isn't the same as write_all", and that's basically it. There's not really a guarantee we can provide here, it's just a suggestion for implementors which is intended to make their lives easier in the majority of situations.