net: UDPConn.SetReadBuffer unexpected behavior on BSD architectures

[sparrc]

When using UDPConn.SetReadBuffer, I expect to be able to set the buffer up to the kernel limit kern.ipc.maxsockbuf, but I am getting failures on Darwin and FreeBSD when trying to do that.

From what I can tell, this should be supported, assuming the code is doing a setsockopt on the SO_RCVBUF constant, from the bsd setsockopt man page:

SO_SNDBUF and SO_RCVBUF are options to adjust the normal buffer sizes
     allocated for output and input buffers, respectively.  The buffer size
     may be increased for high-volume connections, or may be decreased to
     limit the possible backlog of incoming data.  The system places an abso-
     lute maximum on these values, which is accessible through the sysctl(3)
     MIB variable ``kern.ipc.maxsockbuf''.

Versions

$ go version
go version go1.5.1 darwin/amd64
$ uname -srm
Darwin 15.0.0 x86_64

and FreeBSD:

$ uname -srm
FreeBSD 10.1-RELEASE amd64

What did you do?

This script fails on any BSD/Darwin architecture, where ReadBuffer is set equal to kern.ipc.maxsockbuf, error output I get is set udp [::]:55647: setsockopt: no buffer space available

// setreadbuffer.go
package main

import (
    "fmt"
    "net"
)

const (
    // Darwin Limit
    // sysctl kern.ipc.maxsockbuf -> 8388608
    ReadBuffer = 8388608 // - (911 * 1024) // (this works if you subtract 911kb)

    // FreeBSD Limit
    // sysctl kern.ipc.maxsockbuf -> 2097152
    // ReadBuffer = 2097152

    // Linux Limit
    // sysctl net.core.rmem_max -> 212992
    // ReadBuffer = 212992 // (this works on Linux)
)

func main() {
    addr, _ := net.ResolveUDPAddr("udp", ":80891")

    conn, err := net.ListenUDP("udp", addr)
    if err != nil {
        fmt.Printf("Failed to set up UDP listener at address %s: %s", ":8089", err)
        panic(err)
    }
    defer conn.Close()

    err = conn.SetReadBuffer(ReadBuffer) // THIS FAILS
    if err != nil {
        fmt.Printf("Failed to set UDP read buffer to %d: %s",
            ReadBuffer, err)
    }
}

What did you expect to see?

I would expect conn.SetReadBuffer(ReadBuffer) to succeed when ReadBuffer is less than or equal to kern.ipc.maxsockbuf

What did you see instead?

On BSD/Darwin, conn.SetReadBuffer(ReadBuffer) fails when ReadBuffer is equal to kern.ipc.maxsockbuf. From my limited testing it doesn't start passing until you subtract ~12% from the max.

[ianlancetaylor]

Can you test what happens with the equivalent C code?

The Go code really is just calling setsockopt(SO_RCVBUF, bytes); I don't know why Go would behave differently than C here.

[sparrc]

I can test but I'm not very good with C, could you provide any guidance on how I might call the C function directly?

[ianlancetaylor]

Try this. READBUFFER is the size of the read buffer being requested. By the way, in your Go example, you should check the error return of ResolveUDPAddr: ":80891" is not a valid UDP address.

#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>

#define READBUFFER 8388608

static void die(const char *s) {
  fprintf(stderr, "%s: %s\n", s, strerror(errno));
  exit(EXIT_FAILURE);
}

int main() {
  int s;
  struct sockaddr_in sin;
  int size;

  s = socket(AF_INET, SOCK_DGRAM, 0);
  if (s < 0) {
    die("socket");
  }
  memset(&sin, 0, sizeof sin);
  sin.sin_port = htons(8891);
  if (bind(s, (struct sockaddr *)(&sin), sizeof sin) < 0) {
    die("bind");
  }
  size = READBUFFER;
  if (setsockopt(s, SOL_SOCKET, SO_RCVBUF, &size, sizeof size) < 0) {
    die("setsockopt");
  }
  return 0;
}

[sparrc]

Okay, @fenderle helped me figure this out, it looks like BSD makes some sort of adjustment to the actual limit here:

https://github.com/freebsd/freebsd/blob/master/sys/kern/uipc_sockbuf.c#L385-L399

which gets triggered from here:

https://github.com/freebsd/freebsd/blob/master/sys/kern/uipc_sockbuf.c#L420

so the actual buffer limit is not 8388608, it's actually

sb_max_adj = (u_quad_t)sb_max * MCLBYTES / (MSIZE + MCLBYTES);
8388608 * (1 << 11) / (256 + (1 << 11))  =  8388608 * (2048) / (2304)  = 7456540

(where 8388608 == kern.ipc.maxsockbuf)

So I think we can close this issue....I don't think it's within Go's domain to adjust for this, BSD is weird.