x/sys/unix: AIX IOCTL SetWinsize integer overflows

[aymanbagabas]

Go version

go version go1.24.4 darwin/arm64

Output of go env in your module/workspace:

AR='ar'
CC='clang'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='clang++'
GCCGO='gccgo'
GO111MODULE=''
GOARCH='arm64'
GOARM64='v8.0'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/Users/ayman/Library/Caches/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/Users/ayman/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/kl/jr58_v3n31g19ls79g3x4ch40000gn/T/go-build1222044133=/tmp/go-build -gno-record-gcc-switches -fno-common'
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMOD='/tmp/setwinsz/go.mod'
GOMODCACHE='/Users/ayman/.go/pkg/mod'
GONOPROXY='github.com/charmbracelet/ultraviolet,github.com/charmbracelet/bubbletea-internal,github.com/charmbracelet/lipgloss-internal'
GONOSUMDB='github.com/charmbracelet/ultraviolet,github.com/charmbracelet/bubbletea-internal,github.com/charmbracelet/lipgloss-internal'
GOOS='darwin'
GOPATH='/Users/ayman/.go'
GOPRIVATE='github.com/charmbracelet/ultraviolet,github.com/charmbracelet/bubbletea-internal,github.com/charmbracelet/lipgloss-internal'
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/nix/store/rq7irijkj3nhapmjcv9d96xgkisj55x2-go-1.24.4/share/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/Users/ayman/Library/Application Support/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/nix/store/rq7irijkj3nhapmjcv9d96xgkisj55x2-go-1.24.4/share/go/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.24.4'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

Run the following program for GOOS=aix GOARCH=ppc64

package main

import "golang.org/x/sys/unix"

func main() {
	_ = unix.IoctlSetWinsize(0, unix.TIOCSWINSZ, nil)
}

What did you see happen?

./main.go:6:30: cannot use unix.TIOCSWINSZ (untyped int constant 18446744071562622055) as int value in argument to unix.IoctlSetWinsize (overflows)

What did you expect to see?

The program to compile correctly.

[gabyhelp]

Related Issues

• x/sys/unix: IoctlGetPtmget fail to get ptsname on NetBSD #66871

Related Code Changes

• internal/syscall/unix, net: improve interface_aix.go

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[mrkfrmn]

CC @golang/runtime from owners.

[ianlancetaylor]

What is the value of TIOCSWINSZ in the C header files? It's showing up as 0xffffffff80087467, which suggests that there may be some unexpected sign extension. Or maybe that is the correct value, I don't know.

Similarly for TIOCREMOTE, TIOCSETC, and so forth.

CC @golang/aix

[ayappanec]

Looked into AIX header files.,
The value of TIOCSWINZ doesn't seems to be straight forward.

#define TIOCSWINSZ  _IOW('t', 103, struct winsize)  /* set window size */
#define _IOW(x,y,t) (IOC_IN|((sizeof(t)&IOCPARM_MASK)<<16)|(x<<8)|y)

It expands like this ((0x40000000<<1)|((sizeof(struct winsize)&0x7f)<<16)|('t'<<8)|103)

[dohrisalim]

I'll start working on this issue