archive/tar: add FileInfoNames interface

[kolyshkin]

Note, Feb 2 2022: The current proposal is in #50102 (comment).

---

Abstract

archive/tar function FileInfoHeader does uid -> uname and gid -> name lookups,
which are not always necessary and can sometimes be problematic. A new function,
FileInfoHeaderNoNames, is proposed to address these issues.

Background

Change https://go-review.googlesource.com/59531
(which made its way to Go 1.10) implemented
user/group name lookups in tar/archive's FileInfoHeader.
It fills in tar file info header fields Uname and Gname,
looking up user and group names (from Uid and Gid)
via os/user.LookupId and LookupGroupId functions.

Doing that is not always desirable, and is sometimes problematic:

1. In a chrooted environment, /etc/passwd and /etc/group may be
   absent, or their contents may be entirely different from that of the host.

2. Failed name lookups are not currently cached, which may result in a
   considerable performance regression, caused by re-parsing of
   /etc/passwd and /etc/group for every file entry added to the tar.

3. In case of static linking against glibc, the latter wants to dlopen
   some libraries that might either be unavailable (which results in
   a panic/crash) or (in case of untrusted chroot) a malicious library
   can be substituted by a bad actor.

4. There may be a need to create a tarball without any user/group names
   (only with numeric uids/gids), akin to GNU tar's --numeric-owner option.

5. There may be a need to use custom uid -> name and gid -> name
   lookup functions.

Now, problem 2 can be mitigated by using (indirectly, via os/user Lookup{,Group}Id)
a good C library that does caching, or by caching failed lookups as well.
Problem 3 can be solved by using osusergo build tag, but it's compile unit wide,
meaning it will also affect other os/user uses, not just archive/tar.
Yet it seems impossible to solve both 2 and 3 at the same time.

As far as I know, there are no easy solutions for problems 1 and 5.

In particular, this affects Docker, which performs image unpacking by re-executing
the main binary (dockerd) in the container context (essentially a chroot). As a workaround,
Docker maintains a fork of archive/tar with commit 0564e30 partially reverted
(see moby/moby#42402).

Proposal

Add a function similar to FileInfoHeader, which does not perform any id -> name lookups,
leaving it to a user. The proposed name is FileInfoHeaderNoNames (can also be *NoLookup,
*Num, etc).

Rationale

Adding a new function seems to be the most simple and elegant approach, with very little code to add, and yet solving all the issues raised above.

Alternatives are:

• (for users) to maintain the fork of archive/tar
• (for archive/tar) to implement a build tag which disable lookups (e.g. archivetarnolookups or archivetarnumeric)
• (for archive/tar) to add a way to provide own name lookup function
• (for archive/tar) to add another way to disable lookups (so a user can do tar.NameLookup = false or tar.NameLookup(false)
• to unconditionally remove id -> name lookups from archive/tar (might bring compatibility issues)

Compatibility

Since this is a new API, and the existing functionality of FileInfoHeader is left intact,
there are no compatibility issues.

Implementation

See https://go-review.googlesource.com/c/go/+/371054 for the example code.

[kolyshkin]

Cc @thaJeztah @tianon @cpuguy83 @tonistiigi @errordeveloper

[errordeveloper]

Alternatives are:

• (for users) to maintain the fork of archive/tar

It is not quite feasible as modules don't actually allow vendoring standard library packages. That is only really possible with "more traditional" vendoring tools, some of which are not quite compatible with some features of modules either, so it is a major pain to use those.

Another alternative would be to have a new tar package, with a new import path and potentially some new functionality.

Forking and enforcing archive/tar import path is basically not really feasible as far as I can tell.

• (for archive/tar) to implement a build tag which disable lookups (e.g. archivetarnolookups or archivetarnumeric)

Is it a common pattern in the standard library to control functionality with build tags? I thought tags would more suitable for toggling optimisations or implementation variants (like the netgo tag).

• to unconditionally remove id -> name lookups from archive/tar (might bring compatibility issues)

It might makes sense to add a new function that does what is currently done, and simplify the existing function. But, yes, there is the compatibility question, however perhaps that can waived as the functionality is best effort and hereby proven problematic, so moving it into an optional secondary function might be quite reasonable.

[tonistiigi]

Just throwing out some alternatives: If we want to keep the impact to Go's public API minimal, current FileInfoHeader could do an interface check against.

interface {
  LookupUserName(int) (string, error)
  LookupGroupName(int) (string, error)
}

If fs.FileInfo implements these methods then they would be used instead and the caller can control behavior. It would remain as a very advanced case though because of its hidden nature, but I guess there is no way to move the default path away from current behavior because of the backward compatibility guarantees anyway.

[ianlancetaylor]

CC @dsnet

[errordeveloper]

It might makes sense to add a new function that does what is currently done, and simplify the existing function. But, yes, there is the compatibility question, however perhaps that can waived as the functionality is best effort and hereby proven problematic, so moving it into an optional secondary function might be quite reasonable.

I'm taking this suggestion back now, I've review this and see where compatibility issue is.

[rsc]

How often do you want all the file system-loaded info except the names?
Maybe code should clear the Sys field in the FileInfo and pass that to FileInfoHeader?

That is:

type FileInfoNoSys { fs.FileInfo }
func (FileInfoNoSys) Sys() interface{} { return nil }

blah blah blah FileInfoHeader(FileInfoNoSys(info)) 

It seems weird to carve out this one detail but leave all the other work that statUnix is doing.
Wouldn't Docker rather leave out all the extraneous system info?

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[kolyshkin]

How often do you want all the file system-loaded info except the names?

Every time the tarball is unpacked in an environment where user/group names
resolution (id -> name lookup) is problematic.

Usually, such resolution is system-dependent and can rely on local plain text
/etc/passwd and /etc/group, or some other kind of local database, or a remote
distributed system such as NIS or LDAP.

When cgo is used, Go os/user relies on libc, which implements all these methods,
as configured on the system. Which is all right, except in case of static linking against
GNU libc, as in this case glibc wants to dlopen NSS libraries (meaning the static
binary will segfault on a system where these libraries are not present). While this
is a glibc and not golang issue per se, this makes it impossible to have Go static
binaries linked with glibc (only because archive/tar wants to look up user/group
names).

Another issue with name resolution using glibc is when chroot (or pivot root) into
an untrusted directory is used (such as, chrooting/entering into a user container),
calling FileInfoHeader may result in dlopen-ing some untrusted library. This is a
major security issue.

All that can be solved by ether disabling cgo or using osusergo tag (available
since Go 1.13 or so), in which case os/user switches to native implementation of
parsing plain text /etc/passwd and /etc/group. This is also problematic because

• if ways of user/group name resolution other than /etc/passwd and /etc/group are
  configured on the system, they are not used;
• if chroot is used (see above), the program will end up reading wrong (untrusted
  and potentially malicious) /etc/passwd and /etc/group files;
• disabling cgo or using osusergo come with a price, and it can't be done selectively
  for archve/tar use only.

TL;DR: user and group name resolution is rather complicated subject, this is why
there is a need to make it optional (one way or another).

It seems weird to carve out this one detail but leave all the other work that statUnix is doing.

This was the way it worked before https://go-review.googlesource.com/59531 / Go 1.10.

The thing is, id -> name lookup is the only tricky part here, anything else in statUnix
is rather straightforward. Please see above.

[ianlancetaylor]

Thanks, but I'm not sure that answers the question. We understand the problem with resolving the names. The question is: do you need the other information extracted by the statUnix function? That is, the uid, gid, access time, change time, and device major and minor fields? Is that information valuable when unpacking a tar file in a Docker container?

[kolyshkin]

The question is: do you need the other information extracted by the statUnix function? That is, the uid, gid, access time, change time, and device major and minor fields

(sorry for being vague before)
Yes, we definitely do do. All that is needed in our use case (I'm only unsure about atime) and, I guess, in most other use cases of FileInfoHeader.

[ianlancetaylor]

Thanks.

[rsc]

Every time the tarball is unpacked in an environment where user/group names
resolution (id -> name lookup) is problematic.

Isn't this issue about packing the tar file, not unpacking it?

It is always possible to change the names in the returned Header.
Many use cases will be able to do that.

We're having a hard time coming up with a suitable way
to express 'don't do the name lookups' in a general way.

It seems like the issue here is fundamentally about glibc being broken in certain ways.
In that (fairly rare) case a copy of FileInfoHeader doesn't seem like the worst workaround.