crypto/hkdf: add package

[qmuntal]

Important

Nov 20, 2024: The latest version of the proposal is here

I propose to move the golang.org/x/crypto/hkdf package into the standard library with the name crypto/hkdf. golang.org/x/crypto/hkdf would then be updated to just be a wrapper around crypto/hkdf.

I acknowledge that depending on golang.org/x/crypto/hkdf or crypto/hkdf doesn't make much difference in terms of usability, either the x/crypto package and the standard library promise backwards compatibility and are respected by the Go community. Yet, doing this move will bring two main benefits for the Go standard library and for the niche of users requiring FIPS 140 compliance:

• crypto/tls uses golang.org/x/crypto/hkdf to implement TLS 1.3, and there are some Go forks out there that either already provide FIPS compliant TLS 1.3 via OpenSSL, or plan to do so in the near term. I suppose that Google will eventually also provide it, but this is out of this proposal. Adding this support would be much easier if crypto/hkdf was part of the standard library, in which case it could be patched to forward calls to crypto/internal/boring as needed.
• hkdf is widely used outside the standard library. Users depending on it that also require FIPS 140 compliance will benefit from having it in the standard library as a package backed by BoringCrypto/OpenSSL/CNG, etc.

Worth noting that golang.org/x/crypto/hkdf API has remained the same for more than 5 years, and that its git log only contains 4 commits since it was added in 2014. It seems to already be in good shape, so I don't expect that moving it to the standard library would require much additional maintenance effort.

For completeness, this is the current golang.org/x/crypto/hkdf API that I'm proposing to add to the standard library:

// Expand returns a Reader, from which keys can be read, using the given
// pseudorandom key and optional context info, skipping the extraction step.
//
// The pseudorandomKey should have been generated by Extract, or be a uniformly
// random or pseudorandom cryptographically strong key. See RFC 5869, Section
// 3.3. Most common scenarios will want to use New instead.
func Expand(hash func() hash.Hash, pseudorandomKey, info []byte) io.Reader

// Extract generates a pseudorandom key for use with Expand from an input
// secret and an optional independent salt.
//
// Only use this function if you need to reuse the extracted key with multiple
// Expand invocations and different context values. Most common scenarios,
// including the generation of multiple keys, should use New instead.
func Extract(hash func() hash.Hash, secret, salt []byte) []byte

// New returns a Reader, from which keys can be read, using the given hash,
// secret, salt and context info. Salt and info can be nil.
func New(hash func() hash.Hash, secret, salt, info []byte) io.Reader

Related to #65269.

@golang/security

[qmuntal]

@rolandshoemaker @FiloSottile

[qmuntal]

I've been thinking a bit more about this. The x/crypto/hkdf API might be over-complicated given that most of hkdf.New and hkdf.Expand uses ends up with a single Read or io.ReadFull call.

A simpler and easier to use API could look like this:

// Expand derives a key from the given hash, pseudorandomKey, and optional context info,
// returning a []byte of length keyLen that can be used as cryptographic key.
// The extraction step is skipped.
//
// The pseudorandomKey should have been generated by Extract, or be a uniformly
// random or pseudorandom cryptographically strong key. See RFC 5869, Section
// 3.3. Most common scenarios will want to use Key instead.
func Expand(hash func() hash.Hash, pseudorandomKey, info []byte, keyLen int) ([]byte, error)

// Extract generates a pseudorandom key for use with Expand from an input
// secret and an optional independent salt.
//
// Only use this function if you need to reuse the extracted key with multiple
// Expand invocations and different context values. Most common scenarios,
// including the generation of multiple keys, should use Key instead.
func Extract(hash func() hash.Hash, secret, salt []byte) []byte

// Key derives a key from the given hash, secret, salt and context info,
// returning a []byte of length keyLen that can be used as cryptographic key.
// Salt and info can be nil.
func Key(hash func() hash.Hash, secret, salt, info []byte, keyLen int) ([]byte, error)

[ericlagergren]

@qmuntal Getting rid of the io.Reader is a good change, but not being able to generate a key into an existing slice is rather unfortunate.

[qmuntal]

Getting rid of the io.Reader is a good change, but not being able to generate a key into an existing slice is rather unfortunate.

I proposed the keyLen instead of passing am existing slice for symmetry with other kdf packages in x/crypto, but passing an slice instead also works for me.

[magical]

I was going to suggest using the mid-stack inlining trick to let us use the nice API (accept keyLen, return []byte) while avoiding the allocation, but it appears that that only works when the slice length is a const so it doesn't help in this case.

[qmuntal]

but it appears that that only works when the slice length is a const so it doesn't help in this case.

Yep, this is a known long-standing limitation of the compiler: #29095.

@rolandshoemaker @FiloSottile what do your think about this API proposal: #61477 (comment)

[FiloSottile]

Re: moving the package, we'll definitely want to do that in Go 1.24 for #69536, although I can't promise we'll want to expand the Go+BoringCrypto mode to cover it.

On removing the io.Reader I do like the keyLen API better, but I am worried about

1. leaving users behind by making the transition harder, and
2. not being able to implement x/crypto/hkdf in terms of crypto/hkdf.

(2) is a problem because we'd like to make moved x/crypto packages simple wrappers around the standard library ones.

(1) depends on how applications have been using HKDF. Have they taken to use it as a XOF? As you said, the API has been unchanged for many years...

A compromise could be to leave New and Expand as-is but add Key as the high-level helper. Do applications generally use New (and hence would be able to migrate to Key) or Expand? We could also have ExpandKey but that's starting to feel crowded.

The allocation can actually be largely mitigated by doing something like the following. I haven't tested but I am confident that some variation will work.

func Key(hash func() hash.Hash, secret, salt, info []byte, keyLen int) ([]byte, error) {
    b := make([]byte, 0, 128) // outlined allocation for keyLen <= 128
    return appendKey(b, hash, secret, salt, info, keyLen)
}

While we are making changes, I think I'd like to make info and maybe salt into strings. salt is largely misunderstood, and in many if not most cases it should be a fixed string, but sometimes like in TLS 1.3 it's used for high-entropy values. info is almost always ASCII AFAIK. Making them different types would also help against swapping them.

[mateusz834]

Do we still want to return an io.Reader, why not a concrete exported type like *hkdf.Reader?