External Signing API, or CCID/PIV Support

[awwad]

Purpose

It would be good to make it easier to manage repository metadata with TUF's repository_tool module while using specialized hardware signing mechanisms that do not pass private keys around in memory. Currently, repository_tool expects to be provided private keys that it can use to produce signatures.

Current workaround

The workaround currently involves producing metadata normally (with unchanged TUF code) through repository_tool using dummy keys, and then using external code to, without TUF, replace signatures in the produced JSON metadata, for any signatures to be made with private key data that cannot be provided to TUF. These can just be swapped into the JSON metadata that repository_tool produces. This should be easy. (Caveat: If Snapshot, specifically, has to be signed with an external key, then in addition to replacing that signature, one must also re-calculate Timestamp's hash of snapshot metadata and re-sign Timestamp. One could make TUF calls to do the Timestamp adjustment after externally re-signing Snapshot in that case.)

Options for the Future

Option 1: External Signing Support

A PR has been put forward to suggest an API defining key signers via an extension of the metadata combined with additional handler mappings for all key types, where the mapped-to value is a class that supports a signature-related functions. I think this is unfortunately a little over-engineered for the codebase of the reference implementation, requiring additional classes and metadata changes, and impairing readability. 😔

I have to do some thinking about what a good API will look like if we go the direction of adding general external signing support. I'd prefer something much thinner and less intrusive. Off the top of my head, while I don't love it, here's an example:

• a (dummy if necessary) key is provided that lists a scheme value identifying the (internally) unsupported signing scheme (or some 'custom' value)
• loading the key into TUF is permitted despite the unknown signing scheme (or, permitted if the scheme is 'custom', say)
• when signing with the "key", TUF notes the scheme is unsupported (or perhaps an expected value like 'custom' or 'external') and calls out to an external handler module with a fixed name, providing the bytes to be signed and the (potentially dummy) "key", taking the result as the signature.
• the external module can be provided by a separate dependency and need only support a one-function interface producing a signature through whatever mechanism is appropriate for the key as determined externally based on scheme.

I'm very much open to alternative suggestions. (@lukpueh?)

Option 2: TUF support for CCID / PIV

There also may be additional (to OLL) interest in this. @trishankatdatadog?
It seems to me like it would be reasonable to add some optional dependencies (similar to the PyNaCl and cryptography libraries) that could handle this and would require some non-Python dependencies. It's likely that the API for this would look like dummy private keys, similar to what I suggested above. I'm curious, though, about how in-toto handles this. (Indirectly via gpg...?) @lukpueh @SantiagoTorres

I also have some reading to do about functionality for CCID/PIV support in Notary.

[trishankatdatadog]

@awwad Heck, yes. One possibility I was looking into was using a HSM on the cloud (e.g., Azure HSM) that would sign a hash of your content, so you never see any private / signing key whatsoever. Another possibility closer to home is using a GPG signing key stored on a Yubikey.

[awwad]

@trishankatdatadog Do you think there'd be a way to get both of those through Option 2 (some standard CCID/PIV support -- is there some way to get Azure HSM to play nice with that or something)? Or, if it turns out that cloud-HSM-type things are very interesting to folks but not really accessible via the same interfaces, would you say that that suggests we should go with Option 1 (instead or in addition) and provide general external support?

[awwad]

The notion of cloud signing makes me nervous, but I'm not the client here. o.o

[trishankatdatadog]

I think Option 1 sounds like what would generalize the best, we cannot assume the hardware might be sitting on even the same machine. For an example of how this is done, see the Azure Key Vault documentation.

P.S. A few folks already use Azure Key Vault (Docker, if I'm not mistaken). You could always check that the cloud signed the same hash you gave it.

[awwad]

K. I'll keep general external signing in mind, then, in the ongoing rewrites and reading, and hope for some better designs than the open-dependency sketch above to come to mind or come from folks in this thread.

[lukpueh]

Re "how in-toto handles this":

in-toto has two types of signed metadata files, i.e.layouts (supply chain definition) and links (supply chain evidence), both of which use the same container format as TUF does, i.e.

{
  "signed": <signed payload>,
  "signatures": [<signature>, ...]
}

link metadata is usually generated by calling in_toto_run, which accepts a private key object or a gpg keyid. In the case of gpg, signing is performed in a subprocess and in-toto never sees the private key. If neither a key object nor a gpg keyid is passed to in_toto_run, the resulting metadata is not signed.
The creation of layout metadata is not part of any in-toto routine. Thus, layouts are usually created and signed manually, e.g. by using one of the API methods also used by in_toto_run (see metadata.sign and metadata.sign_gpg).

Long story short, in in-toto metadata signing is fairly decoupled, so it wouldn't need intermediary dummy keys (*) to later sign with external tools. As a matter of fact we have an in-house external signing tool that does exactly that. However, I'm all for a pluggable external signing mechanisms that can be integrated with the in-toto signing API and routines (such as in_toto_run).

--
(*) It might need intermediary dummy keys for in_toto_record_{start,stop}, but those aren't part of the core functionality.

[lukpueh]

Re "alternative suggestions": I'll look into it. :)

[trishankatdatadog]

Yeah, I'm not sure we want to expose intermediate dummy keys, that doesn't sound like the right abstraction

The right abstraction is: here's a piece of metadata ready to be signed by this pubkey / keyid, something implementing a shared interface signs it, then the caller (presumably TUF repo tool) optionally checks whether the signature matches the pubkey and the message

Does this roughly make sense?

[awwad]

I think there's a miscommunication. @trishankatdatadog So, I'm less enthusiastic about that, but I may be unclear on what you mean, and I'm not sure what I meant was clear... so let's make sure! 😀

A dummy private key wouldn't actually be part of the interface; it'd be internal to TUF's operation. You'd just tell repository_tool that you want to use an external signing key (or some integrated CCID/PIV support, either way) to sign something later.

Status Quo, Lower Level Clarification

For clarity, here's the general procedure as it will be once #846 is done (current is very similar). The dummy private key option comes in, internally, during step 2. (You can skim what's not in bold.)

[external repository mgmt code] <--> repository_tool <--> [underlying TUF code]

1. DELEGATE: repo mgmt tells repository_tool what key role1 authorizes role2 to sign with (via public key)
   • repository.<role>.add_verification_key(...) <-- interface not stable
   • (This results in an update to the appropriate "delegation" in a draft metadata role in memory -- in roledb, and in a registration of the public key under the key's keyid in keydb.)
2. QUEUE SIGS: repo mgmt later provides the private signing key to repository_tool, in anticipation of writing signed metadata. You can think of this as queuing signatures. repository_tool should complain if you don't queue signatures from enough authorized keys to meet a threshold (which it mostly does correctly).
   • repository.<role>.load_signing_key(...) <-- interface not stable
   • (This results in the private key being stored in memory in keydb.)
3. SIGN AND WRITE: repo mgmt later instructs repository_tool to sign and write the role(s)
   • repository.write() or repository.writeall()
   • (This results in signatures being calculated over metadata and full roles being written to disk, using private key information from keydb.)

Changes with and without "dummy keys"

Step 1 still has to be done, regardless of how we change things for this issue, so the public key still has to be provided to put in metadata appropriately.

As for steps 2 and 3:
You can continue to call load_signing_key() if you're using a key you can provide private key info for, and if it's a YubiKey or something, you'd instead call, e.g. add_external_signing_key(), providing the keyid or public key, and either a signing scheme we can interpret (say, CCID/PIV if we provide that support) or a module or function to call. There's no constructing dummy keys for the interface.

Internally, TUF could then add a dummy private key to keydb, and when a write command later is given to repository_tool, TUF will see that dummy private key and make the appropriate external call, sending bytes, a scheme, and a keyid or public key.

This way, repository_tool still keeps track of which keys need to sign which roles for you. For external signing, this is a very minimal TUF change. (For CCID/PIV integration, there's a project there, but there shouldn't be much change to existing code, mostly clean additions.)

We can instead restructure things to do away with step 2, and, instead, force the external repository management code to keep track of what key should sign what, and make calls to repository_tool to fetch signable metadata in bytes form and/or dictionary form, then create a signature itself without TUF calls, and call repository_tool to add a signature externally produced.

I think those two are probably the cleanest options I've yet thought of.

[danixeee]

@awwad At some point we will probably need to just load existing signature during process of signing and writing TUF metadata. Let's say we have a root metadata which needs 3 valid signatures in order to change it (threshold: 3). Each key is stored on a different YubiKey at a different location. So, we will need to send a data that needs to be signed and once we collect all signatures, we load them from files. This PR provides a different implementation from one that I initially did (for getting external signatures) and I believe it is closer to your idea from the above comments.

[JustinCappos]

This brings up an issue we had discussed with @justin Cormack <justin.cormack@docker.com> a while back. The discussion was about "partial signatures", i.e., what to do with metadata that is in the process of being signed. I'd be happy to revisit this. I think this will not break backwards compatibility with the 1.0 spec so we might get the other changes in first. Am I understanding this and what do others think about this timing?

…

On Fri, May 3, 2019 at 8:38 AM danixeee ***@***.***> wrote: @awwad <https://github.com/awwad> At some point we will probably need to just load existing signature during process of signing and writing TUF metadata. Let's say we have a root metadata which needs 3 valid signatures in order to change it (threshold: 3). Each key is stored on a different YubiKey at a different location. So, we will need to send a data that needs to be signed and once we collect all signatures, we load them from files. This PR <https://github.com/openlawlibrary/tuf/pull/6/files> provides a different implementation from one that I initially did (for getting external signatures) and I believe it is closer to your idea from the above comments. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#864 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGROD3DQXGDLWIDJXGGN2TPTQW3BANCNFSM4HH4RG2A> .

[lukpueh]

@awwad and I talked this through and agreed that an internal solution that allows external signing as part of the whole TUF metadata signing process (see repository_tool.writeall) is preferable.
The other approach of restructuring things, so that the management code has the freedom to choose whatever signing process it wishes, puts too much responsibility on the management code to sign the right thing at the right time (also Seb's #864 (comment) above for the two approaches).

@danixeee's proposition in openlawlibrary#6 and is actually pretty much what we want. It extends keydb to also store signature provider functions that can be individually defined by the management code as long as they take the data to be signed as argument and return a signature in the desired TUF format. repository_lib.sign_metadata, called in writeall, may then use the passed signing keyids to fetch either a signing private key from keydb and provide the signature itself (status quo) or pass the data to the signature provider function, also stored at a keyid in keydb, and add the returned signature.

If a keyid in the keydb holds both a signing key and a signature provider function, prioritization may be resolved by sign_metadata. But keydb could also allow only one of private key or signature provider function for a given keyid.

On a more general note, I saw that keydb does not differentiate between public and private keys. It is up to the caller to check if a queried signing key does have the private portion, or to not accidentally disclose a private key, when querying an assumed public key. It might make sense to better separate public and private keys in keydb to make such accidents less likely. Note that in the case of signature provider functions, public keys need to be stored separately anyway. What do others think?

@JustinCappos, can you point me to resources about the partial signatures discussion.

[JustinCappos]

There isn't a written discussion about this. It was a few hallway talks with @justincormack. The basic idea is we need a principled way to store partially signed metadata.

We can add more information about this and should write a TAP on this topic anyways. I don't think this is blocking this issue though, right?

[SantiagoTorres]

The basic idea is we need a principled way to store partially signed metadata.

I'm not sure I follow. Either we store the canonicalized bytestream to have an external tool to sign it or we provide this facility. AFAIU there's an intern already working on this.