[API CHANGE] Introduce pubkey and signature types

[sipa]

This introduces a new data type secp256k1_pubkey_t, which is an abstract parsed public key, convertible to/from serialized variable-size public keys.

This avoids all API problems resulting from needing to pass pubkey sizes and pointers to pubkey sizes anywhere public keys are needed.

Advantages:

• No need for secp256k1_ec_pubkey_verify, secp256k1_ec_pubkey_compress, secp256k1_ec_pubkey_decompress
• Reusing the same public key in multiple operations is faster due to avoiding decompression costs on every use.
• Less parameters to pass around.
• No need for separate secp256k1_ecdsa_verify return code to deal with invalid public keys.

Disadvantages:

• Need for API calls secp256k1_ec_pubkey_serialize and secp256k1_ec_pubkey_parse.
• Slightly slower for verification (0.4%) due to extra serialization and weak validation overhead between parsing and verifying.
• Potentially unsafe when passing an uninitialized secp256k1_pubkey_t for verification, though this is mitigated as much as possible by wiping the output secp256k1_pubkey_t any time an invalid one would be returned, and not accepting a wiped one as input.

[sipa]

An alternative to this is having a secp256k1_pubkey_t type that maintains the compressedness, and is just a serialized pubkey, but in a constant-size buffer.

This has as advantages that it does not add any significant performance overhead, and does not risk extra problems when uninitialized secp256k1_pubkey_t's are passed in (though that would still be a risky bug for other reasons), but it also would not avoid the compress/decompress/verify API calls, not offer any performance advantages for repeated pubkey use, and be a bit unintuitive for having parsing of an invalid pubkey potentially succeed.

[sipa]

@afk11 Any comments here? This significantly changes the way public keys are handled, and removes the need for the secp256k1_ec_pubkey_compress function which you added.

[sipa]

Added extra documentation.

[apoelstra]

The indentation in this function is different from all the others.

[apoelstra]

It looks like bench_recover fails:

[apoelstra@titanic secp256k1]$ ./bench_recover 
src/bench_recover.c:26: test condition failed: secp256k1_ecdsa_recover_compact(data->ctx, data->msg, data->sig, &pubkey, 1)
Aborted

[DavidEGrayson]

I was expecting @gmaxwell to point out that _t is reserved by POSIX (see #198).

[sipa]

@DavidEGrayson I thought I already answered, but can't see the post now

@gmaxwell would be right to point that out, but then we should consistently change it throughout the codebase once, not introduce multiple coding standards.

[sipa]

I've changed the implementation of secp256k1_pubkey_load/save now, using secp256k1_ge_storage where possible... and the result is around 0.15% faster now than before the pubkey_t introduction.

[sipa]

I extended the scope of this patch, and introduced a secp256k1_ecdsa_signature_t for ECDSA signatures.

[apoelstra]

Update comment for removal of siglen

[sipa]

done

[apoelstra]

Remove second line under pubkey, entry for pubkeylen, entry for compressed in this comment

[apoelstra]

Under what circumstances is this not 64? You are depending on its exact format in the case that it is, so I'm confused about what you think could change.

[sipa]

Who knows what a compiler can do with integer sizes and alignments. Maybe we want to add debug fields at some point, when VERIFY is enabled.

It's also not relying on its exact representation, only that it can be memcpy'd to and from a 64-byte array, which is faster than going through the b32 representation.

[sipa]

In any case, this probably requires some comments in the code.

[apoelstra]

Yeah, I realized that you weren't depending on the representation down below when I saw you were doing the same thing with scalar_t. I think you should add a comment saying that (assuming the data fits) the API only allows the content of pubkey->data to be a memcpy of a preexisting secp256k1_ge_storage_t, so this is fine.

[sipa]

Added several comments about the representation of pubkey and signature types.

[apoelstra]

I find it a little confusing to have two signature types, especially ones that are so similarly named. It's hard to remember which one is the public one and which is the internal.

The internal one, secp256k1_ecdsa_sig_t, consists of an (r, s) pair, and is used only by the five functions defined in ecdsa.h. If you grep for these five functions you will see that every one of them occurs in only two places: in their passthroughs in the public API, and in the unit tests.

I propose:

• Dropping secp256k1_ecdsa_sig_t and having the ecdsa.h functions take a public secp256k1_ecdsa_signature_t.
• Moving signature_load and signature_save into ecdsa_impl.h, and having them output separate r, s scalars rather than a single struct (since the ecdsa.h functions use the individual scalars and never pass around the complete type anyway).

This would avoid having separate types and remove some double conversions (like calling secp256k1_ecdsa_sig_parse immediately followed by secp256k1_ecdsa_signature_save in secp256k1_ecdsa_signature_parse_der). Also note that recid is already carried around separately from secp256k1_ecdsa_sig_t, so this type is not doing a great job of encapsulation anyway.

I don't feel super strongly about this, it's up to you.

[sipa]

I think that would be a layering violation. the toplevel module defines a data type for external use, and depends on the internal code for operations on it. The lowlevel module would end up depending on the toplevel module if it needed to be passed in the toplevel data type, resulting in a weak circular dependency.

Furthermore, both datatypes have a different use case: one is obscurity and consistency (at least wrt its size), the other is efficiency.

Moving the load and save to ecdsa feels wrong, as it makes the ecdsa module depend on external API peculiarities.

One option is to drop the internal type, and just operate on scalar_t's inside ecdsa. The other is perhaps to rename the internal types to something more obvious (secp256k1_ecdsa_signature_internal_t ?), and maybe move the recid inside it.

[apoelstra]

That's a good point.

Between the two options you listed I'd prefer to just operate on scalar_t's, since there are only two of them and they have standard names.

[afk11]

@sipa concept ACK on this, the performance boost should affect me also since presently I have to serialize my PublicKey types whenever calling secp256k1_* functions. I'll get to testing this today.

[sipa]

@apoelstra Added a commit which removes the internal secp256k1_ecdsa_sig_t.

[apoelstra]

Thanks, I'm much happier with this.

[apoelstra]

ACK