Parallel signing #1468
Parallel signing #1468
Conversation
This is (especially) useful with rekor 2 where the service only responds after log inclusion: We would prefer to get all signatures in the same inclusion batch. The change still affects both rekor v1 and rekor v2. This commit is a rebase of a bunch of Ramons commits. Signed-off-by: Ramon Petgrave <[email protected]> Signed-off-by: Jussi Kukkonen <[email protected]>
* Session thread safety is ambiguous: it may now be safe but situation is unclear * Use a single Session per create_entry() call: This may have a little more overhead but seems like the safest option * Note that RekorLog (v1) other methods may not be thread safe: I only reviewed the create_entry() flow Signed-off-by: Jussi Kukkonen <[email protected]>
It's a little unclear if Session is now thread safe or not: avoid reuse just in case Signed-off-by: Jussi Kukkonen <[email protected]>
* Separate the thread method so it's easier to see potential safety issues
* Using print() with the same file object is generally not thread safe: Avoid
it from the threaded method
The output remains effectively the same except:
* The b64 encoded signature is no longer printed to terminal
* Some print()s are now logger.info(): e.g.
Transparency log entry created at index: 5562
* Other print()s happen in a batch now, after he signing has finished
Signed-off-by: Jussi Kukkonen <[email protected]>
Signed-off-by: Jussi Kukkonen <[email protected]>
Signed-off-by: Jussi Kukkonen <[email protected]>
|
The included test only signs a few artifacts at a time but I have run this on more pathological cases: Runtime on current main branch
Runtime on parallel-sign branch
The time differences here are not that big but I think that's mostly because staging rekor 2 is currently configured with small and quick batches: With longer batching times the rekor2 difference will only grow |
Signed-off-by: Jussi Kukkonen <[email protected]>
woodruffw
added
component:cli
Make every RekorLog have a Session of their own by default. This means RekorClient no longer needs to manage that. Signed-off-by: Jussi Kukkonen <[email protected]>
Colleen confirmed this for me: tessera checkpoint interval is currently configured at 2 seconds in staging, this might get bumped up to 10 seconds The effect of this with the 25 artifact rekor2 test (with the current 10 thread limit) is :
The timings for this branch depend on the max number of threads: current 10 threads means 25 artifacts splits to 3 checkpoints. The takeaways to me are:
|
| cert_pem = signer._signing_cert().public_bytes(Encoding.PEM).decode() | ||
| print(cert_pem) | ||
|
|
||
| # sign in threads: this is relevant for especially Rekor v2 as otherwise we wait |
There was a problem hiding this comment.
(Small typo?)
| # sign in threads: this is relevant for especially Rekor v2 as otherwise we wait | |
| # signing threads: this is relevant for especially Rekor v2 as otherwise we wait |
|
|
||
| # sign in threads: this is relevant for especially Rekor v2 as otherwise we wait | ||
| # for log inclusion for each signature separately | ||
| with futures.ThreadPoolExecutor(max_workers=10) as executor: |
There was a problem hiding this comment.
From the CPython docs:
Changed in version 3.5: If max_workers is None or not given, it will default to the number of processors on the machine, multiplied by 5, assuming that ThreadPoolExecutor is often used to overlap I/O instead of CPU work and the number of workers should be higher than the number of workers for ProcessPoolExecutor.
Given that, I think we want to let Python pick the default here, since we expect to mostly be I/O bound (i.e. mostly waiting on the network). Thoughts?
| with futures.ThreadPoolExecutor(max_workers=10) as executor: | |
| with futures.ThreadPoolExecutor() as executor: |
This uses threads to sign multiple artifacts in parallel. The issue that is being solved here is that rekor v2 only responds to the entry request after the entry has been included in the log and the inclusion proof is available. When entry requests are done in parallel, they all get responses in the next "inclusion batch" (instead of one response per batch), speeding up multi-artifact signing by a lot. A practical example of the use case could be cpython releases where they sign ~10 artifacts.
There are multiple changes:
print()in threaded methodrequestswithurllib3#1040 we can likely get rid of this.Signer.sign_artifact()andSigner.sign_dsse()are now expected to be thread safe.