-
Notifications
You must be signed in to change notification settings - Fork 54
External Signing using CCID/PIV interface. #170
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
External Signing using CCID/PIV interface. #170
Conversation
Glad to see this moving along!
…On Wed, Jun 19, 2019 at 11:42 AM Tanishq Jasoria ***@***.***> wrote:
This aims to add support for hardware security modules, to generate
signatures using keys stored in the modules.
*Fixes issue #*: tuf/issues/864
<theupdateframework/python-tuf#864>
*Description of the changes being introduced by the pull request*:
1. Add PKCS11LIB to securesystemslib/settings.py. It servers as the
PATH of PKCS11 library for
cryptographic tokens
2. Add hsm.py, to support hardware security modules through the
PKCS#11 standard. This module use
PyKCS11, a python wrapper (SWIG) for PKCS#11 modules to communicate
with the cryptographic
tokens using CCID/PIV interface
3. Add hsm_keys.py, to provide a high-level API for using hardware
security modules for various
cryptographic operations
*Please verify and check that the pull request fulfills the following
requirements*:
- The code follows the Code Style Guidelines
<https://github.com/secure-systems-lab/code-style-guidelines#code-style-guidelines>
- Tests have been added for the bug fix or new feature
- Docs have been added for the bug fix or new feature
------------------------------
You can view, comment on, or merge this pull request online at:
#170
Commit Summary
- Add PKCS#11 Library path variable
- Add support for cryptographic tokens
- Add functions to establish communication with HSM
- Add methods to extract various objects from the HSM
- Add methods to genrerate and verify signatures using HSMs
- Create a high level API to perform cryptographic operations using
HSMs
- Add methods to load private key and create signatures
- Add various method for cryptographic operations using public keys
File Changes
- *A* securesystemslib/hsm.py
<https://github.com/secure-systems-lab/securesystemslib/pull/170/files#diff-0>
(448)
- *A* securesystemslib/hsm_keys.py
<https://github.com/secure-systems-lab/securesystemslib/pull/170/files#diff-1>
(213)
- *M* securesystemslib/settings.py
<https://github.com/secure-systems-lab/securesystemslib/pull/170/files#diff-2>
(3)
Patch Links:
- https://github.com/secure-systems-lab/securesystemslib/pull/170.patch
- https://github.com/secure-systems-lab/securesystemslib/pull/170.diff
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#170?email_source=notifications&email_token=AAGROD6H5QDLP3CQMN2NEKDP3JHUPA5CNFSM4HZK6ST2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4G2N364A>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAGRODYLR6UP2HMFQYB2K6DP3JHUPANCNFSM4HZK6STQ>
.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added a few suggestions inline. In addition, could you add some tests for this? That will make the Travis build pass as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the PR, @tanishqjasoria, this is impressive work! :) Don't let the many review comments discourage you, most of them should be very easy to fix.
Please make sure to add tests all of your new functionality. I also think we should make PyKCS11
optional, similar to cryptography
and pynacl
(see extras_require
in setup.py
), since we want to provide a minimal python-only securesystemslib
.
I'm hesitant whether it's a good idea to add libsofthsm2.so
to the repo, since it isn't cross-platform. I'd rather have pointers to resources that explain how to install those native dependencies in our docs, i.e.:
- http://www.swig.org (to install the python binding)
- https://github.com/opendnssec/SoftHSMv2 (for testing)
- https://github.com/OpenSC/OpenSC (for operations)
We can also explain how to then use securesystemslib.settings.PKCS11LIB
or the PYKCS11LIB
envvar to point to those dependencies.
We also need to tell Travis to install swig and SoftHSMv2 in order to run the tests.
Let me know if any of my comments are unclear.
ce7d029
to
5e64921
Compare
Thanks for the changes, @tanishqjasoria! Please ping me when you've fixed the tests and addressed all of my comments, and I'll re-review. And let me know if you need any help. |
Hi @tanishqjasoria, are you planning on finalizing this PR? It would be very much appreciated. :) |
Yeah sure, I would try to finalize this PR by this weekend! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for addressing my comments, @tanishqjasoria! Besides my new inline comments, there are a few more things that we should tackle before merging:
- Add
PyKCS11
toextras_require
insetup.py
for easy optional installation - Remove
libsofthsm2.so
- Add documentation for local testing, i.e. how to install and configure
softhsm
(may be ticketized) - Add usage documentation (may be ticketized)
Regarding your questions (on slack):
- It's okay to not have 100% coverage, we can add more tests in a follow-up PR.
- It's okay if this does not work on older Python versions. However, we should
- mention this in the usage documentation (may be ticketized),
- add version markers to
*-requirements.txt
andsetup.py
(e.g.PyKCS11==1.5.5; python_version > '<X.X>'
), - don't run HSM-related tests on those versions.
Let me know if you want do address these (or some of these) issues, otherwise I am happy to take over.
It would be good to rebase on top of master, because the recent changes enhance automated testing and coverage visualization. I do suggest though to first address the inline comments in my last review, as GitHub understandably has a hard time mapping the comments to the code after a rebase. :) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Two minor nits, but this is looking good :)
a29fa9f
to
078da10
Compare
Finally --- good job, guys!! |
setup.py
Outdated
], | ||
install_requires = ['six>=1.11.0', 'subprocess32; python_version < "3"', | ||
'python-dateutil==2.8.0', 'colorama>=0.3.9'], | ||
'python-dateutil==2.8.0', 'PyKCS11==1.5.5; python_version > "3"', 'colorama>=0.3.9'], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You're pinning here a version of PyKCS11 that is not available anymore on PyPI.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, maybe I'm missing something, but in general we should be using >=
not ==
, no?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, I forgot to change that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
BTW, you have the same version pinning in ci-requirements.txt and test-requirements.txt.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for pointing this out. The various *-requirements.txt are due for revision (see #182).
if PyKCS11.CKK[key_type] == 'CKK_RSA': | ||
mechanism = PyKCS11.RSA_PSS_Mechanism(RSA_PSS_MECH, | ||
RSA_PSS_HASH_SHA256, RSA_PSS_MGF_SHA256, RSA_PSS_SALT_LENGTH) | ||
|
||
elif PyKCS11.CKK[key_type] == 'CKK_EC' or PyKCS11.CKK[key_type] == 'CKK_ECDSA': | ||
mechanism = PyKCS11.Mechanism(PyKCS11.CKM_ECDSA) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The mechanism to be used should be negotiated with the slot to ensure that a supported one (provided by PyKCS11Lib.getMechanismList) gets used. Mine, for example, doesn't support any of the PSS mechanisms.
I'd rather provide a sorted list (being the first the preferred mechanism) and automatically select the first matching one.
BTW this is exactly the same code as in lines 384-389, right? I'd rather extract that code to a function, not only for code economy, but in order to ensure that the same negotiation algorithm gets used.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To get started with the work, We kept it simple at first (worked on the issue according to the mechanisms available with the YubiKey). I can make the required changes to get it supported with various HSM available.
How would we decide the list of mechanisms to be made available?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's okay to restrict the list of supported mechanisms. Ideally we do this via some module level constants or setting. For other signing algorithms/schemes in this library we make the information available, e.g. to the verifier, via the key.
securesystemslib/hsm.py
Outdated
# RSA-PSS with SHA256 hash to be used for signature generation. | ||
RSA_PSS_MECH = PyKCS11.CKM_SHA256_RSA_PKCS_PSS | ||
# SHA256 hash to be used to digest the data. | ||
RSA_PSS_HASH_SHA256 = PyKCS11.CKM_SHA256 | ||
# Mask generating function for SHA256 Hash. | ||
RSA_PSS_MGF_SHA256 = PyKCS11.CKG_MGF1_SHA256 | ||
# Length of salt to be used for hashing. | ||
RSA_PSS_SALT_LENGTH = 32 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Due to various mechanisms available, I just declared the mechanism at the start for consistency within various operations.
5bb604b
to
f7a3bbb
Compare
Hi @tanishqjasoria, you have a couple of unnecessary merge commits in your branch. The preferred way to sync a feature branch with upstream master is to regularly rebase on master (instead of merging master into the feature branch). I cleaned up your commit history by cherry-picking your relevant commits on top of the up-to-date upstream master (resolving many conflicts). You can replace your branch with my cleaned up version using these commands (try to understand what they are doing): # Fetch my version of your branch
git fetch [email protected]:lukpueh/securesystemslib.git hsm_support_signing-clean:hsm_support_signing-clean
# Go to your branch (if you're not already on it)
git checkout hsm_support_signing
# You can diff yours with mine to see that mine does not remove any of your changes
# You should only see additions that come from syncing with upstream master
git diff hsm_support_signing hsm_support_signing-clean
# Replace yours with mine
git reset --hard hsm_support_signing-clean
# Force push to update the branch on GitHub
# (you might have to change "origin" to whatever points to [email protected]:tanishqjasoria/securesystemslib.git)
git push origin hsm_support_signing -f In case you are interested how I cleaned up your history, here's what I did. You don't need to do this, if you used above commands, but it helps to know these commands. # Note down the relevant commit hashes from your PR
# `--no-merges` shows the log without merge commits
# `--first-parent` shows the log without commits that you merged in
git log --oneline --no-merges --first-parent master..hsm_support_signing
# Create a new branch on top of master
# (make sure that master is the laster version of upstream/master !!)
git checkout -b hsm_support_signing-clean && git reset --hard master
# Now use `git cherry-pick <commit ID>` for all commits that you noted above,
# starting at the bottom and going to the top.
# This is actually the hard part, because it requires fixing quite a few conflicts, due
# to upstream master having moved on, which in turn requires knowledge of the
# changes in upstream master.
For future references, please try to avoid merge commits on your feature branch! :) |
f7a3bbb
to
17e4f08
Compare
Thanks a lot for your help @lukpueh! |
In Issue #179 we're working to ensure that any functionality with a native dependency presents meaningful user feedback when the native dependency is not available. It would be great if this PR could implement functionality similar to #200 to ensure the |
Also, how do we test that this works? Docker has tests that somehow tests Yubikeys, not sure how they do it, maybe using emulators. |
""" | ||
|
||
# Refresh the list of available slots for HSM | ||
self.refresh() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a more general design question: what is the intended lifetime behavior of HSM
objects?
My intuition as a consumer of this library is that the list of available HSMs available for use in an HSM
instance is fixed at initialization time, but calling refresh
here makes it possible that the list has changed between object creation and use. If the intended behavior is that any operation that needs to access HSMs always operates on a refreshed list, it might make sense to turn refresh
into a decorator and explicitly document that as the behavior.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The intended behaviour of the HSM
object is something like, get the list of available HSM -> establish a session with the required HSM -> do the intended job or jobs -> close the session.
It's advisable to refresh the list before doing a series of operations as SlotID may change on removal and reinsertion of the HSM.
In hsm_keys
, load_HSM
calls refresh
and get the list of available HSMs to the user. The user chooses from the list of slot available and then pass it as the argument HSM_info
to various functions. Adding refresh
as a decorator won't give the user the functionality to choose from the available HSMs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would it be feasible to make initialization and session handling transparent to the user of the interface? You already do this to some extent, i.e. by each time opening the session in load_public_keys
and load_private_keys
. Similarly you always call login
before loading private keys.
|
||
|
||
|
||
def logout(self): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: Do logout
and close_session
form part of the public API, or are they just abstracted here for calling in close
? If the latter, it might make sense to document them as private and prefix them with underscores (e.g. _logout
and _close_session
) to discourage misuse.
It would be nice if NYU could get one to test it. I only have a YubiKey 5 nano, but haven't tested it with this PR yet. |
I have tested this PR using the Yubikey NEO. |
Let me test this PR with a YubiHSM we have around here. NYU should get one of their own, it's not very expensive (~$650). |
5aa0651
to
d479a3b
Compare
d479a3b
to
4ca1d2e
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very cool stuff, @tanishqjasoria!! I just started playing around with your interface and my YubiKey. Here are a few very superficial things I noticed. More feedback to come ...
""" | ||
<Purpose> | ||
To load a custom library to interact with the hardware tokens. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be nice to have a description of the PKCS11LIB_USER
argument here in the docstring.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see the HSM
class has more info. Maybe you can transfer/copy some of it to the public facing function, and flesh it out a bit.
Note to self: test code with YubiHSM sooner than later... |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for all the hard work you put into this PR, @tanishqjasoria. I finally had the time to play around with your code, using my YubiKey 5 nano. And I did manage to create/verify signatures with keys on the PIV certificate slots. 🎉 💯 😄
Here is my high-level feedback (more details inline):
Usage docs
My Yubikey didn't make this too easy for me, so it might be a good idea to put together some usage docs. We can ticketize this though. Until then, for anyone who wants to try this, use these keys creation instructions and don't forget to add a certificate. Apparently the pkcs11 interface does not find keys on the Yubikey if they don't have a certificate attached. Furthermore, the latests opensc-pkcs11.so
from the OpenSC project did not work for me, instead I used ykcs11
, which is available via the yubico-piv-tool.
(Btw. both of these rather relevant pieces of information are hidden in Yubico's "Using PIV for SSH through PKCS #11" doc.)
Architectural decisions
I appreciate your motivation to keep the high-level user interface (hsm_keys.py
) separated from the low-level code (hsm.py
) that interacts with PyKCS11
. But I still think the code would be easier to understand, maintain, and even refactor (e.g. if we replaced PyKCS11
with python-pkcs11
) without those two levels of abstraction. Right now I have to keep track of the global smartcard
in hsm_keys.py
and also of the attributes (most notably session
) inside the HSM
container pointed to by smartcard
. Maybe we can simplify this?
Tests
Removing one level of abstraction will also make testing easier. Instead of testing two create_signature
functions you only have to test one. In return you can focus on actually testing their functionality (see inline comments).
Let me know if my comments make sense.
# To carry out the tests even when the hardware token is not connected, | ||
# we would be emulating the hardware token using softHSM 2.0. | ||
# To carry out all the tests, SoftHSM needs to be initialized and | ||
# RSD, ECDSA key pairs must be generated on the SoftHSM. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
s/RSD/RSA ?
|
||
install: | ||
- pip install -U tox coveralls | ||
- wget https://dist.opendnssec.org/source/softhsm-2.3.0.tar.gz |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ping
@@ -1,5 +1,6 @@ | |||
# test runtime dependencies (see 'tests_require' field in setup.py) | |||
mock; python_version < "3.3" | |||
PyKCS11>=1.5.5; python_version > '3' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
#209 shuffled the requirements files quite a bit, so there will be a few conflicts on a (necessary) rebase. I can help you with that if you want.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I already did a rebase and resolved the conflicts.
""" | ||
try: | ||
self.session = self.PKCS11.openSession(slot_info['slot_id'], | ||
PyKCS11.CKF_SERIAL_SESSION | PyKCS11.CKF_RW_SESSION) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we generally need write access. Probably only to create test keys.
self.session.login(user_pin) | ||
except PyKCS11.PyKCS11Error as error: | ||
if PyKCS11.CKR[error.value] == "CKR_USER_ALREADY_LOGGED_IN": | ||
logger.warning('Already logged in as CKU_USER.') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please change to info or debug level log message. I unexpectedly see this message when calling load_private_keys
more than once.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 to this.
self.SMARTCARD.login(_USER_PIN) | ||
|
||
self.SMARTCARD.logout() | ||
self.SMARTCARD.close_session() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same as above in test_close_session
""" | ||
|
||
# Refresh the list of available slots for HSM | ||
self.refresh() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would it be feasible to make initialization and session handling transparent to the user of the interface? You already do this to some extent, i.e. by each time opening the session in load_public_keys
and load_private_keys
. Similarly you always call login
before loading private keys.
if PyKCS11.CKK[key_type] == 'CKK_RSA': | ||
mechanism = PyKCS11.RSA_PSS_Mechanism(RSA_PSS_MECH, | ||
RSA_PSS_HASH_SHA256, RSA_PSS_MGF_SHA256, RSA_PSS_SALT_LENGTH) | ||
|
||
elif PyKCS11.CKK[key_type] == 'CKK_EC' or PyKCS11.CKK[key_type] == 'CKK_ECDSA': | ||
mechanism = PyKCS11.Mechanism(PyKCS11.CKM_ECDSA) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's okay to restrict the list of supported mechanisms. Ideally we do this via some module level constants or setting. For other signing algorithms/schemes in this library we make the information available, e.g. to the verifier, via the key.
except: | ||
logger.debug('The public key object does not contain the DER encoded value' | ||
'It needs to be calculated from the Modulus and Exponent.' | ||
'But this functionality is not yet available!') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same as in test_hsm.test_get_public_key_value
""" | ||
|
||
# Use the HSM with the corresponding 'slot_info' | ||
smartcard.get_HSM_session(HSM_info) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you call this function before load_library
you will get NameError: name 'smartcard' is not defined
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same comment applies to other functions that call smartcard
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, there is a particular flow that needs to be followed while using this.
It should give a more meaningful error to the user, I would change that.
@lukpueh I agree on the change of the architecture. We should remove the two levels of abstraction. Apart from all the other benefits, it would make the interaction with HSM more transparent to the user. |
Notes from an out-of-band conversation with @tanishqjasoria about the architecture: The API should look similar to securesystemslib.functions.gpg , that is:
In addition we will need a public function that lists all the available keys from all the available hsms that function should give the user the necessary hsm/key identifiers required by We might want to check how costly (computationally, memory-wise) these operations are, but I guess it’s feasible and definitely makes the interaction a lot nicer for the user. Similarly, loading the I picture something like this: # in module securesystemslib.hsm
import PyKCS11
PKCS11 = PyKCS11.PyKCS11Lib()
# Per default we try loading from PYKCS11LIB environment variable
try:
PKCS11.load()
except PyKCS11.PyKCS11Error as e:
# TODO: maybe add some more debug info
log.debug(e)
def load_pkcs11_library(path=None):
# TODO: Wrap in try/except and add nice error message/user feedback,
# and re-raise as suited exception
PKCS11.load(path)
def create_signature():
# TODO: figure out a way to tell that PKCS11 has not been loaded
if not PKCS11.loaded:
# TODO: Raise suited exception informing user about the PYKCS11LIB envvar
# or the option to use `load_pkcs11_library`
raise
# TODO:
# open session
# login
# sign
# logout
# close session
# TODO: add other functions |
Looks great to me, thanks, Lukas and @tanishqjasoria! |
# For all the available HSMs available, add relevant information | ||
# to the slots dictionary | ||
for slot in slot_list: | ||
slot_dict = dict() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
More of a style nit, but literal construction is a little easier to read and less repetitive:
slot_dict = {
'slot_id': slot,
'slot_info': self.PKCS11.getSlotInfo(slot),
# ...
}
slot_info_list.append(slot_dict)
Superseded by #229. Thanks for the solid groundwork, @tanishqjasoria! Closing here... |
This aims to add support for hardware security modules, to generate signatures using keys stored in the modules.
Fixes issue #: tuf/issues/864
Description of the changes being introduced by the pull request:
cryptographic tokens
PyKCS11, a python wrapper (SWIG) for PKCS#11 modules to communicate with the cryptographic
tokens using CCID/PIV interface
cryptographic operations
Please verify and check that the pull request fulfills the following
requirements: