From 3b14eaa89af0a00bd2f21ca80c7b9feb7a165bfb Mon Sep 17 00:00:00 2001 From: Leo Date: Fri, 19 Jan 2024 10:14:48 -0600 Subject: [PATCH 1/5] Botan Cryptography Community Project Extension initial commit --- proposals/053-cryptography-extension.md | 81 +++++++++++++++++++++++++ 1 file changed, 81 insertions(+) create mode 100644 proposals/053-cryptography-extension.md diff --git a/proposals/053-cryptography-extension.md b/proposals/053-cryptography-extension.md new file mode 100644 index 0000000..a004011 --- /dev/null +++ b/proposals/053-cryptography-extension.md @@ -0,0 +1,81 @@ +# Botan Cryptography Community Project Extension + +> NOTE: This document references an earlier proposal that is ongoing and close to meeting its goals. Language in this document may reflect a future perspective that assumes those goals have been met. + +# Abstract + +The Botan Cryptography Community Project is reaching the end of its funding on Jan 31, 2024. This proposal is for the extension of funding to and addition of new goals for the [Botan Cryptography Community Project](https://github.com/haskellfoundation/tech-proposals/pull/57), which is a previously funded proposal that is ongoing, and nearing completion of its initial goals. These new goals are for the continued refinement of the produced libraries, completion of the X509 C++ work, and many other small features and minutia. + +# Background + +The [Botan Cryptography Community Project](https://github.com/apotheca/botan) is a project that was previously funded by the Haskell Foundation for the purpose of developing a suite of Haskell cryptography libraries and tooling suitable a wide range of uses including data integrity, privacy, security, and networking. + +This project has maintained a [devlog](https://discourse.haskell.org/t/botan-bindings-devlog/6855) and produced several monthly reports [(1)](https://discourse.haskell.org/t/botan-cryptography-monthly-status-report-0/8280) [(2)](https://discourse.haskell.org/t/botan-cryptography-monthly-status-report-1/8497) + +The project has produced 3 libraries: `botan-bindings`, `botan-low`, and `botan`. + +This project also has made a substantial amount of progress towards one of its stretch goals of extending the Botan C FFI to include full X509 support. We were initially optimistic that this stretch goal might be completed by the end of initial funding, but the C++ work required more overhead than expected, and efforts towards it were paused to avoid endangering any required deliverables. + +# Technical Content + +Despite (being on pace for) fulfilling its goals, there is still more work to do, as the scope of the original proposal was deliberately restricted for the purposes of focus and achievability. Now that the original goals have been met, some of the previously out-of-scope goals may now be reconsidered. + +## Continued improvements to `botan` + +- `botan-bindings` and `botan-low` are effectively frozen, but small changes may still occur +- `botan` is likely to see significant churn as we continue to improve and respond to feedback + +As `botan` is a very "fresh" library, it is expected to see some amount of churn in its API as it is developed and refined further. The initial goals of functionality are nominally complete, but there is still a great amount of minutia to be performed that is concerned with improving the safety and user experience. Expected improvements include limited lifespan objects with guaranteed cleanup, auto-zeroing of sensitive memory, and implementing instances from libraries such as `random`. + +## Botan C++ development + +- `Botan C++` is being actively developed, so project will need continued maintenance to achieve parity +- Several valuable features in `Botan C++` are not exposed by the C FFI, and require C / C++ development in order to expose them + +One of the previous stretch goals was to develop the C bindings for X509 certificate architecture, and to [contribute that code back](https://github.com/randombit/botan/issues/3627) to `Botan C++`. Substantial progress was made, but further work is required in order to complete this. + +Another feature missing from the Botan C FFI is Stream Ciphers, and like the X509 bindings, it will require effort to extend the Botan C++ library in order to expose it to Haskell. + +A third feature missing from the Botan C FFI is TLS, as Botan provides a full TLS server implementation, which will also require effort to expose to the FFI. + +## Development of cryptographic abstractions + +There are many 'class-y' improvements that could be made, abstractions that `botan` could implement instances for better ergonomics and better management of keys, nonces, and other cryptographic minutia. While many of these abstractions are not strictly necessary for `botan`, they do hold value, and investigations into the development of an abstract hierarchy and how it might be applied to botan and other cryptographic libraries is ongoing. + +# Goals + +The following are the immediate goals / deliverables of this proposal: + + - To continue to refine the `botan` libraries + - To complete the X509 C / C++ FFI and Haskell bindings + - To begin development of the Stream Cipher C / C++ FFI and Haskell bindings + +The following are intermediate deliverables: + + - Frequent activity update on devlog + - Weekly major update that includes github push + - Monthly report that summarizes what specific items have been or will be worked on, and what challenges have arisen, if any. + +The following are stretch goals / optional deliverables: + + - To complete development of the Stream Cipher C / C++ FFI and Haskell bindings + - To develop the full TLS server C / C++ FFI and Haskell bindings + - To investigate integrating `botan` into libraries such `x509`, `tls` + - To develop abstract classes / hierarchy for cryptography and cryptosystems + - To develop tutorials for integrating with other libraries such as `servant` + +# Timeline and Budget + +We propose a timeline of 3 months for these new goals, with a budget of $7000 USD per month for one full-time engineer, the same as the original proposal. + +# Stakeholders + +The primary stakeholders of this proposal are: + +- Leon Dillinger (Myself) +- Haskell Foundation +- Haskell Cryptography Group + +# Success + +This proposal will be considered a success if the immediate and intermediate deliverables are met, and at least one of the stretch goals has seen progress. From 5c712d9f9e428d9a25d19effce02668490aada4b Mon Sep 17 00:00:00 2001 From: Leo Date: Fri, 16 Feb 2024 11:19:54 -0600 Subject: [PATCH 2/5] Updated botan round 2 proposal --- proposals/053-cryptography-extension.md | 81 ++++++++++++++++++------- 1 file changed, 60 insertions(+), 21 deletions(-) diff --git a/proposals/053-cryptography-extension.md b/proposals/053-cryptography-extension.md index a004011..b2ed37f 100644 --- a/proposals/053-cryptography-extension.md +++ b/proposals/053-cryptography-extension.md @@ -4,33 +4,59 @@ # Abstract -The Botan Cryptography Community Project is reaching the end of its funding on Jan 31, 2024. This proposal is for the extension of funding to and addition of new goals for the [Botan Cryptography Community Project](https://github.com/haskellfoundation/tech-proposals/pull/57), which is a previously funded proposal that is ongoing, and nearing completion of its initial goals. These new goals are for the continued refinement of the produced libraries, completion of the X509 C++ work, and many other small features and minutia. +The Botan Cryptography Community Project has reached the end of its funding on Jan 31, 2024. This proposal is for the extension of funding to and the addition of new goals for the [Botan Cryptography Community Project](https://github.com/haskellfoundation/tech-proposals/pull/57), which is a previously funded proposal. -# Background +These new goals are for the continued refinement of the produced libraries, the continuation of C++ work, the development of cryptographic typeclasses and data families, improvements to the installation & CI pipeline, increased safety measures such as memory scrubbing and conditional compilation for optional modules, data types and instances for libraries such as `random` and `mtl`, and continued support of existing code. + +# Background / Retrospective The [Botan Cryptography Community Project](https://github.com/apotheca/botan) is a project that was previously funded by the Haskell Foundation for the purpose of developing a suite of Haskell cryptography libraries and tooling suitable a wide range of uses including data integrity, privacy, security, and networking. -This project has maintained a [devlog](https://discourse.haskell.org/t/botan-bindings-devlog/6855) and produced several monthly reports [(1)](https://discourse.haskell.org/t/botan-cryptography-monthly-status-report-0/8280) [(2)](https://discourse.haskell.org/t/botan-cryptography-monthly-status-report-1/8497) +This project has maintained a [devlog](https://discourse.haskell.org/t/botan-bindings-devlog/6855) and produced several monthly reports [(1)](https://discourse.haskell.org/t/botan-cryptography-monthly-status-report-0/8280) [(2)](https://discourse.haskell.org/t/botan-cryptography-monthly-status-report-1/8497) [(3)](https://discourse.haskell.org/t/botan-cryptography-3rd-monthly-status-report/8754) + +The project has produced 2 libraries which have been released to hackage: `botan-bindings`[(1)](https://hackage.haskell.org/package/botan-bindings-0.0.1.0), `botan-low` [(2)](https://hackage.haskell.org/package/botan-low-0.0.1.0). + +A third library `botan` [(3)\*](https://hackage.haskell.org/package/botan-0.0.1.0) is entering package candidate status soon, which will complete the set of milestones laid out in the initial proposal. + +> \* Package candidate link to `botan` is not yet live at time of writing. + +This project was a moderate-to-high success, with room for improvement, mostly in terms of scoping and communication. The low-level `botan-bindings` and `botan-low` libraries were completed satisfactorily, with the high-level `botan` experiencing some minor delays due to a few factors. -The project has produced 3 libraries: `botan-bindings`, `botan-low`, and `botan`. +Significant progress was made towards one of the stretch goals (extending the Botan C FFI to include full X509 support) and while we were initially optimistic that this stretch goal might be completed, this effort came at a opportunity cost towards required deliverables, and was ultimatly paused to avoid endangering them. -This project also has made a substantial amount of progress towards one of its stretch goals of extending the Botan C FFI to include full X509 support. We were initially optimistic that this stretch goal might be completed by the end of initial funding, but the C++ work required more overhead than expected, and efforts towards it were paused to avoid endangering any required deliverables. +In addition to this, the algorithm ADT approach (a core component of the high-level `botan` interface) turned out to be insufficient in practice. This was really only evident after having a first implementation, and some time was needed to come up with a better interface with much stronger typing, especially after feedback stressing the desire for per-algorithm type-safety. + +Solving this required bringing back cryptographic typeclasses, which were originally and explicitly removed from the proposal due to being out-of-scope. This caused a noticable increase in scope, as it was necessary to bring them back in due to the insufficiency of the ADTs. + +This has ultimately ended up delaying the `botan` library by a few weeks after feedback indicated a strong desire to see more effort being spent on polishing, documentation, and release of `botan-bindings` and `botan-low` before continuing work on `botan`. + +These issues could have been handled by better prioritization and scope management, and a more clear communication of any effect on the project timeline. As such, this proposal is slightly less aggressive in scope, to allow for a little more flexibility to account for shifting needs and feedback. # Technical Content -Despite (being on pace for) fulfilling its goals, there is still more work to do, as the scope of the original proposal was deliberately restricted for the purposes of focus and achievability. Now that the original goals have been met, some of the previously out-of-scope goals may now be reconsidered. +There is still more work to do, as the scope of the original proposal was deliberately restricted for the purposes of focus and achievability. Now that the original goals have been met, some of the previously out-of-scope goals may now be reconsidered. ## Continued improvements to `botan` -- `botan-bindings` and `botan-low` are effectively frozen, but small changes may still occur -- `botan` is likely to see significant churn as we continue to improve and respond to feedback +While `botan-bindings` and `botan-low` are effectively frozen, small changes may still occur. However, `botan` is likely to see significant churn as we continue to improve and respond to feedback. + +As `botan` is a very "fresh" library, it is expected to see some amount of churn in its API as it is developed and refined further. The initial goals of functionality are nominally complete, but there is still a great amount of ways to improve the safety and user experience. + +Expected improvements include: -As `botan` is a very "fresh" library, it is expected to see some amount of churn in its API as it is developed and refined further. The initial goals of functionality are nominally complete, but there is still a great amount of minutia to be performed that is concerned with improving the safety and user experience. Expected improvements include limited lifespan objects with guaranteed cleanup, auto-zeroing of sensitive memory, and implementing instances from libraries such as `random`. +- Managed / limited lifespan objects with bracket / guaranteed cleanup +- Auto-zeroing of sensitive memory +- Implementing data types and instances for libraries such as `random` and `mtl` +- Conditional compilation suport for optional modules +- Improvements to the `Make / Remake` factory methods used to generate bindings ## Botan C++ development -- `Botan C++` is being actively developed, so project will need continued maintenance to achieve parity -- Several valuable features in `Botan C++` are not exposed by the C FFI, and require C / C++ development in order to expose them +`Botan C++` is being actively developed, so this project will need continued maintenance in order to achieve continued parity. Furthermore, several valuable features in `Botan C++` are not exposed by the C FFI, and require C / C++ development in order to expose them: + +- Stream ciphers +- Complete X509 infrastructure implementation +- Complete TLS server implementation One of the previous stretch goals was to develop the C bindings for X509 certificate architecture, and to [contribute that code back](https://github.com/randombit/botan/issues/3627) to `Botan C++`. Substantial progress was made, but further work is required in order to complete this. @@ -38,30 +64,43 @@ Another feature missing from the Botan C FFI is Stream Ciphers, and like the X50 A third feature missing from the Botan C FFI is TLS, as Botan provides a full TLS server implementation, which will also require effort to expose to the FFI. -## Development of cryptographic abstractions +## Development of cryptographic typeclasses and data families + +Due to the insufficiency of the ADT approach towards managing algorithms, cryptographic typeclasses and data families have been brought back into scope as a deliverable after being declared out-of-scope in the original proposal. + +While a first attempt has already been made to typeclassify the various cryptographic primitives, this interface will require extensive refinement and feedback over the coming months, and is likely to experience significant change before settling down. + +It is expected that these typeclasses will result in improved per-algorithm type safety, and better ergonomics through more consistent handling of cryptographic primitives such as keys, nonces, and ciphertexts. + +## Improvements to installation / CI / Unit tests -There are many 'class-y' improvements that could be made, abstractions that `botan` could implement instances for better ergonomics and better management of keys, nonces, and other cryptographic minutia. While many of these abstractions are not strictly necessary for `botan`, they do hold value, and investigations into the development of an abstract hierarchy and how it might be applied to botan and other cryptographic libraries is ongoing. +One of the largest warts in using these libraries is the installation of the Botan C++ library. Package distributions are available for some users and operating systems via `pkg-config`, but other users currently require manual installation. + +This is a one-time thing, but it still is a significant pain point in what should otherwise be a much smoother process. As such, research into packaging or bundling Botan C++ is ongoing, with multiple potential solutions already having been identified. + +This is a high-priority goal, alongside improvements to CI and unit testing, and will be a major focus. # Goals The following are the immediate goals / deliverables of this proposal: - To continue to refine the `botan` libraries - - To complete the X509 C / C++ FFI and Haskell bindings - - To begin development of the Stream Cipher C / C++ FFI and Haskell bindings - + - To continue improvements to the Botan C FFI and Haskell bindings + - To develop an abstract cryptographic typeclass / data family hierarchy exhibiting per-algorithm safety + - To improve Botan installation / CI ease of use + The following are intermediate deliverables: - Frequent activity update on devlog - Weekly major update that includes github push - Monthly report that summarizes what specific items have been or will be worked on, and what challenges have arisen, if any. -The following are stretch goals / optional deliverables: +The following are stretch goals / optional deliverables to be looked into only after required deliverables are met: - - To complete development of the Stream Cipher C / C++ FFI and Haskell bindings - - To develop the full TLS server C / C++ FFI and Haskell bindings + - Development of the Stream Cipher C / C++ FFI and Haskell bindings + - Completion of the extended X509 C / C++ FFI and Haskell bindings + - Beginning development the full TLS server C / C++ FFI and Haskell bindings - To investigate integrating `botan` into libraries such `x509`, `tls` - - To develop abstract classes / hierarchy for cryptography and cryptosystems - To develop tutorials for integrating with other libraries such as `servant` # Timeline and Budget @@ -78,4 +117,4 @@ The primary stakeholders of this proposal are: # Success -This proposal will be considered a success if the immediate and intermediate deliverables are met, and at least one of the stretch goals has seen progress. +This proposal will be considered a success if the immediate and intermediate deliverables are met, new versions of the libraries are released as necessary, and at least one of the stretch goals has seen progress. From a7fcaa7248950078642fe18a24b060797e0f554d Mon Sep 17 00:00:00 2001 From: Leo Date: Fri, 16 Feb 2024 11:58:02 -0600 Subject: [PATCH 3/5] Added mitigating factors to retrospective --- proposals/053-cryptography-extension.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/proposals/053-cryptography-extension.md b/proposals/053-cryptography-extension.md index b2ed37f..3da8f18 100644 --- a/proposals/053-cryptography-extension.md +++ b/proposals/053-cryptography-extension.md @@ -30,7 +30,11 @@ Solving this required bringing back cryptographic typeclasses, which were origin This has ultimately ended up delaying the `botan` library by a few weeks after feedback indicated a strong desire to see more effort being spent on polishing, documentation, and release of `botan-bindings` and `botan-low` before continuing work on `botan`. -These issues could have been handled by better prioritization and scope management, and a more clear communication of any effect on the project timeline. As such, this proposal is slightly less aggressive in scope, to allow for a little more flexibility to account for shifting needs and feedback. +These issues and confounding factors could have been handled by better prioritization and scope management, and a more clear communication of any effect on the project timeline. + +However, there are mitigating factors as well: significant progress was made towards a high-value stretch goal (Extended X509 suppport), and the deliverables were met despite the size and scope of the project being significantly larger than originally estimated. + +As such, this proposal is slightly less aggressive in scope, to allow for a little more flexibility to account for shifting needs and feedback. # Technical Content From 268ab169f3358815b7824a64daedc49195d057f2 Mon Sep 17 00:00:00 2001 From: Leo Date: Thu, 7 Mar 2024 15:27:39 -0600 Subject: [PATCH 4/5] Proposal significantly rewritten with more details to take into account feedback --- proposals/053-cryptography-extension.md | 213 ++++++++++++++++-------- 1 file changed, 145 insertions(+), 68 deletions(-) diff --git a/proposals/053-cryptography-extension.md b/proposals/053-cryptography-extension.md index 3da8f18..237a1d2 100644 --- a/proposals/053-cryptography-extension.md +++ b/proposals/053-cryptography-extension.md @@ -1,116 +1,122 @@ -# Botan Cryptography Community Project Extension +# Botan Cryptography Community Project #2 -> NOTE: This document references an earlier proposal that is ongoing and close to meeting its goals. Language in this document may reflect a future perspective that assumes those goals have been met. +> NOTE: This document references an [earlier proposal][first proposal]. This new proposal continues with the same long term goals and it is suggested that the reader be familiar with it. A retrospective is also included - see: Appendix: Retrospective. # Abstract -The Botan Cryptography Community Project has reached the end of its funding on Jan 31, 2024. This proposal is for the extension of funding to and the addition of new goals for the [Botan Cryptography Community Project](https://github.com/haskellfoundation/tech-proposals/pull/57), which is a previously funded proposal. +The [Botan Cryptography Community Project][botan repo], a [previously funded proposal][first proposal], successfully concluded having met its immediate goals and deliverables. This new proposal continues with the same long term goals, and a new set of immediate goals and deliverables in furtherance of them. These new goals are for the continued refinement of the produced libraries, the improvement of installation and CI pipeline support for Windows and Linux, the development of a drop-in interface replacement for `crypton`, and continued development of `botan` to include features such as: secure memory erasure, data types and instances for `random` and `mtl`, and the continued development of cryptographic typeclasses and gold-standard (per-algorithm type-safe) modules. -These new goals are for the continued refinement of the produced libraries, the continuation of C++ work, the development of cryptographic typeclasses and data families, improvements to the installation & CI pipeline, increased safety measures such as memory scrubbing and conditional compilation for optional modules, data types and instances for libraries such as `random` and `mtl`, and continued support of existing code. +# Background -# Background / Retrospective +Botan is an comprehensive, open-source, BSD-licenced, C++ cryptography library with a stable C API. It offers a broad variety of functionality and algorithms, including **post-quantum cryptography**, is developed and maintained by an active community, and has been [audited][botan audit] in the past. -The [Botan Cryptography Community Project](https://github.com/apotheca/botan) is a project that was previously funded by the Haskell Foundation for the purpose of developing a suite of Haskell cryptography libraries and tooling suitable a wide range of uses including data integrity, privacy, security, and networking. +By binding to Botan, we solved a significant problem of providing much of the necessary 'cryptographic kitchen sink' via a suitably performant, suitably licensed, open-source library. Furthermore, we did this without imposing a large maintenance burden on the Haskell community, as we are only required to maintain the bindings, and not the Botan cryptography library itself. -This project has maintained a [devlog](https://discourse.haskell.org/t/botan-bindings-devlog/6855) and produced several monthly reports [(1)](https://discourse.haskell.org/t/botan-cryptography-monthly-status-report-0/8280) [(2)](https://discourse.haskell.org/t/botan-cryptography-monthly-status-report-1/8497) [(3)](https://discourse.haskell.org/t/botan-cryptography-3rd-monthly-status-report/8754) +That was the first milestone and a big step towards building a stable foundation for the Haskell cryptography ecosystem, but there's still work to do. With this milestone, the project enters a new phase in the software development lifecycle - maintenance and development. Now that we have something working, with an initial release and users, we need to listen to feedback and keep it working while we continue to further development of the long-term goals. -The project has produced 2 libraries which have been released to hackage: `botan-bindings`[(1)](https://hackage.haskell.org/package/botan-bindings-0.0.1.0), `botan-low` [(2)](https://hackage.haskell.org/package/botan-low-0.0.1.0). +That is what this proposal is about - listening to feedback, improving the user experience, and seeing where the pain points are. Now that the original goals have been met, we'd also like to reconsider some previously out-of-scope goals as optional or stretch goals. Here's what we've heard, and what we're planning for the next three months. -A third library `botan` [(3)\*](https://hackage.haskell.org/package/botan-0.0.1.0) is entering package candidate status soon, which will complete the set of milestones laid out in the initial proposal. +# Technical Content -> \* Package candidate link to `botan` is not yet live at time of writing. +## Improved installation and CI support -This project was a moderate-to-high success, with room for improvement, mostly in terms of scoping and communication. The low-level `botan-bindings` and `botan-low` libraries were completed satisfactorily, with the high-level `botan` experiencing some minor delays due to a few factors. +One of the biggest pieces of feedback that we've received is the need for improved support for the installation of the `botan3` C++ library. This was a recurring item, and we've made it one of our priorities. Installation of the Botan C++ library is one of the largest warts in using these libraries - package distributions are available for some users and operating systems via `pkg-config`, but other users currently require manual installation. This is a one-time process, but it is still a significant pain point in what should otherwise be a much smoother process. -Significant progress was made towards one of the stretch goals (extending the Botan C FFI to include full X509 support) and while we were initially optimistic that this stretch goal might be completed, this effort came at a opportunity cost towards required deliverables, and was ultimatly paused to avoid endangering them. +We'd like to spend a good chunk of time improving the installation process, with a specific focus on Windows[^1] and Linux support, working on the CI + unit tests for improved reliability and reporting overall. -In addition to this, the algorithm ADT approach (a core component of the high-level `botan` interface) turned out to be insufficient in practice. This was really only evident after having a first implementation, and some time was needed to come up with a better interface with much stronger typing, especially after feedback stressing the desire for per-algorithm type-safety. +[^1]: Definitely more highly requested than anticipated -Solving this required bringing back cryptographic typeclasses, which were originally and explicitly removed from the proposal due to being out-of-scope. This caused a noticable increase in scope, as it was necessary to bring them back in due to the insufficiency of the ADTs. +This is a high-priority goal, alongside improvements to CI and unit testing, and will be a major focus. We're already looking into using `build-type: Configure` for bundling Botan C++ as a Haskell package for easy installation on all operating systems - we'd like for usage to be as easy as adding `botan` to your dependencies. -This has ultimately ended up delaying the `botan` library by a few weeks after feedback indicated a strong desire to see more effort being spent on polishing, documentation, and release of `botan-bindings` and `botan-low` before continuing work on `botan`. +## Development of a drop-in interface replacement for `crypton` -These issues and confounding factors could have been handled by better prioritization and scope management, and a more clear communication of any effect on the project timeline. +This was mentioned several times in feedback. `crypton` is a dependency in many important libraries in the Haskell ecosystem, and we would like to build an interface that is as near a drop-in replacement for `crypton` as possible. -However, there are mitigating factors as well: significant progress was made towards a high-value stretch goal (Extended X509 suppport), and the deliverables were met despite the size and scope of the project being significantly larger than originally estimated. +These libraries sit at the root of a lot of production haskell code, and their list of transitive dependents is quite large. Anything involving networking, APIs, and the internet is likely to depend on at least one of them: -As such, this proposal is slightly less aggressive in scope, to allow for a little more flexibility to account for shifting needs and feedback. +- `amazonka-core` +- `hpack` +- `scotty` +- `servant` +- `stack` +- `tls` +- `x509` +- `warp` +- `websockets` -# Technical Content +They are all directly or indirectly dependent on `crypton`, a fork of `cryptonite` containing significant quantities of unaudited C that must be maintained by the Haskell community after the original author became unavailable. This is a problem, and we're trying to change that with bindings to `botan`. Having a solid, reliable, and well-maintained cryptography libraries is a huge benefit to the Haskell ecosystem in general, and is essential for most commercial and industrial use cases. -There is still more work to do, as the scope of the original proposal was deliberately restricted for the purposes of focus and achievability. Now that the original goals have been met, some of the previously out-of-scope goals may now be reconsidered. +There will be some differences, as `botan` doesn't necessarily support everything[^2] in the same way as `crypton` does, but we'll give it our best effort to make migration as simple as possible, through a drop-in replacement, migration guide, or other suitable adapter. -## Continued improvements to `botan` +[^2]: There are a few things that `crypton` supports that `botan` doesn't, but also vice versa - `botan` supports things like modern post-quantum algorithms and `crypton` doesn't. -While `botan-bindings` and `botan-low` are effectively frozen, small changes may still occur. However, `botan` is likely to see significant churn as we continue to improve and respond to feedback. +## Continued development of the cryptographic typeclasses and gold-standard modules -As `botan` is a very "fresh" library, it is expected to see some amount of churn in its API as it is developed and refined further. The initial goals of functionality are nominally complete, but there is still a great amount of ways to improve the safety and user experience. +The development of [cryptographic typeclasses][cryptographic typeclasses] and gold-standard per-algorithm modules is ongoing, in an effort to improve per-algorithm type safety and ergonomics through more consistent handling of cryptographic primitives such as keys, nonces, and ciphertexts. -Expected improvements include: +> This is also a necessity for the `crypton` drop-in replacement. -- Managed / limited lifespan objects with bracket / guaranteed cleanup -- Auto-zeroing of sensitive memory -- Implementing data types and instances for libraries such as `random` and `mtl` -- Conditional compilation suport for optional modules -- Improvements to the `Make / Remake` factory methods used to generate bindings +Due to the inadequacy of the ADT approach towards managing algorithms, cryptographic typeclasses and data families have been brought back into scope as a deliverable after being declared out-of-scope in the original proposal. While a first attempt has already been made to typeclassify the various cryptographic primitives, this interface will require extensive refinement and feedback over the coming months, and is likely to experience significant change before settling down. -## Botan C++ development +It is expected that these typeclasses will result in improved per-algorithm type safety, and better ergonomics through more consistent handling of cryptographic primitives such as keys, nonces, and ciphertexts. -`Botan C++` is being actively developed, so this project will need continued maintenance in order to achieve continued parity. Furthermore, several valuable features in `Botan C++` are not exposed by the C FFI, and require C / C++ development in order to expose them: +## Continued improvements of `botan` -- Stream ciphers -- Complete X509 infrastructure implementation -- Complete TLS server implementation +While `botan-bindings` and `botan-low` are effectively frozen, small changes may still occur. However, `botan` will be seeing significant development as we continue to improve and respond to feedback, as there are still a great number of ways to improve the safety and user experience: -One of the previous stretch goals was to develop the C bindings for X509 certificate architecture, and to [contribute that code back](https://github.com/randombit/botan/issues/3627) to `Botan C++`. Substantial progress was made, but further work is required in order to complete this. +### Improved error handling -Another feature missing from the Botan C FFI is Stream Ciphers, and like the X509 bindings, it will require effort to extend the Botan C++ library in order to expose it to Haskell. +We'd like to reduce the amount of noise, and to use `HasCallstack` appropriately throughout the library. -A third feature missing from the Botan C FFI is TLS, as Botan provides a full TLS server implementation, which will also require effort to expose to the FFI. +### Secure memory erasure and limited lifespan objects with guaranteed cleanup -## Development of cryptographic typeclasses and data families +Ensuring that the instructions to securely erase memory are not compiled out is a historically complex subject. Botan has appropriate functionality for secure memory erasure, but it must be called manually. We'd like to take advantage of the secure memory erasure support by providing functions for creating temporary objects that are automatically zeroed when they go out of scope. This is important, we don't want sensitive data such as keys hanging around in memory after we are done using them. -Due to the insufficiency of the ADT approach towards managing algorithms, cryptographic typeclasses and data families have been brought back into scope as a deliverable after being declared out-of-scope in the original proposal. +### Integration with other libraries -While a first attempt has already been made to typeclassify the various cryptographic primitives, this interface will require extensive refinement and feedback over the coming months, and is likely to experience significant change before settling down. +We'd also like to investigate integrating with other libraries. In particular, we'd like to integrate with `mtl` and `random` in order to provide a `MonadRandomIO / NonDeterministic` monad and take advantage of the `Uniform` class to make sampling / generation of random data easy. This monad already exists in a primitive state in `botan`, but significant refinement is expected. -It is expected that these typeclasses will result in improved per-algorithm type safety, and better ergonomics through more consistent handling of cryptographic primitives such as keys, nonces, and ciphertexts. +### Improvements to the binding generator functions -## Improvements to installation / CI / Unit tests +Finally, we'd also like to continue improving, unifying, and standardizing the `botan-low` factory methods used to generate bindings bindings, for improved low-level consistency overall. The techniques for binding to functions were refined over time, and have not been evenly applied, and there is a significant amount of duplicated code or one-off implementations. It would improve maintenance significantly if all of the buffer handling et al were in one place. -One of the largest warts in using these libraries is the installation of the Botan C++ library. Package distributions are available for some users and operating systems via `pkg-config`, but other users currently require manual installation. +## Development of a high-level libsodium-like interface -This is a one-time thing, but it still is a significant pain point in what should otherwise be a much smoother process. As such, research into packaging or bundling Botan C++ is ongoing, with multiple potential solutions already having been identified. +We'd like to expose a high-level libsodium-like interface of selected best-in-class algorithms in order to make usage dead simple. We don't want you managing primitives yourselves - we want you calling a simple function purely or in an appropriate monad / transformer. This is bit of a stretch goal, however, in favor of focusing our efforts on primary goals such as replacing `crypton`. -This is a high-priority goal, alongside improvements to CI and unit testing, and will be a major focus. +# Timeline and Budget -# Goals +We propose a timeline of 3 months for these new goals, with a budget of $7000 USD per month for one full-time engineer. Please see the [first proposal][first proposal] for more detail. -The following are the immediate goals / deliverables of this proposal: +# Goals and Deliverables - - To continue to refine the `botan` libraries - - To continue improvements to the Botan C FFI and Haskell bindings - - To develop an abstract cryptographic typeclass / data family hierarchy exhibiting per-algorithm safety - - To improve Botan installation / CI ease of use +## Primary deliverables -The following are intermediate deliverables: +The following are the primary deliverables of this proposal, in order of priority: - - Frequent activity update on devlog - - Weekly major update that includes github push - - Monthly report that summarizes what specific items have been or will be worked on, and what challenges have arisen, if any. +- Completion and publishing[^3] version `0.0.1` of `botan`, and continued updates to the library suite as necessary +- Improved installation and CI support for more operating systems, with a focus on Windows and Ubuntu +- Development of one or more suitable ways of assisting users in migrating from `crypton` to `botan`, be it drop-in interface replacement, fork,and/or migration guide -The following are stretch goals / optional deliverables to be looked into only after required deliverables are met: - - - Development of the Stream Cipher C / C++ FFI and Haskell bindings - - Completion of the extended X509 C / C++ FFI and Haskell bindings - - Beginning development the full TLS server C / C++ FFI and Haskell bindings - - To investigate integrating `botan` into libraries such `x509`, `tls` - - To develop tutorials for integrating with other libraries such as `servant` - -# Timeline and Budget +[^3]: `botan` is a package candidate, and is currently undergoing pre-release polishing and refinemenet. Publishing and release of `0.0.1` is expected to occur with a few weeks at time of writing. + +## Secondary deliverables + +The following are secondary deliverables, to be considered as optional stretch goals to be completed if time permits after completion of primary deliverables + +- Continued development of the cryptographic typeclasses and gold-standard algorithm modules + - There is no set completion date on this, but progress should be made if possible +- Completion of the objectives in 'Continued improvements to `botan`' +- Development of a high-level libsodium-like interface + +## Intermediate deliverables + +The following are intermediate deliverables intended to keep the community apprised of any progress that is made, and any challenges that arise, and to seek community feedback: + +- Frequent (2-3x weekly) activity update on devlog +- Weekly major update that includes github push +- Monthly report that summarizes what specific items have been worked on, what items will be worked on, and what challenges have arisen, if any. -We propose a timeline of 3 months for these new goals, with a budget of $7000 USD per month for one full-time engineer, the same as the original proposal. - # Stakeholders The primary stakeholders of this proposal are: @@ -118,7 +124,78 @@ The primary stakeholders of this proposal are: - Leon Dillinger (Myself) - Haskell Foundation - Haskell Cryptography Group +- The Haskell Community +- Current and previous sponsors + + Please see the [first proposal][first proposal] for more detail. + + # Success + +This proposal will be considered a success if the immediate and intermediate deliverables are met, and new versions of the libraries are released as necessary. The secondary stretch goals are not to be considered necessary for success, but should be worked on if time permits after completion of the primary goals. + +# Future work + +There are many things that are out-of-scope or are of lower priority than the the project goals already mentioned. The following are long-term goals that are likely to be revisited in a future proposal: + +- Implementation of advanced algorithms such as Merkle Trees, Distributed JSON, [Signal's Double Ratchet][double ratchet], and [Apple's PQ3][apple pq3] +- Improving APIs with higher-order functions +- To investigate integration / migrating specific libraries to `botan`, such as `x509`, `tls`, `servant`, as an alternative to `crypton` +- Splitting off cryptographic classes as a separate package to be backend agnostic +- Building an application framework that takes care of cryptography & security + +# Appendix: Retrospective + +The [Botan Cryptography Community Project][botan repo] is a project that was [previously funded][first proposal] by the Haskell Foundation with the immediate goals of developing bindings to the [Botan C++ cryptography library][botan cpp], and the long term goals of developing a stable, open-source cryptography ecosystem for the Haskell community, with uses including data integrity, privacy, security, and networking. + +This project successfully maintained a [devlog][devlog], produced several monthly reports [(1)][monthly report 1], [(2)][monthly report 2], [(3)][monthly report 3], 2 libraries which have been released to hackage: `botan-bindings`[(1)][botan-bindings] and `botan-low` [(2)][botan-low], and a third library `botan` [(3)][botan candidate] has entered package candidate status, and completed the set of milestones and deliverables laid out in the initial proposal. + +This project was a moderate-to-high success, with room for improvement, mostly in terms of scoping and communication. The low-level `botan-bindings` and `botan-low` libraries were completed satisfactorily, with the high-level `botan` experiencing some minor delays due to a few factors. + +Significant progress was made towards one of the stretch goals (extending the Botan C FFI to include full X509 support) and while we were initially optimistic that this stretch goal might be completed, this effort came at a opportunity cost towards required deliverables, and was ultimatly paused to avoid endangering them. + +In addition to this, the algorithm ADT approach (a core component of the high-level `botan` interface) turned out to be insufficient in practice. This was really only evident after having a first implementation, and some time was needed to come up with a better interface with much stronger typing, especially after feedback stressing the desire for per-algorithm type-safety. + +Solving this required bringing back cryptographic typeclasses, which were originally and explicitly removed from the proposal due to being out-of-scope. This caused a noticable increase in scope, as it was necessary to bring them back in due to the insufficiency of the ADTs. + +This ultimately ended up delaying work on the `botan` library after feedback indicated a strong desire to see more effort being spent on polishing, documentation, and release of `botan-bindings` and `botan-low` before continuing work on `botan`. + +These issues and confounding factors could have been handled by better prioritization and scope management, and a more clear communication to the community of any effect on the project timeline. + +However, there are mitigating factors as well: significant progress was made towards a high-value stretch goal (Extended X509 suppport), and the deliverables were met despite the size and scope of the project being significantly larger than originally estimated. + +As such, this proposal is slightly less aggressive in scope, to allow for a little more flexibility to account for shifting needs and feedback. + +# Appendix: Links + +- [First proposal][first proposal] +- [Botan repository][botan repo] +- [Botan C++][botan cpp] +- [Devlog][devlog] +- [Monthly report 1][monthly report 1] +- [Monthly report 2][monthly report 2] +- [Monthly report 3][monthly report 3] +- [botan-bindings package][botan-bindings] +- [botan-low package][botan-low] +- [botan package candidate][botan candidate] + + + +[first proposal]: https://github.com/haskellfoundation/tech-proposals/pull/57 "First proposal" + +[botan repo]: https://github.com/haskell-cryptography/botan "Botan repository" +[botan cpp]: https://botan.randombit.net/ "Botan C++" +[botan audit]: https://botan.randombit.net/releases/audit_1.11.18.pdf "Botan audit" + +[devlog]: https://discourse.haskell.org/t/botan-bindings-devlog/6855 "Devlog" + +[monthly report 1]: https://discourse.haskell.org/t/botan-cryptography-monthly-status-report-0/8280 "Monthly report 1" +[monthly report 2]: https://discourse.haskell.org/t/botan-cryptography-monthly-status-report-1/8497 "Monthly report 2" +[monthly report 3]: https://discourse.haskell.org/t/botan-cryptography-3rd-monthly-status-report/8754 "Monthly report 3" -# Success +[botan-bindings]: https://hackage.haskell.org/package/botan-bindings-0.0.1.0 "botan-bindings" +[botan-low]: https://hackage.haskell.org/package/botan-low-0.0.1.0 "botan-low" +[botan candidate]: https://hackage.haskell.org/package/botan-0.0.1.0 "botan candidate" -This proposal will be considered a success if the immediate and intermediate deliverables are met, new versions of the libraries are released as necessary, and at least one of the stretch goals has seen progress. +[cryptographic typeclasses]: https://discourse.haskell.org/t/botan-bindings-devlog/6855/ "Cryptographic typeclasses" +[double ratchet]: https://signal.org/docs/specifications/doubleratchet/ "Double-ratchet" +[apple pq3]: https://security.apple.com/blog/imessage-pq3/ "Apple PQ3" From 7da09bcc8fd9cac76b71978cd7629ee4534ff611 Mon Sep 17 00:00:00 2001 From: Leo Date: Sun, 10 Mar 2024 13:44:38 -0500 Subject: [PATCH 5/5] Moved libsodium-like interface to long-term goals --- proposals/053-cryptography-extension.md | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/proposals/053-cryptography-extension.md b/proposals/053-cryptography-extension.md index 237a1d2..5fcfb44 100644 --- a/proposals/053-cryptography-extension.md +++ b/proposals/053-cryptography-extension.md @@ -26,7 +26,7 @@ We'd like to spend a good chunk of time improving the installation process, with [^1]: Definitely more highly requested than anticipated -This is a high-priority goal, alongside improvements to CI and unit testing, and will be a major focus. We're already looking into using `build-type: Configure` for bundling Botan C++ as a Haskell package for easy installation on all operating systems - we'd like for usage to be as easy as adding `botan` to your dependencies. +This is a high-priority goal, alongside improvements to CI and unit testing, and will be a major focus. We're already looking into using `build-type: Configure` to automatically fetch and/or build the latest compatible version of Botan C++ via Cabal if it is not already installed, in order to streamline the process on all operating systems - we'd like for usage to be as easy as adding `botan` to your dependencies. ## Development of a drop-in interface replacement for `crypton` @@ -80,10 +80,6 @@ We'd also like to investigate integrating with other libraries. In particular, w Finally, we'd also like to continue improving, unifying, and standardizing the `botan-low` factory methods used to generate bindings bindings, for improved low-level consistency overall. The techniques for binding to functions were refined over time, and have not been evenly applied, and there is a significant amount of duplicated code or one-off implementations. It would improve maintenance significantly if all of the buffer handling et al were in one place. -## Development of a high-level libsodium-like interface - -We'd like to expose a high-level libsodium-like interface of selected best-in-class algorithms in order to make usage dead simple. We don't want you managing primitives yourselves - we want you calling a simple function purely or in an appropriate monad / transformer. This is bit of a stretch goal, however, in favor of focusing our efforts on primary goals such as replacing `crypton`. - # Timeline and Budget We propose a timeline of 3 months for these new goals, with a budget of $7000 USD per month for one full-time engineer. Please see the [first proposal][first proposal] for more detail. @@ -107,7 +103,6 @@ The following are secondary deliverables, to be considered as optional stretch g - Continued development of the cryptographic typeclasses and gold-standard algorithm modules - There is no set completion date on this, but progress should be made if possible - Completion of the objectives in 'Continued improvements to `botan`' -- Development of a high-level libsodium-like interface ## Intermediate deliverables @@ -137,6 +132,7 @@ This proposal will be considered a success if the immediate and intermediate del There are many things that are out-of-scope or are of lower priority than the the project goals already mentioned. The following are long-term goals that are likely to be revisited in a future proposal: +- Development of a high-level libsodium-like interface of recommended best-in-class algorithms - Implementation of advanced algorithms such as Merkle Trees, Distributed JSON, [Signal's Double Ratchet][double ratchet], and [Apple's PQ3][apple pq3] - Improving APIs with higher-order functions - To investigate integration / migrating specific libraries to `botan`, such as `x509`, `tls`, `servant`, as an alternative to `crypton`