Refactor capnp-futures and capnp-rpc for Improved no-std Compatibility

[stevefan1999-personal]

Overview

This pull request introduces a significant refactoring of the capnp-futures and capnp-rpc crates to enhance compatibility with no_std environments. The changes replace direct usage of std with alloc and core where possible, while introducing feature flags to control the level of standard library dependency. This enables the crates to be used in embedded systems and other constrained environments that cannot rely on the full standard library.

Key Changes

1. Feature Flag Introduction

• New Features in capnp-futures/Cargo.toml:

  • default = ["std"]: Maintains backward compatibility by defaulting to std-enabled builds
  • std: Enables standard library features, including futures-channel/std, futures-util/std, and embedded-io-async?/std
  • alloc: Enables allocation features for futures/alloc and embedded-io-async?/alloc
  • embedded-io: Adds support for embedded-io-async as an optional dependency for embedded I/O operations

• New Features in capnp-rpc/Cargo.toml:

  • Similar feature structure with default = ["std"], std, and alloc features
  • Conditional compilation based on feature flags

2. Library Configuration Updates

• capnp-futures/src/lib.rs:

  • Added #![cfg_attr(not(feature = "std"), no_std)] for conditional no_std compilation
  • Added extern crate alloc; when the alloc feature is enabled
  • Introduced Clippy lints to enforce proper usage: clippy::std_instead_of_alloc, clippy::std_instead_of_core, clippy::alloc_instead_of_core

• capnp-rpc/src/lib.rs:

  • Similar no_std configuration and alloc extern crate
  • Updated imports to use alloc and core instead of std

3. Import and Type Replacements

• Systematic replacement across multiple files:
  • std::future::Future → core::future::Future
  • std::pin::Pin → core::pin::Pin
  • std::task::{Context, Poll} → core::task::{Context, Poll}
  • std::rc::Rc → alloc::rc::Rc
  • std::sync::Arc → alloc::sync::Arc
  • std::vec::Vec → alloc::vec::Vec
  • std::boxed::Box → alloc::boxed::Box
  • std::string::ToString → alloc::string::ToString
  • std::collections::HashMap → Conditional: std::collections::HashMap (with std) or alloc::collections::BTreeMap (without std)
  • std::collections::BTreeMap → alloc::collections::BTreeMap
  • std::cell::RefCell → core::cell::RefCell
  • std::mem → core::mem
  • std::marker::PhantomData → core::marker::PhantomData
  • std::iter::Iterator → core::iter::Iterator
  • std::fmt::Debug → core::fmt::Debug

4. Conditional Compilation for Collections

• capnp-rpc/src/rpc.rs:
  • CapabilityServerSet now uses std::collections::HashMap when std feature is enabled, and alloc::collections::BTreeMap otherwise
  • Similar conditional logic for other collection types throughout the codebase

5. I/O Abstraction Updates

• capnp-rpc/src/twoparty.rs:
  • Introduced conditional imports for AsyncRead and AsyncWrite
  • When embedded-io feature is enabled, uses embedded_io_async::{Read, Write} instead of futures::{AsyncRead, AsyncWrite}
  • This allows compatibility with embedded I/O traits

6. Error Handling Adjustments

• capnp-rpc/src/rpc.rs:
  • SystemTaskReaper::task_failed now conditionally prints errors only when std feature is available
  • In no_std environments, errors are silently dropped since printing is not available

7. Dependency Updates

• capnp-futures/Cargo.toml:
  • Added embedded-io-async = { version = "0.7.0", optional = true }
  • Modified futures-util to use features = ["io"] instead of ["io", "std"]
  • Made futures-channel default-features false and conditionally enable std

Motivation

The primary motivation for this refactoring is to enable the use of Cap'n Proto RPC in no_std environments, such as embedded systems, microcontrollers, and other constrained platforms. By minimizing dependencies on the standard library and providing feature flags for different levels of functionality, this change opens up new use cases for the library.

Limitations and Current Constraints

Despite these improvements, there are still some limitations to full no_std compatibility:

1. I/O Operations Still Require std: The current implementation still relies on std::io for certain I/O operations. This means that while the core RPC logic can operate without std, actual network communication still requires standard library support.

2. Embedded Platform Compatibility:

   • Embassy: Cannot currently be used with Embassy due to the std::io dependency for I/O operations.
   • ESP-IDF: Technically possible with std support, but limited to bare-metal ESP32 microcontrollers. Full ESP-IDF integration would require additional work to abstract away the I/O layer.

3. Collection Performance: In no_std mode, BTreeMap is used instead of HashMap, which may have performance implications for applications with many capabilities.

Testing and Compatibility

• All existing tests should continue to pass with the default std feature enabled
• New tests should be added to verify no_std + alloc functionality
• The embedded-io feature should be tested with actual embedded I/O implementations

Future Work

To achieve full no_std compatibility, future PRs should focus on:

1. Abstracting I/O operations to work with embedded-io traits throughout the codebase
2. Providing alternative implementations for any remaining std dependencies
3. Optimizing collection usage for no_std environments

This refactoring represents a significant step towards broader platform support while maintaining backward compatibility for existing users.

[codecov]

Codecov Report

❌ Patch coverage is 83.33333% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 50.84%. Comparing base (ab342b3) to head (3afc695).
⚠️ Report is 234 commits behind head on master.

Files with missing lines	Patch %	Lines
capnp-rpc/src/lib.rs	40.00%	3 Missing ⚠️
capnp-rpc/src/rpc.rs	80.00%	2 Missing ⚠️
capnp-futures/src/serialize_packed.rs	90.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #599      +/-   ##
==========================================
- Coverage   51.64%   50.84%   -0.81%     
==========================================
  Files          69       70       +1     
  Lines       33735    32334    -1401     
==========================================
- Hits        17422    16439     -983     
+ Misses      16313    15895     -418     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[dwrensha]

cargo build -p capnp-futures --no-default-features fails.

I don't expect embedded_io_async::Read to work as a simple drop-in replacement for futures_util::AsyncRead. Probably to make that work you would need to add a layer of indirection, similar to what's in https://github.com/capnproto/capnproto-rust/blob/master/capnp/src/io.rs.

[stevefan1999-personal]

cargo build -p capnp-futures --no-default-features fails.

I don't expect embedded_io_async::Read to work as a simple drop-in replacement for futures_util::AsyncRead. Probably to make that work you would need to add a layer of indirection, similar to what's in master/capnp/src/io.rs.

Yep, that's why I made it a draft first, the problem of lacking core::io is hard to dismiss or worked around, because that means core::io::Error will be hard to substitute at the moment because it has platform-dependent error code, and that alone kills a lot stuff such as AsyncRead and AsyncWrite. futures::io has the similar problem except it is for the async ecosystem, but they also use std::io:Error indirectly through std::io::Result

And the biggest problem is that we don't know how to handle alloc. Probably take years to make it no-alloc so that it works on embassy, but if you would like to strap in a simple allocator, maybe that would work if you don't care, just that 200KB or more RAM is the bare minimum for one connection I suppose...