feat: chain source rest sync #526

enigbe · 2025-04-18T10:10:38Z

What this PR does

This PR introduces chain source synchronization with Bitcoin Core's REST API. To do so, it explicitly distinguishes between the bitcoind_rpc_client and the newly introduced bitcoind_sync_client that can sync either via the RPC or REST APIs.

BitcoindRpcClient is maintained and the distinction created because the REST interface exposed by Bitcoin Core is not as robust as the RPC interface. This makes it impossible to send POST request to broadcast a transaction for example.

Related Issues

Fixes #496

cc @tnull @joostjager

Adds: - Sync client configuration with RPC (default) and REST variants - Extend the ChainSource::Bitcoind configuration options - Preliminary implementation for BitcoindSyncClient in RPC mode

Additionally adds the BlockSource implementation for BitcoindSyncClient

ldk-reviews-bot · 2025-04-18T10:10:41Z

👋 Thanks for assigning @tnull as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

tnull

Thanks for looking into this! Before I get into more detailed review, I want to note two things on the chosen approach that stood out to me from a first look:

I'm not quite convinced we should rename the BitcoindRpc client. Wouldn't it be cleaner to just add a (granted, rather duplicative) BitcoindRest variant side-by-side, and then in a second step maybe see how we can DRY up the code. Mushing both variants together seems to result in a rather confusing API and code structure.
I'm a bit confused why we'd now still have RPC credentials on set_chain_source_bitcoind (and elsewhere in downstream from that), even if we want to use the REST backend? Is there any operation that still requires RPC access? Can't we just rely on REST alone?

enigbe · 2025-04-19T11:57:23Z

Thanks for the early feedback.

I'm not quite convinced we should rename the BitcoindRpc client. Wouldn't it be cleaner to just add a (granted, rather duplicative) BitcoindRest variant side-by-side, and then in a second step maybe see how we can DRY up the code. Mushing both variants together seems to result in a rather confusing API and code structure.

I'm a bit confused why we'd now still have RPC credentials on set_chain_source_bitcoind (and elsewhere in downstream from that), even if we want to use the REST backend? Is there any operation that still requires RPC access? Can't we just rely on REST alone?

I understand your concerns about the name and API changes I have introduced but here is a bit of context.

Adding a ChainSource::BitcondRest variant was my initial approach too because I had erroneously thought Bitcoin Core's REST interface exposed (more or less) the same APIs as the RPC interface. However, the REST interface exposes a handful of APIs only for GET requests. It is not possible to broadcast transactions with a pure REST variant as we do with the BitcoindRpcClient, There is no REST API to sendrawtransaction.
For that particular use case, given that we'd need a handle on an RPC client regardless, I thought it best to:

Keep the BitcoindRpcClient as is.
Introduce a configurable REST/RPC sync client. With this, users can decide if syncing should happen either via the REST or RPC APIs.
Rename ChainSource::BitcoindRpc to ChainSource::Bitcoind as the latter leaves no ambiguity about the possibility of a REST variant.

tnull · 2025-04-22T17:14:54Z

I understand your concerns about the name and API changes I have introduced but here is a bit of context.

Adding a ChainSource::BitcondRest variant was my initial approach too because I had erroneously thought Bitcoin Core's REST interface exposed (more or less) the same APIs as the RPC interface. However, the REST interface exposes a handful of APIs only for GET requests. It is not possible to broadcast transactions with a pure REST variant as we do with the BitcoindRpcClient, There is no REST API to sendrawtransaction. For that particular use case, given that we'd need a handle on an RPC client regardless, I thought it best to:
* Keep the `BitcoindRpcClient` as is.

* Introduce a configurable REST/RPC sync client. With this, users can decide if syncing should happen either via the REST or RPC APIs.

* Rename `ChainSource::BitcoindRpc` to `ChainSource::Bitcoind` as the latter leaves no ambiguity about the possibility of a REST variant.

Aaah, okay, then the proposed approach makes more sense. I think it's still a bit awkward to have the mandatory RPC credentials on the set_chain_source_bitcoind method and addtionally have the BitcoindSyncConfig::Rpc variant.

While it then makes sense to reuse the client internally, it might be simpler to just keep the original set_chain_source_bitcoind_rpc method and just add a set_chain_source_bitcoind_rest variant, that also features the mandatory RPC credentials?

Camillarhi · 2025-05-01T13:48:15Z

Hi @enigbe,

I noticed your CI is failing. I had a similar issue, but it was resolved after rebasing onto the latest dev branch (which includes changes from #524).

Could you try rebasing and see if that fixes the problem?

Thanks

- Adds set_chain_source_bitcoind_rest to configure REST synchronization. - Update docs to reflect the configuration order if REST sync is preferred. - Simplify the configuration object to capture only REST-related fields.

chuksys

Looking into this PR has helped me gain a good understanding of how chain sourcing works. I think introducing BitcoindSyncClient is a good idea as the user can still carry out syncing operations using the REST API while the RPC Client is still maintained to carry out operations not supported by the REST API. I have just a few suggestions with regards to error handling.

src/builder.rs

tnull · 2025-05-12T07:16:43Z

@enigbe Please let me know when this is ready for a second look.

enigbe · 2025-05-12T07:52:41Z

Thanks for the feedback @tnull. I revisited this last week and addressed concerns raised here.

Aaah, okay, then the proposed approach makes more sense. I think it's still a bit awkward to have the mandatory RPC credentials on the set_chain_source_bitcoind method and addtionally have the BitcoindSyncConfig::Rpc variant.

This has been removed and we just keep a config for REST sync.

While it then makes sense to reuse the client internally, it might be simpler to just keep the original set_chain_source_bitcoind_rpc method and just add a set_chain_source_bitcoind_rest variant, that also features the mandatory RPC credentials?

Agreed! This seems like a cleaner split. I've added a set_chain_source_bitcoind_rest variant accordingly. However, I think it'd be potentially confusing/awkward for callers to still pass RPC parameters to the REST function. I opted for set_chain_source_bitcoind_rest to take only REST parameters and clarified comments on usage and dependency on the RPC variant.

If you'd still prefer we pass RPC parameters as a required input for the REST variant, happy to adjust.

enigbe · 2025-05-12T07:55:06Z

Hi @enigbe,

I noticed your CI is failing. I had a similar issue, but it was resolved after rebasing onto the latest dev branch (which includes changes from #524).

Could you try rebasing and see if that fixes the problem?

Thanks

Thanks for looking at this. I've rebased and have just the CLN integration tests failing. I tried reproducing the failing test locally but couldn't. I should give it another go today.

tnull · 2025-05-12T07:56:57Z

Thanks for looking at this. I've rebased and have just the CLN integration tests failing. I tried reproducing the failing test locally but couldn't. I should give it another go today.

Yes, I think this is unrelated, so feel free to ignore the flaky CI in this PR. Fixing it is really an unrelated task.

enigbe · 2025-05-12T07:57:46Z

Looking into this PR has helped me gain a good understanding of how chain sourcing works. I think introducing BitcoindSyncClient is a good idea as the user can still carry out syncing operations using the REST API while the RPC Client is still maintained to carry out operations not supported by the REST API. I have just a few suggestions with regards to error handling.

Thanks for the review. Added the suggestions recommended for error handling.

enigbe · 2025-05-12T12:39:18Z

@enigbe Please let me know when this is ready for a second look.

This is ready @tnull.

tnull

Hmm, tbh. I still dislike the current refactorings as they make the codepaths unnecessarily confusing and duplicative. Why keep a BitcoindRpcClient around side-by-side with the new BitcoindSyncClient that could also be an RPC client? If you maintain that merging in the REST codepaths with the RPC client is the way to go, why not extend the enum BitcoindSyncClient variants so that one would only hold the RPC client and the other the RPC and REST clients, which would allow us to add an fn rpc_client accessor that we can use everywhere, i.e., hence would allow us to drop the bitcoind_rpc_client field?

tnull · 2025-05-13T12:53:50Z

src/builder.rs

+	///
+	/// ## Parameters:
+	/// * `rest_host`, `rest_port` - Required parameters for the Bitcoin Core REST connection.
+	pub fn set_chain_source_bitcoind_rest(


Hmm, I really dislike this API design introducing a separate chain source setter here. Basically, the fact that you had to introduce two new error variants to handle the cases where users misuse this API call already tells you that there is room for improvement here. Why not just have an entirely separate set_chain_source_bitcoind_rest call that take all the necessary fields, including the RPC configs?

I'll refactor to include the recommended changes . I thought having to pass RPC config for REST would be confusing but I now agree that it would be clearer and less likely to misuse.

tnull · 2025-05-13T12:55:05Z

src/builder.rs

+		rpc_port: u16,
+		rpc_user: String,
+		rpc_password: String,
+		sync_client_config: Option<BitcoindRestSyncClientConfig>,


Mhh, let's name this rest_client_config then.

tnull · 2025-05-13T13:01:13Z

src/chain/mod.rs

@@ -186,6 +190,52 @@ impl ElectrumRuntimeStatus {
 	}
 }

+pub(crate) enum BitcoindSyncClient {


I'm not sure why this was added here vs bitcoind_rpc.rs.

enigbe added 2 commits April 17, 2025 22:50

feat: initial bitcoind sync client support

822fe46

Adds: - Sync client configuration with RPC (default) and REST variants - Extend the ChainSource::Bitcoind configuration options - Preliminary implementation for BitcoindSyncClient in RPC mode

test: channel full cycle with chain source in REST sync

e4adcfc

Additionally adds the BlockSource implementation for BitcoindSyncClient

enigbe changed the title ~~feat: chain source rest sync with bitcoind~~ feat: chain source rest sync Apr 18, 2025

ldk-reviews-bot requested a review from joostjager April 18, 2025 10:21

tnull requested review from tnull and removed request for joostjager April 18, 2025 10:47

tnull reviewed Apr 18, 2025

View reviewed changes

refactor: configure rest sync

3ceff37

- Adds set_chain_source_bitcoind_rest to configure REST synchronization. - Update docs to reflect the configuration order if REST sync is preferred. - Simplify the configuration object to capture only REST-related fields.

chuksys reviewed May 6, 2025

View reviewed changes

src/builder.rs Outdated Show resolved Hide resolved

src/builder.rs Outdated Show resolved Hide resolved

src/builder.rs Outdated Show resolved Hide resolved

refactor: replace panic statements with appropriate build error

a28d28f

enigbe force-pushed the 2025-04-bitcoind-rest-sync-v2 branch from 9b07570 to a28d28f Compare May 7, 2025 08:28

tnull reviewed May 13, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: chain source rest sync #526

feat: chain source rest sync #526

enigbe commented Apr 18, 2025 •

edited

Loading

ldk-reviews-bot commented Apr 18, 2025 •

edited

Loading

tnull left a comment •

edited

Loading

enigbe commented Apr 19, 2025 •

edited

Loading

tnull commented Apr 22, 2025

Camillarhi commented May 1, 2025

chuksys left a comment

tnull commented May 12, 2025

enigbe commented May 12, 2025

enigbe commented May 12, 2025

tnull commented May 12, 2025

enigbe commented May 12, 2025

enigbe commented May 12, 2025

tnull left a comment

tnull May 13, 2025

enigbe May 14, 2025

tnull May 13, 2025

tnull May 13, 2025

feat: chain source rest sync #526

Are you sure you want to change the base?

feat: chain source rest sync #526

Conversation

enigbe commented Apr 18, 2025 • edited Loading

What this PR does

Related Issues

ldk-reviews-bot commented Apr 18, 2025 • edited Loading

tnull left a comment • edited Loading

Choose a reason for hiding this comment

enigbe commented Apr 19, 2025 • edited Loading

tnull commented Apr 22, 2025

Camillarhi commented May 1, 2025

chuksys left a comment

Choose a reason for hiding this comment

tnull commented May 12, 2025

enigbe commented May 12, 2025

enigbe commented May 12, 2025

tnull commented May 12, 2025

enigbe commented May 12, 2025

enigbe commented May 12, 2025

tnull left a comment

Choose a reason for hiding this comment

tnull May 13, 2025

Choose a reason for hiding this comment

enigbe May 14, 2025

Choose a reason for hiding this comment

tnull May 13, 2025

Choose a reason for hiding this comment

tnull May 13, 2025

Choose a reason for hiding this comment

enigbe commented Apr 18, 2025 •

edited

Loading

ldk-reviews-bot commented Apr 18, 2025 •

edited

Loading

tnull left a comment •

edited

Loading

enigbe commented Apr 19, 2025 •

edited

Loading