Commit(s) pushed to ElementsProject/lightning #1
+665,146
−3,923
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
whitslack
changed the title
Repository owner
locked and limited conversation to collaborators
Oct 24, 2019
cdecker
force-pushed
the
master
branch
2 times, most recently
from
bef4422 to
af4eec7
Compare
Fixes: #4873 In particular, we used to get upset when a peer accepts our channel, if it was too small! We should do reasonable checks first. We no longer try to send requests to delay for 2017 blocks though, so remove that test. Signed-off-by: Rusty Russell <[email protected]> Changelog-Fixed: Protocol: trying to create a channel below our own min-capacity-sat will now fail before asking the peer, not with an error blaming the peer when they accept!
…hot. This way we can be sure they're the same after migration. Signed-off-by: Rusty Russell <[email protected]>
Signed-off-by: Rusty Russell <[email protected]>
Signed-off-by: Rusty Russell <[email protected]>
Note that bookkeeper de-duplicates chain_moves: we need to too! So we add an index to make this efficient. Signed-off-by: Rusty Russell <[email protected]>
…> nonchannel in db when we close it. This avoids us keeping references into closed channels. Signed-off-by: Rusty Russell <[email protected]>
…ables. We change notify_chain_mvt to wallet_save_chain_mvt, and notify_channel_mvt to wallet_save_channel_mvt, which save to the db and call the notifier themselves. Signed-off-by: Rusty Russell <[email protected]>
Signed-off-by: Rusty Russell <[email protected]>
This is mainly for the coming list commands, but it's just more logical to have the common fields at the top. Signed-off-by: Rusty Russell <[email protected]>
Signed-off-by: Rusty Russell <[email protected]>
…nmoves. Iterating through every peer and channel every time can be very slow for large nodes, when calling wallet_coinmoves_extract for listcoinmoves. Signed-off-by: Rusty Russell <[email protected]>
This is where all the previous work pays off: we can access the coinmoves in the db. Changelog-Added: JSON-RPC: `listchainmoves` and `listchannelmoves` commands to access the audit log of coin movements. Signed-off-by: Rusty Russell <[email protected]>
Signed-off-by: Rusty Russell <[email protected]>
Note that the channel account on l1 doesn't account for the onchain-fulfilled HTLC: ``` > assert sum(channel1[channel_id]) == -msats_sent_to_2 E assert -50000000000 == -50100000000 E + where -50000000000 = sum([0, -50000000000]) ``` Lisa points out that this is accounted for in the chain moves, instead. Signed-off-by: Rusty Russell <[email protected]>
This happens if l1 doesn't get a signature, so it doesn't consider the fulfill complete, but l2 does (and thus credits l1). This is a trivial test, but will matter should we later correctly account for channelmoves when onchain: in that case it will actually hard to tell, in general, what HTLC(s) were fulfilled. Signed-off-by: Rusty Russell <[email protected]>
Signed-off-by: Rusty Russell <[email protected]>
Only makes sense to wait on creation, since they neither are deleted nor updated. We also enhance the list commands to take the standard index options. Signed-off-by: Rusty Russell <[email protected]> Changelog-Added: JSON-RPC: `wait`: new subsystems `chainmoves` and `channelmoves`.
And note the other commands in See Also section. Note that this means handling the "outpoint" type. Signed-off-by: Rusty Russell <[email protected]> Changelog-Added: JSON-RPC: `sql` plugin now supports `chainmoves` and `channelmoves` tables.
We don't yet do the other list commands, as they are not append-only: we would need to check deletes and updates. Signed-off-by: Rusty Russell <[email protected]>
It's a unique integer, and very useful for querying changes. Unlike
our generated rowid, it's *stable* across queries.
We still need an explicit rowid column for list commands which don't
(currently) have this.
Here's the documentation diff:
@@ -85,69 +85,69 @@
TABLES
------
-Note that the first column of every table is a unique integer called `rowid`: this is used for related tables to refer to specific rows in their parent. sqlite3 usually has this as an implicit column, but we make it explicit as the implicit version is not allowed to be used as a foreign key.
+Note that tables which have a `created_index` field use that as the primary key (and `rowid` is an alias to this), otherwise an explicit `rowid` integer primary key is generated, whose value changes on each refresh. This field is used for related tables to refer to specific rows in their parent. (sqlite3 usually has this as an implicit column, but we make it explicit as the implicit version is not allowed to be used as a foreign key).
The following tables are currently supported:
- `bkpr_accountevents` (see lightning-bkpr-listaccountevents(7))
@@ -119,14 +119,14 @@
- `payment_id` (type `hex`, sqltype `BLOB`)
- `chainmoves` indexed by `account_id` (see lightning-listchainmoves(7))
- - `created_index` (type `u64`, sqltype `INTEGER`)
+ - `created_index` (type `u64`, sqltype `INTEGER PRIMARY KEY`)
- `account_id` (type `string`, sqltype `TEXT`)
- `credit_msat` (type `msat`, sqltype `INTEGER`)
- `debit_msat` (type `msat`, sqltype `INTEGER`)
- `timestamp` (type `u64`, sqltype `INTEGER`)
- `primary_tag` (type `string`, sqltype `TEXT`)
- related table `chainmoves_extra_tags`
- - `row` (reference to `chainmoves.rowid`, sqltype `INTEGER`)
+ - `row` (reference to `chainmoves.created_index`, sqltype `INTEGER`)
- `arrindex` (index within array, sqltype `INTEGER`)
- `extra_tags` (type `string`, sqltype `TEXT`)
- `peer_id` (type `pubkey`, sqltype `BLOB`)
@@ -139,7 +139,7 @@
- `blockheight` (type `u32`, sqltype `INTEGER`)
- `channelmoves` indexed by `account_id` (see lightning-listchannelmoves(7))
- - `created_index` (type `u64`, sqltype `INTEGER`)
+ - `created_index` (type `u64`, sqltype `INTEGER PRIMARY KEY`)
- `account_id` (type `string`, sqltype `TEXT`)
- `credit_msat` (type `msat`, sqltype `INTEGER`)
- `debit_msat` (type `msat`, sqltype `INTEGER`)
@@ -204,7 +204,7 @@
- `last_stable_connection` (type `u64`, sqltype `INTEGER`)
- `forwards` indexed by `in_channel` and `in_htlc_id` (see lightning-listforwards(7))
- - `created_index` (type `u64`, sqltype `INTEGER`)
+ - `created_index` (type `u64`, sqltype `INTEGER PRIMARY KEY`)
- `in_channel` (type `short_channel_id`, sqltype `TEXT`)
- `in_htlc_id` (type `u64`, sqltype `INTEGER`)
- `in_msat` (type `msat`, sqltype `INTEGER`)
@@ -222,7 +222,7 @@
- `htlcs` indexed by `short_channel_id` and `id` (see lightning-listhtlcs(7))
- `short_channel_id` (type `short_channel_id`, sqltype `TEXT`)
- - `created_index` (type `u64`, sqltype `INTEGER`)
+ - `created_index` (type `u64`, sqltype `INTEGER PRIMARY KEY`)
- `updated_index` (type `u64`, sqltype `INTEGER`)
- `id` (type `u64`, sqltype `INTEGER`)
- `expiry` (type `u32`, sqltype `INTEGER`)
@@ -242,7 +242,7 @@
- `bolt12` (type `string`, sqltype `TEXT`)
- `local_offer_id` (type `hash`, sqltype `BLOB`)
- `invreq_payer_note` (type `string`, sqltype `TEXT`)
- - `created_index` (type `u64`, sqltype `INTEGER`)
+ - `created_index` (type `u64`, sqltype `INTEGER PRIMARY KEY`)
- `updated_index` (type `u64`, sqltype `INTEGER`)
- `pay_index` (type `u64`, sqltype `INTEGER`)
- `amount_received_msat` (type `msat`, sqltype `INTEGER`)
@@ -408,7 +408,7 @@
- `features` (type `hex`, sqltype `BLOB`)
- `sendpays` indexed by `payment_hash` (see lightning-listsendpays(7))
- - `created_index` (type `u64`, sqltype `INTEGER`)
+ - `created_index` (type `u64`, sqltype `INTEGER PRIMARY KEY`)
- `id` (type `u64`, sqltype `INTEGER`)
- `groupid` (type `u64`, sqltype `INTEGER`)
- `partid` (type `u64`, sqltype `INTEGER`)
Changelog-Changed: Plugins: `sql` tables `forwards`, `htlcs`, `invoices`, `sendpays` all use `created_index` as their primary key (and `rowid` is now an alias to this).
Signed-off-by: Rusty Russell <[email protected]>
If we do this, we get a database error (now we try to refresh intelligently, is this is currently only chainmoves / channelmoves). Signed-off-by: Rusty Russell <[email protected]>
Simply wait if there's one going already. This is a minor optimization, but critical for the case where we do partial refreshes asynchonously (rather than deleting everything and reloading). This is currently only coinmoves and chainmoves, but the duplicated effort is a waste everywhere. Signed-off-by: Rusty Russell <[email protected]>
Note that these migrations were inserted for v0.12, so only someone upgrading directly from before that (2022-08-23) would be affected. This avoids having to fix the migrations as we make changes. We are going to mangle the db to allow testing, but then the final step will be to migrate it to the core. Signed-off-by: Rusty Russell <[email protected]>
In this case, we make an immediately-expiring invoice. This correctly blocks any successive requests for invoices, as per the spec requirement. This means we have to handle invoice_requests without reply_path, amounts or quantity *if* they specify invreq_recurrence_cancel. Signed-off-by: Rusty Russell <[email protected]>
Signed-off-by: Rusty Russell <[email protected]>
Deprecated in 24.11, disabled in 25.05. Changelog-Removed: JSON-RPC: `decode` field `blinding` (use `first_path_key` as per modern BOLT naming) Signed-off-by: Rusty Russell <[email protected]>
Changelog-Removed: Plugins: `onion_message_recv` hook `blinding` field (use `first_path_key` as per modern BOLT 4 naming). Signed-off-by: Rusty Russell <[email protected]>
Deprecated 24.11, disabled 25.05 (they're the default now). Signed-off-by: Rusty Russell <[email protected]> Changelog-Removed: Config: --experimental-offers and --experimental-quiesce (default since v24.11)
Signed-off-by: Rusty Russell <[email protected]>
```
lightningd-1 2025-09-22T02:10:10.978Z **BROKEN** plugin-bookkeeper: Unparsable datastore ["bookkeeper","rebalances","1-2"]
```
And, indeed, rebalance is missing:
```
> outbound_ev = only_one([ev for ev in inc_evs if ev['tag'] == 'rebalance_fee'])
tests/test_bookkeeper.py:825:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
arr = []
def only_one(arr):
"""Many JSON RPC calls return an array; often we only expect a single entry
"""
> assert len(arr) == 1
E AssertionError
```
Signed-off-by: Rusty Russell <[email protected]>
Parse key correctly. Signed-off-by: Rusty Russell <[email protected]> Changelog-Fixed: bookkeeper: failed reload of rebalances on restart.
RPC documentation is not syncing on readme server with error `uv: command not found`. Changelog-None.
* tests/test_cln_lsps.py::test_lsps0_listprotocols * tests/test_clnrest.py * tests/test_connection.py::test_wss_proxy Changelog-Fixed: pytest: Tests that require Rust no longer fail if Rust is disabled.
This doesn't happen yet, since we delete all HTLCs when we close a channel. But we're about to change that, so update the wallet_htlcs_first() code to avoid them. Signed-off-by: Rusty Russell <[email protected]>
…rtup. For old channels, this can take a while, and it stops everything. But we are only doing this to save space; it's not a *functional* necessity. A quick and dirty test with 50,000 htlcs shows the htlc deletion took 450msec. I tried adding an index, and changing it to set hstate to HTLC_STATE_INVALID instead of deleting entries, but it still took about 350ms. Whereas the "COUNT(*)" only took 1.7msec, so it's worth keeping. Reported-by: @michael1011 Signed-off-by: Rusty Russell <[email protected]> Changelog-Changed: lightningd: we defer deletion of old htlcs on channel close, to avoid pausing for a long time (we clean them on startup) Fixes: #7962
…it for restart. Signed-off-by: Rusty Russell <[email protected]>
…migration. When we migrate from accounts.db, we use the `account_nonchannel_id` field. But we can replay the block chain and the channel involved is still open, we will use the `account_channel_id` field, and our duplicate detection fails. As a result, we can end up with duplicate entries in the database, which make accounting incorrect. Signed-off-by: Rusty Russell <[email protected]> Changelog-Fixed: JSON-RPC: `listchainmoves` could contain bogus duplicate entries after 25.09 bookkeeper migration.
Signed-off-by: Rusty Russell <[email protected]>
Signed-off-by: Rusty Russell <[email protected]>
This is based on a real database, which values changed. Signed-off-by: Rusty Russell <[email protected]>
The bitcoin.rpc.DEFAULT_HTTP_TIMEOUT of 30 seconds may not be enough time to generate a block when the test machine is under load. Pass pyln.testing.utils.TIMEOUT to bitcoin.rpc.RawProxy to allow extra time: currently 60 seconds by default or 180 seconds if SLOW_MACHINE is set. Changelog-None
This reverts commit 084b033.
We need to remove the feature bits set via a plugins get_manifest response when the init response disables the plugin. Changelog-Fixed Remove feature bits set by a plugin when the plugin disables itself during init. Signed-off-by: Peter Neuroth <[email protected]>
for python 3.3+ we don’t need an __init__.py at namespace package to allow extensions instead we can just get rid of them
duplicate definition isn’t required as we already defined them at workspace level sources
prevent updates to lock file which might leave working directory dirty resulting in a modded cln version build
This allows compatibility with python 3.14.0 which coincurve 21.0.0 did not support. The next coincurve release should restore compatibility. Fixes: #8591 Changelog-changed: pyln-testing requires python>=3.9.2
…entonion. Signed-off-by: Rusty Russell <[email protected]>
…nionmessage. In this case we have a failmsg, so we should use that. Otherwise we can have both failmsg and failonion NULL in the call to injectonion_fail, which is not valid. ``` DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-chan#1: Removing out HTLC 1 state RCVD_REMOVE_ACK_REVOCATION WIRE_INVALID_ONION_HMAC **BROKEN** lightningd: FATAL SIGNAL 11 (version v25.09-135-g19a3bbc-modded) **BROKEN** lightningd: backtrace: common/daemon.c:41 (send_backtrace) 0x6220e8fe0080 **BROKEN** lightningd: backtrace: common/daemon.c:78 (crashdump) 0x6220e8fe00cf **BROKEN** lightningd: backtrace: ./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0 ((null)) 0x73614bc4532f **BROKEN** lightningd: backtrace: lightningd/pay.c:1701 (injectonion_fail) 0x6220e8f951c0 **BROKEN** lightningd: backtrace: lightningd/pay.c:330 (tell_waiters_failed) 0x6220e8f943be **BROKEN** lightningd: backtrace: lightningd/pay.c:656 (payment_failed) 0x6220e8f98db1 **BROKEN** lightningd: backtrace: lightningd/peer_htlcs.c:313 (fail_out_htlc) 0x6220e8fa1d04 **BROKEN** lightningd: backtrace: lightningd/peer_htlcs.c:1988 (remove_htlc_out) 0x6220e8fa271b **BROKEN** lightningd: backtrace: lightningd/peer_htlcs.c:2086 (update_out_htlc) 0x6220e8fa2904 **BROKEN** lightningd: backtrace: lightningd/peer_htlcs.c:2095 (changed_htlc) 0x6220e8fa2c24 **BROKEN** lightningd: backtrace: lightningd/peer_htlcs.c:2608 (peer_got_revoke) 0x6220e8fa6e5a **BROKEN** lightningd: backtrace: lightningd/channel_control.c:1555 (channel_msg) 0x6220e8f62725 **BROKEN** lightningd: backtrace: lightningd/subd.c:560 (sd_msg_read) 0x6220e8fb2eed **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:60 (next_plan) 0x6220e90a3335 **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:422 (do_plan) 0x6220e90a3806 **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:439 (io_ready) 0x6220e90a38c3 **BROKEN** lightningd: backtrace: ccan/ccan/io/poll.c:455 (io_loop) 0x6220e90a524f **BROKEN** lightningd: backtrace: lightningd/io_loop_with_timers.c:22 (io_loop_with_timers) 0x6220e8f7d1c7 **BROKEN** lightningd: backtrace: lightningd/lightningd.c:1496 (main) 0x6220e8f82db2 **BROKEN** lightningd: backtrace: ../sysdeps/nptl/libc_start_call_main.h:58 (__libc_start_call_main) 0x73614bc2a1c9 **BROKEN** lightningd: backtrace: ../csu/libc-start.c:360 (__libc_start_main_impl) 0x73614bc2a28a **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x6220e8f53b64 **BROKEN** lightningd: backtrace: (null):0 ((null)) 0xffffffffffffffff ``` Reported-by: @michael1011 Changelog-Fixed: lightningd: potential crash when we receive a malformed onion complain from our first peer when using sendonion / injectpaymentonion. Signed-off-by: Rusty Russell <[email protected]>
Show the work we're doing (at debug level) and every 10 seconds print progress (at INFO level):x ``` lightningd-1 2025-10-08T05:13:07.973Z INFO lightningd: Creating database lightningd-1 2025-10-08T05:13:10.987Z DEBUG lightningd: Transferring 6166 chain_events lightningd-1 2025-10-08T05:13:11.780Z DEBUG lightningd: Transferring 1660043 channel_events ``` It's the inserting channel_events which takes a long time, slowing down exponentially: ``` lightningd-1 2025-10-08T05:13:18.034Z INFO lightningd: Inserted 26690/1660043 channel_events lightningd-1 2025-10-08T05:13:28.034Z INFO lightningd: Inserted 47086/1660043 channel_events lightningd-1 2025-10-08T05:13:38.035Z INFO lightningd: Inserted 61699/1660043 channel_events lightningd-1 2025-10-08T05:13:48.035Z INFO lightningd: Inserted 73743/1660043 channel_events lightningd-1 2025-10-08T05:13:58.035Z INFO lightningd: Inserted 83244/1660043 channel_events ... lightningd-1 2025-10-08T05:35:18.286Z INFO lightningd: Inserted 466720/1660043 channel_events lightningd-1 2025-10-08T05:35:29.074Z INFO lightningd: Inserted 468437/1660043 channel_events lightningd-1 2025-10-08T05:35:39.079Z INFO lightningd: Inserted 470130/1660043 channel_events lightningd-1 2025-10-08T05:35:49.081Z INFO lightningd: Inserted 471871/1660043 channel_events ``` Signed-off-by: Rusty Russell <[email protected]>
…n migrations. Before db is complete, ld->wallet->db is NULL. Signed-off-by: Rusty Russell <[email protected]>
Testing a large db shows Postgres slowing down exponentially as it inserts the channel_events. Rather than updating the index in the db every time, do it at the end, for spectacular speedup: ``` lightningd-1 2025-10-08T05:39:44.333Z INFO lightningd: Creating database lightningd-1 2025-10-08T05:39:47.581Z DEBUG lightningd: Transferring 6166 chain_events lightningd-1 2025-10-08T05:39:48.455Z DEBUG lightningd: Transferring 1660043 channel_events lightningd-1 2025-10-08T05:39:54.390Z INFO lightningd: Inserted 103100/1660043 channel_events lightningd-1 2025-10-08T05:40:04.390Z INFO lightningd: Inserted 283280/1660043 channel_events lightningd-1 2025-10-08T05:40:14.390Z INFO lightningd: Inserted 464065/1660043 channel_events lightningd-1 2025-10-08T05:40:24.390Z INFO lightningd: Inserted 629559/1660043 channel_events lightningd-1 2025-10-08T05:40:34.390Z INFO lightningd: Inserted 800659/1660043 channel_events lightningd-1 2025-10-08T05:40:44.390Z INFO lightningd: Inserted 975433/1660043 channel_events lightningd-1 2025-10-08T05:40:54.390Z INFO lightningd: Inserted 1134719/1660043 channel_events lightningd-1 2025-10-08T05:41:04.390Z INFO lightningd: Inserted 1290549/1660043 channel_events lightningd-1 2025-10-08T05:41:14.390Z INFO lightningd: Inserted 1443304/1660043 channel_events lightningd-1 2025-10-08T05:41:24.390Z INFO lightningd: Inserted 1590013/1660043 channel_events lightningd-1 2025-10-08T05:41:29.148Z INFO lightningd: bookkeeper migration complete: migrated 6166 chainmoves, 1660043 channelmoves, 132481 descriptions ``` Now we complete the entire migration in 1 minute 45 seconds. Thanks to @Michael1101 for reporting this. Signed-off-by: Rusty Russell <[email protected]> Changelog-Fixed: db: migration from v25.09 on a reasonable size account database could take almost infinite time.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
15 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR exists only to generate email notifications upon pushes to
ElementsProject/lightning:master.