TESTNET: unable to broadcast justice tx ... Witness program hash mismatch

Background

Testing/learning lightning and the various tx's on TESTNET.

Having a channel between a c-lightning and a lnd node on my lan. A revoked commit-tx with htlc-output was captured manually from the c-lightning node and subsequently broadcasted. Lnd detects the breach, constructs the penalty-tx ... but fails to broadcast it because the Witness script hash for the htlc-output does not match. Had the same error yesterday which I could reproduced today with clean settings and latest builds.

Funding-tx ID: 487f43bca6862b7259557936f65ad3854011bf2c6e50b94030f61676f3dccc32
Revoked commit-tx ID: ad7961bb2f99358c743cfe2f90d0dd39fb34a308a7d3b16ad99dcaacdc941c29

Here is part of the lnd logfile about penalty-tx:

2018-03-29 15:36:06.152 [DBG] BRAR: Broadcasting justice tx: (*wire.MsgTx)(0xc4206d8580)({
 Version: (int32) 2,
 TxIn: ([]*wire.TxIn) (len=3 cap=15) {
  (*wire.TxIn)(0xc4206a6ea0)({
   PreviousOutPoint: (wire.OutPoint) ad7961bb2f99358c743cfe2f90d0dd39fb34a308a7d3b16ad99dcaacdc941c29:0,
   SignatureScript: ([]uint8) <nil>,
   Witness: (wire.TxWitness) (len=2 cap=2) {
    ([]uint8) (len=72 cap=144) {
     00000000  30 45 02 21 00 b5 b2 60  4a 46 3a d2 ea 6e 36 66  |0E.!...`JF:..n6f|
     00000010  87 18 e9 44 a0 25 ff eb  57 ed 53 8d 7f 99 8a 3a  |...D.%..W.S....:|
     00000020  dd 47 48 a9 3b 02 20 41  04 b5 61 ba cc 89 c9 07  |.GH.;. A..a.....|
     00000030  a4 2b 0e 42 97 63 2e ea  6c bb 5b 86 b7 11 c4 a3  |.+.B.c..l.[.....|
     00000040  f1 69 7e 30 be d8 2d 01                           |.i~0..-.|
    },
    ([]uint8) (len=33 cap=33) {
     00000000  03 9e e6 6b b4 11 ed 36  dd c0 5c 0e b3 d8 13 db  |...k...6..\.....|
     00000010  b2 cb 63 d3 87 9f ce 14  13 18 bd a0 38 31 be 81  |..c.........81..|
     00000020  0a                                                |.|
    }
   },
   Sequence: (uint32) 0
  }),
  (*wire.TxIn)(0xc4206a6f00)({
   PreviousOutPoint: (wire.OutPoint) ad7961bb2f99358c743cfe2f90d0dd39fb34a308a7d3b16ad99dcaacdc941c29:2,
   SignatureScript: ([]uint8) <nil>,
   Witness: (wire.TxWitness) (len=3 cap=3) {
    ([]uint8) (len=71 cap=144) {
     00000000  30 44 02 20 21 b2 31 85  ea ca ba 0a e9 65 e1 7c  |0D. !.1......e.||
     00000010  15 5e 75 4c 14 fe 27 7c  30 c1 c5 df 1b 11 7c e9  |.^uL..'|0.....|.|
     00000020  d7 6e 0a 02 02 20 08 29  f6 a2 14 06 c1 e6 d2 99  |.n... .)........|
     00000030  79 0b 32 36 59 a0 1c 61  c2 de a2 86 4e 65 0a dc  |y.26Y..a....Ne..|
     00000040  82 01 93 b7 06 3f 01                              |.....?.|
    },
    ([]uint8) (len=1 cap=1) {
     00000000  01                                                |.|
    },
    ([]uint8) (len=77 cap=77) {
     00000000  63 21 02 50 a3 9e 34 59  83 75 ae 4d 46 5b e8 5d  |c!.P..4Y.u.MF[.]|
     00000010  28 09 c2 2e 91 cc 6e f7  69 7a 57 8f d6 81 0d a8  |(.....n.izW.....|
     00000020  ac c4 51 67 02 90 00 b2  75 21 03 3e 5a 65 70 d4  |..Qg....u!.>Zep.|
     00000030  98 f3 54 a6 3f 0b f2 cc  32 46 3f 69 a2 a4 b3 b1  |..T.?...2F?i....|
     00000040  cd d3 e7 50 be 4e 29 e1  8a da 82 68 ac           |...P.N)....h.|
    }
   },
   Sequence: (uint32) 0
  }),
  (*wire.TxIn)(0xc4206a6f60)({
   PreviousOutPoint: (wire.OutPoint) ad7961bb2f99358c743cfe2f90d0dd39fb34a308a7d3b16ad99dcaacdc941c29:1,
   SignatureScript: ([]uint8) <nil>,
   Witness: (wire.TxWitness) (len=3 cap=3) {
    ([]uint8) (len=72 cap=144) {
     00000000  30 45 02 21 00 f3 d6 80  b3 0e f4 cd 31 72 a2 91  |0E.!........1r..|
     00000010  10 43 e6 03 7e 40 44 0f  dd 3a 2c 4f 7f 15 15 c5  |.C..~@D..:,O....|
     00000020  59 a5 0e b5 23 02 20 54  39 32 3c c2 2f 62 10 97  |Y...#. T92<./b..|
     00000030  1a 9b 26 76 7b 87 71 e8  b7 e1 7e db 35 9d a4 28  |..&v{.q...~.5..(|
     00000040  64 e8 11 e4 d8 52 9f 01                           |d....R..|
    },
    ([]uint8) (len=33 cap=33) {
     00000000  02 50 a3 9e 34 59 83 75  ae 4d 46 5b e8 5d 28 09  |.P..4Y.u.MF[.](.|
     00000010  c2 2e 91 cc 6e f7 69 7a  57 8f d6 81 0d a8 ac c4  |....n.izW.......|
     00000020  51                                                |Q|
    },
    ([]uint8) (len=133 cap=133) {
     00000000  76 a9 14 06 60 c4 ae 39  dc 32 5f 97 54 f7 91 97  |v...`..9.2_.T...|
     00000010  6e 80 1d 8b 02 2f df 87  63 ac 67 21 02 b4 94 ce  |n..../..c.g!....|
     00000020  e9 b3 ca d2 04 72 fa 5e  03 78 3e 44 0b 85 8f fc  |.....r.^.x>D....|
     00000030  ff 15 b1 f2 de 00 81 e8  3f fe d5 87 af 7c 82 01  |........?....|..|
     00000040  20 87 64 75 52 7c 21 03  d3 95 16 2f 93 92 78 49  | .duR|!..../..xI|
     00000050  ac 58 66 ab 9a 19 97 d1  e7 34 af 78 be c8 6b 4c  |.Xf......4.x..kL|
     00000060  b3 10 09 d7 48 a1 dc 1d  52 ae 67 a9 14 94 e1 9b  |....H...R.g.....|
     00000070  25 70 98 c4 d6 7e 3d a4  59 54 1c fa fb dc e1 cb  |%p...~=.YT......|
     00000080  c7 88 ac 68 68                                    |...hh|
    }
   },
   Sequence: (uint32) 0
  })
 },
 TxOut: ([]*wire.TxOut) (len=1 cap=15) {
  (*wire.TxOut)(0xc4201f57e0)({
   Value: (int64) 557669,
   PkScript: ([]uint8) (len=22 cap=500) {
    00000000  00 14 81 9c bf 57 e9 53  5f e2 0e 40 44 40 f8 54  |.....W.S_..@D@.T|
    00000010  95 2e 08 55 11 86                                 |...U..|
   }
  })
 },
 LockTime: (uint32) 0
})

2018-03-29 15:36:06.155 [INF] LNWL: Inserting unconfirmed transaction 78ca2785440bdcccdf9a1580e85fa2a03557e08219bc198379370c7e4767196d
2018-03-29 15:36:06.266 [ERR] BRAR: unable to broadcast justice tx: -26: 64: non-mandatory-script-verify-flag (Witness program hash mismatch)

From above log result, for outpoint ad7961bb2f99358c743cfe2f90d0dd39fb34a308a7d3b16ad99dcaacdc941c29:1 the wscript=76a9140660c4ae39dc325f9754f791976e801d8b022fdf8763ac672102b494cee9b3cad20472fa5e03783e440b858ffcff15b1f2de0081e83ffed587af7c820120876475527c2103d395162f93927849ac5866ab9a1997d1e734af78bec86b4cb31009d748a1dc1d52ae67a91494e19b257098c4d67e3da459541cfafbdce1cbc788ac6868
which hashes to sha256(wscript)=fc31809b74fccdbe9b82c8a2fb875e7b5a9cd999aac58b3c1f93e3295a6c75f4

Where it should be: a93f67d3e83bdf68670d7b92003a884c721b34f03e174d8898d9aae9bc3e4fc2
according to the corresponding redeemscript in the broadcasted revoked commit-tx. Other two hashes seem correct.

Your environment

• version of lndversion 0.4, did 'git pull' and build yesterday, but also a 'git pull origin master' today (I wasn't not so sure about git)
• Linux linux-tb68 4.4.114-42-default Fix name typo in README #1 SMP Tue Feb 6 10:58:10 UTC 2018 (b6ee9ae) x86_64 x86_64 x86_64 GNU/Linux
• Bitcoin Core RPC client version v0.16.0
• ./lnd --bitcoind.rpcuser=xxx --bitcoind.rpcpass=mysecret --bitcoind.zmqpath=tcp://127.0.0.1:18501 --bitcoin.defaultchanconfs=1
• two c-lightning nodes on other machine (Debian) connected to lnd's machine via lan

Steps to reproduce

1. Run two c-lightning nodes on one machine, call them Debian1 and Debian3, each node listening on a separate port.
2. Run lnd node on another machine (OpenSuse).
3. Create channels: Debian1-->Lenovo and Debian3-->Lenovo.
4. Do some payments between the three nodes.
5. During a 2-hop payment from Debian1-->Debian3:

cli/lightning-cli pay lntb1200u1pdtezlcpp592xglzmrn2pc3ttn9hn95lzuyuwf5uac72up97gxwz7qdr6qzqnsdq2w3jhxazl8qcqpxrpqtpe0cle3r43mc0t8qlr6skx7hr5ujpcqadh2avjrg6tpagfu45uhsvdhl5pfmqxp3nv9zptsvsqpc9suuam3u5mdcaalpewasf8gpalsqzp

capture commit-tx on Debian1 by running in another terminal simultaneously:

while true; do cli/lightning-cli dev-sign-last-tx 02b5941e79c1858ba8f339d4a386c1f62af49f3159d7de7044b39958b8c377ae71; sleep 0.01; done > signed-last-commit-tx8c

6. Do another payment from Debian1 --> Lenovo

cli/lightning-cli pay lntb1m1pdte9jcpp5k8gf35d4dwx8vycv96z9ufp0pveczshr7wnqxc5t3p7hw9e383dsdqvw3jhxazl8p3scqpx5qu28f02gwjcvtc8ycjczv2ycz4mltupzy78tseh2lpyw4njj609dygkl0lnphpm45nu6d6q2x7v0mfm027wlyq6xtfup270us4uracpp4w3cd

7. Broadcast the the commit-tx (the one with htlc-output) captured in step 5
8. Wait halve a day for confirmation of 7 ... some testnet-miner censorship ?
9. Wait for htlc-timeout to expire and Debian1 can reverse payment 6 ?

Beside above issue, I had other issues with lnd node.

• after or before revoked commit-tx got confrmed, lncli would not repsond

[lncli] rpc error: code = Unknown desc = Post http://localhost:18332: dial tcp [::1]:18332: connect: connection refused

bitcoind was crashed, restarting bitcoind and lnd solved above.

Not sure if related but in above lnd's version I changed in fundingmanager.go:

+       feePerVSize, err := f.cfg.FeeEstimator.EstimateFeePerVSize(10) //temporary for testnet's unstable estimatesmartfee results, by simon

To get a more stable (low) feerate on testnet, but this was done and build after creating above channels.

[halseth]

@SimonVrouwe This is a very interesting issue! Any chance you got more of the lnd logs? (or if you can repro try capturing as much of the logs as possible)

Here are log files, the part I pasted above is from lnd.log.14
lnd.log.15.gz
lnd.log.14.gz
lnd.log.13.gz
lnd.log

older log files are missing, are they auto cleaned-up?

[edit] Forgot something, after step 6 and before step 7 (broadcasting): shutdown Debian1 node

[Roasbeef]

bitcoind was crashed, restarting bitcoind and lnd solved above

Why did bitcoind crash? What error did it encounter?

older log files are missing, are they auto cleaned-up?

Log file atm are rotated automatically. However, there's an issue to make this optional.

Decoding the script we get:

{
  "asm": "OP_DUP OP_HASH160 0660c4ae39dc325f9754f791976e801d8b022fdf OP_EQUAL OP_IF OP_CHECKSIG OP_ELSE 02b494cee9b3cad20472fa5e03783e440b858ffcff15b1f2de0081e83ffed587af OP_SWAP OP_SIZE 20 OP_EQUAL OP_NOTIF OP_DROP 2 OP_SWAP 03d395162f93927849ac5866ab9a1997d1e734af78bec86b4cb31009d748a1dc1d 2 OP_CHECKMULTISIG OP_ELSE OP_HASH160 94e19b257098c4d67e3da459541cfafbdce1cbc7 OP_EQUALVERIFY OP_CHECKSIG OP_ENDIF OP_ENDIF",
  "type": "nonstandard",
  "p2sh": "32QYNTJSqD4njaRezdFfSZNb7PBVr8SWs1"
}

This is actually the sender's HTLC script, meaning you captured lnd's commitment transaction, rather than the commitment transaction of c-lightning. So this is akin to tricking lnd itself to broadcast its prior commitment transaction.

If we take a look at this fragment: https://github.com/lightningnetwork/lnd/blob/master/lnwallet/channel.go#L1990-L2014

You'll see that we generate the script based on if this was an incoming our outgoing HTLC. The description of your setup above was a bit unclear, but it seems like:

• lnd gets forwarded incoming HTLC
• its commitment carries the receiver's HTLC script as a result
• tx gets breached, but the breach is actually the commitment transaction of lnd
• we go to find which script to use
• our commitment was broadcast, but we think it's theirs, so we generate the sender's script
• when we go to broadcast, this is actually incorrect, as we're generating things from the PoV of the remote commitment

So I'm not sure if this is actually a bug or not, as we wouldn't have been able to sweep the outputs, since the the transaction should actually be invalid all together since we can't execute our own revocation clause. I could be wrong, but this is my current understanding, will revisit this in a bit to ensure I have the correct picture.

[Roasbeef]

Were all 3 nodes sharing the same bitcoind?

[Roasbeef]

Also it just seems that there's a lack of segwit enabled miners on testnet atm, so that would explain why it took so long for the transaction to get mined.

[Roasbeef]

Also the hash of the script above is actually: f0119fdd51e86b8eed1205501ca92244f4c2b50ed537f515e597b5c546d9f6d1.

[Roasbeef]

Do you know what block height this was at? I'm trying to reconstruct what the script for the receiver's HTLC would have been.

[Roasbeef]

Actually it wouldn't be possible for clightning to broadcast our prior state. So still not sure what happened here, partially due to the ambiguity in the report w.r.t the set up of the nodes (though the rest of the report was fantastic!).

[Roasbeef]

Also, are you able to repro on the current master?

I must say that my setup is a bit messy/chaotic and a lot of things go wrong. High probability that I made a mistake somewhere or forgot to mention a step or just don't know what I am doing. But because this was the second time I encountered this error I reported it anyway. Will try to sort it out.

Were all 3 nodes sharing the same bitcoind?

no, the c-lightning nodes share one bitcoind and lnd has its own

Also the hash of the script above is actually: f0119fdd51e86b8eed1205501ca92244f4c2b50ed537f515e597b5c546d9f6d1

for the wscript for outpoint n=1 (witness_v0_scripthash), I got sha256(wscript)=fc31809b74fccdbe9b82c8a2fb875e7b5a9cd999aac58b3c1f93e3295a6c75f4
(using sha256 on anyhash.com) Am I doing something wrong here?

Do you know what block height this was at? I'm trying to reconstruct what the script for the receiver's HTLC would have been.

the revoked commit-tx was mined in block (testnet): 1289539

The commit-tx's where captured using the dev-sign-last-tx command of c-lightning cli. Not so sure about this, but that command takes the other party's (latest) half-signed commit-tx it has in possession and just adds it's own signature so it can be broadcasted?

[edit] in step 2, "OpenSuse" should be "Lenovo"
Lenovo is the name of the lnd node.

[Roasbeef]

The commit-tx's where captured using the dev-sign-last-tx command of c-lightning cli. Not so sure about this, but that command takes the other party's (latest) half-signed commit-tx it has in possession and just adds it's own signature so it can be broadcasted?

This isn't possible. You can't take the other party's commitment, sign it and broadcast it. Otherwise, it would be possible to make the remote party violate the contract at any point. The remote party only sends you sigs of your commitment. I'll need to go through their source to see exactly what the command does.