[Bridge] qanet and testnet bridges keep crashing with no transactions has no signatures error

[hossnys]

Describe the bug

qanet and testnet bridges keep crashing with transactions has no signatures error

To Reproduce

Steps to reproduce the behavior:

1. try to bridge some tfts on qanet or testnet
2. check the bridge pod on that network
3. Scroll down to latest logs
4. You'll see this error and then the bridge will crash and restart

Expected

Transferred tft amount reach it's destination .

Logs:

{"level":"info","version":1,"source":{"Instance_public_key":"GA2CWNBUHX7NZ3B5GR4I23FMU7VY5RPA77IUJTIXTTTGKYSKDSV6LUA4","Bridge_wallet_address":"GA2CWNBUHX7NZ3B5GR4I23FMU7VY5RPA77IUJTIXTTTGKYSKDSV6LUA4","Stellar_network":"production","Tfchain_url":"wss://tfchain.test.grid.tf"},"trace_id":"11889","event_action":"withdraw_proposed","event_kind":"event","category":"withdraw","metadata":{"amount":10000000,"tx_id":"11889","to":"GAL3ZIMKRDU2EJRFO7KX3YIMC7MIDDNQP5EKZWD6XI2DT6GOIBAUNYG7"},"time":"2025-06-02T12:55:12Z","message":"a withdraw has proposed with the target stellar address of GAL3ZIMKRDU2EJRFO7KX3YIMC7MIDDNQP5EKZWD6XI2DT6GOIBAUNYG7"}
{"level":"fatal","version":1,"source":{"Instance_public_key":"GA2CWNBUHX7NZ3B5GR4I23FMU7VY5RPA77IUJTIXTTTGKYSKDSV6LUA4","Bridge_wallet_address":"GA2CWNBUHX7NZ3B5GR4I23FMU7VY5RPA77IUJTIXTTTGKYSKDSV6LUA4","Stellar_network":"production","Tfchain_url":"wss://tfchain.test.grid.tf"},"error":"an error occurred while handling WithdrawReadyEvents: transaction has no signatures","event_action":"bridge_unexpectedly_exited","event_kind":"error","category":"availability","time":"2025-06-02T12:55:12Z","message":"the bridge instance has exited unexpectedly"}

{"Instance_public_key":"GAQH7XXFBRWXT2SBK6AHPOLXDCLXVFAKFSOJIRMRNCDINWKHGI6UYVKM","Bridge_wallet_address":"GAQH7XXFBRWXT2SBK6AHPOLXDCLXVFAKFSOJIRMRNCDINWKHGI6UYVKM","Stellar_network":"testnet","Tfchain_url":"wss://tfchain.qa.grid.tf"},"trace_id":"ec4b3d078de16a45f89cea0476eebe1a66fc5b6b2a50ff04e8f3ce357f5626e2","event_action":"refund_proposed","event_kind":"event","category":"refund","metadata":{"reason":""},"time":"2025-06-02T17:55:06Z","message":"a refund has proposed due to "}
{"level":"info","version":1,"source":{"Instance_public_key":"GAQH7XXFBRWXT2SBK6AHPOLXDCLXVFAKFSOJIRMRNCDINWKHGI6UYVKM","Bridge_wallet_address":"GAQH7XXFBRWXT2SBK6AHPOLXDCLXVFAKFSOJIRMRNCDINWKHGI6UYVKM","Stellar_network":"testnet","Tfchain_url":"wss://tfchain.qa.grid.tf"},"trace_id":"8988a32efba23f019d759ec28e9f91e22a395a07894e6b8ec3fb3744fe21873e","event_action":"refund_proposed","event_kind":"event","category":"refund","metadata":{"reason":""},"time":"2025-06-02T17:55:12Z","message":"a refund has proposed due to "}
{"level":"info","version":1,"source":{"Instance_public_key":"GAQH7XXFBRWXT2SBK6AHPOLXDCLXVFAKFSOJIRMRNCDINWKHGI6UYVKM","Bridge_wallet_address":"GAQH7XXFBRWXT2SBK6AHPOLXDCLXVFAKFSOJIRMRNCDINWKHGI6UYVKM","Stellar_network":"testnet","Tfchain_url":"wss://tfchain.qa.grid.tf"},"time":"2025-06-02T17:55:12Z","message":"horizon error: \"Transaction Failed\" (tx_bad_seq) - check horizon.Error.Problem for more information"}
{"level":"error","version":1,"source":{"Instance_public_key":"GAQH7XXFBRWXT2SBK6AHPOLXDCLXVFAKFSOJIRMRNCDINWKHGI6UYVKM","Bridge_wallet_address":"GAQH7XXFBRWXT2SBK6AHPOLXDCLXVFAKFSOJIRMRNCDINWKHGI6UYVKM","Stellar_network":"testnet","Tfchain_url":"wss://tfchain.qa.grid.tf"},"error":"horizon error: \"Transaction Failed\" (tx_bad_seq) - check horizon.Error.Problem for more information","time":"2025-06-02T17:55:12Z","message":"error while submitting transaction map[envelope_xdr:AAAAAgAAAAAgf97lDG156kFXgHe5dxiXepQKLJyURZFohobZRzI9TAABhqAABSxPAAAY7QAAAAEAAAAAAAAAAAAAAAAAAAAAAAAABJU+jAY/uPONKu4eXeCMQ9stKNHBOhfenlfbBw5krKZ7AAAAAQAAAAEAAAAAIH/e5QxteepBV4B3uXcYl3qUCiyclEWRaIaG2UcyPUwAAAABAAAAACB/3uUMbXnqQVeAd7l3GJd6lAosnJRFkWiGhtlHMj1MAAAAAVRGVAAAAAAAOfxkG3qLTLHrhsPS6JsSUB7+ZjU/J4oT1YBMKb/3n2QAAAAAAJiWgAAAAAAAAAABRzI9TAAAAECJSwxhJEt9j8fCfyUDcS0SLeNzGs3islm3YBArC8BhBWd/KmBoEOZVTA3Op4qjNH0+iCoI7hiAK9Gbj1GvKnsM result_codes:map[transaction:tx_bad_seq] result_xdr:AAAAAAAAAGT////7AAAAAA==]"}
{"level":"info","version":1,"source":{"Instance_public_key":"GAQH7XXFBRWXT2SBK6AHPOLXDCLXVFAKFSOJIRMRNCDINWKHGI6UYVKM","Bridge_wallet_address":"GAQH7XXFBRWXT2SBK6AHPOLXDCLXVFAKFSOJIRMRNCDINWKHGI6UYVKM","Stellar_network":"testnet","Tfchain_url":"wss://tfchain.qa.grid.tf"},"time":"2025-06-02T17:55:12Z","message":"resetting account sequence"}
{"level":"fatal","version":1,"source":{"Instance_public_key":"GAQH7XXFBRWXT2SBK6AHPOLXDCLXVFAKFSOJIRMRNCDINWKHGI6UYVKM","Bridge_wallet_address":"GAQH7XXFBRWXT2SBK6AHPOLXDCLXVFAKFSOJIRMRNCDINWKHGI6UYVKM","Stellar_network":"testnet","Tfchain_url":"wss://tfchain.qa.grid.tf"},"error":"an error occurred while handling RefundReadyEvents: an error occurred while submitting the transaction: horizon error: \"Transaction Failed\" (tx_bad_seq) - check horizon.Error.Problem for more information","event_action":"bridge_unexpectedly_exited","event_kind":"error","category":"availability","time":"2025-06-02T17:55:12Z","message":"the bridge instance has exited unexpectedly"}

[AhmedHanafy725]

This occurs due to a race condition while the bridge is trying to execute the ready events and the expiration of these events on TFChain.

• The ready events took 20 blocks on TFChain (2 mins) to expire, which means TFChain will remove the signatures from the transaction.
• As the bridge runs sequentially, it attempts to serve the events one by one. So if there is a crowd of transactions, it might take more than 2 mins to handle these ready events
• So that the ready transactions must contain signatures to continue submitting the transaction, but it will be gone if the bridge exceeds 2 mins to handle it.
• This will result in an unhandled exception that crashes the main process.

To resolve that:

• In case the signatures are gone from the transaction while handling the ready events, we should just ignore the transaction, as the expiration events will be emitted to re-handle this transaction

ready events: [RefundTransactionReady, BurnTransactionReady]
expired events: [RefundTransactionExpired, BurnTransactionExpired]

[AlaaElattar]

Work Completed:

• Handled the ready transactions to be skipped, if no signatures found.

[sameh-farouk]

Seems this wasn't enough.
It’s primarily a scalability and efficiency issue, rather than just a random race condition that we attempted to mitigate through retries.
Since the bridge no longer crashes but still has issues handling the accumulated work, I openeda new bug report related to this one
#1053