Deals randomly fail with 'failed to persist deal state: database is locked' error after publish

[cerblue]

Checklist

• This is not a question or a support request. If you have any boost related questions, please ask in the discussion forum.
• This is not a new feature or enhancement request. If it is, please open a new idea discussion instead. New feature and enhancement requests would be entertained by the boost team after a thorough discussion only.
• I have searched on the issue tracker and the discussion forum, and there is no existing related issue or discussion.
• I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
• I did not make any code changes to boost.

Boost component

• boost daemon - storage providers
• boost client
• boost UI
• boost data-transfer
• boost index-provider
• Other

Boost Version

Daemon: boostd version 1.2.0+git.c355880

Describe the Bug

Deals randomly fail with error: failed to persist deal state: database is locked after deal publish step. The deal will go into error state and cannot be retried. This happens intermittently; there are cases where all deals in a batch succeed, and cases where 3 out 5 deals fail.

My Boost daemon is accepting 20 deals a day, in few bursts of 510 deals. Boost will batch up to 10 deals before publishing the PublishStorageDeals message, but usually flushes the batch with 5~8 deals.

Logging Information

Jul 20 02:54:08 purumine-miner boostd[1271904]: {"level":"info","ts":"2022-07-20T02:54:08.131Z","logger":"boost-storage-deal","caller":"logs/log.go:40","msg":"sending deal to deal publisher","id":"a163925b-08df-4c8c-a06d-0d7d070bdbfc"}
Jul 20 03:18:06 purumine-miner boostd[1271904]: {"level":"error","ts":"2022-07-20T03:18:06.726Z","logger":"boost-storage-deal","caller":"logs/log.go:53","msg":"deal failed","id":"a163925b-08df-4c8c-a06d-0d7d070bdbfc","err":"failed to persist deal state: database is locked"}
Jul 20 03:18:11 purumine-miner boostd[1271904]: {"level":"warn","ts":"2022-07-20T03:18:11.730Z","logger":"boost-storage-deal","caller":"logs/log.go:77","msg":"failed to persist deal log","id":"a163925b-08df-4c8c-a06d-0d7d070bdbfc","err":"database is locked"}
Jul 20 03:18:17 purumine-miner boostd[1271904]: {"level":"info","ts":"2022-07-20T03:18:17.584Z","logger":"boost-storage-deal","caller":"logs/log.go:40","msg":"cleaning up deal","id":"a163925b-08df-4c8c-a06d-0d7d070bdbfc"}
Jul 20 03:18:22 purumine-miner boostd[1271904]: {"level":"warn","ts":"2022-07-20T03:18:22.588Z","logger":"boost-storage-deal","caller":"logs/log.go:77","msg":"failed to persist deal log","id":"a163925b-08df-4c8c-a06d-0d7d070bdbfc","err":"database is locked"}
Jul 20 03:18:26 purumine-miner boostd[1271904]: {"level":"info","ts":"2022-07-20T03:18:26.705Z","logger":"boost-storage-deal","caller":"logs/log.go:40","msg":"deal finished","id":"a163925b-08df-4c8c-a06d-0d7d070bdbfc"}
Jul 20 03:18:26 purumine-miner boostd[1271904]: {"level":"info","ts":"2022-07-20T03:18:26.864Z","logger":"boost-storage-deal","caller":"logs/log.go:40","msg":"untagged funds for deal as deal finished","id":"a163925b-08df-4c8c-a06d-0d7d070bdbfc","untagged publish":"100000000000000000","untagged collateral":"6995810500887622","err":null}
Jul 20 03:18:26 purumine-miner boostd[1271904]: {"level":"info","ts":"2022-07-20T03:18:26.999Z","logger":"boost-storage-deal","caller":"logs/log.go:40","msg":"untagged storage space for deal","id":"a163925b-08df-4c8c-a06d-0d7d070bdbfc"}
Jul 20 03:18:27 purumine-miner boostd[1271904]: {"level":"info","ts":"2022-07-20T03:18:27.060Z","logger":"boost-storage-deal","caller":"logs/log.go:40","msg":"finished cleaning up deal","id":"a163925b-08df-4c8c-a06d-0d7d070bdbfc"}
Jul 20 03:18:27 purumine-miner boostd[1271904]: {"level":"info","ts":"2022-07-20T03:18:27.118Z","logger":"boost-storage-deal","caller":"logs/log.go:40","msg":"deal go-routine finished execution","id":"a163925b-08df-4c8c-a06d-0d7d070bdbfc"}

Repo Steps

1. Run boostd daemon
2. Wait for deals to come in
3. Wait until deals are published
4. See some of deals fail

[nonsense]

Fixed at #679 ; Thank you @cerblue !!!