3.13: Quorum queue fails to open a segment file with `enoent` #14713

ulagnes123-crypto · 2025-10-08T12:53:27Z

ulagnes123-crypto
Oct 8, 2025

Describe the bug

In our 3 node HA cluster, we were getting repeated quorum queues disappearing followed by leader elections to create new queues. This kept going on in a loop (queue disappear, new queue leader elections) until we restarted the RMQ servers.
Note: Auto-delete is NOT enabled for the queues and there are not applications or operator commands being run to delete these queues.

There was also a stretch of time where some segment write errors were thrown.

2025-10-03 10:38:41.869864-06:00 [info] <0.2201081.0> ra: started cluster %2F_ABC_PRIVATE_SINGLE_DEVICEOPERATION with 3 servers
2025-10-03 10:38:41.869864-06:00 [info] <0.2201081.0> 0 servers failed to start: []
2025-10-03 10:38:41.869864-06:00 [info] <0.2201081.0> Leader: {'%2F_ABC_PRIVATE_SINGLE_DEVICEOPERATION','rabbit@rabbitmq-03-r2'}
2025-10-03 10:38:41.870750-06:00 [debug] <0.2201284.0> queue 'ABC_PRIVATE_SINGLE_DEVICEOPERATION' in vhost '/': enabling ra cluster changes in 1, index 1
2025-10-03 10:38:41.871091-06:00 [debug] <0.2201284.0> queue 'ABC_PRIVATE_SINGLE_DEVICEOPERATION' in vhost '/': applying new machine version 3 current 0
2025-10-03 10:38:41.876859-06:00 [debug] <0.2201113.0> Will start up to 3 replicas for quorum queue 'ABC_PRIVATE_DASHBOARD' in vhost '/' with leader on node 'rabbit@rabbitmq-03-r2'
2025-10-03 10:38:41.879897-06:00 [debug] <0.2201294.0> queue 'ABC_PRIVATE_DASHBOARD' in vhost '/': ra_log:init recovered last_index_term {0,0} first index 0
2025-10-03 10:38:41.881422-06:00 [debug] <0.2201294.0> queue 'ABC_PRIVATE_DASHBOARD' in vhost '/': post_init -> recover in term: 0 machine version: 3
2025-10-03 10:38:41.881502-06:00 [debug] <0.2201294.0> queue 'ABC_PRIVATE_DASHBOARD' in vhost '/': recovering state machine version 0:3 from index 0 to 0
2025-10-03 10:38:41.881520-06:00 [debug] <0.2201294.0> queue 'ABC_PRIVATE_DASHBOARD' in vhost '/': recovery of state machine version 0:3 from index 0 to 0 took 0ms
2025-10-03 10:38:41.881565-06:00 [debug] <0.2201294.0> queue 'ABC_PRIVATE_DASHBOARD' in vhost '/': scanning for cluster changes 1:0
2025-10-03 10:38:41.881607-06:00 [debug] <0.2201294.0> queue 'ABC_PRIVATE_DASHBOARD' in vhost '/': recover -> recovered in term: 0 machine version: 3
2025-10-03 10:38:41.881673-06:00 [debug] <0.2201294.0> queue 'ABC_PRIVATE_DASHBOARD' in vhost '/': recovered -> follower in term: 0 machine version: 3
2025-10-03 10:38:41.882192-06:00 [debug] <0.2201294.0> queue 'ABC_PRIVATE_DASHBOARD' in vhost '/': election triggered by <0.2201113.0>
2025-10-03 10:38:41.882214-06:00 [debug] <0.2201294.0> queue 'ABC_PRIVATE_DASHBOARD' in vhost '/': pre_vote election called for in term 0
2025-10-03 10:38:41.883431-06:00 [debug] <0.2201294.0> queue 'ABC_PRIVATE_DASHBOARD' in vhost '/': follower -> pre_vote in term: 0 machine version: 3
2025-10-03 10:38:41.883488-06:00 [debug] <0.2201294.0> queue 'ABC_PRIVATE_DASHBOARD' in vhost '/': pre_vote granted #Ref<0.2971604610.3918004225.171752> for term 0 votes 1
2025-10-03 10:38:41.883639-06:00 [debug] <0.2201294.0> queue 'ABC_PRIVATE_DASHBOARD' in vhost '/': pre_vote granted #Ref<0.2971604610.3918004225.171752> for term 0 votes 2
2025-10-03 10:38:41.883682-06:00 [debug] <0.2201294.0> queue 'ABC_PRIVATE_DASHBOARD' in vhost '/': election called for in term 1
2025-10-03 10:38:41.885203-06:00 [debug] <0.2201294.0> queue 'ABC_PRIVATE_DASHBOARD' in vhost '/': pre_vote -> candidate in term: 1 machine version: 3
2025-10-03 10:38:41.885246-06:00 [debug] <0.2201294.0> queue 'ABC_PRIVATE_DASHBOARD' in vhost '/': vote granted for term 1 votes 1
2025-10-03 10:38:41.888300-06:00 [debug] <0.2201294.0> queue 'ABC_PRIVATE_DASHBOARD' in vhost '/': vote granted for term 1 votes 2
2025-10-03 10:38:41.888381-06:00 [notice] <0.2201294.0> queue 'ABC_PRIVATE_DASHBOARD' in vhost '/': candidate -> leader in term: 1 machine version: 3
2025-10-03 10:38:41.888509-06:00 [info] <0.2201113.0> ra: started cluster %2F_ABC_PRIVATE_DASHBOARD with 3 servers
2025-10-03 10:38:41.888509-06:00 [info] <0.2201113.0> 0 servers failed to start: []
2025-10-03 10:38:41.888509-06:00 [info] <0.2201113.0> Leader: {'%2F_ABC_PRIVATE_DASHBOARD','rabbit@rabbitmq-03-r2'}
2025-10-03 10:38:41.889501-06:00 [debug] <0.2201294.0> queue 'ABC_PRIVATE_DASHBOARD' in vhost '/': enabling ra cluster changes in 1, index 1
2025-10-03 10:38:41.889844-06:00 [debug] <0.2201294.0> queue 'ABC_PRIVATE_DASHBOARD' in vhost '/': applying new machine version 3 current 0
2025-10-03 10:38:41.898857-06:00 [debug] <0.2198250.0> Will start up to 3 replicas for quorum queue 'ABC_PRIVATE' in vhost '/' with leader on node 'rabbit@rabbitmq-03-r2'
2025-10-03 10:38:41.902026-06:00 [debug] <0.2201308.0> queue 'ABC_PRIVATE' in vhost '/': ra_log:init recovered last_index_term {0,0} first index 0
2025-10-03 10:38:41.904315-06:00 [debug] <0.2201308.0> queue 'ABC_PRIVATE' in vhost '/': post_init -> recover in term: 0 machine version: 3
2025-10-03 10:38:41.904383-06:00 [debug] <0.2201308.0> queue 'ABC_PRIVATE' in vhost '/': recovering state machine version 0:3 from index 0 to 0
2025-10-03 10:38:41.904407-06:00 [debug] <0.2201308.0> queue 'ABC_PRIVATE' in vhost '/': recovery of state machine version 0:3 from index 0 to 0 took 0ms
2025-10-03 10:38:41.904435-06:00 [debug] <0.2201308.0> queue 'ABC_PRIVATE' in vhost '/': scanning for cluster changes 1:0
2025-10-03 10:38:41.904466-06:00 [debug] <0.2201308.0> queue 'ABC_PRIVATE' in vhost '/': recover -> recovered in term: 0 machine version: 3
2025-10-03 10:38:41.904511-06:00 [debug] <0.2201308.0> queue 'ABC_PRIVATE' in vhost '/': recovered -> follower in term: 0 machine version: 3
2025-10-03 10:38:41.905000-06:00 [debug] <0.2201308.0> queue 'ABC_PRIVATE' in vhost '/': election triggered by <0.2198250.0>
2025-10-03 10:38:41.905041-06:00 [debug] <0.2201308.0> queue 'ABC_PRIVATE' in vhost '/': pre_vote election called for in term 0
2025-10-03 10:38:41.905990-06:00 [debug] <0.2201308.0> queue 'ABC_PRIVATE' in vhost '/': follower -> pre_vote in term: 0 machine version: 3
2025-10-03 10:38:41.906035-06:00 [debug] <0.2201308.0> queue 'ABC_PRIVATE' in vhost '/': pre_vote granted #Ref<0.2971604610.3908304898.235552> for term 0 votes 1
2025-10-03 10:38:41.906187-06:00 [debug] <0.2201308.0> queue 'ABC_PRIVATE' in vhost '/': pre_vote granted #Ref<0.2971604610.3908304898.235552> for term 0 votes 2
2025-10-03 10:38:41.906217-06:00 [debug] <0.2201308.0> queue 'ABC_PRIVATE' in vhost '/': election called for in term 1
2025-10-03 10:38:41.907953-06:00 [debug] <0.2201308.0> queue 'ABC_PRIVATE' in vhost '/': pre_vote -> candidate in term: 1 machine version: 3
2025-10-03 10:38:41.907991-06:00 [debug] <0.2201308.0> queue 'ABC_PRIVATE' in vhost '/': vote granted for term 1 votes 1
2025-10-03 10:38:41.910651-06:00 [debug] <0.2201308.0> queue 'ABC_PRIVATE' in vhost '/': vote granted for term 1 votes 2
2025-10-03 10:38:41.910733-06:00 [notice] <0.2201308.0> queue 'ABC_PRIVATE' in vhost '/': candidate -> leader in term: 1 machine version: 3
2025-10-03 10:38:41.910821-06:00 [info] <0.2198250.0> ra: started cluster %2F_ABC_PRIVATE with 3 servers
2025-10-03 10:38:41.910821-06:00 [info] <0.2198250.0> 0 servers failed to start: []
2025-10-03 10:38:41.910821-06:00 [info] <0.2198250.0> Leader: {'%2F_ABC_PRIVATE','rabbit@rabbitmq-03-r2'}
2025-10-03 10:38:41.911592-06:00 [debug] <0.2201308.0> queue 'ABC_PRIVATE' in vhost '/': enabling ra cluster changes in 1, index 1
2025-10-03 10:38:41.911854-06:00 [debug] <0.2201308.0> queue 'ABC_PRIVATE' in vhost '/': applying new machine version 3 current 0
2025-10-03 10:38:41.917817-06:00 [debug] <0.2198275.0> Will start up to 3 replicas for quorum queue 'ABC_PRIVATE_DEVICEOPERATION' in vhost '/' with leader on node 'rabbit@rabbitmq-03-r2'
2025-10-03 10:38:41.920964-06:00 [debug] <0.2201313.0> queue 'ABC_PRIVATE_DEVICEOPERATION' in vhost '/': ra_log:init recovered last_index_term {0,0} first index 0
2025-10-03 10:38:41.922506-06:00 [debug] <0.2201313.0> queue 'ABC_PRIVATE_DEVICEOPERATION' in vhost '/': post_init -> recover in term: 0 machine version: 3
2025-10-03 10:38:41.922560-06:00 [debug] <0.2201313.0> queue 'ABC_PRIVATE_DEVICEOPERATION' in vhost '/': recovering state machine version 0:3 from index 0 to 0
2025-10-03 10:38:41.922625-06:00 [debug] <0.2201313.0> queue 'ABC_PRIVATE_DEVICEOPERATION' in vhost '/': recovery of state machine version 0:3 from index 0 to 0 took 0ms
2025-10-03 10:38:41.922670-06:00 [debug] <0.2201313.0> queue 'ABC_PRIVATE_DEVICEOPERATION' in vhost '/': scanning for cluster changes 1:0
2025-10-03 10:38:41.922699-06:00 [debug] <0.2201313.0> queue 'ABC_PRIVATE_DEVICEOPERATION' in vhost '/': recover -> recovered in term: 0 machine version: 3
2025-10-03 10:38:41.922744-06:00 [debug] <0.2201313.0> queue 'ABC_PRIVATE_DEVICEOPERATION' in vhost '/': recovered -> follower in term: 0 machine version: 3
2025-10-03 10:38:41.922874-06:00 [debug] <0.2201313.0> queue 'ABC_PRIVATE_DEVICEOPERATION' in vhost '/': election triggered by <0.2198275.0>
2025-10-03 10:38:41.922895-06:00 [debug] <0.2201313.0> queue 'ABC_PRIVATE_DEVICEOPERATION' in vhost '/': pre_vote election called for in term 0
2025-10-03 10:38:41.924094-06:00 [debug] <0.2201313.0> queue 'ABC_PRIVATE_DEVICEOPERATION' in vhost '/': follower -> pre_vote in term: 0 machine version: 3
2025-10-03 10:38:41.924136-06:00 [debug] <0.2201313.0> queue 'ABC_PRIVATE_DEVICEOPERATION' in vhost '/': pre_vote granted #Ref<0.2971604610.3908304898.235633> for term 0 votes 1
2025-10-03 10:38:41.924242-06:00 [debug] <0.2201313.0> queue 'ABC_PRIVATE_DEVICEOPERATION' in vhost '/': pre_vote granted #Ref<0.2971604610.3908304898.235633> for term 0 votes 2
2025-10-03 10:38:41.924286-06:00 [debug] <0.2201313.0> queue 'ABC_PRIVATE_DEVICEOPERATION' in vhost '/': election called for in term 1
2025-10-03 10:38:41.925762-06:00 [debug] <0.2201313.0> queue 'ABC_PRIVATE_DEVICEOPERATION' in vhost '/': pre_vote -> candidate in term: 1 machine version: 3
2025-10-03 10:38:41.925804-06:00 [debug] <0.2201313.0> queue 'ABC_PRIVATE_DEVICEOPERATION' in vhost '/': vote granted for term 1 votes 1
2025-10-03 10:38:41.928910-06:00 [debug] <0.2201313.0> queue 'ABC_PRIVATE_DEVICEOPERATION' in vhost '/': vote granted for term 1 votes 2
2025-10-03 10:38:41.928959-06:00 [notice] <0.2201313.0> queue 'ABC_PRIVATE_DEVICEOPERATION' in vhost '/': candidate -> leader in term: 1 machine version: 3
2025-10-03 10:38:41.929033-06:00 [info] <0.2198275.0> ra: started cluster %2F_ABC_PRIVATE_DEVICEOPERATION with 3 servers
2025-10-03 10:38:41.929033-06:00 [info] <0.2198275.0> 0 servers failed to start: []
2025-10-03 10:38:41.929033-06:00 [info] <0.2198275.0> Leader: {'%2F_ABC_PRIVATE_DEVICEOPERATION','rabbit@rabbitmq-03-r2'}
2025-10-03 10:38:41.929904-06:00 [debug] <0.2201313.0> queue 'ABC_PRIVATE_DEVICEOPERATION' in vhost '/': enabling ra cluster changes in 1, index 1
2025-10-03 10:38:41.930137-06:00 [debug] <0.2201313.0> queue 'ABC_PRIVATE_DEVICEOPERATION' in vhost '/': applying new machine version 3 current 0
2025-10-03 10:38:41.937794-06:00 [debug] <0.2201328.0> Will start up to 3 replicas for quorum queue 'ABC_PRIVATE_2' in vhost '/' with leader on node 'rabbit@rabbitmq-03-r2'
2025-10-03 10:38:41.940682-06:00 [debug] <0.2201346.0> queue 'ABC_PRIVATE_2' in vhost '/': ra_log:init recovered last_index_term {0,0} first index 0
2025-10-03 10:38:41.942129-06:00 [debug] <0.2201346.0> queue 'ABC_PRIVATE_2' in vhost '/': post_init -> recover in term: 0 machine version: 3
2025-10-03 10:38:41.942171-06:00 [debug] <0.2201346.0> queue 'ABC_PRIVATE_2' in vhost '/': recovering state machine version 0:3 from index 0 to 0
2025-10-03 10:38:41.942229-06:00 [debug] <0.2201346.0> queue 'ABC_PRIVATE_2' in vhost '/': recovery of state machine version 0:3 from index 0 to 0 took 0ms
2025-10-03 10:38:41.942290-06:00 [debug] <0.2201346.0> queue 'ABC_PRIVATE_2' in vhost '/': scanning for cluster changes 1:0
2025-10-03 10:38:41.942327-06:00 [debug] <0.2201346.0> queue 'ABC_PRIVATE_2' in vhost '/': recover -> recovered in term: 0 machine version: 3
2025-10-03 10:38:41.942401-06:00 [debug] <0.2201346.0> queue 'ABC_PRIVATE_2' in vhost '/': recovered -> follower in term: 0 machine version: 3
2025-10-03 10:38:41.942516-06:00 [debug] <0.2201346.0> queue 'ABC_PRIVATE_2' in vhost '/': election triggered by <0.2201328.0>
2025-10-03 10:38:41.942544-06:00 [debug] <0.2201346.0> queue 'ABC_PRIVATE_2' in vhost '/': pre_vote election called for in term 0
2025-10-03 10:38:41.943584-06:00 [debug] <0.2201346.0> queue 'ABC_PRIVATE_2' in vhost '/': follower -> pre_vote in term: 0 machine version: 3
2025-10-03 10:38:41.943626-06:00 [debug] <0.2201346.0> queue 'ABC_PRIVATE_2' in vhost '/': pre_vote granted #Ref<0.2971604610.3918004225.172104> for term 0 votes 1
2025-10-03 10:38:41.943768-06:00 [debug] <0.2201346.0> queue 'ABC_PRIVATE_2' in vhost '/': pre_vote granted #Ref<0.2971604610.3918004225.172104> for term 0 votes 2
2025-10-03 10:38:41.943789-06:00 [debug] <0.2201346.0> queue 'ABC_PRIVATE_2' in vhost '/': election called for in term 1
2025-10-03 10:38:41.945507-06:00 [debug] <0.2201346.0> queue 'ABC_PRIVATE_2' in vhost '/': pre_vote -> candidate in term: 1 machine version: 3
2025-10-03 10:38:41.945543-06:00 [debug] <0.2201346.0> queue 'ABC_PRIVATE_2' in vhost '/': vote granted for term 1 votes 1
2025-10-03 10:38:41.948449-06:00 [debug] <0.2201346.0> queue 'ABC_PRIVATE_2' in vhost '/': vote granted for term 1 votes 2
2025-10-03 10:38:41.948505-06:00 [notice] <0.2201346.0> queue 'ABC_PRIVATE_2' in vhost '/': candidate -> leader in term: 1 machine version: 3
2025-10-03 10:38:41.948583-06:00 [info] <0.2201328.0> ra: started cluster %2F_ABC_PRIVATE_2 with 3 servers
2025-10-03 10:38:41.948583-06:00 [info] <0.2201328.0> 0 servers failed to start: []
2025-10-03 10:38:41.948583-06:00 [info] <0.2201328.0> Leader: {'%2F_ABC_PRIVATE_2','rabbit@rabbitmq-03-r2'}
2025-10-03 10:38:41.949328-06:00 [debug] <0.2201346.0> queue 'ABC_PRIVATE_2' in vhost '/': enabling ra cluster changes in 1, index 1
2025-10-03 10:38:41.949540-06:00 [debug] <0.2201346.0> queue 'ABC_PRIVATE_2' in vhost '/': applying new machine version 3 current 0
2025-10-03 10:38:41.963820-06:00 [debug] <0.2201394.0> Will start up to 3 replicas for quorum queue 'ABC_PRIVATE_CWMPCLUSTERCHANGE' in vhost '/' with leader on node 'rabbit@rabbitmq-03-r2'
2025-10-03 10:38:41.967393-06:00 [debug] <0.2201405.0> queue 'ABC_PRIVATE_CWMPCLUSTERCHANGE' in vhost '/': ra_log:init recovered last_index_term {0,0} first index 0
2025-10-03 10:38:41.968898-06:00 [debug] <0.2201405.0> queue 'ABC_PRIVATE_CWMPCLUSTERCHANGE' in vhost '/': post_init -> recover in term: 0 machine version: 3
2025-10-03 10:38:41.968953-06:00 [debug] <0.2201405.0> queue 'ABC_PRIVATE_CWMPCLUSTERCHANGE' in vhost '/': recovering state machine version 0:3 from index 0 to 0
2025-10-03 10:38:41.968992-06:00 [debug] <0.2201405.0> queue 'ABC_PRIVATE_CWMPCLUSTERCHANGE' in vhost '/': recovery of state machine version 0:3 from index 0 to 0 took 0ms
2025-10-03 10:38:41.969026-06:00 [debug] <0.2201405.0> queue 'ABC_PRIVATE_CWMPCLUSTERCHANGE' in vhost '/': scanning for cluster changes 1:0
2025-10-03 10:38:41.969081-06:00 [debug] <0.2201405.0> queue 'ABC_PRIVATE_CWMPCLUSTERCHANGE' in vhost '/': recover -> recovered in term: 0 machine version: 3
2025-10-03 10:38:41.969118-06:00 [debug] <0.2201405.0> queue 'ABC_PRIVATE_CWMPCLUSTERCHANGE' in vhost '/': recovered -> follower in term: 0 machine version: 3
2025-10-03 10:38:41.969450-06:00 [debug] <0.2201405.0> queue 'ABC_PRIVATE_CWMPCLUSTERCHANGE' in vhost '/': election triggered by <0.2201394.0>
2025-10-03 10:38:41.969508-06:00 [debug] <0.2201405.0> queue 'ABC_PRIVATE_CWMPCLUSTERCHANGE' in vhost '/': pre_vote election called for in term 0
2025-10-03 10:38:41.970719-06:00 [debug] <0.2201405.0> queue 'ABC_PRIVATE_CWMPCLUSTERCHANGE' in vhost '/': follower -> pre_vote in term: 0 machine version: 3
2025-10-03 10:38:41.970798-06:00 [debug] <0.2201405.0> queue 'ABC_PRIVATE_CWMPCLUSTERCHANGE' in vhost '/': pre_vote granted #Ref<0.2971604610.3905683459.139205> for term 0 votes 1
2025-10-03 10:38:41.970938-06:00 [debug] <0.2201405.0> queue 'ABC_PRIVATE_CWMPCLUSTERCHANGE' in vhost '/': pre_vote granted #Ref<0.2971604610.3905683459.139205> for term 0 votes 2
2025-10-03 10:38:41.970972-06:00 [debug] <0.2201405.0> queue 'ABC_PRIVATE_CWMPCLUSTERCHANGE' in vhost '/': election called for in term 1
2025-10-03 10:38:41.972432-06:00 [debug] <0.2201405.0> queue 'ABC_PRIVATE_CWMPCLUSTERCHANGE' in vhost '/': pre_vote -> candidate in term: 1 machine version: 3
2025-10-03 10:38:41.972493-06:00 [debug] <0.2201405.0> queue 'ABC_PRIVATE_CWMPCLUSTERCHANGE' in vhost '/': vote granted for term 1 votes 1
2025-10-03 10:38:41.975546-06:00 [debug] <0.2201405.0> queue 'ABC_PRIVATE_CWMPCLUSTERCHANGE' in vhost '/': vote granted for term 1 votes 2
2025-10-03 10:38:41.975611-06:00 [notice] <0.2201405.0> queue 'ABC_PRIVATE_CWMPCLUSTERCHANGE' in vhost '/': candidate -> leader in term: 1 machine version: 3
2025-10-03 10:38:41.975715-06:00 [info] <0.2201394.0> ra: started cluster %2F_ABC_PRIVATE_CWMPCLUSTERCHANGE with 3 servers
2025-10-03 10:38:41.975715-06:00 [info] <0.2201394.0> 0 servers failed to start: []
2025-10-03 10:38:41.975715-06:00 [info] <0.2201394.0> Leader: {'%2F_ABC_PRIVATE_CWMPCLUSTERCHANGE','rabbit@rabbitmq-03-r2'}
2025-10-03 10:38:41.976520-06:00 [debug] <0.2201405.0> queue 'ABC_PRIVATE_CWMPCLUSTERCHANGE' in vhost '/': enabling ra cluster changes in 1, index 1
2025-10-03 10:38:41.976717-06:00 [debug] <0.2201405.0> queue 'ABC_PRIVATE_CWMPCLUSTERCHANGE' in vhost '/': applying new machine version 3 current 0
2025-10-03 10:38:41.982793-06:00 [debug] <0.2201424.0> Will start up to 3 replicas for quorum queue 'ABC_PRIVATE_TRENDING' in vhost '/' with leader on node 'rabbit@rabbitmq-03-r2'
2025-10-03 10:38:41.985860-06:00 [debug] <0.2201434.0> queue 'ABC_PRIVATE_TRENDING' in vhost '/': ra_log:init recovered last_index_term {0,0} first index 0
2025-10-03 10:38:41.987281-06:00 [debug] <0.2201434.0> queue 'ABC_PRIVATE_TRENDING' in vhost '/': post_init -> recover in term: 0 machine version: 3
2025-10-03 10:38:41.987370-06:00 [debug] <0.2201434.0> queue 'ABC_PRIVATE_TRENDING' in vhost '/': recovering state machine version 0:3 from index 0 to 0
2025-10-03 10:38:41.987390-06:00 [debug] <0.2201434.0> queue 'ABC_PRIVATE_TRENDING' in vhost '/': recovery of state machine version 0:3 from index 0 to 0 took 0ms
2025-10-03 10:38:41.987416-06:00 [debug] <0.2201434.0> queue 'ABC_PRIVATE_TRENDING' in vhost '/': scanning for cluster changes 1:0
2025-10-03 10:38:41.987456-06:00 [debug] <0.2201434.0> queue 'ABC_PRIVATE_TRENDING' in vhost '/': recover -> recovered in term: 0 machine version: 3
2025-10-03 10:38:41.987513-06:00 [debug] <0.2201434.0> queue 'ABC_PRIVATE_TRENDING' in vhost '/': recovered -> follower in term: 0 machine version: 3
2025-10-03 10:38:41.987685-06:00 [debug] <0.2201434.0> queue 'ABC_PRIVATE_TRENDING' in vhost '/': election triggered by <0.2201424.0>
2025-10-03 10:38:41.987710-06:00 [debug] <0.2201434.0> queue 'ABC_PRIVATE_TRENDING' in vhost '/': pre_vote election called for in term 0
2025-10-03 10:38:41.988851-06:00 [debug] <0.2201434.0> queue 'ABC_PRIVATE_TRENDING' in vhost '/': follower -> pre_vote in term: 0 machine version: 3
2025-10-03 10:38:41.988906-06:00 [debug] <0.2201434.0> queue 'ABC_PRIVATE_TRENDING' in vhost '/': pre_vote granted #Ref<0.2971604610.3908304898.236282> for term 0 votes 1
2025-10-03 10:38:41.989039-06:00 [debug] <0.2201434.0> queue 'ABC_PRIVATE_TRENDING' in vhost '/': pre_vote granted #Ref<0.2971604610.3908304898.236282> for term 0 votes 2
2025-10-03 10:38:41.989068-06:00 [debug] <0.2201434.0> queue 'ABC_PRIVATE_TRENDING' in vhost '/': election called for in term 1
2025-10-03 10:38:41.990489-06:00 [debug] <0.2201434.0> queue 'ABC_PRIVATE_TRENDING' in vhost '/': pre_vote -> candidate in term: 1 machine version: 3
2025-10-03 10:38:41.990536-06:00 [debug] <0.2201434.0> queue 'ABC_PRIVATE_TRENDING' in vhost '/': vote granted for term 1 votes 1
2025-10-03 10:38:41.993407-06:00 [debug] <0.2201434.0> queue 'ABC_PRIVATE_TRENDING' in vhost '/': vote granted for term 1 votes 2
2025-10-03 10:38:41.993453-06:00 [notice] <0.2201434.0> queue 'ABC_PRIVATE_TRENDING' in vhost '/': candidate -> leader in term: 1 machine version: 3
2025-10-03 10:38:41.993546-06:00 [info] <0.2201424.0> ra: started cluster %2F_ABC_PRIVATE_TRENDING with 3 servers
2025-10-03 10:38:41.993546-06:00 [info] <0.2201424.0> 0 servers failed to start: []
2025-10-03 10:38:41.993546-06:00 [info] <0.2201424.0> Leader: {'%2F_ABC_PRIVATE_TRENDING','rabbit@rabbitmq-03-r2'}
2025-10-03 10:38:41.994645-06:00 [debug] <0.2201434.0> queue 'ABC_PRIVATE_TRENDING' in vhost '/': enabling ra cluster changes in 1, index 1
2025-10-03 10:38:41.994854-06:00 [debug] <0.2201434.0> queue 'ABC_PRIVATE_TRENDING' in vhost '/': applying new machine version 3 current 0
2025-10-03 10:38:42.010719-06:00 [debug] <0.2201449.0> Will start up to 3 replicas for quorum queue 'ABC_PRIVATE_VALUECHANGE' in vhost '/' with leader on node 'rabbit@rabbitmq-03-r2'
2025-10-03 10:38:42.013773-06:00 [debug] <0.2201458.0> queue 'ABC_PRIVATE_VALUECHANGE' in vhost '/': ra_log:init recovered last_index_term {0,0} first index 0
2025-10-03 10:38:42.015304-06:00 [debug] <0.2201458.0> queue 'ABC_PRIVATE_VALUECHANGE' in vhost '/': post_init -> recover in term: 0 machine version: 3
2025-10-03 10:38:42.015374-06:00 [debug] <0.2201458.0> queue 'ABC_PRIVATE_VALUECHANGE' in vhost '/': recovering state machine version 0:3 from index 0 to 0
2025-10-03 10:38:42.015396-06:00 [debug] <0.2201458.0> queue 'ABC_PRIVATE_VALUECHANGE' in vhost '/': recovery of state machine version 0:3 from index 0 to 0 took 0ms
2025-10-03 10:38:42.015431-06:00 [debug] <0.2201458.0> queue 'ABC_PRIVATE_VALUECHANGE' in vhost '/': scanning for cluster changes 1:0
2025-10-03 10:38:42.015472-06:00 [debug] <0.2201458.0> queue 'ABC_PRIVATE_VALUECHANGE' in vhost '/': recover -> recovered in term: 0 machine version: 3
2025-10-03 10:38:42.015524-06:00 [debug] <0.2201458.0> queue 'ABC_PRIVATE_VALUECHANGE' in vhost '/': recovered -> follower in term: 0 machine version: 3
2025-10-03 10:38:42.016050-06:00 [debug] <0.2201458.0> queue 'ABC_PRIVATE_VALUECHANGE' in vhost '/': election triggered by <0.2201449.0>
2025-10-03 10:38:42.016110-06:00 [debug] <0.2201458.0> queue 'ABC_PRIVATE_VALUECHANGE' in vhost '/': pre_vote election called for in term 0
2025-10-03 10:38:42.017115-06:00 [debug] <0.2201458.0> queue 'ABC_PRIVATE_VALUECHANGE' in vhost '/': follower -> pre_vote in term: 0 machine version: 3
2025-10-03 10:38:42.017160-06:00 [debug] <0.2201458.0> queue 'ABC_PRIVATE_VALUECHANGE' in vhost '/': pre_vote granted #Ref<0.2971604610.3918004225.172283> for term 0 votes 1
2025-10-03 10:38:42.017243-06:00 [debug] <0.2201458.0> queue 'ABC_PRIVATE_VALUECHANGE' in vhost '/': pre_vote granted #Ref<0.2971604610.3918004225.172283> for term 0 votes 2
2025-10-03 10:38:42.017270-06:00 [debug] <0.2201458.0> queue 'ABC_PRIVATE_VALUECHANGE' in vhost '/': election called for in term 1
2025-10-03 10:38:42.018931-06:00 [debug] <0.2201458.0> queue 'ABC_PRIVATE_VALUECHANGE' in vhost '/': pre_vote -> candidate in term: 1 machine version: 3
2025-10-03 10:38:42.018967-06:00 [debug] <0.2201458.0> queue 'ABC_PRIVATE_VALUECHANGE' in vhost '/': vote granted for term 1 votes 1
2025-10-03 10:38:42.022106-06:00 [debug] <0.2201458.0> queue 'ABC_PRIVATE_VALUECHANGE' in vhost '/': vote granted for term 1 votes 2
2025-10-03 10:38:42.022154-06:00 [notice] <0.2201458.0> queue 'ABC_PRIVATE_VALUECHANGE' in vhost '/': candidate -> leader in term: 1 machine version: 3
2025-10-03 10:38:42.022227-06:00 [info] <0.2201449.0> ra: started cluster %2F_ABC_PRIVATE_VALUECHANGE with 3 servers
2025-10-03 10:38:42.022227-06:00 [info] <0.2201449.0> 0 servers failed to start: []
2025-10-03 10:38:42.022227-06:00 [info] <0.2201449.0> Leader: {'%2F_ABC_PRIVATE_VALUECHANGE','rabbit@rabbitmq-03-r2'}
2025-10-03 10:38:42.023052-06:00 [debug] <0.2201458.0> queue 'ABC_PRIVATE_VALUECHANGE' in vhost '/': enabling ra cluster changes in 1, index 1
2025-10-03 10:38:42.023291-06:00 [debug] <0.2201458.0> queue 'ABC_PRIVATE_VALUECHANGE' in vhost '/': applying new machine version 3 current 0
2025-10-03 10:38:42.041563-06:00 [debug] <0.2201500.0> Will start up to 3 replicas for quorum queue 'ABC_PRIVATE_DASHBOARD_2' in vhost '/' with leader on node 'rabbit@rabbitmq-03-r2'
2025-10-03 10:38:42.044708-06:00 [debug] <0.2201505.0> queue 'ABC_PRIVATE_DASHBOARD_2' in vhost '/': ra_log:init recovered last_index_term {0,0} first index 0
2025-10-03 10:38:42.046648-06:00 [debug] <0.2201505.0> queue 'ABC_PRIVATE_DASHBOARD_2' in vhost '/': post_init -> recover in term: 0 machine version: 3
2025-10-03 10:38:42.046733-06:00 [debug] <0.2201505.0> queue 'ABC_PRIVATE_DASHBOARD_2' in vhost '/': recovering state machine version 0:3 from index 0 to 0
2025-10-03 10:38:42.046761-06:00 [debug] <0.2201505.0> queue 'ABC_PRIVATE_DASHBOARD_2' in vhost '/': recovery of state machine version 0:3 from index 0 to 0 took 0ms
2025-10-03 10:38:42.046796-06:00 [debug] <0.2201505.0> queue 'ABC_PRIVATE_DASHBOARD_2' in vhost '/': scanning for cluster changes 1:0
2025-10-03 10:38:42.046837-06:00 [debug] <0.2201505.0> queue 'ABC_PRIVATE_DASHBOARD_2' in vhost '/': recover -> recovered in term: 0 machine version: 3
2025-10-03 10:38:42.046875-06:00 [debug] <0.2201505.0> queue 'ABC_PRIVATE_DASHBOARD_2' in vhost '/': recovered -> follower in term: 0 machine version: 3
2025-10-03 10:38:42.047184-06:00 [debug] <0.2201505.0> queue 'ABC_PRIVATE_DASHBOARD_2' in vhost '/': election triggered by <0.2201500.0>
2025-10-03 10:38:42.047227-06:00 [debug] <0.2201505.0> queue 'ABC_PRIVATE_DASHBOARD_2' in vhost '/': pre_vote election called for in term 0
2025-10-03 10:38:42.048225-06:00 [debug] <0.2201505.0> queue 'ABC_PRIVATE_DASHBOARD_2' in vhost '/': follower -> pre_vote in term: 0 machine version: 3
2025-10-03 10:38:42.048294-06:00 [debug] <0.2201505.0> queue 'ABC_PRIVATE_DASHBOARD_2' in vhost '/': pre_vote granted #Ref<0.2971604610.3908304898.236866> for term 0 votes 1
2025-10-03 10:38:42.048420-06:00 [debug] <0.2201505.0> queue 'ABC_PRIVATE_DASHBOARD_2' in vhost '/': pre_vote granted #Ref<0.2971604610.3908304898.236866> for term 0 votes 2
2025-10-03 10:38:42.048451-06:00 [debug] <0.2201505.0> queue 'ABC_PRIVATE_DASHBOARD_2' in vhost '/': election called for in term 1
2025-10-03 10:38:42.050109-06:00 [debug] <0.2201505.0> queue 'ABC_PRIVATE_DASHBOARD_2' in vhost '/': pre_vote -> candidate in term: 1 machine version: 3
2025-10-03 10:38:42.050156-06:00 [debug] <0.2201505.0> queue 'ABC_PRIVATE_DASHBOARD_2' in vhost '/': vote granted for term 1 votes 1
2025-10-03 10:38:42.053142-06:00 [debug] <0.2201505.0> queue 'ABC_PRIVATE_DASHBOARD_2' in vhost '/': vote granted for term 1 votes 2
2025-10-03 10:38:42.053196-06:00 [notice] <0.2201505.0> queue 'ABC_PRIVATE_DASHBOARD_2' in vhost '/': candidate -> leader in term: 1 machine version: 3
2025-10-03 10:38:42.053331-06:00 [info] <0.2201500.0> ra: started cluster %2F_ABC_PRIVATE_DASHBOARD_2 with 3 servers
2025-10-03 10:38:42.053331-06:00 [info] <0.2201500.0> 0 servers failed to start: []
2025-10-03 10:38:42.053331-06:00 [info] <0.2201500.0> Leader: {'%2F_ABC_PRIVATE_DASHBOARD_2','rabbit@rabbitmq-03-r2'}
2025-10-03 10:38:42.054099-06:00 [debug] <0.2201505.0> queue 'ABC_PRIVATE_DASHBOARD_2' in vhost '/': enabling ra cluster changes in 1, index 1
2025-10-03 10:38:42.054354-06:00 [debug] <0.2201505.0> queue 'ABC_PRIVATE_DASHBOARD_2' in vhost '/': applying new machine version 3 current 0
2025-10-03 10:38:42.060321-06:00 [debug] <0.2201517.0> Will start up to 3 replicas for quorum queue 'RPC_PRIVATE' in vhost '/' with leader on node 'rabbit@rabbitmq-03-r2'
2025-10-03 10:38:42.063167-06:00 [debug] <0.2201526.0> queue 'RPC_PRIVATE' in vhost '/': ra_log:init recovered last_index_term {0,0} first index 0
2025-10-03 10:38:42.064754-06:00 [debug] <0.2201526.0> queue 'RPC_PRIVATE' in vhost '/': post_init -> recover in term: 0 machine version: 3
2025-10-03 10:38:42.064817-06:00 [debug] <0.2201526.0> queue 'RPC_PRIVATE' in vhost '/': recovering state machine version 0:3 from index 0 to 0
2025-10-03 10:38:42.064893-06:00 [debug] <0.2201526.0> queue 'RPC_PRIVATE' in vhost '/': recovery of state machine version 0:3 from index 0 to 0 took 0ms
2025-10-03 10:38:42.064920-06:00 [debug] <0.2201526.0> queue 'RPC_PRIVATE' in vhost '/': scanning for cluster changes 1:0
2025-10-03 10:38:42.064949-06:00 [debug] <0.2201526.0> queue 'RPC_PRIVATE' in vhost '/': recover -> recovered in term: 0 machine version: 3
2025-10-03 10:38:42.064996-06:00 [debug] <0.2201526.0> queue 'RPC_PRIVATE' in vhost '/': recovered -> follower in term: 0 machine version: 3
2025-10-03 10:38:42.065106-06:00 [debug] <0.2201526.0> queue 'RPC_PRIVATE' in vhost '/': election triggered by <0.2201517.0>
2025-10-03 10:38:42.065133-06:00 [debug] <0.2201526.0> queue 'RPC_PRIVATE' in vhost '/': pre_vote election called for in term 0
2025-10-03 10:38:42.066072-06:00 [debug] <0.2201526.0> queue 'RPC_PRIVATE' in vhost '/': follower -> pre_vote in term: 0 machine version: 3
2025-10-03 10:38:42.066108-06:00 [debug] <0.2201526.0> queue 'RPC_PRIVATE' in vhost '/': pre_vote granted #Ref<0.2971604610.3908304898.236996> for term 0 votes 1
2025-10-03 10:38:42.066300-06:00 [debug] <0.2201526.0> queue 'RPC_PRIVATE' in vhost '/': pre_vote granted #Ref<0.2971604610.3908304898.236996> for term 0 votes 2
2025-10-03 10:38:42.066342-06:00 [debug] <0.2201526.0> queue 'RPC_PRIVATE' in vhost '/': election called for in term 1
2025-10-03 10:38:42.068446-06:00 [debug] <0.2201526.0> queue 'RPC_PRIVATE' in vhost '/': pre_vote -> candidate in term: 1 machine version: 3
2025-10-03 10:38:42.068505-06:00 [debug] <0.2201526.0> queue 'RPC_PRIVATE' in vhost '/': vote granted for term 1 votes 1
2025-10-03 10:38:42.071243-06:00 [debug] <0.2201526.0> queue 'RPC_PRIVATE' in vhost '/': vote granted for term 1 votes 2
2025-10-03 10:38:42.071313-06:00 [notice] <0.2201526.0> queue 'RPC_PRIVATE' in vhost '/': candidate -> leader in term: 1 machine version: 3
2025-10-03 10:38:42.071449-06:00 [info] <0.2201517.0> ra: started cluster %2F_RPC_PRIVATE with 3 servers
2025-10-03 10:38:42.071449-06:00 [info] <0.2201517.0> 0 servers failed to start: []
2025-10-03 10:38:42.071449-06:00 [info] <0.2201517.0> Leader: {'%2F_RPC_PRIVATE','rabbit@rabbitmq-03-r2'}
2025-10-03 10:38:42.072339-06:00 [debug] <0.2201526.0> queue 'RPC_PRIVATE' in vhost '/': enabling ra cluster changes in 1, index 1
2025-10-03 10:38:42.072583-06:00 [debug] <0.2201526.0> queue 'RPC_PRIVATE' in vhost '/': applying new machine version 3 current 0
2025-10-03 10:38:44.857587-06:00 [debug] <0.285.0> wal: opening new file 00000183.wal open mem tables: #Ref<0.2971604610.3874357249.210850>
2025-10-03 10:38:44.869087-06:00 [debug] <0.2201641.0> segment_writer: failed to open segment file /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-03-r2/quorum/rabbit@rabbitmq-03-r2/2F_RPC4C2ALZ290U7C/00000001.segment error: enoent
2025-10-03 10:38:44.869105-06:00 [debug] <0.2201637.0> segment_writer: failed to open segment file /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-03-r2/quorum/rabbit@rabbitmq-03-r2/2F_RPCG0M5VUV16W3R/00000001.segment error: enoent
2025-10-03 10:38:44.869182-06:00 [debug] <0.2201644.0> segment_writer: failed to open segment file /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-03-r2/quorum/rabbit@rabbitmq-03-r2/2F_ABC2VB8XZA412J6/00000001.segment error: enoent
2025-10-03 10:38:44.869158-06:00 [debug] <0.2201641.0> segment_writer: skipping segment as directory /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-03-r2/quorum/rabbit@rabbitmq-03-r2/2F_RPC4C2ALZ290U7C does not exist
2025-10-03 10:38:44.869164-06:00 [debug] <0.2201639.0> segment_writer: failed to open segment file /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-03-r2/quorum/rabbit@rabbitmq-03-r2/2F_ABCFGUKIOFO5BAF/00000001.segment error: enoent
2025-10-03 10:38:44.869173-06:00 [debug] <0.2201649.0> segment_writer: failed to open segment file /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-03-r2/quorum/rabbit@rabbitmq-03-r2/2F_ABCVBIC12SEI350/00000001.segment error: enoent
2025-10-03 10:38:44.869179-06:00 [debug] <0.2201647.0> segment_writer: failed to open segment file /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-03-r2/quorum/rabbit@rabbitmq-03-r2/2F_RPCC4A119NTT1YY/00000001.segment error: enoent
2025-10-03 10:38:44.869166-06:00 [debug] <0.2201637.0> segment_writer: skipping segment as directory /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-03-r2/quorum/rabbit@rabbitmq-03-r2/2F_RPCG0M5VUV16W3R does not exist
2025-10-03 10:38:44.869172-06:00 [debug] <0.2201640.0> segment_writer: failed to open segment file /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-03-r2/quorum/rabbit@rabbitmq-03-r2/2F_ABC8AWI8H1BNFEP/00000001.segment error: enoent
2025-10-03 10:38:44.869263-06:00 [debug] <0.2201641.0> ra_log_segment_writer: cleaning closed table for '2F_RPC4C2ALZ290U7C' range: 0-24
2025-10-03 10:38:44.869177-06:00 [debug] <0.2201645.0> segment_writer: failed to open segment file /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-03-r2/quorum/rabbit@rabbitmq-03-r2/2F_ABCASMNUKWGARBM/00000001.segment error: enoent
2025-10-03 10:38:44.869207-06:00 [debug] <0.2201644.0> segment_writer: skipping segment as directory /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-03-r2/quorum/rabbit@rabbitmq-03-r2/2F_ABC2VB8XZA412J6 does not exist
2025-10-03 10:38:44.869294-06:00 [debug] <0.2201647.0> segment_writer: skipping segment as directory /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-03-r2/quorum/rabbit@rabbitmq-03-r2/2F_RPCC4A119NTT1YY does not exist
2025-10-03 10:38:44.869245-06:00 [debug] <0.2201639.0> segment_writer: skipping segment as directory /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-03-r2/quorum/rabbit@rabbitmq-03-r2/2F_ABCFGUKIOFO5BAF does not exist
2025-10-03 10:38:44.869328-06:00 [debug] <0.2201637.0> ra_log_segment_writer: cleaning closed table for '2F_RPCG0M5VUV16W3R' range: 0-4
2025-10-03 10:38:44.869437-06:00 [debug] <0.2201639.0> ra_log_segment_writer: cleaning closed table for '2F_ABCFGUKIOFO5BAF' range: 0-4
2025-10-03 10:38:44.869404-06:00 [debug] <0.2201647.0> ra_log_segment_writer: cleaning closed table for '2F_RPCC4A119NTT1YY' range: 0-4
2025-10-03 10:38:44.869287-06:00 [debug] <0.2201649.0> segment_writer: skipping segment as directory /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-03-r2/quorum/rabbit@rabbitmq-03-r2/2F_ABCVBIC12SEI350 does not exist
2025-10-03 10:38:44.869337-06:00 [debug] <0.2201640.0> segment_writer: skipping segment as directory /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-03-r2/quorum/rabbit@rabbitmq-03-r2/2F_ABC8AWI8H1BNFEP does not exist
2025-10-03 10:38:44.869363-06:00 [debug] <0.2201645.0> segment_writer: skipping segment as directory /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-03-r2/quorum/rabbit@rabbitmq-03-r2/2F_ABCASMNUKWGARBM does not exist
2025-10-03 10:38:44.869398-06:00 [debug] <0.2201644.0> ra_log_segment_writer: cleaning closed table for '2F_ABC2VB8XZA412J6' range: 0-4
2025-10-03 10:38:44.869512-06:00 [debug] <0.2201645.0> ra_log_segment_writer: cleaning closed table for '2F_ABCASMNUKWGARBM' range: 0-4
2025-10-03 10:38:44.869501-06:00 [debug] <0.2201649.0> ra_log_segment_writer: cleaning closed table for '2F_ABCVBIC12SEI350' range: 0-4
2025-10-03 10:38:44.869539-06:00 [debug] <0.2201640.0> ra_log_segment_writer: cleaning closed table for '2F_ABC8AWI8H1BNFEP' range: 0-26
2025-10-03 10:38:44.869713-06:00 [debug] <0.2201646.0> segment_writer: failed to open segment file /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-03-r2/quorum/rabbit@rabbitmq-03-r2/2F_ABCAH8NAE0HRJ18/00000001.segment error: enoent
2025-10-03 10:38:44.869745-06:00 [debug] <0.2201646.0> segment_writer: skipping segment as directory /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-03-r2/quorum/rabbit@rabbitmq-03-r2/2F_ABCAH8NAE0HRJ18 does not exist
2025-10-03 10:38:44.869844-06:00 [debug] <0.2201646.0> ra_log_segment_writer: cleaning closed table for '2F_ABCAH8NAE0HRJ18' range: 0-4
2025-10-03 10:38:44.869845-06:00 [debug] <0.2201655.0> segment_writer: failed to open segment file /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-03-r2/quorum/rabbit@rabbitmq-03-r2/2F_ABCI6T32OLQOYIB/00000001.segment error: enoent
2025-10-03 10:38:44.869883-06:00 [debug] <0.2201650.0> segment_writer: failed to open segment file /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-03-r2/quorum/rabbit@rabbitmq-03-r2/2F_RPCGD01IMWZVXTZ/00000001.segment error: enoent
2025-10-03 10:38:44.869884-06:00 [debug] <0.2201661.0> segment_writer: failed to open segment file /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-03-r2/quorum/rabbit@rabbitmq-03-r2/2F_ABC8X7VBH0OI9XY/00000001.segment error: enoent
2025-10-03 10:38:44.869869-06:00 [debug] <0.2201665.0> segment_writer: failed to open segment file /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-03-r2/quorum/rabbit@rabbitmq-03-r2/2F_RPCEAENO24OUXTA/00000001.segment error: enoent
2025-10-03 10:38:44.869871-06:00 [debug] <0.2201648.0> segment_writer: failed to open segment file /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-03-r2/quorum/rabbit@rabbitmq-03-r2/2F_ABCDKUGSQYYIBPQ/00000001.segment error: enoent
2025-10-03 10:38:44.869873-06:00 [debug] <0.2201655.0> segment_writer: skipping segment as directory /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-03-r2/quorum/rabbit@rabbitmq-03-r2/2F_ABCI6T32OLQOYIB does not exist

This happened during two stretches of time, and after 2 restarts of RMQ servers over weekend, the cluster is now stable with regards to these problems. But does anyone know how we can debug this any further or be prepared for more information the next time it happens?

Does this remind anyone of a similar issue?

Reproduction steps

In 3 node HA clusters create quorum queues (no auto-deletes).
All nodes running version 3.13.7

Expected behavior

No disappearing queues.

Additional context

No response

Answered by michaelklishin

Oct 8, 2025

RabbitMQ 3.13.x has been out of community support for over a year. Time to upgrade.

View full answer

michaelklishin · 2025-10-08T14:02:37Z

michaelklishin
Oct 8, 2025
Maintainer

RabbitMQ 3.13.x has been out of community support for over a year. Time to upgrade.

0 replies

michaelklishin · 2025-10-08T14:05:30Z

michaelklishin
Oct 8, 2025
Maintainer

enoent when opening a segment file is more likely to be an issue with the filesystem and not a bug in quorum queues. Quorum queues (well, Ra) do not do anything creative when it comes to creating/opening files.

In any case, for non-paying users, 3.13.x is entirely out of support.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

3.13: Quorum queue fails to open a segment file with `enoent` #14713

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 2 comments

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

3.13: Quorum queue fails to open a segment file with enoent #14713

Uh oh!

Uh oh!

ulagnes123-crypto Oct 8, 2025

Describe the bug

Reproduction steps

Expected behavior

Additional context

Replies: 2 comments

Uh oh!

michaelklishin Oct 8, 2025 Maintainer

Uh oh!

michaelklishin Oct 8, 2025 Maintainer

3.13: Quorum queue fails to open a segment file with `enoent` #14713

ulagnes123-crypto
Oct 8, 2025

michaelklishin
Oct 8, 2025
Maintainer

michaelklishin
Oct 8, 2025
Maintainer