Skip to content

Conversation

@tomjkidd
Copy link

@tomjkidd tomjkidd commented Dec 28, 2021

First off, wanted to say thank you for creating this library!

While trying to evaluate it as a solution for a use case of mine, I ran into some unexpected results. While debugging, it became clear that take-batch is not working as I had expected.

I've created a minimal test case to highlight the issue I'm seeing (this test is currently failing!), and it has to do with the query being issued via locking-take-batch . It appears that the underlying union all query is introducing duplicate items into the batch result. I started to analyze the query, but wanted to make sure that my expectations are indeed how it was designed to work before going too far down that path.

Test output


lein test pgqueue.core-test

lein test :only pgqueue.core-test/take-batch

FAIL in (take-batch) (core_test.clj:114)
take-batch
The batch doesn't produce more items than calls to put
expected: (= item-count (count batch))
  actual: (not (= 5 6))

lein test :only pgqueue.core-test/take-batch

FAIL in (take-batch) (core_test.clj:116)
take-batch
The items are returned in the correct order
expected: (= (range item-count) batch)
  actual: (not (= (0 1 2 3 4) (0 1 2 3 4 1)))

Ran 3 tests containing 134 assertions.
2 failures, 0 errors.
Tests failed.

Related, but not fully relevant

In case it helps, my use case is that I want to be able to take a set of horizontally scaled servers and have each of them put onto the same queue, and to wait for a small window, and then to let them race and have a single winner recover the items. I had tried to use count to guide how many items to pull from take-batch, but was running into some issues based on the race puts, so I decided to just provide the take-batch argument directly, asking for 10 items (since I have fewer than 10 servers I plan on using). The idea is that this should take all of the available items, and that there should be only as many as were put on the queue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant