Skip to content

Add performance tests to compare crud and vshard #251

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

DifferentialOrange
Copy link
Member

@DifferentialOrange DifferentialOrange commented Dec 15, 2021

This patch adds new cases for performance tests: select for equal
conditions for primary and secondary indexes (including known bucket_id
case from #220), and adds corresponding vshard test cases to compare
performance. Comparison may be not exactly precise since vshard test
functions use naive mergers, but should at least estimate basic
differences.

Test run on HP ProBook 440 G7 i7/16Gb/256SSD shows that CRUD module is
3-6 times slower than vshard calls for prepared functions for select and
1.6 times slower for insert.

HP ProBook 440 G7 i7/16Gb/256SSD

SUCCESS REQUESTS PER SECOND

(The higher the better)

vshard crud (stats disabled) crud (stats disabled, known bucket_id)
select by pk 81824.72 23936.52 23992.12
select gt by pk (limit 10) 15201.35 4305.91
select eq by secondary (limit 10) 14783.89 4835.79
select eq by sharding secondary 79791.30 13689.91 18376.29
insert 84113.60 51847.18

AVERAGE CALL TIME

(The lower the better)

vshard crud (stats disabled) crud (stats disabled, known bucket_id)
select by pk 2.444 ms 8.355 ms 8.335 ms
select gt by pk (limit 10) 13.156 ms 46.446 ms
select eq by secondary (limit 10) 13.528 ms 41.357 ms
select eq by sharding secondary 2.506 ms 14.604 ms 10.881 ms
insert 7.132 ms 11.571 ms

MAX CALL TIME

(The lower the better)

vshard crud (stats disabled) crud (stats disabled, known bucket_id)
select by pk 17.277 ms 37.793 ms 36.457 ms
select gt by pk (limit 10) 41.015 ms 103.195 ms
select eq by secondary (limit 10) 43.169 ms 98.459 ms
select eq by sharding secondary 15.934 ms 71.610 ms 58.771 ms
insert 66.039 ms 72.328 ms

CI Runner

See https://github.com/tarantool/crud/runs/5421561722?check_suite_focus=true

Several minor fixes also was introduced.

Fix replicaset alias in test cluster. Disable checks for perf tests.
Return tuples from pairs perf case.

Fix running separate tests. After this patch, developer can run luatest
with -p flag for perf tests to get results only for requested tests.
Example:

PERF_MODE_ON=true ./.rocks/bin/luatest -c -p perf.test_crud_pairs_gt_with_stats_disabled
  • Tests
  • Changelog (I don't think it should bother users)
  • Documentation (There's nothing to document)

Closes #225

@DifferentialOrange DifferentialOrange force-pushed the DifferentialOrange/gh-225-vshard-performance branch 3 times, most recently from e976a4a to 580ba0a Compare December 17, 2021 09:19
@DifferentialOrange DifferentialOrange force-pushed the DifferentialOrange/gh-224-operation-stats branch from 23c6445 to 3332a01 Compare December 17, 2021 14:04
@DifferentialOrange DifferentialOrange marked this pull request as draft December 17, 2021 15:57
@DifferentialOrange DifferentialOrange force-pushed the DifferentialOrange/gh-225-vshard-performance branch from 580ba0a to 21420dd Compare December 20, 2021 08:03
@DifferentialOrange DifferentialOrange marked this pull request as ready for review December 20, 2021 09:55
@DifferentialOrange
Copy link
Member Author

PR is ready for review, but since it depends on #244 improvements, I will not request any reviews for now (until #244 is merged).

@DifferentialOrange DifferentialOrange force-pushed the DifferentialOrange/gh-224-operation-stats branch 2 times, most recently from 5deebbd to dc06b81 Compare December 20, 2021 14:40
@DifferentialOrange DifferentialOrange force-pushed the DifferentialOrange/gh-225-vshard-performance branch 2 times, most recently from 10733cb to 859a624 Compare December 20, 2021 14:51
@DifferentialOrange DifferentialOrange force-pushed the DifferentialOrange/gh-224-operation-stats branch 7 times, most recently from bea37df to afec422 Compare December 24, 2021 20:52
@DifferentialOrange DifferentialOrange force-pushed the DifferentialOrange/gh-224-operation-stats branch from 23cf464 to 8e6e5f9 Compare January 18, 2022 09:46
@DifferentialOrange DifferentialOrange force-pushed the DifferentialOrange/gh-224-operation-stats branch 8 times, most recently from df98d90 to b2dd54c Compare February 4, 2022 10:00
@DifferentialOrange DifferentialOrange force-pushed the DifferentialOrange/gh-224-operation-stats branch from b2dd54c to c34491e Compare February 17, 2022 10:26
@DifferentialOrange DifferentialOrange force-pushed the DifferentialOrange/gh-224-operation-stats branch 7 times, most recently from 8c054db to 0ed9215 Compare February 25, 2022 09:18
Base automatically changed from DifferentialOrange/gh-224-operation-stats to master February 25, 2022 09:41
@DifferentialOrange DifferentialOrange force-pushed the DifferentialOrange/gh-225-vshard-performance branch 3 times, most recently from b0b9690 to 920523e Compare February 25, 2022 12:18
Copy link
Member

@Totktonada Totktonada left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Quite interesting results, thanks!

I left some minor comments. Feel free to fix or go ahead.

Fix replicaset alias in test cluster. Disable checks for perf tests.
Return tuples from pairs perf case.

Fix running separate tests. After this patch, developer can run luatest
with -p flag for perf tests to get results only for requested tests.
Example:
```
PERF_MODE_ON=true ./.rocks/bin/luatest -c -p perf.test_crud_pairs_gt_with_stats_disabled
```
@DifferentialOrange DifferentialOrange force-pushed the DifferentialOrange/gh-225-vshard-performance branch from 920523e to fc249e9 Compare March 4, 2022 11:35
This patch adds new cases for performance tests: select for equal
conditions for primary and secondary indexes (including known bucket_id
case from #220), and adds corresponding vshard test cases to compare
performance. Comparison may be not exactly precise since approaches to
vshard stored procedures may vary, but it should at least estimate basic
differences.

Test run on HP ProBook 440 G7 i7/16Gb/256SSD shows that CRUD module is
3-6 times slower than vshard calls for prepared functions for select and
1.6 times slower for insert.

Closes #225
@DifferentialOrange DifferentialOrange force-pushed the DifferentialOrange/gh-225-vshard-performance branch from fc249e9 to 3c53164 Compare March 4, 2022 12:23
@DifferentialOrange DifferentialOrange merged commit a315a67 into master Mar 4, 2022
@DifferentialOrange DifferentialOrange deleted the DifferentialOrange/gh-225-vshard-performance branch March 4, 2022 13:47
@Totktonada Totktonada mentioned this pull request Jun 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Create a test with comparison vshard.call() and CRUD performance
2 participants