Skip to content

Commit 625ea91

Browse files
committed
Implementation of batch replace
Batch upsert is mostly used for operation with one bucket / one Tarantool node in a transaction. In this case batch replace is more efficient then replacing tuple-by-tuple. Right now CRUD cannot provide batch replace with full consistency. CRUD offers batch upsert with partial consistency. That means that full consistency can be provided only on single replicaset using `box` transactions. Part of #193
1 parent a75a1b7 commit 625ea91

File tree

9 files changed

+2597
-0
lines changed

9 files changed

+2597
-0
lines changed

CHANGELOG.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,11 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
88
## [Unreleased]
99

1010
### Added
11+
* Batch insert/upsert operation
12+
`crud.insert_many()`/`crud.insert_object_many()`/
13+
`crud.upsert_many()`/`crud.upsert_object_many()`
14+
`crud.replace_many()`/`crud.replace_object_many()`
15+
with partial consistency
1116

1217
### Changed
1318

README.md

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ It also provides the `crud-storage` and `crud-router` roles for
2121
- [Update](#update)
2222
- [Delete](#delete)
2323
- [Replace](#replace)
24+
- [Replace many](#replace-many)
2425
- [Upsert](#upsert)
2526
- [Upsert many](#upsert-many)
2627
- [Select](#select)
@@ -520,6 +521,142 @@ crud.replace_object('customers', {
520521
...
521522
```
522523

524+
### Replace many
525+
526+
```lua
527+
-- Replace batch of tuples
528+
local result, err = crud.replace_many(space_name, tuples, opts)
529+
-- Replace batch of objects
530+
local result, err = crud.replace_object_many(space_name, objects, opts)
531+
```
532+
533+
where:
534+
535+
* `space_name` (`string`) - name of the space to insert/replace an object
536+
* `tuples` / `objects` (`table`) - array of tuples/objects to insert
537+
* `opts`:
538+
* `timeout` (`?number`) - `vshard.call` timeout (in seconds)
539+
* `fields` (`?table`) - field names for getting only a subset of fields
540+
* `stop_on_error` (`?boolean`) - stop on a first error and report error
541+
regarding the failed operation and error about what tuples were not
542+
performed, default is `false`
543+
* `rollback_on_error` (`?boolean`) - any failed operation will lead to
544+
rollback on a storage, where the operation is failed, report error
545+
about what tuples were rollback, default is `false`
546+
547+
Returns metadata and array with inserted/replaced rows, array of errors.
548+
Each error object can contain field `operation_data`.
549+
550+
`operation_data` field can contain:
551+
* tuple for which the error occurred;
552+
* object with an incorrect format;
553+
* tuple the operation on which was performed but
554+
operation was rollback;
555+
* tuple the operation on which was not performed
556+
because operation was stopped by error.
557+
558+
Right now CRUD cannot provide batch replace with full consistency.
559+
CRUD offers batch replace with partial consistency. That means
560+
that full consistency can be provided only on single replicaset
561+
using `box` transactions.
562+
563+
**Example:**
564+
565+
```lua
566+
crud.replace_many('developers', {
567+
{1, box.NULL, 'Elizabeth', 'lizaaa'},
568+
{2, box.NULL, 'Anastasia', 'iamnewdeveloper'},
569+
})
570+
---
571+
- metadata:
572+
- {'name': 'id', 'type': 'unsigned'}
573+
- {'name': 'bucket_id', 'type': 'unsigned'}
574+
- {'name': 'name', 'type': 'string'}
575+
- {'name': 'login', 'type': 'string'}
576+
rows:
577+
- [1, 477, 'Elizabeth', 'lizaaa']
578+
- [2, 401, 'Anastasia', 'iamnewdeveloper']
579+
...
580+
crud.replace_object_many('developers', {
581+
{id = 1, name = 'Inga', login = 'mylogin'},
582+
{id = 10, name = 'Anastasia', login = 'qwerty'},
583+
})
584+
---
585+
- metadata:
586+
- {'name': 'id', 'type': 'unsigned'}
587+
- {'name': 'bucket_id', 'type': 'unsigned'}
588+
- {'name': 'name', 'type': 'string'}
589+
- {'name': 'age', 'type': 'number'}
590+
rows:
591+
- [1, 477, 'Inga', 'mylogin']
592+
- [10, 569, 'Anastasia', 'qwerty']
593+
594+
-- Partial success
595+
-- Let's say login has unique secondary index
596+
local res, errs = crud.replace_object_many('developers', {
597+
{id = 22, name = 'Alex', login = 'pushkinn'},
598+
{id = 3, name = 'Anastasia', login = 'qwerty'},
599+
{id = 5, name = 'Sergey', login = 's.petrenko'},
600+
})
601+
---
602+
res
603+
- metadata:
604+
- {'name': 'id', 'type': 'unsigned'}
605+
- {'name': 'bucket_id', 'type': 'unsigned'}
606+
- {'name': 'name', 'type': 'string'}
607+
- {'name': 'age', 'type': 'number'}
608+
rows:
609+
- [5, 1172, 'Sergey', 's.petrenko'],
610+
- [22, 655, 'Alex', 'pushkinn'],
611+
612+
#errs -- 1
613+
errs[1].class_name -- ReplaceManyError
614+
errs[1].err -- 'Duplicate key exists <...>'
615+
errs[1].tuple -- {3, 2804, 'Anastasia', 'qwerty'}
616+
617+
-- Partial success with stop and rollback on error
618+
-- stop_on_error = true, rollback_on_error = true
619+
-- two error on one storage with rollback, inserts stop by error on this storage
620+
-- inserts before error are rollback
621+
local res, crud.replace_object_many('developers', {
622+
{id = 6, name = 'Alex', login = 'alexpushkin'},
623+
{id = 92, name = 'Artur', login = 'AGolden'},
624+
{id = 11, name = 'Anastasia', login = 'qwerty'},
625+
{id = 4, name = 'Sergey', login = 's.smirnov'},
626+
{id = 9, name = 'Anna', login = 'AnnaBlack'},
627+
{id = 17, name = 'Oksana', login = 'OKonov'},
628+
}, {
629+
stop_on_error = true,
630+
rollback_on_error = true,
631+
})
632+
res
633+
- metadata:
634+
- {'name': 'id', 'type': 'unsigned'}
635+
- {'name': 'bucket_id', 'type': 'unsigned'}
636+
- {'name': 'name', 'type': 'string'}
637+
- {'name': 'age', 'type': 'number'}
638+
rows:
639+
- [4, 1161, 'Sergey', 's.smirnov'],
640+
- [6, 1064, 'Alex', 'alexpushkin'],
641+
#errs -- 4
642+
errs[1].class_name -- ReplaceManyError
643+
errs[1].err -- 'Duplicate key exists <...>'
644+
errs[1].tuple -- {11, 2652, "Anastasia", "qwerty"}
645+
646+
errs[2].class_name -- NotPerformedError
647+
errs[2].err -- 'Operation with tuple was not performed'
648+
errs[2].tuple -- {9, 1644, "Anna", "AnnaBlack"}
649+
650+
errs[3].class_name -- NotPerformedError
651+
errs[3].err -- 'Operation with tuple was not performed'
652+
errs[3].tuple -- {17, 2900, "Oksana", "OKonov"}
653+
654+
errs[4].class_name -- NotPerformedError
655+
errs[4].err -- 'Operation with tuple was rollback'
656+
errs[4].tuple -- {92, 2040, "Artur", "AGolden"}
657+
...
658+
```
659+
523660
### Upsert
524661

525662
```lua

crud.lua

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ local cfg = require('crud.cfg')
66
local insert = require('crud.insert')
77
local insert_many = require('crud.insert_many')
88
local replace = require('crud.replace')
9+
local replace_many = require('crud.replace_many')
910
local get = require('crud.get')
1011
local update = require('crud.update')
1112
local upsert = require('crud.upsert')
@@ -53,6 +54,14 @@ crud.replace = stats.wrap(replace.tuple, stats.op.REPLACE)
5354
-- @function replace_object
5455
crud.replace_object = stats.wrap(replace.object, stats.op.REPLACE)
5556

57+
-- @refer replace_many.tuples
58+
-- @function replace_many
59+
crud.replace_many = replace_many.tuples
60+
61+
-- @refer replace_many.objects
62+
-- @function replace_object_many
63+
crud.replace_object_many = replace_many.objects
64+
5665
-- @refer update.call
5766
-- @function update
5867
crud.update = stats.wrap(update.call, stats.op.UPDATE)
@@ -145,6 +154,7 @@ function crud.init_storage()
145154
insert_many.init()
146155
get.init()
147156
replace.init()
157+
replace_many.init()
148158
update.init()
149159
upsert.init()
150160
upsert_many.init()

0 commit comments

Comments
 (0)