Skip to content

Commit 3bd5684

Browse files
authored
Add idempotency (#6748)
* added idempotency router and middleware * added idempotency rules for routes classes, functions, jobs, installaions, users * fixed typo * ignore requests without header * removed unused var * enabled feature only for MongoDB * changed code comment * fixed inconsistend storage adapter specification * Trigger notification * Travis CI trigger * Travis CI trigger * Travis CI trigger * rebuilt option definitions * fixed incorrect import path * added new request ID header to allowed headers * fixed typescript typos * add new system class to spec helper * fixed typescript typos * re-added postgres conn parameter * removed postgres conn parameter * fixed incorrect schema for index creation * temporarily disabling index creation to fix postgres issue * temporarily disabling index creation to fix postgres issue * temporarily disabling index creation to fix postgres issue * temporarily disabling index creation to fix postgres issue * temporarily disabling index creation to fix postgres issue * temporarily disabling index creation to fix postgres issue * temporarily disabling index creation to fix postgres issue * trying to fix postgres issue * fixed incorrect auth when writing to _Idempotency * trying to fix postgres issue * Travis CI trigger * added test cases * removed number grouping * fixed test description * trying to fix postgres issue * added Github readme docs * added change log * refactored tests; fixed some typos * fixed test case * fixed default TTL value * Travis CI Trigger * Travis CI Trigger * Travis CI Trigger * added test case to increase coverage * Trigger Travis CI * changed configuration syntax to use regex; added test cases * removed unused vars * removed IdempotencyRouter * Trigger Travis CI * updated docs * updated docs * updated docs * updated docs * update docs * Trigger Travis CI * fixed coverage * removed code comments
1 parent cbf9da5 commit 3bd5684

21 files changed

+976
-533
lines changed

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
### master
44
[Full Changelog](https://github.com/parse-community/parse-server/compare/4.2.0...master)
5+
- NEW (EXPERIMENTAL): Idempotency enforcement for client requests. This deduplicates requests where the client intends to send one request to Parse Server but due to network issues the server receives the request multiple times. **Caution, this is an experimental feature that may not be appropriate for production.** [#6744](https://github.com/parse-community/parse-server/issues/6744). Thanks to [Manuel Trezza](https://github.com/mtrezza).
56

67
### 4.2.0
78
[Full Changelog](https://github.com/parse-community/parse-server/compare/4.1.0...4.2.0)

README.md

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -388,6 +388,39 @@ Parse Server allows developers to choose from several options when hosting files
388388

389389
`GridFSBucketAdapter` is used by default and requires no setup, but if you're interested in using S3 or Google Cloud Storage, additional configuration information is available in the [Parse Server guide](http://docs.parseplatform.org/parse-server/guide/#configuring-file-adapters).
390390

391+
### Idempodency Enforcement
392+
393+
**Caution, this is an experimental feature that may not be appropriate for production.**
394+
395+
This feature deduplicates identical requests that are received by Parse Server mutliple times, typically due to network issues or network adapter access restrictions on mobile operating systems.
396+
397+
Identical requests are identified by their request header `X-Parse-Request-Id`. Therefore a client request has to include this header for deduplication to be applied. Requests that do not contain this header cannot be deduplicated and are processed normally by Parse Server. This means rolling out this feature to clients is seamless as Parse Server still processes request without this header when this feature is enbabled.
398+
399+
> This feature needs to be enabled on the client side to send the header and on the server to process the header. Refer to the specific Parse SDK docs to see whether the feature is supported yet.
400+
401+
Deduplication is only done for object creation and update (`POST` and `PUT` requests). Deduplication is not done for object finding and deletion (`GET` and `DELETE` requests), as these operations are already idempotent by definition.
402+
403+
#### Configuration example
404+
```
405+
let api = new ParseServer({
406+
idempotencyOptions: {
407+
paths: [".*"], // enforce for all requests
408+
ttl: 120 // keep request IDs for 120s
409+
}
410+
}
411+
```
412+
#### Parameters
413+
414+
| Parameter | Optional | Type | Default value | Example values | Environment variable | Description |
415+
|-----------|----------|--------|---------------|-----------|-----------|-------------|
416+
| `idempotencyOptions` | yes | `Object` | `undefined` | | PARSE_SERVER_EXPERIMENTAL_IDEMPOTENCY_OPTIONS | Setting this enables idempotency enforcement for the specified paths. |
417+
| `idempotencyOptions.paths`| yes | `Array<String>` | `[]` | `.*` (all paths, includes the examples below), <br>`functions/.*` (all functions), <br>`jobs/.*` (all jobs), <br>`classes/.*` (all classes), <br>`functions/.*` (all functions), <br>`users` (user creation / update), <br>`installations` (installation creation / update) | PARSE_SERVER_EXPERIMENTAL_IDEMPOTENCY_PATHS | An array of path patterns that have to match the request path for request deduplication to be enabled. The mount path must not be included, for example to match the request path `/parse/functions/myFunction` specifiy the path pattern `functions/myFunction`. A trailing slash of the request path is ignored, for example the path pattern `functions/myFunction` matches both `/parse/functions/myFunction` and `/parse/functions/myFunction/`. |
418+
| `idempotencyOptions.ttl` | yes | `Integer` | `300` | `60` (60 seconds) | PARSE_SERVER_EXPERIMENTAL_IDEMPOTENCY_TTL | The duration in seconds after which a request record is discarded from the database. Duplicate requests due to network issues can be expected to arrive within milliseconds up to several seconds. This value must be greater than `0`. |
419+
420+
#### Notes
421+
422+
- This feature is currently only available for MongoDB and not for Postgres.
423+
391424
### Logging
392425

393426
Parse Server will, by default, log:

resources/buildConfigDefinitions.js

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,9 @@ function getENVPrefix(iface) {
5252
if (iface.id.name === 'LiveQueryOptions') {
5353
return 'PARSE_SERVER_LIVEQUERY_';
5454
}
55+
if (iface.id.name === 'IdempotencyOptions') {
56+
return 'PARSE_SERVER_EXPERIMENTAL_IDEMPOTENCY_';
57+
}
5558
}
5659

5760
function processProperty(property, iface) {
@@ -170,6 +173,13 @@ function parseDefaultValue(elt, value, t) {
170173
});
171174
literalValue = t.objectExpression(props);
172175
}
176+
if (type == 'IdempotencyOptions') {
177+
const object = parsers.objectParser(value);
178+
const props = Object.keys(object).map((key) => {
179+
return t.objectProperty(key, object[value]);
180+
});
181+
literalValue = t.objectExpression(props);
182+
}
173183
if (type == 'ProtectedFields') {
174184
const prop = t.objectProperty(
175185
t.stringLiteral('_User'), t.objectPattern([

spec/Idempotency.spec.js

Lines changed: 247 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,247 @@
1+
'use strict';
2+
const Config = require('../lib/Config');
3+
const Definitions = require('../lib/Options/Definitions');
4+
const request = require('../lib/request');
5+
const rest = require('../lib/rest');
6+
const auth = require('../lib/Auth');
7+
const uuid = require('uuid');
8+
9+
describe_only_db('mongo')('Idempotency', () => {
10+
// Parameters
11+
/** Enable TTL expiration simulated by removing entry instead of waiting for MongoDB TTL monitor which
12+
runs only every 60s, so it can take up to 119s until entry removal - ain't nobody got time for that */
13+
const SIMULATE_TTL = true;
14+
// Helpers
15+
async function deleteRequestEntry(reqId) {
16+
const config = Config.get(Parse.applicationId);
17+
const res = await rest.find(
18+
config,
19+
auth.master(config),
20+
'_Idempotency',
21+
{ reqId: reqId },
22+
{ limit: 1 }
23+
);
24+
await rest.del(
25+
config,
26+
auth.master(config),
27+
'_Idempotency',
28+
res.results[0].objectId);
29+
}
30+
async function setup(options) {
31+
await reconfigureServer({
32+
appId: Parse.applicationId,
33+
masterKey: Parse.masterKey,
34+
serverURL: Parse.serverURL,
35+
idempotencyOptions: options,
36+
});
37+
}
38+
// Setups
39+
beforeEach(async () => {
40+
if (SIMULATE_TTL) { jasmine.DEFAULT_TIMEOUT_INTERVAL = 200000; }
41+
await setup({
42+
paths: [
43+
"functions/.*",
44+
"jobs/.*",
45+
"classes/.*",
46+
"users",
47+
"installations"
48+
],
49+
ttl: 30,
50+
});
51+
});
52+
// Tests
53+
it('should enforce idempotency for cloud code function', async () => {
54+
let counter = 0;
55+
Parse.Cloud.define('myFunction', () => {
56+
counter++;
57+
});
58+
const params = {
59+
method: 'POST',
60+
url: 'http://localhost:8378/1/functions/myFunction',
61+
headers: {
62+
'X-Parse-Application-Id': Parse.applicationId,
63+
'X-Parse-Master-Key': Parse.masterKey,
64+
'X-Parse-Request-Id': 'abc-123'
65+
}
66+
};
67+
expect(Config.get(Parse.applicationId).idempotencyOptions.ttl).toBe(30);
68+
await request(params);
69+
await request(params).then(fail, e => {
70+
expect(e.status).toEqual(400);
71+
expect(e.data.error).toEqual("Duplicate request");
72+
});
73+
expect(counter).toBe(1);
74+
});
75+
76+
it('should delete request entry after TTL', async () => {
77+
let counter = 0;
78+
Parse.Cloud.define('myFunction', () => {
79+
counter++;
80+
});
81+
const params = {
82+
method: 'POST',
83+
url: 'http://localhost:8378/1/functions/myFunction',
84+
headers: {
85+
'X-Parse-Application-Id': Parse.applicationId,
86+
'X-Parse-Master-Key': Parse.masterKey,
87+
'X-Parse-Request-Id': 'abc-123'
88+
}
89+
};
90+
await expectAsync(request(params)).toBeResolved();
91+
if (SIMULATE_TTL) {
92+
await deleteRequestEntry('abc-123');
93+
} else {
94+
await new Promise(resolve => setTimeout(resolve, 130000));
95+
}
96+
await expectAsync(request(params)).toBeResolved();
97+
expect(counter).toBe(2);
98+
});
99+
100+
it('should enforce idempotency for cloud code jobs', async () => {
101+
let counter = 0;
102+
Parse.Cloud.job('myJob', () => {
103+
counter++;
104+
});
105+
const params = {
106+
method: 'POST',
107+
url: 'http://localhost:8378/1/jobs/myJob',
108+
headers: {
109+
'X-Parse-Application-Id': Parse.applicationId,
110+
'X-Parse-Master-Key': Parse.masterKey,
111+
'X-Parse-Request-Id': 'abc-123'
112+
}
113+
};
114+
await expectAsync(request(params)).toBeResolved();
115+
await request(params).then(fail, e => {
116+
expect(e.status).toEqual(400);
117+
expect(e.data.error).toEqual("Duplicate request");
118+
});
119+
expect(counter).toBe(1);
120+
});
121+
122+
it('should enforce idempotency for class object creation', async () => {
123+
let counter = 0;
124+
Parse.Cloud.afterSave('MyClass', () => {
125+
counter++;
126+
});
127+
const params = {
128+
method: 'POST',
129+
url: 'http://localhost:8378/1/classes/MyClass',
130+
headers: {
131+
'X-Parse-Application-Id': Parse.applicationId,
132+
'X-Parse-Master-Key': Parse.masterKey,
133+
'X-Parse-Request-Id': 'abc-123'
134+
}
135+
};
136+
await expectAsync(request(params)).toBeResolved();
137+
await request(params).then(fail, e => {
138+
expect(e.status).toEqual(400);
139+
expect(e.data.error).toEqual("Duplicate request");
140+
});
141+
expect(counter).toBe(1);
142+
});
143+
144+
it('should enforce idempotency for user object creation', async () => {
145+
let counter = 0;
146+
Parse.Cloud.afterSave('_User', () => {
147+
counter++;
148+
});
149+
const params = {
150+
method: 'POST',
151+
url: 'http://localhost:8378/1/users',
152+
body: {
153+
username: "user",
154+
password: "pass"
155+
},
156+
headers: {
157+
'X-Parse-Application-Id': Parse.applicationId,
158+
'X-Parse-Master-Key': Parse.masterKey,
159+
'X-Parse-Request-Id': 'abc-123'
160+
}
161+
};
162+
await expectAsync(request(params)).toBeResolved();
163+
await request(params).then(fail, e => {
164+
expect(e.status).toEqual(400);
165+
expect(e.data.error).toEqual("Duplicate request");
166+
});
167+
expect(counter).toBe(1);
168+
});
169+
170+
it('should enforce idempotency for installation object creation', async () => {
171+
let counter = 0;
172+
Parse.Cloud.afterSave('_Installation', () => {
173+
counter++;
174+
});
175+
const params = {
176+
method: 'POST',
177+
url: 'http://localhost:8378/1/installations',
178+
body: {
179+
installationId: "1",
180+
deviceType: "ios"
181+
},
182+
headers: {
183+
'X-Parse-Application-Id': Parse.applicationId,
184+
'X-Parse-Master-Key': Parse.masterKey,
185+
'X-Parse-Request-Id': 'abc-123'
186+
}
187+
};
188+
await expectAsync(request(params)).toBeResolved();
189+
await request(params).then(fail, e => {
190+
expect(e.status).toEqual(400);
191+
expect(e.data.error).toEqual("Duplicate request");
192+
});
193+
expect(counter).toBe(1);
194+
});
195+
196+
it('should not interfere with calls of different request ID', async () => {
197+
let counter = 0;
198+
Parse.Cloud.afterSave('MyClass', () => {
199+
counter++;
200+
});
201+
const promises = [...Array(100).keys()].map(() => {
202+
const params = {
203+
method: 'POST',
204+
url: 'http://localhost:8378/1/classes/MyClass',
205+
headers: {
206+
'X-Parse-Application-Id': Parse.applicationId,
207+
'X-Parse-Master-Key': Parse.masterKey,
208+
'X-Parse-Request-Id': uuid.v4()
209+
}
210+
};
211+
return request(params);
212+
});
213+
await expectAsync(Promise.all(promises)).toBeResolved();
214+
expect(counter).toBe(100);
215+
});
216+
217+
it('should re-throw any other error unchanged when writing request entry fails for any other reason', async () => {
218+
spyOn(rest, 'create').and.rejectWith(new Parse.Error(0, "some other error"));
219+
Parse.Cloud.define('myFunction', () => {});
220+
const params = {
221+
method: 'POST',
222+
url: 'http://localhost:8378/1/functions/myFunction',
223+
headers: {
224+
'X-Parse-Application-Id': Parse.applicationId,
225+
'X-Parse-Master-Key': Parse.masterKey,
226+
'X-Parse-Request-Id': 'abc-123'
227+
}
228+
};
229+
await request(params).then(fail, e => {
230+
expect(e.status).toEqual(400);
231+
expect(e.data.error).toEqual("some other error");
232+
});
233+
});
234+
235+
it('should use default configuration when none is set', async () => {
236+
await setup({});
237+
expect(Config.get(Parse.applicationId).idempotencyOptions.ttl).toBe(Definitions.IdempotencyOptions.ttl.default);
238+
expect(Config.get(Parse.applicationId).idempotencyOptions.paths).toBe(Definitions.IdempotencyOptions.paths.default);
239+
});
240+
241+
it('should throw on invalid configuration', async () => {
242+
await expectAsync(setup({ paths: 1 })).toBeRejected();
243+
await expectAsync(setup({ ttl: 'a' })).toBeRejected();
244+
await expectAsync(setup({ ttl: 0 })).toBeRejected();
245+
await expectAsync(setup({ ttl: -1 })).toBeRejected();
246+
});
247+
});

spec/ParseQuery.Aggregate.spec.js

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1440,7 +1440,7 @@ describe('Parse.Query Aggregate testing', () => {
14401440
['location'],
14411441
'geoIndex',
14421442
false,
1443-
'2dsphere'
1443+
{ indexType: '2dsphere' },
14441444
);
14451445
// Create objects
14461446
const GeoObject = Parse.Object.extend('GeoObject');

spec/helper.js

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -230,6 +230,7 @@ afterEach(function(done) {
230230
'_Session',
231231
'_Product',
232232
'_Audience',
233+
'_Idempotency'
233234
].indexOf(className) >= 0
234235
);
235236
}

src/Adapters/Storage/Mongo/MongoStorageAdapter.js

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -692,26 +692,28 @@ export class MongoStorageAdapter implements StorageAdapter {
692692
fieldNames: string[],
693693
indexName: ?string,
694694
caseInsensitive: boolean = false,
695-
indexType: any = 1
695+
options?: Object = {},
696696
): Promise<any> {
697697
schema = convertParseSchemaToMongoSchema(schema);
698698
const indexCreationRequest = {};
699699
const mongoFieldNames = fieldNames.map((fieldName) =>
700700
transformKey(className, fieldName, schema)
701701
);
702702
mongoFieldNames.forEach((fieldName) => {
703-
indexCreationRequest[fieldName] = indexType;
703+
indexCreationRequest[fieldName] = options.indexType !== undefined ? options.indexType : 1;
704704
});
705705

706706
const defaultOptions: Object = { background: true, sparse: true };
707707
const indexNameOptions: Object = indexName ? { name: indexName } : {};
708+
const ttlOptions: Object = options.ttl !== undefined ? { expireAfterSeconds: options.ttl } : {};
708709
const caseInsensitiveOptions: Object = caseInsensitive
709710
? { collation: MongoCollection.caseInsensitiveCollation() }
710711
: {};
711712
const indexOptions: Object = {
712713
...defaultOptions,
713714
...caseInsensitiveOptions,
714715
...indexNameOptions,
716+
...ttlOptions,
715717
};
716718

717719
return this._adaptiveCollection(className)

src/Adapters/Storage/Postgres/PostgresStorageAdapter.js

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1209,6 +1209,7 @@ export class PostgresStorageAdapter implements StorageAdapter {
12091209
'_GlobalConfig',
12101210
'_GraphQLConfig',
12111211
'_Audience',
1212+
'_Idempotency',
12121213
...results.map((result) => result.className),
12131214
...joins,
12141215
];
@@ -2576,9 +2577,9 @@ export class PostgresStorageAdapter implements StorageAdapter {
25762577
fieldNames: string[],
25772578
indexName: ?string,
25782579
caseInsensitive: boolean = false,
2579-
conn: ?any = null
2580+
options?: Object = {},
25802581
): Promise<any> {
2581-
conn = conn != null ? conn : this._client;
2582+
const conn = options.conn !== undefined ? options.conn : this._client;
25822583
const defaultIndexName = `parse_default_${fieldNames.sort().join('_')}`;
25832584
const indexNameOptions: Object =
25842585
indexName != null ? { name: indexName } : { name: defaultIndexName };

src/Adapters/Storage/StorageAdapter.js

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -93,7 +93,7 @@ export interface StorageAdapter {
9393
fieldNames: string[],
9494
indexName?: string,
9595
caseSensitive?: boolean,
96-
indexType?: any
96+
options?: Object,
9797
): Promise<any>;
9898
ensureUniqueness(
9999
className: string,

0 commit comments

Comments
 (0)