Reversed order of add post (INS6) and add like to post (INS2)

[luoxiaojian]

In the updateStream_0_26_forum.csv file of the dataset, when listing all updates related to post-70368991551179 through

$ grep -w 70368991551179 updateStream_0_26_forum.csv
1349790181752|1278281267146|2|6597069905389|70368991551179|1349790181752
1349790181752|1289052676356|6|70368991551179||1349790181752|24.32.177.170|Internet Explorer|uz|About Niki Lauda, er Formula One racing driver and three-time F1 World Champion. More recently an aviati|104|10995116536216|21990277706324|57|164
1349790237878|1291770543934|2|10995116326590|70368991551179|1349790237878
1349790272484|1327084360170|2|26388279332421|70368991551179|1349790272484
1349790380456|1322943558393|2|24189256307723|70368991551179|1349790380456

(Vertex post-70368991551179 doesn't appear in any other streams and the initial snapshot.)

The first insert query related to post-70368991551179 is adding a like edge (INS2) to this post, and the second one is adding the post (INS6).

Though these two queries have a same timestamp, they will be issued by driver sequentially.

So there will be an adding edge query to the post vertex before the vertex is added.

Similar problem also found in the sf300 dataset when adding like edge to a comment (INS3).

[ArnauPrat]

I'm 99% sure the issue is at this line

ldbc_snb_datagen_hadoop/src/main/java/ldbc/snb/datagen/generator/generators/LikeGenerator.java

Line 73 in 9355de4

long minDate = message.creationDate() > memberships.get(startIndex + i).creationDate() ? message

or in this line

ldbc_snb_datagen_hadoop/src/main/java/ldbc/snb/datagen/generator/generators/LikeGenerator.java

Line 75 in 9355de4

long date = Dictionaries.dates.randomDate(random, minDate, Dictionaries.dates

where DatagenParams.deltaTime should be added, similar to other generators. Note also how the assert at line

ldbc_snb_datagen_hadoop/src/main/java/ldbc/snb/datagen/generator/generators/LikeGenerator.java

Line 77 in 9355de4

assert ((membership.person().creationDate() + DatagenParams.deltaTime) < date);

should be extended like
assert ((membership.person().creationDate() + DatagenParams.deltaTime) < date && message.creationDate() < date);

So for the particular example updates, my hypothesis is that the minDate is set to be the POST.creationDate (since this is created after the person membership to the forum), and we are so unlucky that the call to random.nextDouble() used to add an offset to that minDate returns 0.0.

[ArnauPrat]

I've made a pull request with a fix here #25

[szarnyasg]

Hello, I ran the generator with Arnau's patch and it fixed the problem:

$ grep -w 70368991551179 updateStream_0_26_forum.csv | head -n 5
1349790181752|1289052676356|6|70368991551179||1349790181752|24.32.177.170|Internet Explorer|uz|About Niki Lauda, er Formula One racing driver and three-time F1 World Champion. More recently an aviati|104|10995116536216|21990277706324|57|164
1349790191752|1278281267146|2|6597069905389|70368991551179|1349790191752
1349790247878|1291770543934|2|10995116326590|70368991551179|1349790247878
1349790282484|1327084360170|2|26388279332421|70368991551179|1349790282484
1349790390456|1322943558393|2|24189256307723|70368991551179|1349790390456

[szarnyasg]

I've been thinking because I was quite baffled how this error never surfaced before. I think the reason is that this timestamp (corresponding to events on 2012-10-12) is quite far back in the update stream whose events occur between 2010-09-13 and 2012-12-31. So most systems never made it this far because they ran for 2+ hours with updates before this date.

[ArnauPrat]

Yeah, and on top of that, we were very unlucky with the random generator producing a 0.0. Well, I don't know if we were unlucky since the larger the SF, the more likely it will happen. It could also be that other datasets had dependent likes with a timestamp difference less than deltaTime, but larger than 0

[szarnyasg]

In any case, the patch alters all data sets by changing the creationdates of the posts. I'm regenerating all data sets (with various serializers) and update streams (with different numparts between 1 and 1024).