Fix racing condition on adding directories

[daviddias]

A racing condition was introduced when adding directories 'recursively'

#312 (comment)

It doesn't happen always, we need to make a test that always detects it and fix it

[dignifiedquire]

This a351eb0 should fix the issue, but I am not a 100% sure as I couldn't reproduce the referenced race myself.

[daviddias]

Can you add a test with a lots of tiny files (like 100) and at least 10 levels of nest'ness, just to be sure? Thank you!

[dignifiedquire]

Can you add a test with a lots of tiny files (like 100) and at least 10 levels

All the files :D

[daviddias]

Note that sorting on the client should not matter, go-ipfs doesn't expect the things to come sorted through the http-api. Files are hashed first and them the dir nodes are constructed after, once there is no more files to add to the dir (flush: false) or keep getting recreated (flush: true)

[dignifiedquire]

I added the sorting to make it easier to test, as it makes the output deterministic.

[daviddias]

A good test to add is to add the same dir, but by a different order.

[dignifiedquire]

Race is still there :( seems to be related to how the unixfs stream module thing works rather than how we use it from js-ipfs :(

[daviddias]

@dignifiedquire is this still a bug with the new pull-streams based unixfs engine?

[dignifiedquire]

Never seen it in pull-streams land