Support for UnixFS Sharding + support balanced & trickle dag

[daviddias]

When spec is ready

[hackergrrl]

Is there a wip spec? Could you link to it (and/or any other relevant docs)?

[daviddias]

There you go. Gist is that we figured out how to do the Sharding while at NYC, waiting on @jbenet to lay down the spec.

• Sharding - unixfs support for large directories ipfs/specs#32
• Distinguish fanout from non-fanout links ipfs/specs#10
• Sharding IPLD objects ipfs/notes#76
• captain.log - go-ipfs ipfs/kubo#2247 (comment)

[daviddias]

@whyrusleeping can you share how is the 'file wrapping' happening in go-ipfs currently? If I'm not mistaken, it is something like ' each time we reach 174 links in a file dag node, we wrap it in another dagnode and add the chunks to the new one'

[daviddias]

@whyrusleeping is there a spec in progress for the sharding? plus probably some impl details to make it faster? It would also be good to have the trickle dag documented and something saying when we are moving to it so that we can prepare.

[dignifiedquire]

Does it make sense to do this before IPLD, as far as I understood the spec would change with IPLD?

[whyrusleeping]

The changes to the unixfs protobuf are the most relevant.

Beyond that, a few notes:

• using murmur3-64 over the directory entry names
• i'm using the unixfs Data field to hold the bitfield.
• All link names are prefixed with the hex representation of their index within the bitfield. Nested shards link names are just the hex number.
• When creating new directories, we leave them unsharded up until some threshold which is at the moment arbitrarily set to 1000

And after that, it just follows the HAMT algorithms

[jbenet]

• using murmur3-64 over the directory entry names

must be configurable.

@whyrusleeping do you recall where we put the HAMT notes we took on a pad once? it had the construction in IPLD and the mapping form protobuf to IPLD.