readableStream broken with raw leaves

[parkan]

• Version: v0.30.0
• Platform: any
• Subsystem: files

Type: Bug-ish

Severity: Moderate

Description

The browser-readablestream demo is a very nice way of handling large video files for seeking. Unfortunately, it appears to break when the files are added using --raw-leaves (required for filestore/urlstore) because it attempts to unmarshal the leaf nodes as unixfs protobufs.

(confirmed my suspicion about this with @achingbrain)

Steps to reproduce the error:

add video file with --raw-leaves, attempt to load with readableStream

Note

This is required for the IA-over-IPFS demo at DWeb Summit on August 1, let me know how I can help get it working by then

[alanshaw]

@parkan do you have an error message or stack trace to help us track this one down?

[parkan]

error is:

decode.js:80 Uncaught TypeError: Cannot read property 'length' of undefined
    at Object.decode (decode.js:80)
    at Function.Data.unmarshal (index.js:101)
    at visitor (file.js:88)
    at eval (index.js:41)
    at drain (index.js:19)
    at eval (index.js:45)
    at dag.get (file.js:117)
    at _get (index.js:143)
    at eval (once.js:12)
    at next (waterfall.js:21)

with zdj7Wc9BBA2kar84oo8S6VotYc9PySAnmc8ji6kzKAFjqMxHS (or QmV3voixu7NXC6VChMYC2AmammEj5EBzsPhEKWmavPo47W in v0)

[daviddias]

@alanshaw this happens when we try to do a unixfs.decode in unixfs-engine and the node is just a raw-leaf. //cc @achingbrain

@parkan can you write some interop tests that catch this?

[parkan]

@diasdavid sure, but I won't have any time before DWeb, can we merge a hotfix and get it covered after aug 3rd?

[parkan]

mfs support for raw-leaves got merged: ipfs-inactive/js-ipfs-mfs#5 🎉

what remains to take this over the finish line?