Content Encryption

[whyrusleeping]

Currently, ipfs does not support any sort of encryption in-line. Users can always encrypt content before adding it into ipfs, which is a solid solution (and a good separation of concerns) but it leaves a bit to be desired for people who just want to use ipfs directly.

There are two main ways that we can have encrypted content in ipfs, both encrypt the actual content, but the first is essentially 'encrypt the data, then add it to ipfs'. The second is 'add the data to ipfs, then encrypt each dag node individually'.

Encrypt First

When encrypting the content first, the dag structure is still plaintext. This has several implications, primarily:

• Attackers can glean information about the size of the file, its chunking algorithm, and its importer layout parameters without having the encryption keys.
• Content can be replicated through the network by nodes who don't know the encryption keys.

Being able to have other peers replicate your content without the encryption keys is a very nice feature to have.

Encrypt Second (or Encrypt the Dag)

To add a bit more secrecy to your data, you can encrypt the raw dag nodes. This definitely prevents anyone without the right keys from gleaning much information about the object in question (does it have links? what is the object type? how big is the whole structure?)

This sort of encryption would be required to encrypt things other than files, like directories, or arbitrary ipld nodes.

UX

So the previous part needs a fair amount of thought, but once we get an idea of how we want to do that, we need a way of actually interacting with encrypted objects. A proposal I have is to adopt something like:

ipfs cat QmFooBarEncrypted#enckey:1bf72d9ef

Where 1bf72d9ef is the key to decrypt the data referenced by QmFooBarEncrypted.

This is nice because with a browser extension, you could achieve the same effect in a browser without leaking the keys to anyone else (since the hash fragment is never sent to the server).

This is definitely a WIP proposal, i'll be updating it as I think through things more and discuss.

[Kubuxu]

We should consider how it would work with the rest of the toolkit. For example GC of the repo.

One of the solutions for it might be having two part encryption key, one for links one for the content but this gets problematic with IPLD.

[whyrusleeping]

Yeah, that gets really problematic with ipld. I also didnt consider GC when thinking about this... that makes things rather painful...

[keks]

One solution is to only store the plaintext, annotated with encryption nonce and key, and encrypt the data on the fly once it leaves the host.
An alternative would be to store the ciphertetx and decrypt on the fly to do batch processing like GC.
A third alternative would be to have an index service that stores indexes for all important queries for both encrypted and unencrypted blobs. The indexer operates on the decrypted blob, but doesn't store it - only metadata that won't leave the system.

I probably wouldn't go down the multiple keys path as that seams like it solve only one very specific problem.

Is there supposed to be a database of known keys, or should the user pass a key and subsequent keys will be in encrypted DAG element? Because if we have a database I think it's simplest (and probably fast enough) to decrypt on the fly.

[ianopolous]

My 2 cents:

If you're only adding a single file then there is no way to hide the size without padding the file pre encyption.

Assuming that you want to store actual sub-trees with files and folders you can hide the size of individual files and also the directory contents, filenames etc if you want.

To do this properly I expect you will reinvent what we do in Peergos. A file tree for us is stored in a merkle-btree, giving you a single root you can pin for gc purposes. Each directory and file gets it's own encryption key. each directory consists of encrypted links to the children (and encrypted directory name). The actual structure is called cryptree (paper is in our repo) and was invented by Wuala in 2007. Using different keys for every file allows you to selectively grant read access, but for your purposes you might be okay with a single key for the entire sub-tree. We also chunk files into 5 MiB sections which each get independently encrypted and added to the btree under a random label. This means the network can see the total size of all files, and roughly the total number of directories (large directories are chunked as well), but not the individual metadata like names, modification date, path, or indeed the topology of the tree.

Then to read the tree you just need the encryption key, and bree label of the root dir.

Another thing I'd advise is to not use asymmetric encryption. The reason being that all standard asymmetric encryption is trivially broken by a sufficiently large quantum computer (which we expect in 5 - 15 years). The cryptree structure is nice because it allows you to create the whole fine grained structure without any asymmetric crypto.

One nice bonus feature would be do do what we do to obtain the root key, which is to generate it from a memory hard hashing function like scrypt. Then you can optionally password protect files, which may result in better UX than a full 32 byte key. (Obviously the encryption is only as secure as the password though)

[richardschneider]

I like the Encrypt First approach. Most use cases are about protecting the data not about hiding minor metadata. I agree that Allowing the content to be replicated through the network (without having the key) is a very good.

My proposal is simple

• the "content" becomes a PKCS 7/CMS message, when the data is encrypted and/or signed
• an optional field is added to the DAG to indicate that Data is CMS

Please read and comment on my notes Using PKCS 7 (Crypto Message Syntax) and the DAG.

The above assumes that a peer has a secure keystore to manage the encrypting and signing keys.

[richardschneider]

@ianopolous my understanding is that CMS allows password based encryption.

I'm interested in your warning about asymmetric encryption. Could you provide some links to the cryptree structure?

[Stebalien]

Personally, I'm not convinced that the encrypt second approach is really worth anything if peers won't be able to replicate the content. We might as well just encrypt the data locally and use authentication to control who gets access. That's a lot more flexible.

---

If we want to be able to replicate and completely encrypt, we wouldn't be able to lazily fetch files (data access patterns will reveal the file structure). Instead, we'd have to store the entire filesystem in some form of append-only log (possibly with checkpointing). Unfortunately, this would be very inefficient.

There are probably ways to do this with differential privacy but I don't know any off the top of my head.

[mitra42]

Interesting discussion, been working on this a bit in our Dweb Library, and use it in our Versioning demo.

The approach taken is that content is encrypted symetrically, then the key is stored encrypted with the public key of any authorised viewer on an AccessControlList. Content is encrypted/decrypted as it goes through the transport layer. When encrypted content is received, the transport retrieves the ACL, and looks for matching keys between there and any of the user's keys. Similar to @richardschneider 's suggestion we wrap the encrypted content in the necessary fields and then store in the DAG.

That's glossing over the details, but there is also:

• a google doc describing it in a lot more detail.
• An example UI that walks through the key management at example_keys.html
• An example of a trivial publishing tool that tracks versions that uses it for its encryption at example_versions.html
• Git repo;

Its entirely browser/JS based, the transporting peers/content servers have no knowledge of the fact content is encrypted nor access to the content, and private keys aren't stored unless themselves encrypted with some higher level key. An earlier version also works in Python, but obviously not over IPFS.

Its transport agnostic, and runs over both IPFS and our internet HTTPS contenthash servers, the default is to use both, but because of the IPFS crashing bugs in Pubsub it will crash every few minutes on IPFS, so until that is fixed, you can use the HTTPS only version by adding transport=HTTP to the URL e.g. keys; and versions; Add "&verbose=true" to each of these as well if you want to look at the console to see what is happening.

(It doesn't tackle the file length concerns of @Stebalien as we don't think that level of encryption/hiding is needed for 90% of applications out there, and the other 10% probably need something much more complex).

[RangerMauve]

@mitra42 Do you encrypt the DAG nodes themselves?

Does your approach just store files, or do you also store folders?

[ianopolous]

@richardschneider Here is the cryptree paper: https://github.com/Peergos/Peergos/blob/master/papers/wuala-cryptree.pdf

In my opinion granting access is outside the scope of this for ipfs (do one thing and do it well). You have access if you have the appropriate symmetric key/passphrase. Individual applications can then use their own key sharing mechanism on top of this.

@Stebalien In our approach to "encrypt second", everything including all metadata, directory structure and topology is encrypted and can be safely shared and duplicated around the network. In terms of hiding the access pattern that is also achievable (if you want it) but is a separate concern to defending the data at rest. We have this implemented already in Java and Javascript for Peergos, see the following:
For the two cryptree nodes files and directories.
And for the merkle-btree which stores them and the actual encrypted file fragments:
https://github.com/Peergos/Peergos/blob/master/src/peergos/shared/merklebtree/MerkleBTree.java

[Stebalien]

@mitra42

Content is encrypted/decrypted as it goes through the transport layer.

Our transports are already encrypted and authenticated, that's why I suggested authentication. Unlike simple encryption, authentication is revocable (and generally more flexible). A simple solution would be to have a separate "capabilities" service. If peer A makes a request to peer B that requires capability C, peer A would ask its capabilities service if peer B has capability C. The two peers' capability services would then run an authentication protocol to actually determine if B has capability C. This allows us to keep authentication/capabilities entirely separate from bitswap, pubsub, etc.

separate concern to defending the data at rest.

Data at rest is really the concern of the OS (e.g., full disk encryption).

[ianopolous]

@Stebalien By data at rest I mean the data structure as visible to the entire network (as ipfs is essentially a global file system) - so nothing to do with your harddisk that you happen to be running ipfs on.

I would strongly advise against authentication (in the form of asymmetric crypto) for the reason i discussed above. Symmetric encryption + hashing are all you need.

[mitra42]

Good discussion …

Sorry if I confused things, When I refer to “encryption in the Transport layer”, I’m talking about it from the app’s perspective, and mean from the point where something is stored in IPFS, to the point where its retrieved by another node. Its great that IPFS is encrypted on the wire between two peers, but in a distributed file system that doesn't protect data since any peer can retrieve it, so another layer is required not just on-the-wire (bitswap) encryption. I think in that sense I'm referring to the same thing as @ianopolous

@RangerMauve - you ask “Do you encrypt the DAG nodes themselves” and I’m not quite clear what you are asking. We encrypt JSON data structures, wrap in another structure (so you know what you’ve got) and then store with dag.put

We are currently storing data structures, I’d use the same approach for files, but haven’t done so yet.

@ianopolous - I agree you “can” build a key sharing mechanism, the challenge is to actually do so in a distributed system. (From historical perspective its one reason why PGP didn’t achieve wide adoption for email - key sharing was so hard that in practice didn’t get done). In our model, the encrypted data carries a pointer to the Access Control List, so that the transport knows how to decrypt it. I’d suggest that that mechanism is made self-describing as well, so for example someone retrieving a DAG node could see that it was encrypted with Peergos, or with some other mechanism so that it knew if it could decrypt it.

Peergos looks interesting, is the API defined somewhere ? And I see a comment on a cross-compile to Javascript but cant find it on the repository.

@Stebalien - IPFS is “data at rest”, as Ian says IPFS is a global shared file system, so data in it should be encrypted one way or another. Also … you can’t presume a negotiation between Peer A and Peer B during the decryption process. You have to presume that Peer B posted the data (and authentication info) to IPFS and then disconnected, and is not available in any way during decryption. Peer A has to interact entirely with the data stored in the global file system.