You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Feb 12, 2024. It is now read-only.
fix: do not load all of a DAG into memory when pinning (#2372)
Given a `CID`, the `dag. _getRecursive` method returns a list of all descendents of the node with the passed `CID`. This can cause enormous memory usage when importing large datasets.
Where this method is invoked the results are either a) disgarded or b) used to calculate the `CID`s of the nodes which is then bad for memory *and* CPU usage.
This PR removes the buffering and `CID` recalculating for a nice speedup when adding large datasets.
In my (non-representative, may need all the other unfinished async/iterator stuff) testing, importing folder of 4MB files totalling about 5GB files with content from `/dev/urandom` into a fresh repo with a daemon running in the background is now:
```
go-ipfs
real 3m43.741s
user 0m31.955s
sys 0m31.959s
```
```
js-ipfs
real 3m40.725s
user 0m7.352s
sys 0m4.489s
```
Which is nice.
fixes#2310
0 commit comments