Included laion400m image embeddings dataset generator and links #44
+135
−1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
filipecosta90
force-pushed
the
laion-400m-img-emb
branch
from
e5db863 to
4bb8a8a
Compare
mpozniak95
reviewed
Oct 20, 2023
| scikit-learn | ||
| jinja2==2.10.1 | ||
| pandas==1.1.5 | ||
| datasets==2.14.5 |
There was a problem hiding this comment.
There are 2 packages missing in the requirements.txt files:
- click
- wget
mpozniak95
reviewed
Oct 20, 2023
|
|
||
|
|
||
| def calc(bf, test, neighbors, distances, count): | ||
| Parallel(n_jobs=multiprocessing.cpu_count(), require="sharedmem")( |
There was a problem hiding this comment.
While creating hdf5 files I encountered an issue:
train size: 100000 * 512
test size: 10000 * 512
0/10000...
BLAS : Program is Terminated. Because you tried to allocate too many memory regions.
BLAS : Program is Terminated. Because you tried to allocate too many memory regions.
BLAS : Program is Terminated. Because you tried to allocate too many memory regions.
…
Segmentation fault (core dumped)
The system that I was using is Ubuntu 22.04 with 144 cpu_count and python3.8.10. I was able to workaround this error with hardcoding the n_jobs to 64.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This dataset is based on LAION-400-MILLION OPEN DATASET's dataset of LAION-400-MILLION OPEN DATASET and it
We use the clip embedings as input to produce the required hdf5 files.
The clip embeddings are stored in NPY files. Each NPY file stores 1M samples, and uses around 1GB disk space. There are a total of 400 such files.
By using the create_laion_ds.py we can specify the train an test size, up to 400M vectors.
By default, it produces a 1M train size and a 10K test size.