Compressed data dumps for works

[several27]

Hi, I was trying to get all of the works using the works API endpoint and paging using the cursor. However, I noticed it would take weeks with the current response time for rows parameter set to 1000. Is there a way to speed it up, or maybe do you provide data dumps, for cases when someone needs all of the works metadata?

Thank you :)

[moqri]

Currently it takes 70 hours to get all the 90 million records with a good internet connection but I totally agree some kind of data dump would help both the users and the server

[danielopitz]

I would be interested in a full dump too, even if it would be updated once a month. Once a week would be just perfect, but there's no reason to get greedy :)

You could compress it greatly and / or host somewhere else to reduce the bandwith.

Reasoning for updates: Sometimes you add new fields or fix something in the whole dataset and it needs to be re-indexed, changing the only reliable date field for updates and resulting in a full download.

[several27]

Oke, so it took me about 90 hours to get all the "works". That's about 170GB of pure JSON data. I used a cheap AWS instance located in the East of USA. If anyone's interested: very short gist to get it efficiently, also can give "requester pays" type access to S3 with the zipped dataset, but can't promise any updates at that moment.

[moqri]

Please add some wait time between failed attempts to not overload the server:
https://github.com/CrossRef/rest-api-doc#rate-limits

[moqri]

In regards to the current API Outage/Slowdown, I think providing a dump would help your API traffic too.

[renaud]

@several27 i'm interested in your dump in S3, could you give me access to it? thanks, Renaud at apache dot org

[kjw]

This is a feature we plan on providing, but I don't want to do "one off" dumps - if we are to do this I want an obvious and workable way for people to keep their copies up-to-date.

[renaud]

@kjw agreed. One way that e.g. PubMed solves it is by providing daily update files that can be downloaded separately. See this for more info: ftp://ftp.ncbi.nlm.nih.gov/pubmed/updatefiles/README.txt

Another more pragmatic way would be for CrossRef to release dumps for each year, and in the meantime consumers would get fresh data from the API.

[several27]

@renaud It's a little bit outdated (1st sept), but for everyone interested in full works data dump, here it is: s3://researchably-shared/crossref/2017-09-01.zip. At least until crossref guys do it, feel free to use it. It's only 30GB compressed, and it's a requester-pays type file, so the download cost is paid by the person downloading. (it's the same strategy to what arxiv does - https://arxiv.org/help/bulk_data_s3). As arXiv mentions, the easiest way to download is to use s3cmd, e.g. s3cmd get --requester-pays s3://researchably-shared/crossref/2017-09-01.zip.

Later in the month we'll be probably updating it, if crossref still doesn't have their data dumps, we can share the newer version as well (if they are fine with that) :)

[renaud]

thanks a lot @several27 !

[moqri]

Monthly dumps would be great!

[mjmehta15]

@several27: I downloaded works data dump from s3, which has 68,907,000 records, whereas the crossref dashboard says it has 91,869,582 records. I understand that this zip file on s3, has data only till Sept. 1st but after reducing number for Sep. & Oct. this dump is missing approximately 22,577,780 records. Can you please check and tell how many records you have in your actual 170GB data or is something wrong at my end.

[several27]

@mjmehta15 Nice catch! I have downloaded this file myself and you're right there's something wrong. Especially that the last line is:

{"status":"error","message-type":"exception","message-version":"1.0.0","message":{"name":"class java.lang.IllegalArgumentException","description":"java.lang.IllegalArgumentException: Vector arg to map conj must be a pair","message":"Vector arg to map conj must be a pair","stack":["clojure.lang.ATransientMap.conj(ATransientMap.java:37)","clojure.lang.ATransientMap.conj(ATransientMap.java:17)","clojure.core$conj_BANG_.invokeStatic(core.clj:3257)","clojure.core$conj_BANG_.invoke(core.clj:3249)","clojure.core.protocols$naive_seq_reduce.invokeStatic(protocols.clj:62)","clojure.core.protocols$interface_or_naive_red....

I doubt it was a compression error, maybe something on crossref side? Anyway, I'm trying to get the rest now, as the last cursor still seems to be working. Unfortunately, the API seems to be very slow now (15s per request) so not sure, how much time will it take.