Record not followed by newline (conversion error)

[mw0000]

Hi, how to deal with such an error? I'm trying to convert a real old ARCs to use in SolrWayback

mw@webarch:~/solrwayback/indexing/warcs1$ warcio recompress test2.arc.gz test2.warc.gz
    WARNING: Record not followed by newline, perhaps Content-Length is invalid
    Offset: 52006972
    Remainder: b'http://www.omega.poznet.pl:80/rekin.html 212.126.5.228 200101211835 text/html 4274\n'
Recompress Failed: test2.arc.gz could not be read as a WARC or ARC

[ikreymer]

Can you share the ARC file that is causing the error? It may be using a format that was not supported so far..

[mistydemeo]

Not sure if this is your exact problem, but I encountered a similar problem with an identical error message. In my case it turned out to be a .arc.gz that had been passed through gunzip at some point; the content length ends up being different than what warcio expects since there's a difference between block gzipped files vs uncompressed files.

For example:

1. Download https://github.com/commoncrawl/arc2warc-conversion/blob/main/test/resources/arc/crawl-002_2009_09_17_12_1253241189984_12-4827319.arc.gz
2. gunzip crawl-002_2009_09_17_12_1253241189984_12-4827319.arc.gz
3. gzip crawl-002_2009_09_17_12_1253241189984_12-4827319.arc

And warcio check on the new .gz will produce a "Record not followed by newline" error.

[wumpus]

warcio recompress exists for that reason. I don't know if it works with arcs.