dart:io File readAsBytes() doesn't handle large files

[sortie]

dart:io's File class doesn't handle large files (files at least 2 GIB) properly in readAsBytes() and (readAsBytesSync() has related if different problems). Instead it attempts to return an invalid array of negative size (-4096).

For instance, make a big file:

truncate -s 4387733994 big

Then this program shows the problem:

import 'dart:io';

main() async {
  print(await new File("big").readAsBytes());
}

Which fails like this:

tcmalloc: large alloc 4387741696 bytes == 0x55ded9554000 @  0x55ded653caa7 0x55ded6544819 0x55ded652ad41 0x55ded67ba171 0x55ded67b713a 0x55ded67b78cc 0x55ded6904e3d 0x55ded6904cd4 0x55ded6860019 0x7ff96cfa2494
../../runtime/vm/object.cc: 22026: error: Fatal error in ExternalTypedData::New: invalid len -4096

Dumping native stack trace for thread 7e72
  [0x000055ded6863a2f] dart::Profiler::DumpStackTrace(void*)
  [0x000055ded6863a2f] dart::Profiler::DumpStackTrace(void*)
  [0x000055ded6a993b2] dart::Assert::Fail(char const*, ...)
  [0x000055ded68400c0] dart::ExternalTypedData::ToCString() const
  [0x000055ded687e4c6] dart::ExternalTypedData::ReadFrom(dart::SnapshotReader*, long, long, dart::Snapshot::Kind, bool)
  [0x000055ded68e3943] dart::SnapshotReader::ReadObjectImpl(long, bool, long, long)
  [0x000055ded68e5a6c] dart::SnapshotReader::ArrayReadFrom(long, dart::Array const&, long, long)
  [0x000055ded687cee0] dart::Array::ReadFrom(dart::SnapshotReader*, long, long, dart::Snapshot::Kind, bool)
  [0x000055ded68e3e0c] dart::SnapshotReader::ReadObjectImpl(long, bool, long, long)
  [0x000055ded68e26f7] dart::SnapshotReader::ReadObject()
  [0x000055ded6785139] dart::IsolateMessageHandler::HandleMessage(dart::Message*)
  [0x000055ded67b713a] dart::MessageHandler::HandleMessages(dart::MonitorLocker*, bool, bool)
  [0x000055ded67b78cc] dart::MessageHandler::TaskCallback()
  [0x000055ded6904e3d] dart::ThreadPool::Worker::Loop()
  [0x000055ded6904cd4] dart::ThreadPool::Worker::Main(unsigned long)
  [0x000055ded6860019] Unknown symbol
-- End of DumpStackTrace

[eikob]

Any news on this?

Here's a quick workaround to circumvent the crash (OS Error: Invalid argument, errno = 22):

  final file = File('somefile');
  final length = file.lengthSync();
  final bytes = Uint8List(length);
  final raFile = file.openSync();
  int pos = 0;
  const chunkSize = 1000000000;
  while (pos < length) {
    raFile.readIntoSync(bytes, pos, (pos+chunkSize).clamp(0, length));
    pos += chunkSize;
  }
  raFile.closeSync();

Used to replace readSync(), but it would probably work with the async methods as well.

[2S-Software]

Dart can't read large files (over 500 MB), which is very cumbersome for desktop applications.
Is there anyone who can help and fix this?

[brianquinlan]

This is a duplicate of #51071, which has a fix in progress.