GZIP header warnings

[bpinsard]

Testing the new validator, I do get these new warnings on BIDS generated with heudiconv/dcm2niix:

	[WARNING] GZIP_HEADER_MTIME The gzip header contains a non-zero timestamp.
This may leak sensitive information or indicate a non-reproducible conversion process.

		/sub-01/ses-001/anat/sub-01_ses-001_rec-norm_T2w.nii.gz
		/sub-01/ses-001/anat/sub-01_ses-001_T2w.nii.gz

		17 more files with the same issue

	Please visit https://neurostars.org/search?q=GZIP_HEADER_MTIME for existing conversations about this issue.

	[WARNING] GZIP_HEADER_FILENAME The gzip header contains a non-empty filename.
This may leak sensitive information or indicate a non-reproducible conversion process.

		/sub-01/ses-001/anat/sub-01_ses-001_rec-norm_T2w.nii.gz
		/sub-01/ses-001/anat/sub-01_ses-001_T2w.nii.gz

		17 more files with the same issue

	Please visit https://neurostars.org/search?q=GZIP_HEADER_FILENAME for existing conversations about this issue.

I think the "sensitive information leak" warning might make some people freak-out.
I guess that such header info are not necessarily sensitive (time of compression? path to uncompressed file), but could be if BIDS conversion is automated after acquisition.

I also see the value of being able to get identical files when running dcm2niix at different times.
Are there GZIP /pigz options to not ship such info in the gzip header? And a way for dcm2niix to use that option?
@neurolabusc ?
If so, we could maybe set heudiconv (@yarikoptic ) /dcm2bids (@arnaudbore) ... to use such options ?

[effigies]

gzip -n

[bpinsard]

So the specs requires the timestamp https://github.com/bids-standard/bids-specification/blob/514f28722bd6d33b24c6814b9c67e92def91a522/src/schema/meta/context.yaml#L295, and the value to be 0 https://github.com/bids-standard/bids-specification/blob/514f28722bd6d33b24c6814b9c67e92def91a522/src/schema/rules/checks/privacy.yaml#L13

pigz should be called with --no-time and --no-name, maybe if dcm2niix could set these options when -ba is used (default when -b is used), or maybe allow passing custom gzip flags to allow adjusting for the different gzip tools in dcm2niix.

[neurolabusc]

dcm2niix has never stored the filename. The miniz internal compressor stores neither file name or conversion date. I have added a commit to the development branch so pigz is called with --no-time. A curious aspect of pigz is that my variants use the alias -m while others claim to use -T. Therefore, I use the full argument name.

@bpinsard you also open an issue for dcm2niix, to provide an audit trail. Close both issues if my commit resolves your issue.

Here is a little script to see whether a gz file stores a file name and date:

import gzip
import time

with open("x.nii.gz", "rb") as f:
    # Verify GZip magic number
    magic = f.read(2)
    if magic != b"\x1f\x8b":
        raise ValueError("Not a valid GZip file.")
    # Compression method (not used here)
    compression_method = f.read(1)
    # Flags (to check for FNAME flag)
    flags = f.read(1)[0]
    # Timestamp
    timestamp_bytes = f.read(4)
    timestamp = int.from_bytes(timestamp_bytes, byteorder="little")
    # Skip extra header fields
    f.read(2)  # Extra flags and OS fields
    # Check for original filename
    original_filename = None
    if flags & 0x08:  # FNAME flag is set
        filename_bytes = b""
        while True:
            byte = f.read(1)
            if byte == b"\x00":
                break
            filename_bytes += byte
        original_filename = filename_bytes.decode("utf-8")
    print("Timestamp:", timestamp)
    if timestamp != 0:
        print("Date:", time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(timestamp)))
    print(f"Original filename: {original_filename}")

[bpinsard]

Thanks @neurolabusc following up on dcm2niix repo.

[bpinsard]

Wait, I was completely off, the files had in fact been edited with nibabel (defacing).
Very sorry, my bad!
However, you are very right about pigz changing the flags across versions, something to be careful about on environments with old OS/pigz.

There is DeterministicGzipFile since nipy/nibabel#1024 in nibabel to deal with that, so I don't understand why it didn't work in my case. Maybe the defacing ran with an old nibabel version.