Conversion of RAW file to mzML fails when "Failed to get reaction when parsing precursor"

[mroemer]

When converting a Thermo RAW file to indexed mzML with the Docker containers 1.4.0 and 1.4.1 on Ubuntu 18.04, the conversion fails (see output below).
With version 1.3.4, the conversion works.

With 1.4.0 and 1.4.1 I get a valid mzML file that ends with the scan that produced the warning (819 in this case).

It seems like the stream is closed when the warning is reported and then it cannot write to the closed stream which causes the "Stream has been closed" error.

I guess the offending commit was this one: 7b7854d

2022-11-15 07:29:35 INFO Started parsing /data/rawfile.raw
2022-11-15 07:29:38 INFO Processing 5936 MS scans
10% 2022-11-15 07:29:40 WARN Failed to get reaction when parsing precursor 819
2022-11-15 07:29:40 ERROR An unexpected error occured while parsing file:/data/rawfile.raw
2022-11-15 07:29:40 ERROR System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'Stream has been closed'.
  at System.IO.FileStream.Write (System.Byte[] array, System.Int32 offset, System.Int32 count) [0x0000d] in <7cd4deb8302a478d9d393f24754841da>:0
  at System.Security.Cryptography.CryptoStream.FlushFinalBlock () [0x00070] in <7cd4deb8302a478d9d393f24754841da>:0
  at System.Security.Cryptography.CryptoStream.Dispose (System.Boolean disposing) [0x0000b] in <7cd4deb8302a478d9d393f24754841da>:0
  at System.IO.Stream.Close () [0x00000] in <7cd4deb8302a478d9d393f24754841da>:0
  at ThermoRawFileParser.Writer.MzMlSpectrumWriter.Write (ThermoFisher.CommonCore.Data.Interfaces.IRawDataPlus rawFile, System.Int32 firstScanNumber, System.Int32 lastScanNumber) [0x00faa] in <7a789c83cb9d4db2bd14ef83a0b964a9>:0
  at ThermoRawFileParser.RawFileParser.ProcessFile (ThermoRawFileParser.ParseInput parseInput) [0x000e9] in <7a789c83cb9d4db2bd14ef83a0b964a9>:0
  at ThermoRawFileParser.RawFileParser.TryProcessFile (ThermoRawFileParser.ParseInput parseInput) [0x00000] in <7a789c83cb9d4db2bd14ef83a0b964a9>:0

[caetera]

Hi @mroemer,

thank you for reporting this error.
The warning itself should not affect the processing, i.e. should not lead to closing the file stream. I suspect that missing precursor might lead to some exception in the downstream processing that is not properly handled.
I do not have any files that generate the same behavior. Will it be possible to share the raw file that produces the error so I can debug it further?

[caetera]

If necessary, I can provide you with the link to a resource allowing sharing large files.

[mroemer]

I'm sorry, but I cannot share the raw file that produces the error.

[caetera]

Ok. I will try to debug it as it is.
Meanwhile, I figured out that the Stream has been closed error is due to the index-writing routine, namely, TRFP still wants to write the mzML index even if the processing has been aborted. That, of course, has to be fixed, but it also means that the reported exception "shadows" the real cause of the problem. Could you, please, run the same processing with plain mzML format (-f mzml or -f 1) that might help in figuring out what was the primary cause.

[mroemer]

Ok. I will try to debug it as it is. Meanwhile, I figured out that the Stream has been closed error is due to the index-writing routine, namely, TRFP still wants to write the mzML index even if the processing has been aborted. That, of course, has to be fixed, but it also means that the reported exception "shadows" the real cause of the problem. Could you, please, run the same processing with plain mzML format (-f mzml or -f 1) that might help in figuring out what was the primary cause.

I ran it with -f 1, same result:

$ docker run -i -t -v $DIR:/data quay.io/biocontainers/thermorawfileparser:1.4.1--ha8f3691_0 ThermoRawFileParser.sh -i /data/rawfile.raw -o /data/ -f 1
2022-11-16 16:49:11 INFO Started parsing /data/rawfile.raw
2022-11-16 16:49:14 INFO Processing 5936 MS scans
10% 2022-11-16 16:49:17 WARN Failed to get reaction when parsing precursor 819
2022-11-16 16:49:17 ERROR An unexpected error occured (see below)
2022-11-16 16:49:17 ERROR System.NullReferenceException: Object reference not set to an instance of an object
  at ThermoRawFileParser.Writer.SpectrumWriter.CalculateSelectedIonMz (ThermoFisher.CommonCore.Data.Business.IReaction reaction, System.Nullable`1[T] monoisotopicMz, System.Nullable`1[T] isolationWidth) [0x00000] in <d8889af6238044b48ab06f99e7b486ca>:0
  at ThermoRawFileParser.Writer.MzMlSpectrumWriter.ConstructPrecursorList (System.Int32 precursorScanNumber, ThermoFisher.CommonCore.Data.Interfaces.IScanEventBase scanEvent, System.Nullable`1[T] charge, System.Nullable`1[T] monoisotopicMz, System.Nullable`1[T] isolationWidth, System.Collections.Generic.List`1[T] SPSMasses, System.Int32& reactionCount) [0x000d8] in <d8889af6238044b48ab06f99e7b486ca>:0
  at ThermoRawFileParser.Writer.MzMlSpectrumWriter.ConstructMSSpectrum (System.Int32 scanNumber) [0x00456] in <d8889af6238044b48ab06f99e7b486ca>:0
  at ThermoRawFileParser.Writer.MzMlSpectrumWriter.Write (ThermoFisher.CommonCore.Data.Interfaces.IRawDataPlus rawFile, System.Int32 firstScanNumber, System.Int32 lastScanNumber) [0x008f1] in <d8889af6238044b48ab06f99e7b486ca>:0
  at ThermoRawFileParser.RawFileParser.ProcessFile (ThermoRawFileParser.ParseInput parseInput) [0x000e7] in <d8889af6238044b48ab06f99e7b486ca>:0
  at ThermoRawFileParser.RawFileParser.TryProcessFile (ThermoRawFileParser.ParseInput parseInput) [0x00000] in <d8889af6238044b48ab06f99e7b486ca>:0
2022-11-16 16:49:17 INFO Processing completed 1 errors, 1 warnings

[caetera]

I think I could fix this problem (generated a workaround to be more exact).
I would prefer not to do the release until it is clear that the fix is working, thus, there won't be a docker container. I have generated an updated executable - it is available from https://syddanskuni-my.sharepoint.com/:u:/g/personal/vgor_bmb_sdu_dk/EZXUNMZPFPlIiv1P05yu6acB3wjV0xqYWie9cEDNFG8mww
You can either run it natively (having mono-complete or mono-devel packages should be enough to run it on Ubuntu) or map the contents to the container. I hope that will be sufficient.

It is still not clear why the Failed to get reaction ... is happening. I am afraid that something could have been broken when the support of all MSn levels has been added (version 1.4.0).
The new version generates a warning for every such event that contains some extra debug info. I would appreciate it if you can forward the output here. Thank you.

The issue with the attempt to write to the closed stream should be fixed as well.

[mroemer]

I used the provided 1.4.2_beta and copied the contents to /usr/local/bin in the Docker container for 1.4.1.

The file is now converted and looks okay to me. This is the output:

$ThermoRawFileParser.sh -i /data/rawfile.raw -o /data/ -f 2
2022-11-17 09:38:40 INFO Started parsing /data/rawfile.raw
2022-11-17 09:38:43 INFO Processing 5936 MS scans
10% 2022-11-17 09:38:45 WARN Failed creating precursor list for scan# 820; Cannot get reaction at index 1 from FTMS + p NSI d Full ms2 507.3024@cid35.00 [134.0000-1350.0000]; precursor information for this and dependent scans will be empty
... snip many warnings ...
100%
2022-11-17 09:39:04 INFO Finished parsing /data/rawfile.raw
2022-11-17 09:39:04 INFO Processing completed 0 errors, 1320 warnings

Just FYI, previously the warning was :

WARN Failed to get reaction when parsing precursor 819

Now it is:

WARN Failed creating precursor list for scan# 820; Cannot get reaction at index 1 from FTMS + p NSI d Full ms2 507.3024@cid35.00 [134.0000-1350.0000]; precursor information for this and dependent scans will be empty

"precursor 819" is no longer part of warning, not sure if that was by design.

[caetera]

Thank you.
Yes, the change in the warning is by design - the previous message referenced the parent scan, and the current one - the fragmentation scan (the latter is more useful, IMO).

There indeed seems to be a problem with finding precursors in your file - i.e. it should get reaction 0 instead of 1.
Unfortunately, I cannot reproduce this error on any RAW file at hand. Does this warning (1320) happen to all MS2 scans in your file (the following command can be used to count MS2 scans in the mzML file without any special tools grep -o 'MSn spectrum' file.mzML | wc -l)? Is it the only file that behaves that way? Can you disclose the instrument you used (so I can check some raw files from the same instrument in PRIDE)? Do you use some extra activation, such as in-source fragmentation? Would it be possible for you to send me the dump of scan headers of your file?

You can use this tool - https://bitbucket.org/caetera/scanheadsman/src/master/ All the necessary dependencies should be present in the TRFP docker container, thus you can just copy it somewhere and run as mono ScanHeadsman.exe {path to raw file} It will generate a CSV file (in the same folder as raw file) with all scan headers (only headers, such as scan number, filter string, ms level, and other technical information, but not actual scan data - m/z and intensities).

[mroemer]

I have gotten permission to share the file. It's 312M, how can I send it to you?

[caetera]

You can use FileSender link https://filesender.deic.dk/?s=upload&vid=61fe1192-a050-6aac-e76a-9de9432058a0

[mroemer]

I have sent the file via the link.

[caetera]

Thank you. Received.
I have checked scan 820 and I think I know what the problem is. This scan reports that its parent is scan 819 which is already an MS2 scan, thus, TRFP expects that 820 is an MS3 scan and tries to parse it accordingly (i.e. looking for the second precursor and so on). I believe, it is because of the decision tree used, but, still, it is strange that the instrument reports 819 as the parent scan.
However, I don't have the final decision on how to deal with such cases, if it is a faulty raw file or instrument, then the way it is done now is, likely good enough, but if all decision tree methods produce these types of scans we need to add proper support for such cases, especially, since we use parent scan information in some other places. I plan to inspect other raw files with decision trees to get deeper into this.

[caetera]

I can confirm, that same pattern (reporting "deciding" scan as a the parent one) is observed on other tribrid systems and, thus, a proper fix has to be implemented.