Reducing mgf file size

[antortjim]

Hi Niels

I was wondering why are the output mgf files so big compared to the raw files. If I convert the raw file from #4 , which is 2.8GB big, I get an mgf output file with 13GB. I have checked that most of the rows correspond to peaks with intensity equal to 0.0000000000, and removing them with grep
grep -v 0.0000000000 PD7505-GDTHP1-A_C2.mgf > PD7505-GDTHP1-A_C2_non_zero.mgf
leaves a 3.2 GB file.
There are still many peaks with very low intensities like 0.0000000277 that I am guessing really are noise and thus make the file unnecessarily big. I tried running msconvert on it

msconvert PD7505-GDTHP1-A_C2_non_zero.mgf --filter "peakPicking true [2,3] zeroSamples removeExtra" --mgf -o msconvert_output

but the file size remains the same.

If I further trim the mgf file to keep intensities > 0.1 with awk by printing lines starting with a capital letter (to keep spectra headers) or where the second field is > 0.1, I get a 1.7 GB file.

awk '{ if ($2 > 0.1 || $1 ~ /^[A-Z]/) {print} }' msconvert_output/PD7505-GDTHP1-A_C2_non_zero.mgf > trimmed.mgf

While I achieved a lot of file size reduction, this is still much bigger than what one gets by running the msconvertGUI on Windows with the same raw file. If I run the program with filters:

Filter	Parameter
peakPicking	vendor msLevel=1-
zeroSamples	removeExtra 1-

I get a 297 MB file.

From ThermoRawFileParser's README.md

It takes a thermo RAW file as input and outputs a metadata file and the MS2 spectra (centroided) in MGF format.

Could you please provide more information about how the tool performs the centroiding of the spectra and how to exactly emulate the output one would get by running msconvert on Windows? Ideally, mgf files with sizes similar to those produced by standard msconvert calls should be easy to produce, either using ThermoRawFileParser alone or in combination with msconvert filtering.

Thank you very much for your help!

Cheers

Antonio

[nielshulstaert]

Hi Antonio,

Could you provide the link to the specified RAW file? The one from the previous issue seems a lot smaller. I'm comparing my output to msconvert and indeed I'll have to add some argument options.

Right now I only consider MS2 scans:

• if the scan has a centroid stream, take those values
• if the scan has no centroid stream, take the segmented scan (low res and profile scan data was mentioned in a comment I found in the thermo library example code). If I don't include these scans, the output of the example RAW file from your previous issue (20070920_CL_Orbi2_XIC_Hela60_Ecoli10_Offgel_green_Frac4_02.RAW) is empty.

I'll test the code with the RAW file your'e talking about if you send me the link.

Best regards,

Niels

[antortjim]

Hi Niels

Thanks for your answer. I am so sorry I directed to the file from #4 when they are not actually the same. I got confused.

What is a centroid stream?
If I repeat my postprocessing with the file from #4 I get a 105MB file with MSconvert using the filters above and 104MB with ThermoRawFileParser. They are thus really close in file size. They both have 13352 spectra (i.e grep finds BEGIN IONS 13352 times)

I can send the link to a zip file containing the raw file I am talking about and the target mgf, but I need to do that through e-mail. Is it ok?

Best regards,
Antonio

[nielshulstaert]

Hi Antonio,

I'm not an expert in thermo RAW files. This is an excerpt from the Thermo library documentation:

Simple instruments (such as a single quad MS) have a scan format which returns one data type, either "profile" or "centroid".
The scans type code (scan event) indicates which of these formats is used.
Profiles represent a set of raw samples from the instrument, with no analysis. These are generally shown as a continuous line of a plot.
Centroids are internally greeted by running a "center of mass" algorithm on the profile (within the instrument firmware).
Some instruments will log only one or the other of these data types, while others can store both.
Instruments can scan continuously across a mass range (called a full scan) or over smaller SIM or SRM windows. These windows are referred to as "segments".
So: An MS scan may have one or more segments (of the mass range), with a full scan having only one segment.
Segments may not overlap.

Right now I only consider centroid MS2 data, and for the scans that don't have centroid stream I take the segmented values. I'll ask an expert about the filtering of low intensity peaks.

Best regards,

Niels

[chrishuges]

I might be able to help a bit here. The file from the example you are trying to process (#4) is an LTQ Orbitrap file. There are 13352 MS2 scans in the file that were acquired in the ion trap (LTQ) in centroid mode. If I process it with RawQuant I get an 87MB file with none of the zero intensity values you mention.

In regards to the profile and centroid stream, the raw file will contain data dependent on whatever you have specified in your method (e.g. profile or centroid). However, for Orbitrap scans we have found that the raw file will typically contain both a profile and centroid scan regardless of what format it was acquired in (e.g. both centroid and profile data will be present, even if profile is specified in the method). For ion trap data, it will only be whatever you specify in the method. So, if you have acquired your data in ion trap profile mode (which I suspect is the case here) there is no centroid stream to be reported by RawFileParser and there is going to be a lot of other data reported (and possibly many zero values) as it just spits out the profile data. I don't think the parser tool here has a built-in centroiding function. When you push it through msconvert with peakPicking you are centroiding the data, which is why you see the size reduction compared to RawFileParser.

Cheers,
Chris

[nielshulstaert]

Hi Antonio, Chris,

Thanks again for your input, it's very useful to me. Indeed the library doesn't do any centroiding on profile data, only the centroiding done by the instrument and provided by the Thermo library. I'm playing around with msconvert to see if their peak picking makes a difference in the file size. It's not my intention to remake msconvert though ;)

I've encountered 3 types of RAW files during testing

• The ''normal" use case: MS2 scans are in centroid mode and with a centroid stream from the Thermo library.
• The raw file from issue 4 (20070920_CL_Orbi2_XIC_Hela60_Ecoli10_Offgel_green_Frac4_02.RAW): MS2 scans are in centroid mode but without a centroid stream from the Thermo library. I look for the segmented data for each scan in that case. The MGF file size is acceptable.
• And the one that Antonio has send me: MS2 scans are in profile mode so I assume a centroid stream is by default not available and writing the segmented data to MGF results in a huge file. It's basically converting the binary raw file to MGF ;)

From the Thermo documentation:

Segments are useful for high resolution SIM or SRM data, as it is possible to see the peak groups (centroid) or profiles within each mass window, using the common core plotting tools.
An application could also choose to do peak integration based on "the data in a particular segment".

So this is what I can do:

• Add a filter flag for filtering out low intensity peaks with a value provided by the user. If no value is given, no filtering occurs. I think I would implement a filter that removes peaks that are lower than x percent of the highest peak in the scan. Setting a cutoff across all scans doesn't make much sense IMHO.
• Add a flag to include MS2 profile scans or not. Or I just don't include these and output a message that no centroid MS2 scans were found. I had a quick chat with a masspec operator and she told me that the MS2 profile mode can be used for MS2 level quant so trying to centroid that data is probably defeating the purpose of the profile.

Thoughts and suggestions are more than welcome.

Best regards,

Niels

[antortjim]

Hi Niels and Chris

Thank you for taking the time to look into this. Very appreciated!
I find really frustrating that dealing with RAW files is so difficult in Linux. Taking them to an open format, may it be MGF or mzML, is a big bottleneck.

I think a flag for filtering peaks lower than x percent of the highest peak in the scan would be a nice feature. But I think including MS2 data is essential right? You need it not just for MS2 level quantification, but also for the peptide-to-spectrum matching if I am correct. Therefore there would be no way we can omit this.

In any case, I think the way to go for now is setting an msconvert command line that takes files like mine in mgf format and produces another simplified mgf. If possible, the output should be similar to what one would get when feeding the RAW file from the beginning (the reason why we don't feed the RAW being of course that it does not work in Linux, and that's why we need ThermoRawFileParser).

The one I came up with does not really work though, as explained in the first comment.
msconvert redundant.mgf --filter "peakPicking true [2,3] zeroSamples removeExtra" --mgf -o msconvert_output

This all sounds like something msconvert should take care of themselves. Maybe I should open an issue in their repo too? 👼

Thanks once again!
Best regards

Antonio

[chrishuges]

Hi Antonio,

I agree it is an important issue to get these types of tools working on Linux, but I think there is some confusion here. Both ThermoRawFileParser and RawQuant do various processing tasks and MGF conversion of raw files on Windows/Mac/Linux without issue. The problem being encountered with both our tool and ThermoRawFileParser is that your MS2 spectra are acquired in an ion trap and are in 'profile' format. In this situation, both of these tools will spit out all of the profile data (that contains many zeroes due to the nature of profile data). The reason you are getting smaller files with msconvert is because you have enabled peakPicking...this will centroid the data and reduce file size significantly. I imagine if you processed your file through msconvert with no filter parameters specified the result would look the same as it does for ThermoRawFileParser.

The problem here is not a file conversion issue, it is just that neither ThermoRawFileParser or RawQuant have centroiding algorithms built into their code. For Orbitrap spectra this doesn't matter because centroid and profile data are stored in the raw file by default, but for ion trap spectra only the specified type is stored.

I can only speak for our tool, but we have discussed adding a centroiding algorithm for ion trap data. However, it is near the bottom of our priority list because this is a very uncommon data type (most ion trap data is acquired in centroid mode).

Re: your command above - there is no need to filter low abundance ions if you have enabled peakPicking. Centroiding will eliminate a lot of the redundant data and noise in your file just due to the nature of the way it works.

Cheers,
Chris

[antortjim]

Hi Chris

Thanks for your comment!
Then if I got it right, the way to go in Linux would be to use ThermoRawFileParser / RawQuant to get an mgf file in Linux, and then use the peakPicking algorithm from msconvert using as input the mgf file. This would skip the impossibility of processing RAW files in Linux while still performing centroiding. Am I correct?
This would be the command line for performing the last task
msconvert redundant.mgf --filter "peakPicking true 1-" --mgf -o msconvert_output

1- meaning that all MS spectra, starting from level 1 i.e all the spectra registered in the first, second, etc MS analyzer, will be filtered using peakPicing.

Thanks for your explanation again!

Cheers,
Antonio

[chrishuges]

Hi Antonio,

It depends what your analysis goal is. I honestly cannot speak to how well converting to an MGF and then processing that MGF with a different program will work. Also, I believe ThermoRawFileParser (and RawQuant) will only insert MS2 spectra into an MGF file as these are typically only used for database searching. I am not sure if msconvert is going to be able to decipher the 'order' in these MGF data (I imagine not).

I would suggest just converting the files to MGF directly in msconvert on Windows for the time being (with peakPicking 2-) as this requires the least data manipulation and chance for error. If you really want Linux compatibility, I would consider acquiring your ion trap data in centroid mode moving forward as this will make this transition a bit easier.

Note - I think this issue can probably be closed. Sorry for hijacking!

Cheers,
Chris

[antortjim]

Hi Chris

Thanks for your answer. You are probably right, it might be too messy to do all this program patching just for the sake of doing it in Linux. It's just not possible yet in a clearly systematic and defined way. But hopefully it will come soon 😄

Thanks for all your help Chris and Niels!

Best regards,
Antonio