20221013 developer call notes

[tomwhite]

20221013

Pre-notes

• Notes from previous meeting
• Ben’s work on tskit/tsinfer and sgkit

PRs

Issues

• sgkit#893 Optional genomic-scale for popgen stats
• sgkit#924 VCF export
• sgkit#926 Output plink
• sgkit#925 maximise lossless compression of vcf_to_zarr

Notes

Attendees

1. Tom White
2. Jerome Kelleher
3. Ben Jeffery
4. Rafał Wojdyła

Discussion

• Ben’s work on tskit/tsinfer and sgkit:
  • JK: BJ is going to work on sgkit/tskit for next 6 months. Data ingest. MalariaGEN later.
  • JK: tsinfer has an API for importing VCF.
  • JK: use sgkit zarr format, not tsinfer one
  • JK: Patchy docs - so fill those gaps.
  • TW: Anything else after that?
  • JK: There are going to be bugs to fix when importing viral and bacterial VCFs, since they have lots of non-standard things in them.
• sgkit#893 Optional genomic-scale for popgen stats
  • TW: Came from Genome-wide average Fst #888 (Alexis Simon) wanting to compute genome-wide average Fst.
  • TW: Question is - is that a single window, or just a one-off special call that doesn’t mention windows? Doesn’t sound like there’s no reason not to call it a window.
  • JK: it’s like the window by variant case, with the number of variants stretching across the whole genome. A bit unusual to have windows cross contig boundaries.
• sgkit#924 VCF export
  • JK: Issue came up when Ben Elsworth (Our Future Health) started asking about using sgkit. A requirement for them is to export VCF. A big gap - not realistic to just keep in sgkit. Need VCF and plink.
  • JK: want to aim for 100MB/s, uncompressed VCF. Fast enough so that htslib parses it more slowly than we produce it.
  • TW: hard bit is formatting ints and floats
  • JK: option 1, we could update cyvcf2 with an API for writing out
  • TW: a possibility, but don’t have an idea of how hard this would be. Choice between cython or numba.
  • TW: write BCF? Since format fields are not interleaved. But a lot harder than writing VCF text, so will try doing that first, using numba to speed things up.
  • TW: hopefully plink will just work
  • JK: dealing with suite of extra files will be the hard bit.
• sgkit#925 maximise lossless compression of vcf_to_zarr
  • TW: Ben Elsworth found that DS field took up a lot of space - caused zarr to be bigger than compressed VCF. After adding some filters to Zarr I got the size down considerably - a lot smaller than compressed VCF, and 25% bigger than genozip.
  • JK: why is genozip so small?
  • TW: probably using something like PBWT. Also bzip2 (which I also tried). Trade off between storage cost and speed for analysis. We want to be able to do analysis on the data. So I stopped going further.
  • TW: we should change default, eg for POS. Using delta encoding is a no brainer.
  • JK: we will evolve a collection of default compressors/filters for VCF fields. That’s OK.
  • JK: storing limited number of decimal places is accepted for floats in format fields. So could do by default. Good for new users who would otherwise think sgkit compression is not very good.
  • JK: quantize on phred scores? E.g. DP is usual < 100. GL, GP, GQ, HQ, etc. Worth looking at?
  • RW: sgkit#80 related?
  • TW: yes, still valid and would be a good small project for someone.