Sharded BAM reader and a sample read counting pipeline #34

iliat · 2015-03-02T07:24:10Z

@wbrockman @jean-philippe-martin @dionloy - here is the first version of the sharded BAM reader. Works in the cloud with a workaround for credentials factory.

Most of the common files are under readers/bam.
The CountReads pipeline is there as an example of using the reader. It can read from the API or BAM file.
I added a GCSOptions class to capture what is needed to access GCS.
Some files added under htsjdk/samtools because I needed to deal with visibility restrictions in HTSJDK classes (not pretty but fwiw GATK code base does the same).
Also added dependency on HTSJDK in pom.xml

coveralls · 2015-03-02T07:26:24Z

Coverage decreased (-31.58%) to 0.0% when pulling d7ba30a on iliat:dev-broad into 408ed35 on googlegenomics:master.

pgrosu · 2015-03-02T14:54:51Z

Congrats Ilia! This is a huge step forward!

~p

wbrockman · 2015-03-02T15:26:19Z

src/main/java/com/google/cloud/genomics/dataflow/pipelines/CountReads.java

Typo - GATK

iliat · 2015-03-05T22:45:11Z

@dionloy @deflaux I fill fix the comment typo but LMK if there are any other comments, concerns before I merge.

iliat · 2015-03-07T19:37:14Z

@dionloy @deflaux Ok, I simplified credentials passing, using OflineAuth - no need for factory and side inputs. Will work on adding more tests a bit later.

pgrosu · 2015-03-07T19:46:23Z

Hey Ilia,

Travis is not happy :(

[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /home/travis/build/googlegenomics/dataflow-java/src/main/java/com/google/cloud/genomics/dataflow/pipelines/CountReads.java:[39,48] cannot find symbol
symbol: class WorkerCredentialFactoryWorkaround
location: package com.google.cloud.genomics.dataflow.utils
[ERROR] /home/travis/build/googlegenomics/dataflow-java/src/main/java/com/google/cloud/genomics/dataflow/pipelines/CountReads.java:[116,5] cannot find symbol
symbol: variable WorkerCredentialFactoryWorkaround
location: class com.google.cloud.genomics.dataflow.pipelines.CountReads
[ERROR] /home/travis/build/googlegenomics/dataflow-java/src/main/java/com/google/cloud/genomics/dataflow/pipelines/CountReads.java:[149,17] no suitable constructor found for ReadReader(com.google.cloud.genomics.utils.GenomicsFactory.OfflineAuth)
constructor com.google.cloud.genomics.dataflow.readers.ReadReader.ReadReader(com.google.cloud.genomics.utils.GenomicsFactory.OfflineAuth,com.google.cloud.genomics.utils.Paginator.ShardBoundary) is not applicable
(actual and formal argument lists differ in length)
constructor com.google.cloud.genomics.dataflow.readers.ReadReader.ReadReader(com.google.cloud.genomics.utils.GenomicsFactory.OfflineAuth,com.google.cloud.genomics.utils.Paginator.ShardBoundary,java.lang.String) is not applicable
(actual and formal argument lists differ in length)
[ERROR] /home/travis/build/googlegenomics/dataflow-java/src/main/java/com/google/cloud/genomics/dataflow/readers/bam/BAMShard.java:[70,11] cannot assign a value to final variable end
[ERROR] /home/travis/build/googlegenomics/dataflow-java/src/main/java/com/google/cloud/genomics/dataflow/readers/bam/BAMShard.java:[82,13] cannot assign a value to final variable end
[ERROR] /home/travis/build/googlegenomics/dataflow-java/src/main/java/com/google/cloud/genomics/dataflow/readers/bam/Sharder.java:[140,13] cannot assign a value to final variable end
[INFO] 6 errors
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 6.717 s
[INFO] Finished at: 2015-03-07T19:33:50+00:00
[INFO] Final Memory: 19M/136M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) on project google-genomics-dataflow: Compilation failure: Compilation failure:
[ERROR] /home/travis/build/googlegenomics/dataflow-java/src/main/java/com/google/cloud/genomics/dataflow/pipelines/CountReads.java:[39,48] cannot find symbol
[ERROR] symbol: class WorkerCredentialFactoryWorkaround
[ERROR] location: package com.google.cloud.genomics.dataflow.utils
[ERROR] /home/travis/build/googlegenomics/dataflow-java/src/main/java/com/google/cloud/genomics/dataflow/pipelines/CountReads.java:[116,5] cannot find symbol
[ERROR] symbol: variable WorkerCredentialFactoryWorkaround
[ERROR] location: class com.google.cloud.genomics.dataflow.pipelines.CountReads
[ERROR] /home/travis/build/googlegenomics/dataflow-java/src/main/java/com/google/cloud/genomics/dataflow/pipelines/CountReads.java:[149,17] no suitable constructor found for ReadReader(com.google.cloud.genomics.utils.GenomicsFactory.OfflineAuth)
[ERROR] constructor com.google.cloud.genomics.dataflow.readers.ReadReader.ReadReader(com.google.cloud.genomics.utils.GenomicsFactory.OfflineAuth,com.google.cloud.genomics.utils.Paginator.ShardBoundary) is not applicable
[ERROR] (actual and formal argument lists differ in length)
[ERROR] constructor com.google.cloud.genomics.dataflow.readers.ReadReader.ReadReader(com.google.cloud.genomics.utils.GenomicsFactory.OfflineAuth,com.google.cloud.genomics.utils.Paginator.ShardBoundary,java.lang.String) is not applicable
[ERROR] (actual and formal argument lists differ in length)
[ERROR] /home/travis/build/googlegenomics/dataflow-java/src/main/java/com/google/cloud/genomics/dataflow/readers/bam/BAMShard.java:[70,11] cannot assign a value to final variable end
[ERROR] /home/travis/build/googlegenomics/dataflow-java/src/main/java/com/google/cloud/genomics/dataflow/readers/bam/BAMShard.java:[82,13] cannot assign a value to final variable end
[ERROR] /home/travis/build/googlegenomics/dataflow-java/src/main/java/com/google/cloud/genomics/dataflow/readers/bam/Sharder.java:[140,13] cannot assign a value to final variable end
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
The command "eval mvn install -DskipTests=true -Dmaven.javadoc.skip=true -B -V" failed 3 times.
The command "mvn install -DskipTests=true -Dmaven.javadoc.skip=true -B -V" failed and exited with 1 during .
Your build has been stopped.

Paul

iliat · 2015-03-07T19:47:59Z

@pgrosu Yeah, I see that. Looking

pgrosu · 2015-03-07T19:50:57Z

@iliat, Cool - yeah they're minor fixes - thanks man :)

iliat · 2015-03-07T20:24:15Z

@pgrosu @deflaux @dionloy Now compiles :)

pgrosu · 2015-03-07T20:36:21Z

Awesome! Thanks Ilia :)

wbrockman · 2015-03-09T16:46:22Z

src/main/java/com/google/cloud/genomics/dataflow/pipelines/CountReads.java

Is there really no builtin operation for this? Rats?

I could not find anything here: https://cloud.google.com/dataflow/java-sdk/JavaDoc/com/google/cloud/dataflow/sdk/transforms/package-summary
The only other alternative is to use a Coders and TextIO.Write.withCoder
but unfortunately the only prepackaged coder that can help is
TextualIntegerCoder and its not suitable for Longs.

Yup, thanks for the context - I concur there's no ToString transform that I can find at present. I've checked in with Jeff G from the Dataflow team to make sure I'm not missing anything.

So why not just take this out and put it into a common directory for all the extended functionality for Dataflow until it is implemented to make the code more readable? I get the feeling it will be reused.

coveralls · 2015-03-10T18:49:10Z

Coverage decreased (-7.79%) to 23.67% when pulling 653ae81 on iliat:dev-broad into 4b57fc1 on googlegenomics:master.

lbergelson · 2015-03-11T15:04:43Z

src/main/java/com/google/cloud/genomics/dataflow/utils/GCSOptions.java

typo in storgae

wbrockman · 2015-03-11T15:16:55Z

LGTM - let's get the unit tests in a separate commit, as you say. Thanks for updating comments on the subtle question of why this code does find all the reads. Many thanks, and nice work - it's outstanding to have a full BAM reader available! Looking forward to experimenting with this further and perhaps iterating on the code design a bit once there are tests available.

pgrosu · 2015-03-11T15:23:22Z

Definitely +1 :)

Sharded BAM reader and a sample read counting pipeline

wbrockman · 2015-03-12T20:06:31Z

Hurray! Thank you!

On Thu, Mar 12, 2015 at 3:59 PM, Ilia Tulchinsky [email protected]
wrote:

Merged #34 #34.

—
Reply to this email directly or view it on GitHub
#34 (comment).

pgrosu · 2015-03-12T20:33:57Z

Things should get very interesting :)

Sharded BAM reader and a sample read counting pipeline

wbrockman reviewed Mar 2, 2015
View reviewed changes

src/main/java/com/google/cloud/genomics/dataflow/pipelines/CountReads.java Outdated

Copy link

Contributor

wbrockman Mar 2, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo - GATK

iliat force-pushed the dev-broad branch from d7ba30a to 377f927 Compare March 7, 2015 19:31

iliat force-pushed the dev-broad branch from 377f927 to 78d279a Compare March 7, 2015 19:46

iliat force-pushed the dev-broad branch 2 times, most recently from 0a661e4 to 447361c Compare March 7, 2015 20:19

wbrockman reviewed Mar 9, 2015
View reviewed changes

This was referenced Mar 9, 2015

Read a bam file into a PCollection broadinstitute/gatk#277

Closed

Count reads with lots of workers broadinstitute/gatk#278

Closed

Count reads pipeline

653ae81

iliat force-pushed the dev-broad branch from 447361c to 653ae81 Compare March 10, 2015 03:56

lbergelson reviewed Mar 11, 2015
View reviewed changes

src/main/java/com/google/cloud/genomics/dataflow/utils/GCSOptions.java

Copy link

Contributor

lbergelson Mar 11, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo in storgae

iliat added a commit that referenced this pull request Mar 12, 2015

Merge pull request #34 from iliat/dev-broad

70bc9f7

Sharded BAM reader and a sample read counting pipeline

iliat merged commit 70bc9f7 into googlegenomics:master Mar 12, 2015

jiridanek pushed a commit to jiridanek/dataflow-java that referenced this pull request Jan 18, 2016

Merge pull request googlegenomics#34 from iliat/dev-broad

e0d3e82

Sharded BAM reader and a sample read counting pipeline

Sharded BAM reader and a sample read counting pipeline #34

Sharded BAM reader and a sample read counting pipeline #34

Uh oh!

Conversation

iliat commented Mar 2, 2015

Uh oh!

coveralls commented Mar 2, 2015

Uh oh!

pgrosu commented Mar 2, 2015

Uh oh!

wbrockman Mar 2, 2015

Choose a reason for hiding this comment

Uh oh!

iliat commented Mar 5, 2015

Uh oh!

iliat commented Mar 7, 2015

Uh oh!

pgrosu commented Mar 7, 2015

Uh oh!

iliat commented Mar 7, 2015

Uh oh!

pgrosu commented Mar 7, 2015

Uh oh!

iliat commented Mar 7, 2015

Uh oh!

pgrosu commented Mar 7, 2015

Uh oh!

wbrockman Mar 9, 2015

Choose a reason for hiding this comment

Uh oh!

iliat Mar 10, 2015

Choose a reason for hiding this comment

Uh oh!

wbrockman Mar 11, 2015

Choose a reason for hiding this comment

Uh oh!

pgrosu Mar 11, 2015

Choose a reason for hiding this comment

Uh oh!

coveralls commented Mar 10, 2015

Uh oh!

lbergelson Mar 11, 2015

Choose a reason for hiding this comment

Uh oh!

wbrockman commented Mar 11, 2015

Uh oh!

pgrosu commented Mar 11, 2015

Uh oh!

wbrockman commented Mar 12, 2015

Uh oh!

pgrosu commented Mar 12, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants