VariantSimilarity: Dataflow job creation fails with a 400

[calbach]

$ java -cp target/google-genomics-dataflow-v1beta2-0.2-SNAPSHOT.jar   com.google.cloud.genomics.dataflow.pipelines.VariantSimilarity   --runner=BlockingDataflowPipelineRunner   --project=274435909155   --output=gs://test-bucket88/test.txt   --genomicsSecretsFile=<redacted>  --stagingLocation=gs://test-bucket88/staging     --numWorkers=10

WARNING: Request failed with code 400, will NOT retry: https://www.googleapis.com/dataflow/v1b3/projects/274435909155/jobs
Exception in thread "main" java.lang.RuntimeException: Failed to create a workflow job: Bad Request
    at com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner.run(DataflowPipelineRunner.java:232)
    at com.google.cloud.dataflow.sdk.runners.BlockingDataflowPipelineRunner.run(BlockingDataflowPipelineRunner.java:96)
    at com.google.cloud.dataflow.sdk.runners.BlockingDataflowPipelineRunner.run(BlockingDataflowPipelineRunner.java:47)
    at com.google.cloud.dataflow.sdk.Pipeline.run(Pipeline.java:145)
    at com.google.cloud.genomics.dataflow.pipelines.VariantSimilarity.main(VariantSimilarity.java:58)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request
{
  "code" : 400,
  "errors" : [ {
    "domain" : "global",
    "message" : "Bad Request",
    "reason" : "invalidArgument",
    "debugInfo" : "com.google.api.server.core.Fault: ImmutableErrorDefinition{base=BAD_REQUEST, category=USER_ERROR, cause=null, debugInfo=null, domain=global, extendedHelp=null, httpHeaders={}, httpStatus=badRequest, internalReason=Reason{arguments={}, cause=null, code=canonicalInvalidArgumentFromBackend, createdByBackend=true, debugMessage=Received status: , errorProtoCode=null, errorProtoDomain=null, filteredMessage=null, location=null, message=null, unnamedArguments=[]}, location=null, message=Bad Request, reason=invalidArgument, rpcCode=400} Bad Request\n\tat com.google.api.server.rest.adapter.client.DefaultFaultCreatorFactory.createFaultFromStatus(DefaultFaultCreatorFactory.java:47)\n\tat com.google.api.server.rest.adapter.client.DefaultFaultCreatorFactory.access$000(DefaultFaultCreatorFactory.java:19)\n\tat com.google.api.server.rest.adapter.client.DefaultFaultCreatorFactory$1.apply(DefaultFaultCreatorFactory.java:35)\n\tat com.google.api.server.rest.adapter.client.DefaultFaultCreatorFactory$1.apply(DefaultFaultCreatorFactory.java:32)\n\tat com.google.api.server.rest.adapter.client.Stubby3Client$Stubby3Method$Stubby3Request$CallbackFuture.mapResult(Stubby3Client.java:179)\n\tat com.google.api.server.rest.adapter.client.Stubby3Client$Stubby3Method$Stubby3Request$CallbackFuture.mapResult(Stubby3Client.java:167)\n\tat com.google.net.rpc3.client.util.RpcAbstractCheckedFuture.internalGet(RpcAbstractCheckedFuture.java:202)\n\tat com.google.net.rpc3.client.util.RpcAbstractCheckedFuture.get(RpcAbstractCheckedFuture.java:243)\n\tat com.google.api.server.rest.adapter.client.Stubby3Client$Stubby3Method$Stubby3Response.getOrThrowFault(Stubby3Client.java:231)\n\tat com.google.api.server.rest.adapter.client.Stubby3Client$Stubby3Method$Stubby3Response.getBody(Stubby3Client.java:214)\n\tat com.google.api.server.rest.adapter.stubby3.Stubby3Handler$1.call(Stubby3Handler.java:72)\n\tat com.google.api.server.rest.adapter.stubby3.Stubby3Handler$1.call(Stubby3Handler.java:59)\n\tat com.google.api.server.core.util.CallableFuture.run(CallableFuture.java:62)\n\tat com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:476)\n\tat com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:784)\n\tat com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:743)\n\tat com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:603)\n\tat com.google.api.server.core.util.CallableFuture.run(CallableFuture.java:62)\n\tat com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:476)\n\tat com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)\n\tat com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)\n\tat com.google.net.rpc3.client.util.RpcAbstractCheckedFuture.complete(RpcAbstractCheckedFuture.java:300)\n\tat com.google.net.rpc3.client.util.RpcAbstractCheckedFuture.failure(RpcAbstractCheckedFuture.java:275)\n\tat com.google.net.rpc3.client.RpcStub$RpcCallbackDispatcher$1.runInContext(RpcStub.java:858)\n\tat com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:444)\n\tat com.google.tracing.CurrentContext.runInContext(CurrentContext.java:220)\n\tat com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:308)\n\tat com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:300)\n\tat com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:441)\n\tat com.google.api.server.thread.ThreadTrackers$ThreadTrackingRunnable.run(ThreadTrackers.java:126)\n\tat com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:437)\n\tat com.google.api.server.server.CommonModule$ContextCarryingExecutorService$1.runInContext(CommonModule.java:796)\n\tat com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:444)\n\tat com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:310)\n\tat com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:300)\n\tat com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:441)\n\tat com.google.gse.internal.DispatchQueueImpl$WorkerThread.run(DispatchQueueImpl.java:387)\n"
  } ],
  "message" : "Bad Request"
}
    at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)
    at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
    at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:312)
    at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1049)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:410)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:343)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:460)
    at com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner.run(DataflowPipelineRunner.java:228)
    ... 4 more

[calbach]

Note: this works fine locally.

[pgrosu]

Hmm..., I'm getting two different kinds of errors (one for Linux and one for Windows), but that might be for different reasons:

Under a Linux Cluster

...
Feb 26, 2015 6:33:49 PM com.google.cloud.dataflow.sdk.util.RetryHttpRequestInitializer$LoggingHttpBackoffUnsuccessfulResponseHandler handleResponse
WARNING: Request failed with code 403, will NOT retry: https://www.googleapis.com/upload/storage/v1/b/test-bucket88/o?uploadType=resumable
Feb 26, 2015 6:33:49 PM com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel$UploadOperation run
SEVERE: Upload failure
com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Forbidden",
    "reason" : "forbidden"
  } ],
  "message" : "Forbidden"
}
        at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:423)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:343)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:460)
        at com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel$UploadOperation.run(GoogleCloudStorageWriteChannel.java:167)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
...

Under a Local Windows Machine

...
INFO: Adding request with 17 41196311 to 41277499
Feb 26, 2015 6:42:31 PM com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner fromOptions
INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: [C:\Go
ogle Genomics\gg_dataflow\dataflow-java\target\google-genomics-dataflow-v1beta2-0.2-SNAPSHOT.jar]
Exception in thread "main" java.lang.RuntimeException: Failed to construct instance from factory met
hod com.google.cloud.dataflow.sdk.runners.BlockingDataflowPipelineRunner#fromOptions
        at com.google.cloud.dataflow.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:2
26)
        at com.google.cloud.dataflow.sdk.util.InstanceBuilder.build(InstanceBuilder.java:159)
        at com.google.cloud.dataflow.sdk.runners.PipelineRunner.fromOptions(PipelineRunner.java:52)
        at com.google.cloud.dataflow.sdk.Pipeline.create(Pipeline.java:112)
        at com.google.cloud.genomics.dataflow.pipelines.VariantSimilarity.main(VariantSimilarity.jav
a:51)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at com.google.cloud.dataflow.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:2
19)
        ... 4 more
Caused by: java.lang.RuntimeException: Could not obtain credential using gcloud
        at com.google.cloud.dataflow.sdk.util.Credentials.getCredentialFromGCloud(Credentials.java:1
96)
        at com.google.cloud.dataflow.sdk.util.Credentials.getUserCredential(Credentials.java:159)
        at com.google.cloud.dataflow.sdk.options.GcpOptions$GcpUserCredentialsFactory.create(GcpOpti
ons.java:144)
        at com.google.cloud.dataflow.sdk.options.GcpOptions$GcpUserCredentialsFactory.create(GcpOpti
ons.java:140)
        at com.google.cloud.dataflow.sdk.options.ProxyInvocationHandler.getDefault(ProxyInvocationHa
ndler.java:259)
        at com.google.cloud.dataflow.sdk.options.ProxyInvocationHandler.invoke(ProxyInvocationHandle
r.java:118)
        at com.sun.proxy.$Proxy12.getGcpCredential(Unknown Source)
        at com.google.cloud.dataflow.sdk.util.Transport.newDataflowClient(Transport.java:114)
        at com.google.cloud.dataflow.sdk.options.DataflowPipelineOptions$DataflowClientFactory.creat
e(DataflowPipelineOptions.java:125)
        at com.google.cloud.dataflow.sdk.options.DataflowPipelineOptions$DataflowClientFactory.creat
e(DataflowPipelineOptions.java:122)
        at com.google.cloud.dataflow.sdk.options.ProxyInvocationHandler.getDefault(ProxyInvocationHa
ndler.java:259)
        at com.google.cloud.dataflow.sdk.options.ProxyInvocationHandler.invoke(ProxyInvocationHandle
r.java:118)
        at com.sun.proxy.$Proxy12.getDataflowClient(Unknown Source)
        at com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner.<init>(DataflowPipelineRunne
r.java:146)
        at com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner.fromOptions(DataflowPipeline
Runner.java:141)
        at com.google.cloud.dataflow.sdk.runners.BlockingDataflowPipelineRunner.fromOptions(Blocking
DataflowPipelineRunner.java:87)
        ... 9 more
Caused by: java.io.IOException: Cannot run program "gcloud": CreateProcess error=2, The system canno
t find the file specified
        at java.lang.ProcessBuilder.start(Unknown Source)
        at com.google.cloud.dataflow.sdk.util.GCloudCredential.executeRefreshToken(GCloudCredential.
java:66)
        at com.google.api.client.auth.oauth2.Credential.refreshToken(Credential.java:489)
        at com.google.cloud.dataflow.sdk.util.Credentials.getCredentialFromGCloud(Credentials.java:1
94)
        ... 24 more
Caused by: java.io.IOException: CreateProcess error=2, The system cannot find the file specified
        at java.lang.ProcessImpl.create(Native Method)
        at java.lang.ProcessImpl.<init>(Unknown Source)
        at java.lang.ProcessImpl.start(Unknown Source)
        ... 28 more
...

Paul

[calbach]

The issue was that I was specifying an output directory, rather than a single output file. The remaining question is why the error was so obfuscated. I'm still not sure if this is something in our code or in Dataflow.

Paul, your issues look unrelated. I can't see your commandline for Linux, but I'm guessing you might not be in the alpha whitelist for dataflow: https://cloud.google.com/dataflow/what-is-google-cloud-dataflow.

For Windows, it looks like you don't have gcloud installed, which is part of the dataflow setup https://cloud.google.com/dataflow/getting-started

[jean-philippe-martin]

Was there a long delay before you go the error? If yes, should we change
the pipeline so it checks that the output is not a directory before
starting the cloud job, in order to save time for future users?

On Thu, Feb 26, 2015 at 3:59 PM, CH Albach notifications@github.com wrote:

The issue was that I was specifying an output directory, rather than a
single output file. The remaining question is why the error was so
obfuscated. I'm still not sure if this is something in our code or in
Dataflow.

Paul, your issues look unrelated. I can't see your commandline for Linux,
but I'm guessing you might not be in the alpha whitelist for dataflow:
https://cloud.google.com/dataflow/what-is-google-cloud-dataflow.

For Windows, it looks like you don't have gcloud installed, which is part
of the dataflow setup https://cloud.google.com/dataflow/getting-started

—
Reply to this email directly or view it on GitHub
#31 (comment)
.

[calbach]

It failed quickly, but very mysteriously in the non-local run. I don't understand enough about the Dataflow machinery to say whether this is the result of sloppiness in our code, or sloppiness on the Dataflow side. I'll reopen this so I don't lose track of it. I suspect by the end of the tomorrow I'll be in a better position to answer that question.

[pgrosu]

Would it be possible to be added to the alpha whitelist for dataflow to help out on this?

Thanks,
Paul

[calbach]

Quickest way would be to fill out the form linked at the top of this page for alpha access: https://cloud.google.com/dataflow/what-is-google-cloud-dataflow

[pgrosu]

Hey, thanks CH! I just did.

[pgrosu]

Hi CH (@calbach),

So I got access to Dataflow - thank you for pointing me to it. So kind of excited now to help out on this, and future implementations. So just to be sure it works for me as well, I also changed the output to be a directory and ran it with the following command, but still getting the error:

 java -cp target/google-genomics-dataflow-v1beta2-0.2-SNAPSHOT.jar   com.google.cloud.genomics.dataflow.pipelines.VariantSimilarity   --runner=BlockingDataflowPipelineRunner   --project=274435909155   --output=gs://test-bucket88/   --genomicsSecretsFile=client_secrets.json  --stagingLocation=gs://test-bucket88/staging     --numWorkers=10

Below is what I get:

$  java -cp target/google-genomics-dataflow-v1beta2-0.2-SNAPSHOT.jar   com.google.cloud.genomics.dataflow.pipelines.VariantSimilarity   --runner=BlockingDataflowPipelineRunner   --project=274435909155   --output=gs://test-bucket88/   --genomicsSecretsFile=client_secrets.json  --stagingLocation=gs://test-bucket88/staging     --numWorkers=10
Mar 06, 2015 11:00:37 PM com.google.cloud.genomics.dataflow.utils.GenomicsDatasetOptions$Methods getVariantRequests
INFO: Adding request with 17 41196311 to 41277499
Mar 06, 2015 11:00:37 PM com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner fromOptions
INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: [/home/pgrosu/me/gg_dataflow/v3/dataflow-java/target/google-genomics-dataflow-v1beta2-0.2-SNAPSHOT.jar]
Mar 06, 2015 11:00:37 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coders for genomics classes
Mar 06, 2015 11:00:38 PM org.reflections.Reflections scan
INFO: Reflections took 226 ms to scan 10 urls, producing 1 keys and 58 values
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for ListCoverageBucketsResponse
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for ReadGroupExperiment
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for ImportReadGroupSetsResponse
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for SearchReadGroupSetsResponse
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for SearchReferenceSetsResponse
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for CoverageBucket
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for ListDatasetsResponse
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for SearchReadsResponse
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for CigarUnit
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for JobRequest
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for ReadGroup
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for AlignReadGroupSetsRequest
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for ReferenceBound
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for ReadGroupSet
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for SearchReadGroupSetsRequest
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for SearchVariantsRequest
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for LinearAlignment
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for CallReadGroupSetsResponse
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for AlignReadGroupSetsResponse
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for CallSet
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for Call
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for ExportVariantSetResponse
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for Variant
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for ImportVariantsRequest
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for ImportReadGroupSetsRequest
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for SearchVariantsResponse
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for ExperimentalCreateJobRequest
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for Metadata
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for CallReadGroupSetsRequest
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for SearchCallSetsRequest
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for Position
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for SearchReadsRequest
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for SearchJobsRequest
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for SearchVariantSetsResponse
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for ExportReadGroupSetsRequest
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for ImportVariantsResponse
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for VariantSet
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for ListBasesResponse
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for SearchCallSetsResponse
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for ReferenceSet
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for ExportVariantSetRequest
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for Reference
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for Job
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for MergeVariantsRequest
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for SearchVariantSetsRequest
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for SearchReferencesRequest
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for SearchJobsResponse
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for SearchReferencesResponse
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for Read
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for Range
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for ReadGroupProgram
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for InterleavedFastqSource
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for SearchReferenceSetsRequest
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for ExperimentalCreateJobResponse
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for PairedFastqSource
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for ExportReadGroupSetsResponse
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for FastqMetadata
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coder for Dataset
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds getPCollection
INFO: Turning 1 options into 10.0 workers
Mar 06, 2015 11:00:38 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds getPCollection
INFO: Adding collection with 0 to 1
Mar 06, 2015 11:00:38 PM com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner run
INFO: Executing pipeline on the Dataflow Service, which will have billing implications related to Google Compute Engine usage and other Google Cloud Services.
Mar 06, 2015 11:00:38 PM com.google.cloud.dataflow.sdk.util.PackageUtil stageClasspathElementsToGcs
INFO: Uploading 1 files from PipelineOptions.filesToStage to GCS to prepare for execution in the cloud.
Mar 06, 2015 11:00:39 PM com.google.cloud.dataflow.sdk.util.RetryHttpRequestInitializer$LoggingHttpBackoffUnsuccessfulResponseHandler handleResponse
WARNING: Request failed with code 403, will NOT retry: https://www.googleapis.com/upload/storage/v1/b/test-bucket88/o?uploadType=resumable
Mar 06, 2015 11:00:39 PM com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel$UploadOperation run
SEVERE: Upload failure
com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Forbidden",
    "reason" : "forbidden"
  } ],
  "message" : "Forbidden"
}
        at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:423)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:343)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:460)
        at com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel$UploadOperation.run(GoogleCloudStorageWriteChannel.java:167)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Mar 06, 2015 11:00:47 PM com.google.cloud.dataflow.sdk.util.PackageUtil stageClasspathElementsToGcs
WARNING: Upload attempt failed, will retry staging of classpath: /home/pgrosu/me/gg_dataflow/v3/dataflow-java/target/google-genomics-dataflow-v1beta2-0.2-SNAPSHOT.jar
java.io.IOException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Forbidden",
    "reason" : "forbidden"
  } ],
  "message" : "Forbidden"
}
        at com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel.throwIfUploadFailed(GoogleCloudStorageWriteChannel.java:376)
        at com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel.write(GoogleCloudStorageWriteChannel.java:240)
        at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
        at java.nio.channels.Channels.writeFully(Channels.java:101)
        at java.nio.channels.Channels.access$000(Channels.java:61)
        at java.nio.channels.Channels$1.write(Channels.java:174)
        at com.google.common.io.ByteStreams.copy(ByteStreams.java:74)
        at com.google.common.io.ByteSource.copyTo(ByteSource.java:220)
        at com.google.cloud.dataflow.sdk.util.PackageUtil.copyContent(PackageUtil.java:263)
        at com.google.cloud.dataflow.sdk.util.PackageUtil.stageClasspathElementsToGcs(PackageUtil.java:159)
        at com.google.cloud.dataflow.sdk.util.PackageUtil.stageClasspathElementsToGcs(PackageUtil.java:106)
        at com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner.run(DataflowPipelineRunner.java:180)
        at com.google.cloud.dataflow.sdk.runners.BlockingDataflowPipelineRunner.run(BlockingDataflowPipelineRunner.java:96)
        at com.google.cloud.dataflow.sdk.runners.BlockingDataflowPipelineRunner.run(BlockingDataflowPipelineRunner.java:47)
        at com.google.cloud.dataflow.sdk.Pipeline.run(Pipeline.java:145)
        at com.google.cloud.genomics.dataflow.pipelines.VariantSimilarity.main(VariantSimilarity.java:58)
        Suppressed: java.io.IOException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Forbidden",
    "reason" : "forbidden"
  } ],
  "message" : "Forbidden"
}
                at com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel.throwIfUploadFailed(GoogleCloudStorageWriteChannel.java:376)
                at com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel.close(GoogleCloudStorageWriteChannel.java:271)
                at com.google.cloud.dataflow.sdk.util.PackageUtil.stageClasspathElementsToGcs(PackageUtil.java:160)
                ... 6 more
        Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Forbidden",
    "reason" : "forbidden"
  } ],
  "message" : "Forbidden"
}
                at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)
                at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
                at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
                at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:423)
                at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:343)
                at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:460)
                at com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel$UploadOperation.run(GoogleCloudStorageWriteChannel.java:167)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                at java.lang.Thread.run(Thread.java:745)
        [CIRCULAR REFERENCE:com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Forbidden",
    "reason" : "forbidden"
  } ],
  "message" : "Forbidden"
}]

Mar 06, 2015 11:00:47 PM com.google.cloud.dataflow.sdk.util.RetryHttpRequestInitializer$LoggingHttpBackoffUnsuccessfulResponseHandler handleResponse
WARNING: Request failed with code 403, will NOT retry: https://www.googleapis.com/upload/storage/v1/b/test-bucket88/o?uploadType=resumable
Mar 06, 2015 11:00:47 PM com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel$UploadOperation run
SEVERE: Upload failure
com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Forbidden",
    "reason" : "forbidden"
  } ],
  "message" : "Forbidden"
}
        at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:423)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:343)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:460)
        at com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel$UploadOperation.run(GoogleCloudStorageWriteChannel.java:167)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Mar 06, 2015 11:00:57 PM com.google.cloud.dataflow.sdk.util.PackageUtil stageClasspathElementsToGcs
WARNING: Upload attempt failed, will retry staging of classpath: /home/pgrosu/me/gg_dataflow/v3/dataflow-java/target/google-genomics-dataflow-v1beta2-0.2-SNAPSHOT.jar
java.io.IOException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Forbidden",
    "reason" : "forbidden"
  } ],
  "message" : "Forbidden"
}
        at com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel.throwIfUploadFailed(GoogleCloudStorageWriteChannel.java:376)
        at com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel.write(GoogleCloudStorageWriteChannel.java:240)
        at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
        at java.nio.channels.Channels.writeFully(Channels.java:101)
        at java.nio.channels.Channels.access$000(Channels.java:61)
        at java.nio.channels.Channels$1.write(Channels.java:174)
        at com.google.common.io.ByteStreams.copy(ByteStreams.java:74)
        at com.google.common.io.ByteSource.copyTo(ByteSource.java:220)
        at com.google.cloud.dataflow.sdk.util.PackageUtil.copyContent(PackageUtil.java:263)
        at com.google.cloud.dataflow.sdk.util.PackageUtil.stageClasspathElementsToGcs(PackageUtil.java:159)
        at com.google.cloud.dataflow.sdk.util.PackageUtil.stageClasspathElementsToGcs(PackageUtil.java:106)
        at com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner.run(DataflowPipelineRunner.java:180)
        at com.google.cloud.dataflow.sdk.runners.BlockingDataflowPipelineRunner.run(BlockingDataflowPipelineRunner.java:96)
        at com.google.cloud.dataflow.sdk.runners.BlockingDataflowPipelineRunner.run(BlockingDataflowPipelineRunner.java:47)
        at com.google.cloud.dataflow.sdk.Pipeline.run(Pipeline.java:145)
        at com.google.cloud.genomics.dataflow.pipelines.VariantSimilarity.main(VariantSimilarity.java:58)
        Suppressed: java.io.IOException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Forbidden",
    "reason" : "forbidden"
  } ],
  "message" : "Forbidden"
}
                at com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel.throwIfUploadFailed(GoogleCloudStorageWriteChannel.java:376)
                at com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel.close(GoogleCloudStorageWriteChannel.java:271)
                at com.google.cloud.dataflow.sdk.util.PackageUtil.stageClasspathElementsToGcs(PackageUtil.java:160)
                ... 6 more
        Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Forbidden",
    "reason" : "forbidden"
  } ],
  "message" : "Forbidden"
}
                at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)
                at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
                at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
                at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:423)
                at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:343)
                at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:460)
                at com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel$UploadOperation.run(GoogleCloudStorageWriteChannel.java:167)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                at java.lang.Thread.run(Thread.java:745)
        [CIRCULAR REFERENCE:com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Forbidden",
    "reason" : "forbidden"
  } ],
  "message" : "Forbidden"
}]

Mar 06, 2015 11:00:57 PM com.google.cloud.dataflow.sdk.util.RetryHttpRequestInitializer$LoggingHttpBackoffUnsuccessfulResponseHandler handleResponse
WARNING: Request failed with code 403, will NOT retry: https://www.googleapis.com/upload/storage/v1/b/test-bucket88/o?uploadType=resumable
Mar 06, 2015 11:00:57 PM com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel$UploadOperation run
SEVERE: Upload failure
com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Forbidden",
    "reason" : "forbidden"
  } ],
  "message" : "Forbidden"
}
        at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:423)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:343)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:460)
        at com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel$UploadOperation.run(GoogleCloudStorageWriteChannel.java:167)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Mar 06, 2015 11:01:12 PM com.google.cloud.dataflow.sdk.util.PackageUtil stageClasspathElementsToGcs
WARNING: Upload attempt failed, will retry staging of classpath: /home/pgrosu/me/gg_dataflow/v3/dataflow-java/target/google-genomics-dataflow-v1beta2-0.2-SNAPSHOT.jar
java.io.IOException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Forbidden",
    "reason" : "forbidden"
  } ],
  "message" : "Forbidden"
}
        at com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel.throwIfUploadFailed(GoogleCloudStorageWriteChannel.java:376)
        at com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel.write(GoogleCloudStorageWriteChannel.java:240)
        at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
        at java.nio.channels.Channels.writeFully(Channels.java:101)
        at java.nio.channels.Channels.access$000(Channels.java:61)
        at java.nio.channels.Channels$1.write(Channels.java:174)
        at com.google.common.io.ByteStreams.copy(ByteStreams.java:74)
        at com.google.common.io.ByteSource.copyTo(ByteSource.java:220)
        at com.google.cloud.dataflow.sdk.util.PackageUtil.copyContent(PackageUtil.java:263)
        at com.google.cloud.dataflow.sdk.util.PackageUtil.stageClasspathElementsToGcs(PackageUtil.java:159)
        at com.google.cloud.dataflow.sdk.util.PackageUtil.stageClasspathElementsToGcs(PackageUtil.java:106)
        at com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner.run(DataflowPipelineRunner.java:180)
        at com.google.cloud.dataflow.sdk.runners.BlockingDataflowPipelineRunner.run(BlockingDataflowPipelineRunner.java:96)
        at com.google.cloud.dataflow.sdk.runners.BlockingDataflowPipelineRunner.run(BlockingDataflowPipelineRunner.java:47)
        at com.google.cloud.dataflow.sdk.Pipeline.run(Pipeline.java:145)
        at com.google.cloud.genomics.dataflow.pipelines.VariantSimilarity.main(VariantSimilarity.java:58)
        Suppressed: java.io.IOException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Forbidden",
    "reason" : "forbidden"
  } ],
  "message" : "Forbidden"
}
                at com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel.throwIfUploadFailed(GoogleCloudStorageWriteChannel.java:376)
                at com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel.close(GoogleCloudStorageWriteChannel.java:271)
                at com.google.cloud.dataflow.sdk.util.PackageUtil.stageClasspathElementsToGcs(PackageUtil.java:160)
                ... 6 more
        Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Forbidden",
    "reason" : "forbidden"
  } ],
  "message" : "Forbidden"
}
                at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)
                at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
                at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
                at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:423)
                at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:343)
                at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:460)
                at com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel$UploadOperation.run(GoogleCloudStorageWriteChannel.java:167)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                at java.lang.Thread.run(Thread.java:745)
        [CIRCULAR REFERENCE:com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Forbidden",
    "reason" : "forbidden"
  } ],
  "message" : "Forbidden"
}]

Mar 06, 2015 11:01:12 PM com.google.cloud.dataflow.sdk.util.RetryHttpRequestInitializer$LoggingHttpBackoffUnsuccessfulResponseHandler handleResponse
WARNING: Request failed with code 403, will NOT retry: https://www.googleapis.com/upload/storage/v1/b/test-bucket88/o?uploadType=resumable
Mar 06, 2015 11:01:12 PM com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel$UploadOperation run
SEVERE: Upload failure
com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Forbidden",
    "reason" : "forbidden"
  } ],
  "message" : "Forbidden"
}
        at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:423)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:343)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:460)
        at com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel$UploadOperation.run(GoogleCloudStorageWriteChannel.java:167)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Mar 06, 2015 11:01:38 PM com.google.cloud.dataflow.sdk.util.PackageUtil stageClasspathElementsToGcs
WARNING: Upload attempt failed, will retry staging of classpath: /home/pgrosu/me/gg_dataflow/v3/dataflow-java/target/google-genomics-dataflow-v1beta2-0.2-SNAPSHOT.jar
java.io.IOException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Forbidden",
    "reason" : "forbidden"
  } ],
  "message" : "Forbidden"
}
        at com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel.throwIfUploadFailed(GoogleCloudStorageWriteChannel.java:376)
        at com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel.write(GoogleCloudStorageWriteChannel.java:240)
        at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
        at java.nio.channels.Channels.writeFully(Channels.java:101)
        at java.nio.channels.Channels.access$000(Channels.java:61)
        at java.nio.channels.Channels$1.write(Channels.java:174)
        at com.google.common.io.ByteStreams.copy(ByteStreams.java:74)
        at com.google.common.io.ByteSource.copyTo(ByteSource.java:220)
        at com.google.cloud.dataflow.sdk.util.PackageUtil.copyContent(PackageUtil.java:263)
        at com.google.cloud.dataflow.sdk.util.PackageUtil.stageClasspathElementsToGcs(PackageUtil.java:159)
        at com.google.cloud.dataflow.sdk.util.PackageUtil.stageClasspathElementsToGcs(PackageUtil.java:106)
        at com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner.run(DataflowPipelineRunner.java:180)
        at com.google.cloud.dataflow.sdk.runners.BlockingDataflowPipelineRunner.run(BlockingDataflowPipelineRunner.java:96)
        at com.google.cloud.dataflow.sdk.runners.BlockingDataflowPipelineRunner.run(BlockingDataflowPipelineRunner.java:47)
        at com.google.cloud.dataflow.sdk.Pipeline.run(Pipeline.java:145)
        at com.google.cloud.genomics.dataflow.pipelines.VariantSimilarity.main(VariantSimilarity.java:58)
        Suppressed: java.io.IOException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Forbidden",
    "reason" : "forbidden"
  } ],
  "message" : "Forbidden"
}
                at com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel.throwIfUploadFailed(GoogleCloudStorageWriteChannel.java:376)
                at com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel.close(GoogleCloudStorageWriteChannel.java:271)
                at com.google.cloud.dataflow.sdk.util.PackageUtil.stageClasspathElementsToGcs(PackageUtil.java:160)
                ... 6 more
        Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Forbidden",
    "reason" : "forbidden"
  } ],
  "message" : "Forbidden"
}
                at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)
                at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
                at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
                at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:423)
                at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:343)
                at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:460)
                at com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel$UploadOperation.run(GoogleCloudStorageWriteChannel.java:167)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                at java.lang.Thread.run(Thread.java:745)
        [CIRCULAR REFERENCE:com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Forbidden",
    "reason" : "forbidden"
  } ],
  "message" : "Forbidden"
}]

Mar 06, 2015 11:01:38 PM com.google.cloud.dataflow.sdk.util.RetryHttpRequestInitializer$LoggingHttpBackoffUnsuccessfulResponseHandler handleResponse
WARNING: Request failed with code 403, will NOT retry: https://www.googleapis.com/upload/storage/v1/b/test-bucket88/o?uploadType=resumable
Mar 06, 2015 11:01:38 PM com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel$UploadOperation run
SEVERE: Upload failure
com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Forbidden",
    "reason" : "forbidden"
  } ],
  "message" : "Forbidden"
}
        at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:423)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:343)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:460)
        at com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel$UploadOperation.run(GoogleCloudStorageWriteChannel.java:167)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Mar 06, 2015 11:01:39 PM com.google.cloud.dataflow.sdk.util.PackageUtil stageClasspathElementsToGcs
SEVERE: Upload failed, will NOT retry staging of classpath: /home/pgrosu/me/gg_dataflow/v3/dataflow-java/target/google-genomics-dataflow-v1beta2-0.2-SNAPSHOT.jar
java.io.IOException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Forbidden",
    "reason" : "forbidden"
  } ],
  "message" : "Forbidden"
}
        at com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel.throwIfUploadFailed(GoogleCloudStorageWriteChannel.java:376)
        at com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel.write(GoogleCloudStorageWriteChannel.java:240)
        at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
        at java.nio.channels.Channels.writeFully(Channels.java:101)
        at java.nio.channels.Channels.access$000(Channels.java:61)
        at java.nio.channels.Channels$1.write(Channels.java:174)
        at com.google.common.io.ByteStreams.copy(ByteStreams.java:74)
        at com.google.common.io.ByteSource.copyTo(ByteSource.java:220)
        at com.google.cloud.dataflow.sdk.util.PackageUtil.copyContent(PackageUtil.java:263)
        at com.google.cloud.dataflow.sdk.util.PackageUtil.stageClasspathElementsToGcs(PackageUtil.java:159)
        at com.google.cloud.dataflow.sdk.util.PackageUtil.stageClasspathElementsToGcs(PackageUtil.java:106)
        at com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner.run(DataflowPipelineRunner.java:180)
        at com.google.cloud.dataflow.sdk.runners.BlockingDataflowPipelineRunner.run(BlockingDataflowPipelineRunner.java:96)
        at com.google.cloud.dataflow.sdk.runners.BlockingDataflowPipelineRunner.run(BlockingDataflowPipelineRunner.java:47)
        at com.google.cloud.dataflow.sdk.Pipeline.run(Pipeline.java:145)
        at com.google.cloud.genomics.dataflow.pipelines.VariantSimilarity.main(VariantSimilarity.java:58)
        Suppressed: java.io.IOException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Forbidden",
    "reason" : "forbidden"
  } ],
  "message" : "Forbidden"
}
                at com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel.throwIfUploadFailed(GoogleCloudStorageWriteChannel.java:376)
                at com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel.close(GoogleCloudStorageWriteChannel.java:271)
                at com.google.cloud.dataflow.sdk.util.PackageUtil.stageClasspathElementsToGcs(PackageUtil.java:160)
                ... 6 more
        Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Forbidden",
    "reason" : "forbidden"
  } ],
  "message" : "Forbidden"
}
                at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)
                at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
                at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
                at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:423)
                at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:343)
                at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:460)
                at com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel$UploadOperation.run(GoogleCloudStorageWriteChannel.java:167)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                at java.lang.Thread.run(Thread.java:745)
        [CIRCULAR REFERENCE:com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Forbidden",
    "reason" : "forbidden"
  } ],
  "message" : "Forbidden"
}]

Exception in thread "main" java.lang.RuntimeException: Could not stage classpath element: /home/pgrosu/me/gg_dataflow/v3/dataflow-java/target/google-genomics-dataflow-v1beta2-0.2-SNAPSHOT.jar
        at com.google.cloud.dataflow.sdk.util.PackageUtil.stageClasspathElementsToGcs(PackageUtil.java:176)
        at com.google.cloud.dataflow.sdk.util.PackageUtil.stageClasspathElementsToGcs(PackageUtil.java:106)
        at com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner.run(DataflowPipelineRunner.java:180)
        at com.google.cloud.dataflow.sdk.runners.BlockingDataflowPipelineRunner.run(BlockingDataflowPipelineRunner.java:96)
        at com.google.cloud.dataflow.sdk.runners.BlockingDataflowPipelineRunner.run(BlockingDataflowPipelineRunner.java:47)
        at com.google.cloud.dataflow.sdk.Pipeline.run(Pipeline.java:145)
        at com.google.cloud.genomics.dataflow.pipelines.VariantSimilarity.main(VariantSimilarity.java:58)
Caused by: java.io.IOException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Forbidden",
    "reason" : "forbidden"
  } ],
  "message" : "Forbidden"
}
        at com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel.throwIfUploadFailed(GoogleCloudStorageWriteChannel.java:376)
        at com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel.write(GoogleCloudStorageWriteChannel.java:240)
        at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
        at java.nio.channels.Channels.writeFully(Channels.java:101)
        at java.nio.channels.Channels.access$000(Channels.java:61)
        at java.nio.channels.Channels$1.write(Channels.java:174)
        at com.google.common.io.ByteStreams.copy(ByteStreams.java:74)
        at com.google.common.io.ByteSource.copyTo(ByteSource.java:220)
        at com.google.cloud.dataflow.sdk.util.PackageUtil.copyContent(PackageUtil.java:263)
        at com.google.cloud.dataflow.sdk.util.PackageUtil.stageClasspathElementsToGcs(PackageUtil.java:159)
        ... 6 more
        Suppressed: java.io.IOException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Forbidden",
    "reason" : "forbidden"
  } ],
  "message" : "Forbidden"
}
                at com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel.throwIfUploadFailed(GoogleCloudStorageWriteChannel.java:376)
                at com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel.close(GoogleCloudStorageWriteChannel.java:271)
                at com.google.cloud.dataflow.sdk.util.PackageUtil.stageClasspathElementsToGcs(PackageUtil.java:160)
                ... 6 more
        Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Forbidden",
    "reason" : "forbidden"
  } ],
  "message" : "Forbidden"
}
                at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)
                at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
                at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
                at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:423)
                at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:343)
                at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:460)
                at com.google.cloud.dataflow.sdk.util.gcsio.GoogleCloudStorageWriteChannel$UploadOperation.run(GoogleCloudStorageWriteChannel.java:167)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                at java.lang.Thread.run(Thread.java:745)
        [CIRCULAR REFERENCE:com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "Forbidden",
    "reason" : "forbidden"
  } ],
  "message" : "Forbidden"
}]
$

Thanks,
~p

[calbach]

You'll need to specify an output and staging cloud storage location where you have write access. gs://test-bucket88 is a scratch bucket I created for testing.

[pgrosu]

@calbach, Aha! I thought that was an open GCS staging area to debug GoogleGenomics PRs before they get merged. Would it be possible to open up the permission for that bucket like the alpha for Dataflow to help with the debugging? I am happy to help out with the debug, but a little hesitant to pay to debug. Maybe we can limit the quota so that it can be used only for debugging such PRs, and then files stored there can be automatically removed after a couple of days.

[calbach]

We don't have anything like that at the moment. Note, however, that you can also run it locally:

java -cp target/google-genomics-dataflow-v1beta2-0.2-SNAPSHOT.jar   com.google.cloud.genomics.dataflow.pipelines.VariantSimilarity   --genomicsSecretsFile=client_secrets.json --output=out.txt

[pgrosu]

Thanks @calbach! I modified it to the following which also uses local output, but am I still using Dataflow if I don't specify a runner? I take it BlockingDataflowPipelineRunner requires GCS.

java -cp target/google-genomics-dataflow-v1beta2-0.2-SNAPSHOT.jar com.google.cloud.genomics.dataflow.pipelines.VariantSimilarity --genomicsSecretsFile=client_secrets.json --project=274435909155 --output=outnew.txt --numWorkers=10

On another note, I'm not seeing any posts/code by @cassiedoll. Is everything okay? I miss them :(

Thanks,
~p