FR: [Storage Emulator] Support googleapis/java-storage

[ScottPierce]

This bug was copied from googleapis/java-storage#875 as it was indicated that it might be a bug with the emulator, not the java client.

Environment details

1. Specify the API at the beginning of the title. For example, "BigQuery: ...").
   General, Core, and Other are also allowed as types
2. OS type and version: osX 10.15.7
3. Java version: 15
4. storage version(s): com.google.cloud:libraries-bom:20.4.0

Steps to reproduce

1. Start the firebase emulator suite (including storage) via the npm firebase client.
2. Try to create a file using the firetstore emulator

Code example

    fun createStorage(): Storage {
        val storageHost = System.getenv().getOrDefault(ENV_STORAGE_EMULATOR_HOST, "http://localhost:9199")
        println("Using $storageHost for the Storage host")

        return StorageOptions
            .getDefaultInstance()
            .toBuilder()
            .setHost(storageHost)
            .setCredentials(NoCredentials.getInstance())
            .setProjectId("test")
            .build()
            .service
    }

suspend fun Storage.upload(from: InputStream, to: BlobId): Unit = withContext(Dispatchers.IO) {
    val storage = this@upload

    val toInfo = BlobInfo.newBuilder(to).build()

    storage
        .create(toInfo)
        .writer().use { writeChannel ->
            val byteBuffer = ByteBuffer.allocate(2048)

            from.use { from ->
                Channels.newChannel(from).use { fileChannel ->
                    while (fileChannel.read(byteBuffer) > 0) {
                        writeChannel.write(byteBuffer)
                    }
                }
            }
        }
}

Stack trace

Using localhost:8080 for the Firestore host.
It is being configured as an emulator
Using http://localhost:9199 for the Storage host

400 Bad Request
POST http://localhost:9199/upload/storage/v1/b/test/o?projection=full&uploadType=multipart
Bad Request
com.google.cloud.storage.StorageException: 400 Bad Request
POST http://localhost:9199/upload/storage/v1/b/test/o?projection=full&uploadType=multipart
Bad Request
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:233)
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.create(HttpStorageRpc.java:314)
	at com.google.cloud.storage.StorageImpl$3.call(StorageImpl.java:221)
	at com.google.cloud.storage.StorageImpl$3.call(StorageImpl.java:218)
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
	at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
	at com.google.cloud.storage.StorageImpl.internalCreate(StorageImpl.java:217)
	at com.google.cloud.storage.StorageImpl.create(StorageImpl.java:157)
	at com.example.test.ProcessController$upload$4.invokeSuspend(ProcessorController.kt:118)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request
POST http://localhost:9199/upload/storage/v1/b/test/o?projection=full&uploadType=multipart
Bad Request
	at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
	at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
	at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:532)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:455)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:565)
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.create(HttpStorageRpc.java:311)
	... 14 more

It seems like maybe this library doesn't support the local emulator.

[yuchenshi]

@abeisgoat any thoughts?

[ScottPierce]

I got a response back here - googleapis/java-storage#875 (comment)

It'd be nice if this is something that could be supported. Any chance we could turn this into a feature request?

[yuchenshi]

I've turn this into a feature request. Note that we are unable to promise any timeline for this, but if others also have this request, adding a +1 on this issue can help us prioritize adding this to the roadmap.

(Googler-only internal tracking bug: b/191996012)

[abeisgoat]

Looking at your logs, the requests actually look like they should be valid, the only unusual bit is the projection=full flag which, frankly, I've never seen before. I agree with @yuchenshi it's a feature request since comprehensive Cloud SDK support is out of scope for the moment, however it may not be that hard of fix. I'll do some research and see, but no promises for the time being.

[ScottPierce]

@abeisgoat Never heard back from you on this, so I thought I'd ping you.

[elefeint]

Piping in with additional context from Spring Cloud GCP where we got a question about supporting the Firebase Storage emulator through our Spring integrations (which directly depend on the Java Storage client library). Non-actionable for us, since the Spring integration can only support what the client library supports, but the troubleshooting information may be useful for implementing support in Firebase.

The Java storage client library forms a URL of the form http://localhost:9199/storage/v1/b/MY-BUCKET/o/MY-FILE?projection=full that com.google.cloud.storage.spi.v1.HttpStorageRpc.get(StorageObject object, Map<Option, ?> options) attempts to retrieve, but the Firebase emulator does not find, resulting in a 404.

Accessing the same file through the emulator web UI, the URL that would work is of the form http://localhost:9199/v0/b/MY-BUCKET/o/MY-FILE?token=7ebf1422-02f4-472d-bea4-0b0e9ebbd6f2.

The differences are:

• v1 vs v0 API prefix
• as you already mentioned, projection=full
• authentication token required -- would the client library know how to form this token?

Here is a minimal reproduction case for retrieving a single blob through production vs Firebase emulator.

	@Test
	public void getBlobThroughProductionStorage() throws IOException {

		Storage prodStorage = StorageOptions.newBuilder()
				.setProjectId(projectId)
				.setCredentials(GoogleCredentials.getApplicationDefault())
				.build()
				.getService();

		Blob blob = prodStorage.get(BlobId.of(bucket, filename));
		assertNotNull(blob);
		System.out.println("Found blob: " + blob);	// this works
		System.out.println("Blob contents: " + new String(blob.getContent()));	// this works
	}

	@Test
	public void getBlobThroughFirebaseStorageEmulator() throws IOException {

		String emulatorHostPort = "http://localhost:9199"; // replace with the correct port for your emulator instance

		Storage emulatorStorage = StorageOptions.newBuilder()
				.setProjectId(projectId)
				.setHost(emulatorHostPort)
				.setCredentials(GoogleCredentials.getApplicationDefault())	// NoCredentials does not work either
				.build()
				.getService();

		Blob blob = emulatorStorage.get(BlobId.of(bucket, filename));	// blob is null
		assertNotNull(blob);
	}