[Bug]: env() transformation does not work after upgrading the project from 0.7.6 to 0.9.7

[anielii]

What happened?

Hi,

After upgrading my project from 0.7.6 to 0.9.7, the @env() transformation stopped working. The environment variables are not replaced at all. I have the same problem both locally (CLI update) and in Azure ContainerApps (image update).

In the past, for 0.7.6, when I was developing locally, I set the environment variables in the terminal in the following way and everything worked fine without any problems. I get the same problem when I try to use the .env file.

Is there a new undocumented option to enable environment variable replacement?

$env:DATABASE_NAME = 'abc'
$env:DATABASE_CONNECTION_STRING = 'abc'
$env:GRAPHQL_SCHEMA_PATH = 'dab-config.schema.gql'
$env:JWT_AUDIENCE = 'abc'
$env:JWT_ISSUER = 'abc'

dab start

dab-config.json:

{
  "$schema": "https://github.com/Azure/data-api-builder/releases/download/v0.7.6/dab.draft.schema.json",
  "data-source": {
    "database-type": "cosmosdb_nosql",
    "options": {
      "database": "@env('DATABASE_NAME')",
      "schema": "@env('GRAPHQL_SCHEMA_PATH')"
    },
    "connection-string": "@env('DATABASE_CONNECTION_STRING')"
  },
  "runtime": {
    "rest": {
      "enabled": true,
      "path": "/api"
    },
    "graphql": {
      "allow-introspection": true,
      "enabled": true,
      "path": "/graphql"
    },
    "host": {
      "mode": "development",
      "cors": {
        "origins": [],
        "allow-credentials": false
      },
      "authentication": {
        "provider": "AzureAD",
        "jwt": {
          "audience": "@env('JWT_AUDIENCE')",
          "issuer": "@env('JWT_ISSUER')"
        }
      }
    }
  },
  "entities": {
    "NoteCard": {
      "source": "NoteCards",
      "permissions": [
        {
          "role": "authenticated",
          "actions": [
            "*"
          ]
        }
      ]
    }
  }
}

Version

0.9.7

What database are you using?

CosmosDB NoSQL

What hosting model are you using?

Local (including CLI), Container Apps

Which API approach are you accessing DAB through?

REST, GraphQL

Relevant log output

PS D:\r\DigestFusion\scripts\data-api-builder\cosmos> dab start
Information: Microsoft.DataApiBuilder 0.9.7+e560142426d1c080b9fd7b7fabff51a276f6bf61
Information: Config not provided. Trying to get default config based on DAB_ENVIRONMENT...
Information: Environment variable DAB_ENVIRONMENT is development
Loading config file from dab-config.json.
Information: Loaded config file: dab-config.json
Information: Setting default minimum LogLevel: Debug for Development mode.
Starting the runtime engine...
Loading config file from dab-config.json.
info: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[63]
      User profile is available. Using 'C:\Users\nieli\AppData\Local\ASP.NET\DataProtection-Keys' as key repository and Windows DPAPI to encrypt keys at rest.
fail: Azure.DataApiBuilder.Service.Startup[0]
      Unable to complete runtime initialization. Refer to exception for error details.
      System.IO.FileNotFoundException: The GraphQL schema file at '@env('GRAPHQL_SCHEMA_PATH')' could not be found. Ensure that it is a path relative to the runtime.
         at Azure.DataApiBuilder.Core.Configurations.RuntimeConfigValidator.ValidateDatabaseType(RuntimeConfig runtimeConfig, IFileSystem fileSystem, ILogger logger)
         at Azure.DataApiBuilder.Core.Configurations.RuntimeConfigValidator.ValidateConfig()
         at Azure.DataApiBuilder.Service.Startup.PerformOnConfigChangeAsync(IApplicationBuilder app)
fail: Azure.DataApiBuilder.Service.Startup[0]
      Could not initialize the engine with the runtime config file: dab-config.json
info: Microsoft.Hosting.Lifetime[0]
      Application is shutting down...
Unable to launch the Data API builder engine.
Error: Failed to start the engine.

Code of Conduct

• I agree to follow this project's Code of Conduct

[abhishekkumams]

@anielii, Thanks for pointing out. I checked it and found out replace environment variable is not working for options under data-source sections, but other places like for connection-string and authentication it is working as expected.

WORKAROUND:
directly put the values of DATABASE_NAME and GRAPHQL_SCHEMA_PATH for now, and you can continue to source the other environment variables from .env file or from your system.

[aaronpowell]

Having a look at it and the problem seems to stem from the late-bound deserialization of the options for a data source, which happens here:

data-api-builder/src/Config/ObjectModel/DataSource.cs

Lines 25 to 48 in dacf3bc

	public TOptionType? GetTypedOptions<TOptionType>() where TOptionType : IDataSourceOptions
	{
	HyphenatedNamingPolicy namingPolicy = new();
	if (typeof(TOptionType).IsAssignableFrom(typeof(CosmosDbNoSQLDataSourceOptions)))
	{
	return Options is not null ?
	(TOptionType)(object)new CosmosDbNoSQLDataSourceOptions(
	Database: ReadStringOption(namingPolicy.ConvertName(nameof(CosmosDbNoSQLDataSourceOptions.Database))),
	Container: ReadStringOption(namingPolicy.ConvertName(nameof(CosmosDbNoSQLDataSourceOptions.Container))),
	Schema: ReadStringOption(namingPolicy.ConvertName(nameof(CosmosDbNoSQLDataSourceOptions.Schema))),
	// The "raw" schema will be provided via the controller to setup config, rather than parsed from the JSON file.
	GraphQLSchema: ReadStringOption(namingPolicy.ConvertName(nameof(CosmosDbNoSQLDataSourceOptions.GraphQLSchema))))
	: default;
	}
	if (typeof(TOptionType).IsAssignableFrom(typeof(MsSqlOptions)))
	{
	return (TOptionType)(object)new MsSqlOptions(
	SetSessionContext: ReadBoolOption(namingPolicy.ConvertName(nameof(MsSqlOptions.SetSessionContext))));
	}
	throw new NotSupportedException($"The type {typeof(TOptionType).FullName} is not a supported strongly typed options object");
	}

This is late-bound because the options data structure isn't known to the deserializer, and the options could be any data type so we store them as a Dictionary<string, JsonElement> rather than a bool, string, etc.

The side effect of this is that when we get the string value for the options we don't use the deserialization pipeline, meaning the StringConverter isn't executed and thus the environment variables aren't replaced.

A relatively simple solution would be to make the code that takes a string value and returns it or the env replaced version and make a new method (probably an internal static):

data-api-builder/src/Config/Converters/StringJsonConverterFactory.cs

Line 64 in dacf3bc

return Regex.Replace(value!, ENV_PATTERN, new MatchEvaluator(ReplaceMatchWithEnvVariable));

Then the DataSource could be updated here

data-api-builder/src/Config/ObjectModel/DataSource.cs

Lines 50 to 58 in dacf3bc

	private string? ReadStringOption(string option)
	{
	if (Options is not null && Options.TryGetValue(option, out JsonElement value))
	{
	return value.GetString();
	}
	return null;
	}

and call that new method with the result of value.GetString().

An alternative approach would be to create a JsonSerializer in that method and pass in the StringJsonConverterFactory as part of the options, but that would be quite a bit of overhead IMO as the code would be setting up a lot more "stuff" to parse something that isn't really JSON anymore.