Token System

[skorfmann]

We want to address issues around the Token system and make the experience better.

Note: Still have to fill out the details here about what the actual problem is and how we're going to tackle this.

Issues Resolved by this Feature

• AzureRM CosmosDB connection_strings[0] yields #{Token[TOKEN.0]} instead of first list item #518
• How to get the list of AZ's of a specific region #374
• ContainerPorts in Docker Provider Container getter returns string identifier rather than container ports  #434
• aws_acm_certificate.*.domain_validation_options throws Invalid index #424
• EKS can't get OIDC issuer #663
• getNumberAttribute() returns number instead of reference #679
• Token in Provider isn't properly resolved #898
• Property Mapper should Respect Tokens #835

[jsteinich]

One question to answer when tackling this is whether to just fix a few current issues or to prepare for supporting functions / expressions including #220.

I certainly see value in taking a simple approach to improve the immediate experience as I anticipate full support to be a much larger project.

[jsteinich]

I've been working on a proposal for how to tackle this. It's not complete and I want to make some pocs, but I want to start writing it out to better kick off discussions.

Current problems

• Only supports strings, numbers, and lists of strings
• Doesn't support accessing a single element from a list of strings
• Confusing output when printing
• Inconvenient to reference real values

Other desired expansions

• Support Terraform functions/expressions
• Provide access to complex computed types

Solution preface

• Not planning to implement all expansions here, but would like to set up a system which will support them.
• I think there are some pieces that would only be solvable with Making cdktf dynamic #435 so this will knowingly not solve all issues.

Proposed solution

• Remove the token system as it currently exists
  • It only works for part of what is needed
  • Not very flexible for desired additions for what it does work with
• Create a TerraformAttribute class
  • Similar idea to Tokens, but is more explicit about it
  • Allows for keeping track of references
  • Can retrieve raw value if set
  • Can add built-in terraform functions as methods
• Create subclasses for core types
  • TerraformStringAttribute, TerraformNumberAttribute, TerraformBooleanAttribute, TerraformDynamicAttribute, TerraformListAttribute, TerraformSetAttribute, TerraformMapAttribute, TerraformObjectAttribute
  • This will give the user a more limited set of options for types attributes are
  • Can scope built-in terraform functions that are only designed to work on say collections to appropriate type
  • Possibly do code gen for specific object / collection types to workaround not having generics in jsii
• Collection types can have additional methods to support accessing elements / iterating in a terraform friendly way
  • The alternative idea of building custom collections into jsii would work well in Java and C# (need to switch to list from array), but I don't believe would work in Typescript (I don't believe it's possible to extend the Array type) (I'm not sure about python)
• Generate type aliases to allow for using either "primitive" types or TerraformAttributes
  • Still want it to be easy to initialize values
  • Also want to be able to easily reference other resources / data sources
  • Internally initing/setting would always result in a TerraformAttribute (or subtype) being created if not already using one
  • The getter will be a bit deceptive since you'll never get a primitive value from it
  • feat(jsii): support aliasing type unions aws/jsii#1807 is a prerequisite in my mind; else C# will be unusable
  • Provide accessors to nested computed attributes #480 complicates things since the union types will need to match else there will be quite a bit of friction
• Would be nice to have some language specific libraries to make easier to use
  • One example is having implicit operators in C# to make going between "union type" and "primitive type" easier (partially destructive operation)
• Still a case where the current token system (or something similar) is necessary
  • When using resources in string interpolation
  • Seems common enough to need support
  • Token would retain reference information to attribute
  • Can make a nicer representation that looks basically like a terraform reference

[skorfmann]

That's a great list, thanks for putting this together.

I've been working on a proposal for how to tackle this. It's not complete and I want to make some pocs, but I want to start writing it out to better kick off discussions.

I think even before starting with POCs, it would probably quite helpful to have some pseudo code which maps out the various aspects of the list from a user point of view.

[jsteinich]

I've started working on some examples / proving my proposed solution here: https://github.com/jsteinich/cdktf-ecs-fargate-tokens
I haven't gotten very far, but I'll continue to update and post here once I have something more meaningful.

[jsteinich]

• I've been thinking some more and I can't think of a case where a complex object (class generated for it) is passed from one resource to another, so we shouldn't run into issues with type aliases not matching up.
• We'll definitely want to do code gen for specific object / collection types since it is relatively common to want to access a particular property in order to pass to another resource.
• The biggest user facing impacts I see are:
  • When creating resources, properties will have multiple types that they accept. While the simple type one would expect will still work, it's a small amount additional cognitive load
  • We'll no longer be returning arrays (though the signature will make it look possible). Given that it wasn't almost always unsafe to do anything but pass along, I believe this to be a good change; however, will probably not be what uses expect.
  • There are a few other breaking changes, but other than calling them out in release notes, I don't really expect them to cause issues for most users.

[jsteinich]

When working on adding the list attribute type, I ran into some issues with the returned type:

(publicRouteTable.propagatingVgws as TerraformStringListAttribute).get(0)

since the returned type is TerraformStringList in order to make it match the setter and allow full flexibility.

Some options:

1. Use TS 4.3 (currently in beta) which allows for having different getter/setter types. Jsii doesn't currently support this and will likely not turn out great for at least c#.
2. Remove the setter and just return the actual type. Add a set method instead (ala Java) if users want to set a property after the constructor.
3. Add a differently named getter which always returns the attribute type.

I've ran into this type of situation many times now. I would really like to go with option 2 under the assumption that it is uncommon to set properties after the constructor.