proposal: Go 2: string interpolation evaluating to string and list of expressions

[Cookie04DE]

Author background

• Would you consider yourself a novice, intermediate, or experienced Go programmer?
  Experienced.
• What other languages do you have experience with?
  Java, C#, Kotlin, JavaScript

Related proposals

• Has this idea, or one like it, been proposed before?
  proposal: Go 2: string interpolation #34174 proposes string interpolation similar to other languages like C# and Kolin.
  • If so, how does this proposal differ?
    Instead of formatting on the language level I propose to add a language features that increases readability and maintainability but leaves the actual formatting up to the user's choosing.

Proposal

• What is the proposed change?
  I propose to add "format strings" which are expressions that return a string and []interface{}. They are intended to be used with fmt.Printf and friends. They accept expressions and return the string without the expressions and the results of those in a slice in the order they appeared in.

• Who does this proposal help, and why?
  Consider the following example:
  emailBody := fmt.Sprintf("Hello %s. The item %s you bookmarked on %s is available now. Consider purchasing now (%v) since there are only %d left.", name, product, date, whatever, amount)
  Can you tell at a glance which expression would go inside the brackets?
  If you can: Great; but I would expect many people to take longer.
  Now what if we wanted to change the email? Imagine we wanted the bracket and content to be at the end of the message. Now we have to change the actual string but also take care to remove the correct expression on the right and place it at the end; we have to change two places and keep them in sync. This adds cognitive overhead and is an easy source of bugs (although the tooling helps here).
  Consider how the email looks with the new syntax:
  emailBody := fmt.Sprintf($"Hello %s{name}. The item %s{product} you bookmarked on %s{date} is available now. Consider purchasing now (%v{whatever}) since there are only %d{amount}} left")
  Moving the brackets and content is very easy now and you can tell quickly what gets printed here.

• Please describe as precisely as possible the change to the language.
  I propose to add the following new Go expression: $""
  The runes between the two quotation marks are to be treated like a string inside the regular quotation marks ("") with the exception of the opening and closing curly braces ({ and }). In between those there has to be a valid Go expression that evaluates to a single value (statements aren't allowed).
  If a literal open or closed curly bracket is needed the following two escape sequences are usable: \{ and \}. These are invalid anywhere else.
  I call these formatting strings (FS's for short).
  FS's are expressions that can be either be passed as arguments to a function or be assigned to variables.
  As function arguments:
  The function has to have the following two types as its parameters where the FS is inserted: string and []interface{} or string and ...interface{}. If the last parameter is variadic the FS has to be the last argument and no further may follow.
  As variable assignment:
  The FS has to be assigned to two variables of the following types: string and []interface{}.

The returned string contains everything inside the quotation marks besides the curly brackets and their contents. For example: For $"Hi, I am %s{name} and %d{age} years old" it is "Hi, I am %s and %d years old", while the slice contains the values of name and age.
The slice is never nil but can be empty if no expressions were provided.

• What would change in the language spec?
  It would include the new quotation marks, the two types they evaluate to and explain in which contexts they may be used. As well as the requirements for their use in arguments and variable assignments.
  But most importantly it wouldn't need to even mention the fmt package, since the actual formatting isn't done by the language itself. This also adds flexibility since the processing code doesn't need to be the fmt package.
  Take this sql statement as an example: dbConn.QueryContext(context.Background, $"SELECT name FROM app_user WHERE email = $1{email} AND profile_type = $2{profileType}").
• Please also describe the change informally, as in a class teaching Go.
  Instead of manually writing the format for the sprintf family of functions, we can use format strings which help maintain readability and are easier to modify. They behave just like normal strings with the exception that you can add expressions inside curly braces in them. Because the expressions are right next to where they are used you can easily copy and paste or move them without worrying about accidentally affecting the other expressions.
• Is this change backward compatible?
  Yes.
  • Before
    fmt.Printf("Hello Gopher %s, you are %d years old and you're favorite food is %s", name, age, favoriteFood)
  • After
    fmt.Printf($"Hello Gopher %s{name}, you are %d{years} old and you're favorite food is %s{favoriteFood}")
• Orthogonality: how does this change interact or overlap with existing features?
  There are no such features at language level.

Costs

• Would this change make Go easier or harder to learn, and why?
  It would make Go slightly harder to learn since every new feature has to be learned and understood.
• What is the cost of this proposal? (Every language change has a cost).
  The cost is the added complexity in the compiler and the tooling.
• How many tools (such as vet, gopls, gofmt, goimports, etc.) would be affected?
  All of them, since the change introduces new syntax which need to be recognized. Gofmt also needs to format the expressions inside the curly braces.
• What is the compile time cost?
  The compiler needs to be able to recognize the new quotes and treat the text inside the curly braces as Go expressions instead of pure text.
• What is the run time cost?
  There is no extra runtime cost since the functionality is identical to the existing way.
• Can you describe a possible implementation?
  Since the new syntax is merely cosmetic a transpiler could simply convert it to the old one and compile the result with the current compiler similar to how generic code was handled previously.
• Do you have a prototype? (This is not required.)
  https://github.com/Cookie04DE/gof

[jfesler]

Would this proposal also cover $` (where ` can easy embed double quotes and use multiline strings), or only $"?

[Cookie04DE]

I didn't initially think about the multiline strings but yes, I think the proposal should cover them too.
Although I am not quite sure how you would handle literal curly braces in them, since they don't work with escape sequences like normal double quotes do.

[slycrel]

Thank you for kick-starting this discussion again, and for your proposal!

A few thoughts.

First... I would want to see having %v be an unspecified default. This would essentially allow the assumption of value-based string output, with a possible override if the type needs to be specified. It's also consistent with the existing fmt intent:

For each Printf-like function, there is also a Print function that takes no format and is equivalent to saying %v for every operand.

So in your example:

emailBody := fmt.Sprintf($"Hello %s{name}. The item %s{product} you bookmarked on %s{date} is available now. Consider purchasing now (%v{whatever}) since there are only %d{amount} left")

could be

emailBody := fmt.Sprintf($"Hello %{name}. The item %{product} you bookmarked on %{date} is available now. Consider purchasing now (%{whatever}) since there are only %d{amount} left")

to get the same results.

Second... if this were done as a string and []interface{} combo at the language level via $"", a slight tweak of implementing string and []string* would allow the compiler to optimize/concat that directly... right? If we're going to all the trouble to implement string and []interface{}, it might be worth exploring that a little further and instead create a specific interface that is always resolvable to a string. Maybe that's a tangent, but I think worth bringing up, as this proposal makes string interpolation a second class citizen, with still needing first class language changes.

Third... I'm a little unclear on what this does to { and } within existing strings, and if escaping those only apply to format strings or all strings. This seems like a backwards compatibility issue if you have to escape braces that would not normally be needed for existing strings, especially once you introduce other string variables into the mix.

Fourth... is it going to be an issue to parse out %...{ at the compiler level to allow dropping in code directly? How would that get parsed there that doesn't have all of the inherent need for the fmt semantics at the compiler level? Maybe I am misunderstanding something.

Honestly I'd like to see string interpolation fully at the language level over something sprintf-based. Can you share the reason you see it being better this way? Is this proposal different to simply overcome the resistance of making it a first class feature of the language itself, as mentioned in the referring proposal?

** or an interface array that derives a string is likely better than a []string directly, I think both could work.

[ianlancetaylor]

I would want to see having %v be an unspecified default.

I don't think that works in the context of this proposal. The proposal is specifically not saying anything at all about format specifiers, which is a good thing. A $"" string evaluates to two values: a string and a []interface. The string will have the {} removed. You can choose to pass these values to fmt.Printf if you like, and that will obviously be the most common use case, but it can be used in other ways as well. So a string like $"%{x}" would evaluate to a string of "%" and a []interface{} with the value of x. It wouldn't make sense to pass that to fmt.Printf.

Note that it doesn't work to use a $"" string with fmt.Print, because it won't put the values at the right point in the string. It only works with fmt.Printf and friends.

I'm a little unclear on what this does to { and } within existing strings

It doesn't do anything.

is it going to be an issue to parse out %...{ at the compiler level to allow dropping in code directly

Yes, it absolutely would be an issue, which is why it is good that this proposal doesn't require that.

Honestly I'd like to see string interpolation fully at the language level over something sprintf-based.

It's not simple. See all the discussion at #34174.

[ianlancetaylor]

Because the curly braces may contain any expression, we need to specify the order of evaluation. I suppose the simplest is to say that expressions within curly braces are evaluated as individual operands as described at https://go.dev/ref/spec#Order_of_evaluation.

It's perhaps unfortunate that this doesn't a mechanism to do simple string interpolation, as in $"name: {firstName} {lastName}". Of course we can do that by writing "name: " + firstName + " " + lastName. But if we add this proposal to the language, we're going to be so close to that simple interpolation that I'm concerned that people are going to keep reaching for it and finding that it is not there.

One possibility would be that $"" also produces a third result which is a []int with the byte offsets into the returned string where each {} was found. Then it becomes possible to write a function to do simple interpolation with fmt.Print style formatting. But then simply passing the $"" string to fmt.Printf doesn't work. And overall having to call a special function for string interpolation is a bit awkward though maybe manageable.

[ALTree]

Changing the base language by introducing a new kind of literal that explodes a string in a way that is closely tailored to a specific standard library function (Printf) and makes little sense in any other context certainly feels weird.

It's basically baking a Sprintf-like "macro" in the language that expand a value into something else for the Printf function's convenience. But these $"" would be in the spec, and thus also exist and be allowed everywhere else, even if they don't really make sense outside the context of Printf calls.

IMO a base language feature (especially at a level this low: we're talking about a new kind of literal, and literals are the lowest, most basic "pieces" of a language in the grammar hierarchy) should make sense in every context, and be generally useful, to be worth adding.

[Cookie04DE]

I disagree that it makes little sense in every other context (see the sql statement as an example), but I agree that it is somewhat limited in its usage. Although I think some kind of language feature is necessary to elegantly solve the problem outlined in my proposal.

[jimmyfrasche]

Since the special kind of string really only makes sense if it's used with a function of a specific signature maybe we can go about this differently and have a special kind of function call with a regular string literal.

Rough sketch:

Something like funcName"literal string" where it must be a string literal with no ().

This would be similar to javascript's tagged template literals https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_literals#tagged_templates

There could be functions in fmt that's like Sprint/Sprintln but with the correct signature.

That wouldn't work with the Printf formatting but the idea could be extended to allow passing extra info to the tag function so that something like fmt.Literalf"{x:d}" would be rewritten by the compiler to fmt.Literalf([]string{"", ""}, []any{x}, []string{"d"}) and in this case would have the same output as fmt.Sprintf("%d", x)