text/template: JSEscape generates invalid JSON

[sding3]

What version of Go are you using (go version)?

$ go version
go version go1.14 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/shang/.cache/go-build"
GOENV="/home/shang/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY="*.epic.com"
GONOSUMDB="*.epic.com"
GOOS="linux"
GOPATH="/home/shang/go"
GOPRIVATE="*.epic.com"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build973203171=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Ran this code:

package main

import (
        "log"
        "os"
        "text/template"
)

func main() {
        const plate = `"{{ .Value | js}}"`

        type Vals struct {
                Value string
        }

        vals := &Vals{`=`}

        t := template.Must(template.New("plate").Parse(plate))

        err := t.Execute(os.Stdout, vals)
        if err != nil {
                log.Println("executing template:", err)
        }

}

What did you expect to see?

Expected to see "=", which is the behavior in go1.13.8

What did you see instead?

Insead see "\x3D".

[FiloSottile]

This is intentional as security hardening. The semantic meaning of the two values are the same, so it should not break anything but tests. Escaping those extra characters makes the output robust against misuse, which was reported in the wild.

See #35665 and 94e9a5e.

[mheggeseth]

However "\x3D" is not valid JSON so the js function can no longer be used to escape strings when templating a JSON document. Is there an alternative?

"\u003D" would have been valid in both code and JSON.

[FiloSottile]

JSEscape was already generating \x3C and \x3E so it sounds like it was never valid JSON? It would be nicer if it worked for both, though. @empijei, any thoughts?

[mheggeseth]

JSEscape was already generating \x3C and \x3E so it sounds like it was never valid JSON?

We were just lucky not to hit those characters in our use case which is writing a JSON document with values that are PEM-formatted certificates (< and > aren't in the Base64 character set).

[empijei]

Interesting issue.

I confirm this is intended behavior as JSEscape was only designed to be used to escape JS and not JSON. The fact that you were lucky not to hit escaping with b64 was just a coincidence and was not intended.

That said I undestand your pain and I would like to ask for a little bit more information before I try to address this:

• Why are you using text/template to format JSON? If this has to be served as a webpage it would be better to use html/template with autoescaping: https://play.golang.org/p/0YXfGQooTq0 this will correctly print this as a "="
• When you say "templating a JSON document" what do you mean? Is there any reasons why you are not using the encoding/json package?

[gopherbot]

Change https://golang.org/cl/226097 mentions this issue: html/template: switch to unicode escape instea of hex

[lpar]

Note that JSEscape still generates invalid JSON:

	value := `"I said 'No!'"`
	var buf bytes.Buffer
	template.JSEscape(&buf, []byte(value))
	fmt.Println(buf.String()) // => \"I said \'No!\'\"

I'm guessing maybe @sding3 is wanting to template JSON for the same reason I do, for generating Slack messages?

[sding3]

@lpar , we were using it for escaping PEM encoded certificates. We ended up writing up our own custom template function to do JSON escapae. Sounds like the standard library could benefit from a JSONEscape function, eh?

[FiloSottile]

Huh, CL 226097 looks like it's replacing ' and " with Unicode escapes, too, but the code indeed changed the escape of = and not ' and ". @empijei?