Add initial jq-based templating engine #456

yosifkit · 2025-10-18T00:43:36Z

Based on docker-library/php#1052 and related.

Notably this drops architectures that don't have prebuilt sharp libraries (a dep of ghost that uses libvips and requires a newer version than is in Alpine or Debian to build sharp).

This also rewrites generate-stackbrew-library.sh to use lots of jq instead of bash string replacements. Maybe it is better, maybe it is not. This also brings in Debian suite and Alpine version tags (and then we could have more than one suite/version).

$ diff -u ../official-images/library/ghost <(./generate-stackbrew-library.sh)
--- ../official-images/library/ghost    2025-10-17 17:11:55.328891433 -0700
+++ /dev/fd/63  2025-10-17 17:12:23.172882941 -0700
@@ -1,26 +1,23 @@
-# this file is generated via https://github.com/docker-library/ghost/blob/2237b8ab43ce6b33ea5dbcf4c446c3fbdcea90cb/generate-stackbrew-library.sh
+# this file is generated via https://github.com/docker-library/ghost/blob/2d2ccdaa39a356e261f978a208aab1b5ab49689d/generate-stackbrew-library.sh

 Maintainers: Tianon Gravi <[email protected]> (@tianon),
              Joseph Ferguson <[email protected]> (@yosifkit),
              Austin Burdine <[email protected]> (@acburdine)
 GitRepo: https://github.com/docker-library/ghost.git
+GitCommit: 2d2ccdaa39a356e261f978a208aab1b5ab49689d

-Tags: 6.3.1, 6.3, 6, latest
-Architectures: amd64, arm32v7, arm64v8, ppc64le, s390x
-GitCommit: d9131c1c24594d782e0ddac3a98daae9237a1202
-Directory: 6/debian
+Tags: 6.4.0-bookworm, 6.4-bookworm, 6-bookworm, bookworm, 6.4.0, 6.4, 6, latest
+Directory: 6/bookworm
+Architectures: amd64, arm32v7, arm64v8, s390x

-Tags: 6.3.1-alpine, 6.3-alpine, 6-alpine, alpine
-Architectures: amd64, arm32v6, arm32v7, arm64v8
-GitCommit: d9131c1c24594d782e0ddac3a98daae9237a1202
-Directory: 6/alpine
+Tags: 6.4.0-alpine3.22, 6.4-alpine3.22, 6-alpine3.22, alpine3.22, 6.4.0-alpine, 6.4-alpine, 6-alpine, alpine
+Directory: 6/alpine3.22
+Architectures: amd64, arm64v8

-Tags: 5.130.5, 5.130, 5
-Architectures: amd64, arm32v7, arm64v8, ppc64le, s390x
-GitCommit: fdba3d80f50da610007165f5fe46f9b8af69764b
-Directory: 5/debian
+Tags: 5.130.5-bookworm, 5.130-bookworm, 5-bookworm, 5.130.5, 5.130, 5
+Directory: 5/bookworm
+Architectures: amd64, arm32v7, arm64v8, s390x

-Tags: 5.130.5-alpine, 5.130-alpine, 5-alpine
-Architectures: amd64, arm32v6, arm32v7, arm64v8
-GitCommit: fdba3d80f50da610007165f5fe46f9b8af69764b
-Directory: 5/alpine
+Tags: 5.130.5-alpine3.22, 5.130-alpine3.22, 5-alpine3.22, 5.130.5-alpine, 5.130-alpine, 5-alpine
+Directory: 5/alpine3.22
+Architectures: amd64, arm64v8

tianon

Still working through it, but figured starting with this is probably helpful to you.

tianon · 2025-10-20T17:59:02Z

Dockerfile.template

+{{ if env.version | endswith("-rc") then ( -}}
+	case "$GHOST_VERSION" in *-alpha* | *-beta* | *-rc*) installCmd="$installCmd --channel next" ;; esac; \
+{{ ) else "" end -}}


We could probably do this unilaterally for the -rc and not have the case anymore, but avoiding the change for now is fine too 👍

tianon · 2025-10-20T18:15:09Z

apply-templates.sh

+[ -f versions.json ] # run "versions.sh" first
+
+cd "$(dirname "$(readlink -f "$BASH_SOURCE")")"


We do the -f as a rough sanity check so that we don't have to do the cd 👀
(otherwise they're checking different directories entirely 😅)

Suggested change

[ -f versions.json ] # run "versions.sh" first

cd "$(dirname "$(readlink -f "$BASH_SOURCE")")"

[ -f versions.json ] # run "versions.sh" first

I kept the cd as a defense in depth for the rm -rf $version to not accidentally delete something unexpected. But yeah, the order is wrong.

tianon · 2025-10-20T18:16:42Z

apply-templates.sh

+	if [ -d "$version" ]; then
+		rm -rf "$version"
+	fi


Suggested change

if [ -d "$version" ]; then

rm -rf "$version"

fi

rm -rf "$version"

-f will make sure this doesn't fail when it doesn't exist,

and if it exists but isn't a directory, it would sometimes fail later (depending on the next conditional) so it's easier to manage/think about the drift if we just enforce the state at this level

Suggested change

if [ -d "$version" ]; then

rm -rf "$version"

fi

rm -rf "$version"

-f will make sure this doesn't fail when it doesn't exist,

and if it exists but isn't a directory, it would sometimes fail later (depending on the next conditional) so it's easier to manage/think about the drift if we just enforce the state at this level

Did you decide against this change? 😇

🤷 I don't like either. But I don't have a better solution to protect against bad input. Certain software of the past nuking user's home directories makes me nervous of rm -rf "$var".

Ah, from that perspective rm -rf "$var" is the safest version we could possibly have, and the rm -rf "$var/" I often do is the unsafe version, because if $var ends up empty somehow (and doesn't trigger set -u), we'll run rm -rf '' which arguably should error, but in practice does nothing successfully instead:

$ ls bar foo $ rm -rf ''; echo $? 0 $ ls bar foo

(Those issues come from combining variables in rm -rf like rm -rf "$somedir/$somefile")

which arguably should error

From the perspective of "empty is an invalid filename" it's probably sane that it doesn't though 🤷

$ touch '' touch: cannot touch '': No such file or directory

tianon · 2025-10-20T18:18:42Z

generate-stackbrew-library.sh

-			| xargs -r bashbrew cat --format '["{{ .RepoName }}:{{ .TagName }}"]="{{ join " " .TagEntry.Architectures }}"'
-	)"
-	eval "declare -g -A parentRepoToArches=( $parentRepoToArchesStr )"
+	: "${BASHBREW_LIBRARY:-https://github.com/docker-library/official-images/raw/HEAD/library}/"


Without an export this doesn't apply, and we don't actually support a URL path in BASBHREW_LIBRARY (we assume it points to a directory), which is why it was implemented the way it was in the previous version of this function. 😅

Ah, I was over aggressive on the removal of the find + awk. After testing with BASHBREW_LIBRARY set and unset, I have made an adjustment (coming soon).

tianon · 2025-10-20T18:23:02Z

generate-stackbrew-library.sh

-getArches 'ghost'
+parentArchesJson="$(getArchesJson 'ghost')"
+
+commit="$(fileCommit '*/**')" #


Was this # for a TODO of some kind or something? 😅

tianon · 2025-10-20T18:24:20Z

generate-stackbrew-library.sh

+	jq --raw-output '.[].variants[].from' versions.json \
+		| sort -u \


Suggested change

jq --raw-output '.[].variants[].from' versions.json \

| sort -u \

jq --raw-output '[ .[].variants[].from ] | unique[]' versions.json \

tianon · 2025-10-20T18:25:20Z

generate-stackbrew-library.sh

+
+	jq --raw-output '.[].variants[].from' versions.json \
+		| sort -u \
+		| xargs -r bashbrew cat --format '{ "{{ .RepoName }}:{{ .TagName }}": {{ json .TagEntry.Architectures }} }' \


Suggested change

| xargs -r bashbrew cat --format '{ "{{ .RepoName }}:{{ .TagName }}": {{ json .TagEntry.Architectures }} }' \

| xargs -r bashbrew cat --format '{ {{ join ":" .RepoName .TagName | json }}: {{ json .TagEntry.Architectures }} }' \

tianon · 2025-10-20T18:25:46Z

generate-stackbrew-library.sh

+	jq --raw-output '.[].variants[].from' versions.json \
+		| sort -u \
+		| xargs -r bashbrew cat --format '{ "{{ .RepoName }}:{{ .TagName }}": {{ json .TagEntry.Architectures }} }' \
+		| jq --compact-output --null-input '[ inputs ] | add'


This is just slurp 😅

Suggested change

| jq --compact-output --null-input '[ inputs ] | add'

| jq --compact-output --slurp 'add'

tianon · 2025-10-20T18:30:06Z

generate-stackbrew-library.sh

-		EOE
-	done
-done
+jq --raw-output --argjson parentArches "$parentArchesJson" '


IMO this is long enough that we should move the body of it into a separate generate-stackbrew-library.jq file so we get nice syntax highlighting (from our editors and GitHub). 😇

exec jq \ --raw-output \ --argjson parentArches "$parentArchesJson" \ --from-file generate-stackbrew-library.jq \ versions.json \ --args -- "$@"

tianon · 2025-10-20T18:49:18Z

generate-stackbrew-library.sh

+	# returns a list of tags for this variant
+	# usage: variant_tags("version"; [sortedVersionList]; "variant"; [sortedVariantList])
+	# e.g. variant_tags("1.2.3.4"; ["1.2.3.4", "0.1.2.3"]; "trixie"; ["trixie", "alpine3.22"]) -> ["1.2.3.4-trixie", "1.2.3-trixie", ...]
+	def variant_tags($version; $versionListSorted; $variant; $variantListSorted):


Given that each of these are only used once, what's the benefit to having functions over just embedding the logic below? 😅

I was actually trying to not embed all the logic inline so that it could be easier to understand and simpler to extend to other images. But it has kind of come back into basically one function. I think that it could be improved to be more generic and useful to any image repo we maintain.

Personally I find it a little harder to read this way and I'm not convinced it will actually be as reusable as we want it to be in practice, but I don't feel strongly about it and I'm willing to give it a chance. 😄 👍

tianon · 2025-10-22T22:17:25Z

I checked this out locally so I could redo your original diff now that it's had some changes/updates, and bashbrew list so the tag changes are easier to scan:

`bashbrew cat`:

$ diff -u <(bashbrew cat ghost) <(bashbrew cat <(./generate-stackbrew-library.sh))
--- /dev/fd/63	2025-10-22 15:15:17.266193532 -0700
+++ /dev/fd/62	2025-10-22 15:15:17.266193532 -0700
@@ -1,22 +1,19 @@
 Maintainers: Tianon Gravi <[email protected]> (@tianon), Joseph Ferguson <[email protected]> (@yosifkit), Austin Burdine <[email protected]> (@acburdine)
 GitRepo: https://github.com/docker-library/ghost.git
+GitCommit: 83ce25215e22a36c646c0a6165a531f12d92339f
 
-Tags: 6.5.0, 6.5, 6, latest
-Architectures: amd64, arm32v7, arm64v8, ppc64le, s390x
-GitCommit: f952f7615274894d6efb0fbc765f5368580fb39a
-Directory: 6/debian
+Tags: 6.5.0-bookworm, 6.5-bookworm, 6-bookworm, bookworm, 6.5.0, 6.5, 6, latest
+Architectures: amd64, arm32v7, arm64v8, s390x
+Directory: 6/bookworm
 
-Tags: 6.5.0-alpine, 6.5-alpine, 6-alpine, alpine
-Architectures: amd64, arm32v6, arm32v7, arm64v8
-GitCommit: f952f7615274894d6efb0fbc765f5368580fb39a
-Directory: 6/alpine
+Tags: 6.5.0-alpine3.22, 6.5-alpine3.22, 6-alpine3.22, alpine3.22, 6.5.0-alpine, 6.5-alpine, 6-alpine, alpine
+Architectures: amd64, arm64v8
+Directory: 6/alpine3.22
 
-Tags: 5.130.5, 5.130, 5
-Architectures: amd64, arm32v7, arm64v8, ppc64le, s390x
-GitCommit: fdba3d80f50da610007165f5fe46f9b8af69764b
-Directory: 5/debian
+Tags: 5.130.5-bookworm, 5.130-bookworm, 5-bookworm, 5.130.5, 5.130, 5
+Architectures: amd64, arm32v7, arm64v8, s390x
+Directory: 5/bookworm
 
-Tags: 5.130.5-alpine, 5.130-alpine, 5-alpine
-Architectures: amd64, arm32v6, arm32v7, arm64v8
-GitCommit: fdba3d80f50da610007165f5fe46f9b8af69764b
-Directory: 5/alpine
+Tags: 5.130.5-alpine3.22, 5.130-alpine3.22, 5-alpine3.22, 5.130.5-alpine, 5.130-alpine, 5-alpine
+Architectures: amd64, arm64v8
+Directory: 5/alpine3.22

`bashbrew list`:

$ diff -u <(bashbrew list ghost | cut -d: -f2) <(bashbrew list <(./generate-stackbrew-library.sh) | cut -d: -f2)
--- /dev/fd/63	2025-10-22 15:16:27.434905448 -0700
+++ /dev/fd/62	2025-10-22 15:16:27.434905448 -0700
@@ -1,14 +1,28 @@
+6.5.0-bookworm
+6.5-bookworm
+6-bookworm
+bookworm
 6.5.0
 6.5
 6
 latest
+6.5.0-alpine3.22
+6.5-alpine3.22
+6-alpine3.22
+alpine3.22
 6.5.0-alpine
 6.5-alpine
 6-alpine
 alpine
+5.130.5-bookworm
+5.130-bookworm
+5-bookworm
 5.130.5
 5.130
 5
+5.130.5-alpine3.22
+5.130-alpine3.22
+5-alpine3.22
 5.130.5-alpine
 5.130-alpine
 5-alpine

tianon · 2025-10-22T22:18:36Z

5/bookworm/Dockerfile

Oh hmm, I didn't notice these renames - this implies that we might support more than one version of a given base distribution -- is that an intentional change? (I certainly wasn't intending to support more than one each here 😅)

Yes and no. I did add variants that could support multiple base distribution versions, but mostly to add extra tags. I don't think we should keep more than one Debian suite or Alpine version per ghost version.

yosifkit · 2025-10-23T20:36:48Z

This force push includes a partial rewrite of generate-stackbrew-library.jq based upon hylang/docker-hylang#7.

tianon

Some versions.sh comments/review 😇

tianon · 2025-10-23T22:09:17Z

versions.sh

+					dist: {
+						linux: [
+							# list of debian versions, in descending order
+							#"trixie", # TODO
+							"bookworm",
+							empty # trailing comma
+						],
+						linuxmusl: [
+							# list of alpine versions, in descending order
+							"3.22",
+							empty
+							| "alpine" + .
+						]
+					}[.[0]],


Relevant to the above conversation about supporting multiple variants, I think it's fine to leave the data model here able to support multiple, but we should update the code filling it in to make it more clear we won't accept more than one (unless something really exceptional happens, I guess), and so that we don't accidentally forget in the future; maybe something like this (very open to alternatives, but this is a straw man)?

Suggested change

dist: {

linux: [

# list of debian versions, in descending order

#"trixie", # TODO

"bookworm",

empty # trailing comma

],

linuxmusl: [

# list of alpine versions, in descending order

"3.22",

empty

| "alpine" + .

]

}[.[0]],

dist: {

# each of these should be a single distro version unless something *really* exceptional happens

linux: [ "bookworm" ],

linuxmusl: [ "alpine3.22" ],

}[.[0]],

tianon · 2025-10-23T22:15:30Z

versions.sh

+				| capture("^ *\"@img/sharp-(?<name>linux[a-z0-9-]+)\" \"[0-9.]+\"$")
+				| .name
+				| split("-")


Given the immediate split to take off more data, couldn't this instead massage the regex a little so that .name is the exact value we need?

Suggested change

| capture("^ *\"@img/sharp-(?<name>linux[a-z0-9-]+)\" \"[0-9.]+\"$")

| .name

| split("-")

| capture("^ *\"@img/sharp-(?<dist>linux[a-z]*)-(?<arch>[a-z0-9]+)\" \"[0-9.]+\"$")

(and then using .dist and .arch instead of .[0] and .[1] below - could even be .dist |= { ... }[.] | .arch |= { ... }[.])

tianon · 2025-10-23T22:19:30Z

versions.sh

+					}[.[1]],
+				}
+			) as $item ({}; .[$item.dist[]].arches += [ $item.arch ])
+			| with_entries(.value.arches |= sort)


This could just be .[].arches |= sort:

$ jq -nc '{ foo: { bar: [ "c", "b", "a" ] } } | .[].bar |= sort' {"foo":{"bar":["a","b","c"]}} $ # still works if we get even more whack: $ jq -nc '{ foo: { bar: [ "c", "b", "a" ] } } | .[][] |= sort' {"foo":{"bar":["a","b","c"]}}

However, the order should already be deterministic from yarn.lock, so I'm not convinced it's necessary for us to sort here. 😅

deterministic from yarn.lock

That puts them in alphabetical order based on the "arch" key, so arm > arm64 > s390x > x64. I don't like amd64 at the end so I've moved the sort up so that it is while the lists are created. This order will also control the order generated by generate-stackbrew-library.sh so that we don't rely on parent arch list order.

tianon · 2025-10-23T22:24:03Z

versions.sh

+	# move alpine variants to the end of the object
+	# debian suites and alpine versions are each relatively sorted already
+	| map(.value.variants |= ( to_entries | sort_by(.key | contains("alpine")) | from_entries))


It feels weird to munge variants on these objects so late instead of doing so above when we're generating them 😅

I'll add a suggestion above for a potential way to resolve this without an explicit sort 👀

tianon · 2025-10-23T22:38:33Z

versions.sh

+			reduce (
+				inputs
+				| capture("^ *\"@img/sharp-(?<name>linux[a-z0-9-]+)\" \"[0-9.]+\"$")
+				| .name
+				| split("-")
+				| {
+					dist: {
+						linux: [
+							# list of debian versions, in descending order
+							#"trixie", # TODO
+							"bookworm",
+							empty # trailing comma
+						],
+						linuxmusl: [
+							# list of alpine versions, in descending order
+							"3.22",
+							empty
+							| "alpine" + .
+						]
+					}[.[0]],
+					arch: {
+						x64: "amd64",
+						arm64: "arm64v8",
+						arm: "arm32v7",
+						s390x: "s390x",
+					}[.[1]],
+				}
+			) as $item ({}; .[$item.dist[]].arches += [ $item.arch ])
+			| with_entries(.value.arches |= sort)


Here's one possibility for fixing the ordering of Debian first, Alpine second without an explicit sort (incorporating the above suggestions too):

Suggested change

reduce (

inputs

| capture("^ *\"@img/sharp-(?<name>linux[a-z0-9-]+)\" \"[0-9.]+\"$")

| .name

| split("-")

| {

dist: {

linux: [

# list of debian versions, in descending order

#"trixie", # TODO

"bookworm",

empty # trailing comma

],

linuxmusl: [

# list of alpine versions, in descending order

"3.22",

empty

| "alpine" + .

]

}[.[0]],

arch: {

x64: "amd64",

arm64: "arm64v8",

arm: "arm32v7",

s390x: "s390x",

}[.[1]],

}

) as $item ({}; .[$item.dist[]].arches += [ $item.arch ])

| with_entries(.value.arches |= sort)

reduce (

inputs

| capture("^ *\"@img/sharp-(?<dist>linux[a-z]*)-(?<arch>[a-z0-9]+)\" \"[0-9.]+\"$")

) as $item ({

# this controls the variant ordering

linux: [], # non-Alpine first

linuxmusl: [], # Alpine second

}; .[$item.dist] += [ $item.arch ])

| with_entries(

select(.value | length > 0)

| .key |= {

# each of these should be a single distro version unless something *really* exceptional happens

linux: [ "bookworm" ][],

linuxmusl: [ "alpine3.22" ][],

}[.]

| .value |= { arches: [ {

x64: "amd64",

arm64: "arm64v8",

arm: "arm32v7",

s390x: "s390x",

}[.[]] ] } # TODO maybe a "// empty" here so we ignore anything we are not explicitly handling? error() would be super reasonable too but would be worth swapping this to ".[] | {...}[.]" or "arches: map(...)" instead so we can include the offending value in the error

)

tianon · 2025-10-23T22:43:20Z

versions.sh

+	to_entries
+
+	# sort by version number, descending
+	| sort_by(.value.version | split("[.-]"; "") | map(tonumber? // .) )


There's a few instances of weird parenthesis "cuddling" throughout this PR (where a parenthesis and its pair don't have the same whitespace) 😅

I don't have a strong opinion on individual cases whether they should be func(foo) or func( foo ) or (foo(bar)) or ( foo(bar) ) or ( foo( bar ) ), but I do have a strong opinion that matching pairs should have the same whitespace at each end, and otherwise am fine with whatever's most readable/ergonomic (I personally usually default to "tight" unless things look exceptionally hard to read, but that's often my clue that I need newlines instead 😅)

tianon · 2025-10-23T22:43:31Z

versions.sh

+	| map(.value.variants |= ( to_entries | sort_by(.key | contains("alpine")) | from_entries))
+
+	| from_entries
+'> versions.json


Suggested change

'> versions.json

' > versions.json

tianon · 2025-10-23T22:43:45Z

versions.sh

+	)"
+
+	export fullVersion cliVersion
+	json="$(jq <<<"$json" -c --argjson doc "$doc" '


Suggested change

json="$(jq <<<"$json" -c --argjson doc "$doc" '

json="$(jq <<<"$json" --compact-output --argjson doc "$doc" '

tianon

fresh generate comments 😇

tianon · 2025-10-23T23:03:07Z

generate-stackbrew-library.jq

+| ( $allVariants | map(select(contains("alpine"))))[0] as $latestAlpine
+| ( $allVariants | map(select(contains("alpine") | not)))[0] as $latestDebian


Suggested change

| ( $allVariants | map(select(contains("alpine"))))[0] as $latestAlpine

| ( $allVariants | map(select(contains("alpine") | not)))[0] as $latestDebian

| first( $allVariants[] | select(contains("alpine")) ) as $latestAlpine

| first( $allVariants[] | select(contains("alpine") | not) ) as $latestDebian

I think this is actually slightly more efficient inside jq if we redo the .[].variants | keys_unsorted[] inside the first() because it won't complete the full iteration and will actively short-circuit, but readability definitely suffers if we do that 😅

tianon · 2025-10-23T23:04:12Z

generate-stackbrew-library.jq

@@ -0,0 +1,73 @@
+# latest version is the first one since it is already in version order
+keys_unsorted[0] as $latestMajor


Is this still true if we have pre-releases? Don't we need to filter them out? ie:

Suggested change

keys_unsorted[0] as $latestMajor

first(keys_unsorted[] | select(endswith("-rc") | not)) as $latestMajor

Oh, and doesn't that also have issues if we have, for example, a 7-rc and 7 is null? So this needs to account for the value too?

Suggested change

keys_unsorted[0] as $latestMajor

first(to_entries[] | select(.value and (.key | endswith("-rc") | not)).key) as $latestMajor

tianon · 2025-10-23T23:12:29Z

generate-stackbrew-library.jq

+
+
+# loop over major versions
+| to_entries[]


tianon · 2025-10-23T23:14:54Z

generate-stackbrew-library.jq

+# latest version is the first one since it is already in version order
+keys_unsorted[0] as $latestMajor
+# list of all variants in version.json order (debian suites followed by alpine versions, each in descending order)
+| [ .[].variants | keys_unsorted[] ] as $allVariants


I guess arguably, this should actually be .[$latestMajor].variants explicitly, right? Because we don't want to "default" any other aliases (like if a pre-release version had a new Alpine or Debian version that the latest release doesn't have yet). That also makes it less useful/meaningful/DRY to have $allVariants because that's a pretty static lookup. 😅

... and yes, this is in direct contrast to my suggestion that we move up the to_entries because then we don't have trivial lookup here unless we save the entire $latestMajor key/value pair instead of just the key 😂

So maybe that'd look something like this?

to_entries | first(.[] | select(.value and (.key | endswith("-rc") | not))) as $latestMajor | first( $latestMajor.value.variants[] | select(contains("alpine")) ) as $latestAlpine | first( $latestMajor.value.variants[] | select(contains("alpine") | not) ) as $latestDebian | $latestMajor.key as $latestMajor | .[] | .key as $major | .value ...

tianon · 2025-10-23T23:37:52Z

generate-stackbrew-library.jq

+# e.g. "1.2.3.4" -> [ "1.2.3.4", "1.2.3", "1.2", "1", "" ]
+| (
+	.version
+	# TODO improve rc suffix support (e.g. this won't work for -beta or -rc.1 versioned software)


Maybe this is really something like if $major | endswith("-rc") then [ ., $major ] else ... instead?

tianon · 2025-10-23T23:39:27Z

generate-stackbrew-library.jq

+			if $variant == $latestAlpine then
+				($variant | sub("[0-9.]+$"; "")) # "alpine3.22" -> "alpine"
+			else empty end,


This is a really verbose way to spell "alpine" 😅

Why the sub? 🤔

Suggested change

if $variant == $latestAlpine then

($variant | sub("[0-9.]+$"; "")) # "alpine3.22" -> "alpine"

else empty end,

if $variant == $latestAlpine then "alpine" else empty end,

yosifkit · 2025-10-28T17:49:04Z

Recent force push simplifies the optional dependencies logic by just ensuring that their system dependencies are installed so that they succeed to install during the ghost install. Then we don't need any extra yarn add that mess up the other dependencies like Error: Cannot find module 'wrap-ansi'.

(Working on the build failure: Error: Cannot find module 'sqlite3')

yosifkit · 2025-10-29T21:39:35Z

Dockerfile.template

+	gosu --version; \
+	gosu nobody true
+{{ if is_alpine then ( -}}
+RUN set -eux; ln -svf gosu /usr/local/bin/su-exec; su-exec nobody true # backwards compatibility (TODO remove in Ghost 6+)


This was added in #417. Should we follow the comment and drop it for the 6+ images or wait for a different release? In either case, we should adjust the template to do it automatically, so we don't forget again.

Oof, oops. I think maybe we missed the boat? or maybe it's still defensible? I'm OK with either direction, but yeah, we should fix the template so we actually do it either now or later.

I went with the "remove it at the next release" on both 5 and 6. The image has been using gosu in Alpine images for over a year so users have had time to migrate off of su-exec.

yosifkit · 2025-10-29T22:32:30Z

Dockerfile.template

+	apk add --no-cache --virtual .build-deps-ghost g++ linux-headers make python3 py3-setuptools; \
+{{ ) else ( -}}
+	savedAptMark="$(apt-mark showmanual)"; \
+	apt-get update; \
+	apt-get install -y --no-install-recommends g++ make python3; \


Possible TODO: add libvips-dev or equivalent package to build deps plus a scanelf/ldd dance to keep linked libraries. This would allow the ghost install line to try building libvips on architectures that don't have prebuilt sharp/vips linking. And would maybe let us add back arm32vX Alpine images and ppc64le for Debian if we are on Trixie. That assumes that the sharp version in ghost's yarn.lock does not change from v0.34.2 because anything newer requires a newer libvips than either Trixie or Alpine 3.22 have.

The scanelf/ldd dance is complicated by the fact that some node modules contain libraries not valid for the current user space so you have things linked against libc.musl-armv7.so.1, libc.so.6, libresolv.so.2, and libvips-cpp.so.8.16.1 in an Alpine container and so they fail when passed to apk add --no-network:

+ apk add --no-network --virtual .ghost-rundeps so:ld-linux-armhf.so.3 so:libc.musl-armv7.so.1 so:libc.so.6 so:libdl.so.2 so:libgcc_s.so.1 so:libglib-2.0.so.0 so:libgobject-2.0.so.0 so:libm.so.6 so:libpthread.so.0 so:libresolv.so.2 so:libstdc++.so.6 so:libvips-cpp.so.42 so:libvips-cpp.so.8.16.1 so:libvips.so.42 ... so:ld-linux-armhf.so.3 (no such package): required by: .ghost-rundeps-20251029.205402[so:ld-linux-armhf.so.3] so:libc.so.6 (no such package): required by: .ghost-rundeps-20251029.205402[so:libc.so.6] so:libdl.so.2 (no such package): required by: .ghost-rundeps-20251029.205402[so:libdl.so.2] so:libm.so.6 (no such package): required by: .ghost-rundeps-20251029.205402[so:libm.so.6] so:libpthread.so.0 (no such package): required by: .ghost-rundeps-20251029.205402[so:libpthread.so.0] so:libresolv.so.2 (no such package): required by: .ghost-rundeps-20251029.205402[so:libresolv.so.2] so:libvips-cpp.so.8.16.1 (no such package): required by: .ghost-rundeps-20251029.205402[so:libvips-cpp.so.8.16.1]

I feel all kinds of ways reading this, especially that latter half.

Given the historically aggressive relationship between Sharp and libvips, I'm very comfortable saying that if Sharp doesn't pre-build for it, we don't support it. I think that's a sane stance that still gets us a good, reasonable set of supported architectures, and even more importantly a set that is arguably supported by upstream (Sharp).

tianon

LGTM (some minor comments, but nothing worth blocking on and I'm happy if you want to merge this as-is)

tianon · 2025-10-31T18:03:20Z

Dockerfile.template

+	apt-mark showmanual | xargs apt-mark auto > /dev/null; \
+	[ -z "$savedAptMark" ] || apt-mark manual $savedAptMark; \
+	apt-get purge -y --auto-remove; \
+	rm -rf /var/lib/apt/lists/*; \


Suggested change

apt-mark showmanual | xargs apt-mark auto > /dev/null; \

[ -z "$savedAptMark" ] || apt-mark manual $savedAptMark; \

apt-get purge -y --auto-remove; \

rm -rf /var/lib/apt/lists/*; \

apt-mark auto '.*' > /dev/null; \

[ -z "$savedAptMark" ] || apt-mark manual $savedAptMark > /dev/null; \

apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false; \

{{ clean_apt }}; \

tianon · 2025-10-31T18:03:56Z

Dockerfile.template

+{{ ) else ( -}}
+	apt-mark auto '.*' > /dev/null; \
+	[ -z "$savedAptMark" ] || apt-mark manual $savedAptMark > /dev/null; \
+	apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false; \


We should technically move clean_apt down here:

Suggested change

apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false; \

apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false; \

{{ clean_apt }}; \

tianon · 2025-10-31T18:05:58Z

Dockerfile.template

+WORKDIR $GHOST_INSTALL
+VOLUME $GHOST_CONTENT
+
+COPY docker-entrypoint.sh /usr/local/bin


Suggested change

COPY docker-entrypoint.sh /usr/local/bin

COPY docker-entrypoint.sh /usr/local/bin/

tianon · 2025-10-31T18:09:15Z

apply-templates.sh

+	if [ -d "$version" ]; then
+		rm -rf "$version"
+	fi


Suggested change

if [ -d "$version" ]; then

rm -rf "$version"

fi

rm -rf "$version"

-f will make sure this doesn't fail when it doesn't exist,

and if it exists but isn't a directory, it would sometimes fail later (depending on the next conditional) so it's easier to manage/think about the drift if we just enforce the state at this level

Did you decide against this change? 😇

tianon · 2025-10-31T18:12:39Z

generate-stackbrew-library.jq

+
+| (
+	"", # newline between library entries
+	"Tags: \( $tags | join(", "))",


Suggested change

"Tags: \( $tags | join(", "))",

"Tags: \($tags | join(", "))",

or

Suggested change

"Tags: \( $tags | join(", "))",

"Tags: \( $tags | join(", ") )",

tianon · 2025-10-31T20:20:45Z

generate-stackbrew-library.sh

-getArches 'ghost'
+parentArchesJson="$(getArchesJson 'ghost')"

+commit="$(fileCommit '*/**')" # last commit that changed files related to a build context


Oh, this matches .github/**. Hmm.

Suggested change

commit="$(fileCommit '*/**')" # last commit that changed files related to a build context

commit="$(git log -1 --format='format:%H' HEAD -- '[^.]*/**')" # last commit that changed files related to a build context

tianon · 2025-10-31T20:28:34Z

generate-stackbrew-library.sh

 cat <<-EOH
 # this file is generated via https://github.com/docker-library/ghost/blob/$(fileCommit "$self")/$self


If we do this too, we can drop fileCommit entirely:

Suggested change

cat <<-EOH

# this file is generated via https://github.com/docker-library/ghost/blob/$(fileCommit "$self")/$self

selfCommit="$(git log -1 --format='format:%H' HEAD -- "$self")"

cat <<-EOH

# this file is generated via https://github.com/docker-library/ghost/blob/$selfCommit/$self

tianon reviewed Oct 20, 2025

View reviewed changes

yosifkit force-pushed the jq-template branch 2 times, most recently from 95d3768 to 83ce252 Compare October 22, 2025 15:49

tianon approved these changes Oct 22, 2025

View reviewed changes

tianon reviewed Oct 22, 2025

View reviewed changes

yosifkit force-pushed the jq-template branch from 83ce252 to 94140a5 Compare October 23, 2025 20:34

tianon reviewed Oct 23, 2025

View reviewed changes

yosifkit force-pushed the jq-template branch from 94140a5 to 20c6047 Compare October 24, 2025 22:24

tianon mentioned this pull request Oct 27, 2025

Rename update.sh/target-pipeline to legacy-pipeline docker-library/oi-janky-groovy#48

Merged

yosifkit force-pushed the jq-template branch from 20c6047 to bc1626c Compare October 28, 2025 17:38

yosifkit force-pushed the jq-template branch 2 times, most recently from d4633e7 to 616959c Compare October 28, 2025 17:58

yosifkit marked this pull request as ready for review October 28, 2025 18:30

yosifkit commented Oct 29, 2025

View reviewed changes

yosifkit force-pushed the jq-template branch from 616959c to 009abb9 Compare October 30, 2025 23:30

tianon approved these changes Oct 31, 2025

View reviewed changes

tianon reviewed Oct 31, 2025

View reviewed changes

Add initial jq-based templating engine

dd86bb1

yosifkit force-pushed the jq-template branch from 009abb9 to dd86bb1 Compare October 31, 2025 23:18

		[ -f versions.json ] # run "versions.sh" first

		cd "$(dirname "$(readlink -f "$BASH_SOURCE")")"

		jq --raw-output '.[].variants[].from' versions.json \
		\| sort -u \

	jq --raw-output '.[].variants[].from' versions.json \
	\| sort -u \
	jq --raw-output '[ .[].variants[].from ] \| unique[]' versions.json \

	\| xargs -r bashbrew cat --format '{ "{{ .RepoName }}:{{ .TagName }}": {{ json .TagEntry.Architectures }} }' \
	\| xargs -r bashbrew cat --format '{ {{ join ":" .RepoName .TagName \| json }}: {{ json .TagEntry.Architectures }} }' \

	\| jq --compact-output --null-input '[ inputs ] \| add'
	\| jq --compact-output --slurp 'add'

	json="$(jq <<<"$json" -c --argjson doc "$doc" '
	json="$(jq <<<"$json" --compact-output --argjson doc "$doc" '

		\| ( $allVariants \| map(select(contains("alpine"))))[0] as $latestAlpine
		\| ( $allVariants \| map(select(contains("alpine") \| not)))[0] as $latestDebian

		@@ -0,0 +1,73 @@
		# latest version is the first one since it is already in version order
		keys_unsorted[0] as $latestMajor

	keys_unsorted[0] as $latestMajor
	first(keys_unsorted[] \| select(endswith("-rc") \| not)) as $latestMajor

	keys_unsorted[0] as $latestMajor
	first(to_entries[] \| select(.value and (.key \| endswith("-rc") \| not)).key) as $latestMajor

	apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false; \
	apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false; \
	{{ clean_apt }}; \

	COPY docker-entrypoint.sh /usr/local/bin
	COPY docker-entrypoint.sh /usr/local/bin/

	"Tags: \( $tags \| join(", "))",
	"Tags: \($tags \| join(", "))",

	"Tags: \( $tags \| join(", "))",
	"Tags: \( $tags \| join(", ") )",

	commit="$(fileCommit '/*')" # last commit that changed files related to a build context
	commit="$(git log -1 --format='format:%H' HEAD -- '[^.]/*')" # last commit that changed files related to a build context

		cat <<-EOH
		# this file is generated via https://github.com/docker-library/ghost/blob/$(fileCommit "$self")/$self

Add initial jq-based templating engine #456

Are you sure you want to change the base?

Add initial jq-based templating engine #456

Uh oh!

Conversation

yosifkit commented Oct 18, 2025

Uh oh!

tianon left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tianon commented Oct 22, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yosifkit commented Oct 23, 2025

Uh oh!

tianon left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tianon left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!