--runtime-config.file path behaviour is broken in in 1.18

[KrisBuytaert]

Describe the bug

Since my upgrade to cortex 1.18 the --runtime-config.file parameter which on 1.15 correctly worked now prepends /var/lib/cortex/ to the path of the file , this breaks the configuration .

To Reproduce
Steps to reproduce the behavior:

start cortex .. e.g

/usr/local/bin/cortex -target=compactor --config.file /etc/cortex/compactor.yaml --runtime-config.file=/etc/cortex/runtime.yaml
ts=2024-09-18T11:14:49.664309188Z caller=main.go:199 level=info msg="Starting Cortex" version="(version=1.18.0, branch=release-1.18, revision=85511746c)"
ts=2024-09-18T11:14:49.665081612Z caller=server.go:319 level=info http=10.4.0.95:9015 grpc=10.4.0.95:9915 msg="server listening on addresses"
ts=2024-09-18T11:14:49.666242068Z caller=module_service.go:64 level=info msg=initialising module=server
ts=2024-09-18T11:14:49.6664001Z caller=module_service.go:64 level=info msg=initialising module=runtime-config
ts=2024-09-18T11:14:49.666663303Z caller=cortex.go:447 level=error msg="module failed" module=runtime-config err="invalid service state: Failed, expected: Running, failure: failed to load runtime config: read file: open file: stat /var/lib/cortex/etc/cortex/runtime.yaml: stat /var/lib/cortex/etc/cortex/runtime.yaml: no such file or directory"

Expected behavior
A clear and concise description of what you expected to happen.

The 1.15 binary with the same config clearly worked

]$ /usr/local/bin/cortex -target=compactor --config.file /etc/cortex/compactor.yaml --runtime-config.file=/etc/cortex/runtime.yaml
level=info ts=2024-09-18T11:14:14.582945025Z caller=main.go:194 msg="Starting Cortex" version="(version=1.15.3, branch=release-1.15, revision=21e836605)"
level=info ts=2024-09-18T11:14:14.584000908Z caller=server.go:323 http=10.4.0.95:9015 grpc=10.4.0.95:9915 msg="server listening on addresses"
level=info ts=2024-09-18T11:14:14.585541585Z caller=module_service.go:64 msg=initialising module=server
level=info ts=2024-09-18T11:14:14.585704046Z caller=module_service.go:64 msg=initialising module=memberlist-kv
level=info ts=2024-09-18T11:14:14.58575076Z caller=module_service.go:64 msg=initialising module=runtime-config
level=info ts=2024-09-18T11:14:14.585926847Z caller=module_service.go:64 msg=initialising module=compactor

Environment:

• Infrastructure: bare-metal
• Deployment tool: manual

Additional Context

This happens for all targets,
A quick fix was to symlink the file,
This might be related to the changes introduced by #5292

[yeya24]

That's weird... We are using this feature but I haven't seen the behavior so far. Can you please share the content of your config file? /etc/cortex/compactor.yaml

[KrisBuytaert]

It's not just the compactor .. it really is all targets .. it feels like it takes /var/lib/cortex as the start of a chroot , however as you can see above it is just the same commandline with different behaviour.

`---

This is the etcd style config

global settings valid for all modules

auth_enabled: true

server:
http_listen_port: 9015
http_listen_address: 10.4.1.3
grpc_listen_port: 9915
grpc_listen_address: 10.4.1.3
grpc_server_max_concurrent_streams: 10000
log_level: info

limits:
max_metadata_per_user: 0
max_metadata_per_metric: 0
accept_ha_samples: true
max_label_names_per_series: 50
max_series_per_metric: 100000
max_series_per_user: 10000000
compactor_blocks_retention_period: 712d
ingestion_rate: 52000

storage configuration

storage:
engine: blocks

blocks_storage:
backend: s3
s3:
bucket_name: o11y-cortex-prod
insecure: false
endpoint: s3.gra.io.cloud.ovh.net
secret_access_key: "nope"
access_key_id: "nope"
http:
idle_conn_timeout: 3m0s
response_header_timeout: 4m0s

bucket_store:
ignore_deletion_mark_delay: 1h
bucket_index:
enabled: true
tsdb:
retention_period: 405m

per-module config settings

alertmanager:
enable_api: true
data_dir: /var/lib/cortex/alertmanager
external_url: http://localhost:9017/alertmanager
sharding_enabled: false
sharding_ring:
kvstore:
store: etcd
etcd:
endpoints: ["http://10.4.1.1:2379", "http://10.4.1.2:2379", "http://10.4.1.3:23:2379"]
ping-without-stream-allowed: false

alertmanager_storage:
backend: s3
s3:
bucket_name: default
insecure: false
endpoint: s3.gra.io.cloud.ovh.net
secret_access_key: "geheim"
access_key_id: "none"

compactor:
data_dir: /var/lib/cortex/compactor
compaction_concurrency: 2
sharding_enabled: false
deletion_delay: 30m

distributor:
shard_by_all_labels: true
ha_tracker:
enable_ha_tracker: true
kvstore:
store: etcd
etcd:
endpoints: ["http://10.4.1.1:2379", "http://10.4.1.2:2379", "http://10.4.1.3:23:2379"]
ping-without-stream-allowed: false

frontend:
instance_interface_names:
- enp1s0

frontend_worker:
frontend_address: "10.4.1.3:9914"

ingester:

We want our ingesters to flush chunks at the same time to optimise

deduplication opportunities.

lifecycler:
ring:
replication_factor: 3
kvstore:
store: etcd
etcd:
endpoints: ["http://10.4.1.1:2379", "http://10.4.1.2:2379", "http://10.4.1.3:23:2379"]
ping-without-stream-allowed: false
num_tokens: 512
# The ingester needs some time before joining to observe the ring state
join_after: 5s
observe_period: 5s
min_ready_duration: 0s
final_sleep: 0s
tokens_file_path: /var/lib/cortex/wal/tokens
# The address to advertise for this ingester. Will be autodiscovered by
# looking up address on eth0 or en0; can be specified if this fails.
address: 10.4.1.3

ruler:
enable_api: true
external_labels:
cortex: o11yprod
component: ruler
external_url:
alertmanager_url:
rule_path: /var/lib/cortex/ruler
ring:
kvstore:
store: etcd
etcd:
endpoints: ["http://10.4.1.1:2379", "http://10.4.1.2:2379", "http://10.4.1.3:23:2379"]
ping-without-stream-allowed: false

ruler_storage:
backend: s3
s3:
bucket_name: default
insecure: false
endpoint: s3.gra.io.cloud.ovh.net
secret_access_key: "geheim"
access_key_id: "none"

store_gateway:
sharding_enabled: true
sharding_ring:
kvstore:
store: etcd
etcd:
endpoints: ["http://10.4.1.1:2379", "http://10.4.1.2:2379", "http://10.4.1.3:23:2379"]
ping-without-stream-allowed: false
instance_interface_names:
- enp1s0

ingester_client:
grpc_client_config:
# Configure the client to allow messages up to 100MB.
max_recv_msg_size: 104857600
max_send_msg_size: 104857600
grpc_compression: gzip
`

[CharlieTLe]

Issue is probably here https://github.com/cortexproject/cortex/pull/5303/files#diff-ccddce6888f2cbf749cdd878e42e84bdcd0a84a720423c8321229a4984cb4634L147-R170.

We started using a BucketClient for the filesystem where the default directory is an empty string:

cortex/pkg/storage/bucket/filesystem/config.go

Line 17 in af9e20c

f.StringVar(&cfg.Directory, prefix+"filesystem.dir", "", "Local filesystem storage directory.")

When we create the new bucket,

cortex/vendor/github.com/thanos-io/objstore/providers/filesystem/filesystem.go

Line 46 in 526a6d9

absDir, err := filepath.Abs(rootDir)

uses filepath.Abs("") which will use the current working directory ("/var/lib/cortex") as the root directory.

Setting the default filesystem directory to "/" would probably fix this.