Multi-target pattern endpoint /probe does not report same metrics as /metrics

[mierea]

What did you do?
I ran pg_exporter in kubernetes quay.io/prometheuscommunity/postgres-exporter:v0.12.0-rc.0
I have accessed /metrics and all looks good.
I have accessed /probe?target=postgres:[email protected]:5432?sslmode=disable

What did you expect to see?
Same metrics that are available under /metrics should be available under /probe

What did you see instead? Under which circumstances?
A shorter metric list.

/probe metrics

# HELP pg_database_size_bytes Disk space used by the database
# TYPE pg_database_size_bytes gauge
pg_database_size_bytes{datname="userdb"} 3.1859491e+07
pg_database_size_bytes{datname="postgres"} 8.815395e+06
pg_database_size_bytes{datname="template0"} 8.602115e+06
pg_database_size_bytes{datname="template1"} 8.602115e+06
# HELP pg_scrape_collector_duration_seconds postgres_exporter: Duration of a collector scrape.
# TYPE pg_scrape_collector_duration_seconds gauge
pg_scrape_collector_duration_seconds{collector="bgwriter"} 0.034231546
pg_scrape_collector_duration_seconds{collector="database"} 0.048092705
# HELP pg_scrape_collector_success postgres_exporter: Whether a collector succeeded.
# TYPE pg_scrape_collector_success gauge
pg_scrape_collector_success{collector="bgwriter"} 1
pg_scrape_collector_success{collector="database"} 1
# HELP pg_stat_bgwriter_buffers_alloc_total Number of buffers allocated
# TYPE pg_stat_bgwriter_buffers_alloc_total counter
pg_stat_bgwriter_buffers_alloc_total 4268
# HELP pg_stat_bgwriter_buffers_backend_fsync_total Number of times a backend had to execute its own fsync call (normally the background writer handles those even when the backend does its own write)
# TYPE pg_stat_bgwriter_buffers_backend_fsync_total counter
pg_stat_bgwriter_buffers_backend_fsync_total 0
# HELP pg_stat_bgwriter_buffers_backend_total Number of buffers written directly by a backend
# TYPE pg_stat_bgwriter_buffers_backend_total counter
pg_stat_bgwriter_buffers_backend_total 2410
# HELP pg_stat_bgwriter_buffers_checkpoint_total Number of buffers written during checkpoints
# TYPE pg_stat_bgwriter_buffers_checkpoint_total counter
pg_stat_bgwriter_buffers_checkpoint_total 2080
# HELP pg_stat_bgwriter_buffers_clean_total Number of buffers written by the background writer
# TYPE pg_stat_bgwriter_buffers_clean_total counter
pg_stat_bgwriter_buffers_clean_total 0
# HELP pg_stat_bgwriter_checkpoint_sync_time_total Total amount of time that has been spent in the portion of checkpoint processing where files are synchronized to disk, in milliseconds
# TYPE pg_stat_bgwriter_checkpoint_sync_time_total counter
pg_stat_bgwriter_checkpoint_sync_time_total 429
# HELP pg_stat_bgwriter_checkpoint_write_time_total Total amount of time that has been spent in the portion of checkpoint processing where files are written to disk, in milliseconds
# TYPE pg_stat_bgwriter_checkpoint_write_time_total counter
pg_stat_bgwriter_checkpoint_write_time_total 46543
# HELP pg_stat_bgwriter_checkpoints_req_total Number of requested checkpoints that have been performed
# TYPE pg_stat_bgwriter_checkpoints_req_total counter
pg_stat_bgwriter_checkpoints_req_total 5
# HELP pg_stat_bgwriter_checkpoints_timed_total Number of scheduled checkpoints that have been performed
# TYPE pg_stat_bgwriter_checkpoints_timed_total counter
pg_stat_bgwriter_checkpoints_timed_total 112
# HELP pg_stat_bgwriter_maxwritten_clean_total Number of times the background writer stopped a cleaning scan because it had written too many buffers
# TYPE pg_stat_bgwriter_maxwritten_clean_total counter
pg_stat_bgwriter_maxwritten_clean_total 0
# HELP pg_stat_bgwriter_stats_reset_total Time at which these statistics were last reset
# TYPE pg_stat_bgwriter_stats_reset_total counter
pg_stat_bgwriter_stats_reset_total 1.677823427e+09
# HELP probe_duration_seconds Returns how long the probe took to complete in seconds
# TYPE probe_duration_seconds gauge
probe_duration_seconds 2.4865e-05
# HELP probe_success Displays whether or not the probe was a success
# TYPE probe_success gauge
probe_success 0

/metrics metrics (truncated)

# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 0
go_gc_duration_seconds{quantile="1"} 0
go_gc_duration_seconds_sum 0
go_gc_duration_seconds_count 0
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 16
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.18.5"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 3.730096e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 3.730096e+06
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 4611
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 0
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 3.579728e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 3.730096e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 4.071424e+06
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 3.760128e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 20864
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 4.071424e+06
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 7.831552e+06
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 0
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 20864
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 9600
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 15600
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 63104
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 65280
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 4.194304e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 1.138381e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 557056
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 557056
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 1.3192208e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 6
# HELP pg_database_size_bytes Disk space used by the database
# TYPE pg_database_size_bytes gauge
pg_database_size_bytes{datname="userdb"} 3.1687459e+07
pg_database_size_bytes{datname="postgres"} 8.815395e+06
pg_database_size_bytes{datname="template0"} 8.602115e+06
pg_database_size_bytes{datname="template1"} 8.602115e+06
# HELP pg_exporter_last_scrape_duration_seconds Duration of the last scrape of metrics from PostgreSQL.
# TYPE pg_exporter_last_scrape_duration_seconds gauge
pg_exporter_last_scrape_duration_seconds 1.154516362
# HELP pg_exporter_last_scrape_error Whether the last scrape of metrics from PostgreSQL resulted in an error (1 for error, 0 for success).
# TYPE pg_exporter_last_scrape_error gauge
pg_exporter_last_scrape_error 0
# HELP pg_exporter_scrapes_total Total number of times PostgreSQL was scraped for metrics.
# TYPE pg_exporter_scrapes_total counter
pg_exporter_scrapes_total 1
# HELP pg_exporter_user_queries_load_error Whether the user queries file was loaded and parsed successfully (1 for error, 0 for success).
# TYPE pg_exporter_user_queries_load_error gauge
pg_exporter_user_queries_load_error{filename="/etc/config.yaml",hashsum="e434d710fafb146b0661d71831a94aa48a1b4a8803c815dfde873b444762d4bb"} 0
# HELP pg_locks_count Number of locks
# TYPE pg_locks_count gauge
pg_locks_count{datname="userdb",mode="accessexclusivelock",server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
pg_locks_count{datname="userdb",mode="accesssharelock",server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
pg_locks_count{datname="userdb",mode="exclusivelock",server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
pg_locks_count{datname="userdb",mode="rowexclusivelock",server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
pg_locks_count{datname="userdb",mode="rowsharelock",server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
pg_locks_count{datname="userdb",mode="sharelock",server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
pg_locks_count{datname="userdb",mode="sharerowexclusivelock",server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
pg_locks_count{datname="userdb",mode="shareupdateexclusivelock",server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
pg_locks_count{datname="userdb",mode="sireadlock",server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
pg_locks_count{datname="postgres",mode="accessexclusivelock",server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
pg_locks_count{datname="postgres",mode="accesssharelock",server="svc-postgres.test-namespace.svc.cluster.local:5432"} 1
pg_locks_count{datname="postgres",mode="exclusivelock",server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
pg_locks_count{datname="postgres",mode="rowexclusivelock",server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
pg_locks_count{datname="postgres",mode="rowsharelock",server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
pg_locks_count{datname="postgres",mode="sharelock",server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
pg_locks_count{datname="postgres",mode="sharerowexclusivelock",server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
pg_locks_count{datname="postgres",mode="shareupdateexclusivelock",server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
pg_locks_count{datname="postgres",mode="sireadlock",server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
pg_locks_count{datname="template0",mode="accessexclusivelock",server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
pg_locks_count{datname="template0",mode="accesssharelock",server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
pg_locks_count{datname="template0",mode="exclusivelock",server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
pg_locks_count{datname="template0",mode="rowexclusivelock",server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
pg_locks_count{datname="template0",mode="rowsharelock",server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
pg_locks_count{datname="template0",mode="sharelock",server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
pg_locks_count{datname="template0",mode="sharerowexclusivelock",server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
pg_locks_count{datname="template0",mode="shareupdateexclusivelock",server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
pg_locks_count{datname="template0",mode="sireadlock",server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
pg_locks_count{datname="template1",mode="accessexclusivelock",server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
pg_locks_count{datname="template1",mode="accesssharelock",server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
pg_locks_count{datname="template1",mode="exclusivelock",server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
pg_locks_count{datname="template1",mode="rowexclusivelock",server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
pg_locks_count{datname="template1",mode="rowsharelock",server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
pg_locks_count{datname="template1",mode="sharelock",server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
pg_locks_count{datname="template1",mode="sharerowexclusivelock",server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
pg_locks_count{datname="template1",mode="shareupdateexclusivelock",server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
pg_locks_count{datname="template1",mode="sireadlock",server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
# HELP pg_postmaster_start_time_seconds Time at which postmaster started
# TYPE pg_postmaster_start_time_seconds gauge
pg_postmaster_start_time_seconds{server="svc-postgres.test-namespace.svc.cluster.local:5432"} 1.677845578e+09
# HELP pg_replication_lag Replication lag behind master in seconds
# TYPE pg_replication_lag gauge
pg_replication_lag{server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
# HELP pg_scrape_collector_duration_seconds postgres_exporter: Duration of a collector scrape.
# TYPE pg_scrape_collector_duration_seconds gauge
pg_scrape_collector_duration_seconds{collector="bgwriter"} 0.029669799
pg_scrape_collector_duration_seconds{collector="database"} 0.037533204
# HELP pg_scrape_collector_success postgres_exporter: Whether a collector succeeded.
# TYPE pg_scrape_collector_success gauge
pg_scrape_collector_success{collector="bgwriter"} 1
pg_scrape_collector_success{collector="database"} 1
# HELP pg_settings_allow_in_place_tablespaces Allows tablespaces directly inside pg_tblspc, for testing.
# TYPE pg_settings_allow_in_place_tablespaces gauge
pg_settings_allow_in_place_tablespaces{server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
# HELP pg_settings_allow_system_table_mods Allows modifications of the structure of system tables.
# TYPE pg_settings_allow_system_table_mods gauge
pg_settings_allow_system_table_mods{server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
# HELP pg_settings_archive_timeout_seconds Forces a switch to the next WAL file if a new file has not been started within N seconds. [Units converted to seconds.]
# TYPE pg_settings_archive_timeout_seconds gauge
pg_settings_archive_timeout_seconds{server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0
# HELP pg_settings_array_nulls Enable input of NULL elements in arrays.
# TYPE pg_settings_array_nulls gauge
pg_settings_array_nulls{server="svc-postgres.test-namespace.svc.cluster.local:5432"} 1
# HELP pg_settings_authentication_timeout_seconds Sets the maximum allowed time to complete client authentication. [Units converted to seconds.]
# TYPE pg_settings_authentication_timeout_seconds gauge
pg_settings_authentication_timeout_seconds{server="svc-postgres.test-namespace.svc.cluster.local:5432"} 60
# HELP pg_settings_autovacuum Starts the autovacuum subprocess.
# TYPE pg_settings_autovacuum gauge
pg_settings_autovacuum{server="svc-postgres.test-namespace.svc.cluster.local:5432"} 1
# HELP pg_settings_autovacuum_analyze_scale_factor Number of tuple inserts, updates, or deletes prior to analyze as a fraction of reltuples.
# TYPE pg_settings_autovacuum_analyze_scale_factor gauge
pg_settings_autovacuum_analyze_scale_factor{server="svc-postgres.test-namespace.svc.cluster.local:5432"} 0.1
# HELP pg_settings_autovacuum_analyze_threshold Minimum number of tuple inserts, updates, or deletes prior to analyze.
# TYPE pg_settings_autovacuum_analyze_threshold gauge
pg_settings_autovacuum_analyze_threshold{server="svc-postgres.test-namespace.svc.cluster.local:5432"} 50
# HELP pg_settings_autovacuum_freeze_max_age Age at which to autovacuum a table to prevent transaction ID wraparound.
# TYPE pg_settings_autovacuum_freeze_max_age gauge
pg_settings_autovacuum_freeze_max_age{server="svc-postgres.test-namespace.svc.cluster.local:5432"} 2e+08
# HELP pg_settings_autovacuum_max_workers Sets the maximum number of simultaneously running autovacuum worker processes.
# TYPE pg_settings_autovacuum_max_workers gauge

Environment

• System information:

kubernetes 1.23
helm3

• postgres_exporter version:

  postgres-exporter:v0.12.0-rc.0

• postgres_exporter flags:

/bin/postgres_exporter --extend.query-path=/etc/config.yaml --web.listen-address

• PostgreSQL version:

PostgreSQL 14.7 (Debian 14.7-1.pgdg110+1)

• Logs:

ts=2023-03-03T15:43:32.435Z caller=probe.go:42 level=info msg="no auth_module specified, using default"```

[sysadmind]

I believe that this was fixed by #683. It will be resolved in the next release.

[mierea]

Thank you very much! :)