[infra] Observability overhaul

[zivkovicmilos]

Description

This issue proposes an observability overhaul to the currently running production cluster:

• Fixing the Grafana dashboards so metrics are displayed correctly and nicely (https://github.com/gnolang/gno/tree/master/misc/telemetry/grafana)
• Implement Loki as a log aggregator, data source for Grafana
• Utilize InfluxDB as the Grafana datastore
• Utilize Telegraf as the system metrics scraper
• Utilize Promtail as the log scraper for Loki

Definition of done:

• Metrics and logs have proper labeling on Grafana
• Grafana metrics are valid, updated regularly and displayed properly
• Logs are fetchable and searchable from inside Grafana
• The deployments for these services are either in the monorepo, or on a separate gnolang/infrastructure repository

cc @sw360cab @mazzy89 please challenge this issue and propose alternatives if any 💪