MSSQL Exporter - Negative Serial Number Errors

[jmbecker]

Describe the bug
The MSSQL exporter runs without issue after being set up and then starts throwing TLS handshake failed errors some time later.

To Reproduce
Steps to reproduce the behavior:

1. Set up prometheus.exporter.mssql with a configuration like the following in Grafana Alloy:

     prometheus.exporter.mssql "standard_exporter" {
       connection_string = "sqlserver://user:pass@sql-server:1433?encrypt=true&trustservercertificate=true"
       query_config = local.file.mssql_standard.content
     }

2. Metrics begin populating as expected.
3. Wait several hours (up to 24).
4. See invalid metric errors that state that a TLS handshake error has occurred:

ts=2025-04-21T14:22:09.549504734Z level=error msg="Invalid metric description." component_path=/ component_id=prometheus.exporter.mssql.standard_exporter err="[mssqlintegration,collector=mssql_standard,query=mssql_user_errors_total] TLS Handshake failed: tls: failed to parse certificate from server: x509: negative serial number"

Expected behavior
Once the MSSQL exporter has been set up, it should continue to work without interruption as long as the sql server being scraped is up and ready to receive the queries. Since I'm trusting the certificate, I'm expecting TLS to continue working.

Configuration
We're using the exporter via Grafana Alloy inside of a kubernetes cluster. We deploy the Grafana Alloy collector with the flag --feature.community-components.enabled. We're using other prometheus-type components without issue.

Additional context

• Both the standard exporter we have set up AND a custom exporter are having the same issue.
• This issue seems to have stemmed from a change in behavior of Go 1.23+. This issue in the Go repository mentions that it will probably not be fixed. The workaround would be to add a godebug flag like so: godebug x509negativeserial=1
• The crypto/x509 package calls out this change in behaviour here
• Interestingly, scaling the MSSQL container down and back up, followed by changing the SQL scrape configuration appears to allow the scrape to work again (until it stops working).

[burningalchemist]

Hey @jmbecker, thanks for reporting the issue.

As this problem originates from the Microsoft side (in our case it's the way the certificate is generated), it won't be solved by Golang, indeed. Generally, it's a violation of RFC 5280 section 4.1.2.2, which states that

The serial number MUST be a positive integer assigned by the CA to each certificate.

sql_exporter uses official microsoft/go-mssql driver, so the connection handling (including TLS) happens there, it's nothing custom on our end.

The issue was accepted and fixed by the Microsoft team and has been reflected in the update - https://learn.microsoft.com/en-us/troubleshoot/sql/releases/sqlserver-2022/cumulativeupdate18#3867855. So I guess you might need to check for a new image with the fix applied.

Also, one of the solutions suggested by the community is to generate a new certificate and make sure the negative numbers aren't used (microsoft/mssql-docker#895 (comment)).

If none of the above works for you, the GODEBUG workaround still can be applied without any code changes. You need to pass the setting as an environment variable to your container such as GODEBUG=x509negativeserial=1. Since you use Grafana Alloy (which is not affiliated with this project), that'd be the right container to add to. Golang runtime will pickup the setting after the restart.

Given the fact that the problem was confirmed and solved by Microsoft, I don't plan to make any changes to the code. Moreover, it won't have any effect on Grafana Alloy installations as they use sql_exporter as an external library, so the runtime settings have to be handled there.

Since it's not a bug of sql_exporter, I'll move it to discussions. But I'm happy to assist you further in solving your problem, please let me know. 👍

[jmbecker]

Hey @burningalchemist - thanks for the quick response. Apologies for putting this in the wrong place initially.

Your suggestions were exactly what I was looking for. Our version of MSSQL is older than SQL2022 CU18, which fixes the issue. I could have chosen to use a new certificate as well, but I ultimately went with the workaround of adding the environment variable to the Alloy container. After defining it in Alloy's helm chart with alloy.extraEnv, the scrapes are working without issue.

If anyone stumbles upon this and is using the Alloy helm chart like I am, adding the following values did the trick for me:

alloy:
  extraEnv:
  - name: "GODEBUG"
     value: "x509negativeserial=1"

[burningalchemist]

hey @jmbecker, no worries at all. I'm happy that you've resolved the issue. Thanks for posting the solution! 😃👍