Windows Certificate Store Issue

[JSanford42]

Windows Dart 2.14.0-93-dev and later.

The majority of this issue was discussed in the pub repo at dart-lang/pub#2990 and it appears to be an issue with dart.

A recent change to the runtime made all certificate stores on Windows accessible to dart. I am in an environment with an internal CA that issues certificates to the proxy servers. When the root certificates for a site are not known to the proxy, it uses the internal certificates for the communication between the workstation and the proxy. The internal certificates are regularly pushed to the workstations to prevent invalid certificate errors.

The issue is that the expired certificates remain in the store and dart is attempting to use them. The runtime appears to be using the first certificate it encounters. If it is invalid for any reason, it throws an exception and moves on. This happens even when it does find a valid certificate.

import 'dart:io';

void main(List<String> arguments) {
  var client = HttpClient();

  client.badCertificateCallback = (cert, host, port) {
    print('Bad certificate connecting to $host:$port:');
    _printCertificate(cert);
    print('');
    return true;
  };
  client
      .getUrl(Uri.parse('https://pub.dartlang.org/api/packages/pedantic'))
      .then((request) => request.close())
      .then((response) {
    print('Response certificate:');
    _printCertificate(response.certificate);
    response.drain();
    client.close();
  });
}

void _printCertificate(X509Certificate? cert) {
  if (cert != null) {
    print('${cert.issuer}');
    print('${cert.subject}');
    print('${cert.startValidity}');
    print('${cert.endValidity}');
  } else {
    print('Certificate is null');
  }
}

Running this code produces this result. (I had to mask some of the names with XXX due to policy.)

Bad certificate connecting to pub.dartlang.org:443:
/C=XXX/O=XXX/OU=XXX/OU=WCF PKI/CN=XXX WCF Intermediate CA 1
/C=XXX/O=XXX/OU=XXX/OU=WCF PKI/CN=XXX WCF Signing CA 4
2020-01-06 17:59:49.000Z
2020-07-06 17:59:49.000Z

Bad certificate connecting to pub.dartlang.org:443:
/C=XXX/O=XXX/OU=XXX/OU=WCF PKI/CN=XXX WCF Signing CA 4
/CN=pub.dartlang.org
2021-05-04 01:59:01.000Z
2021-07-10 14:07:57.000Z

Response certificate:
/C=XXX/O=XXX/OU=XXX/OU=WCF PKI/CN=XXX WCF Signing CA 4
/CN=pub.dartlang.org
2021-05-04 01:59:01.000Z
2021-07-10 14:07:57.000Z

The first certificate displayed is expired but the second one is not. I have tried this URL in the browser and the valid certificate here is the one being used.

This issue can be replicated using the --root-certs-file option for dart. I used a hand crafted certificate bundle that removed all expired certificates and this code worked. I tried using it with pub and was getting local issuer certificate errors because I did not have the time to hand craft a certificate bundle that included 100+ items from my environment.

[aam]

The first certificate displayed is expired but the second one is not. I

There got to be a reason why second certificate is deemed to be bad by boringssl/openssl.
Is it a root certificate, if not, are all certificates in its certificate chain are valid?
Which store are they in? (personal, local machine) (roots, intermediate cas...)

It is surprising you got two Bad certificate callbacks too - first one seems to have been sufficient since it returns true(meaning it can be trusted).

[JSanford42]

There got to be a reason why second certificate is deemed to be bad by boringssl/openssl.

I hand crafted a certificate bundle with that cert and all certs in the chain. I used it with the --root-certs-file option to dart. The bad certificate callback was not called.

Is it a root certificate, if not, are all certificates in its certificate chain are valid?

When I navigate to the URL in the browser, all certificates in the chain are valid.

Which store are they in? (personal, local machine) (roots, intermediate cas...)

The certificates are in the local machine store. The certs listed in the output are intermediate. The root certs for them are stored in root and are trusted and valid.

It is surprising you got two Bad certificate callbacks too - first one seems to have been sufficient since it returns true(meaning it can be trusted).

I thought it was odd as well.

[aam]

I hand crafted a certificate bundle with that cert and all certs in the chain

Have you been able to confirm that any(or all) of these certificates that you bundled are present in your Windows localmachine/currentuser certificate store(via certmgr.msc)?

It is surprising you got two Bad certificate callbacks too - first one seems to have been sufficient since it returns true(meaning it can be trusted).

I thought it was odd as well.

This is actually working as intended as per https://www.openssl.org/docs/man1.0.2/man3/SSL_CTX_set_verify.html the callback is called for every certificate in the chain that failed openssl builtin verification.

I am in an environment with an internal CA that issues certificates to the proxy servers. When the root certificates for a site are not known to the proxy, it uses the internal certificates for the communication between the workstation and the proxy. The internal certificates are regularly pushed to the workstations to prevent invalid certificate errors.

I am not familiar with this process. pub.dev certificate chain has certain certificates in it's chain. In order for dart to connect to pub.dev, certificates that pub.dev shows need to be present in Windows certificate store.

[aam]

The first certificate displayed is expired but the second one is not.

So if the first certificate printed(which seems to be an intermediate certificate in the chain) has expired/is invalid, then whole certificate won't be trusted neither.
Question is how does this intermediate certificate become part of the chain that dart app sees? Is it part of the chain that you can see in the browser when you check certificates for pub.dev website too?

[JSanford42]

Have you been able to confirm that any(or all) of these certificates that you bundled are present in your Windows localmachine/currentuser certificate store(via certmgr.msc)?

I have verified that all certificates are in the local machine store.

I am not familiar with this process. pub.dev certificate chain has certain certificates in it's chain. In order for dart to connect to pub.dev, certificates that pub.dev shows need to be present in Windows certificate store.

The proxy servers in my environment do not always use the certificates of the host site for secure communication with the workstations. For example, if I navigate to microsoft.com, the browser shows that the certs I am using are the ones from microsoft.com. When I navigate to pub.dev, the browser shows that the certs are from the internal CA. All of the certificates are valid and present in the local machine cert store.

Question is how does this intermediate certificate become part of the chain that dart app sees? Is it part of the chain that you can see in the browser when you check certificates for pub.dev website too?

When I open the URL in the browser, an intermediate certificate with the same name is in the chain but it is valid. Several intermediate certificates with the same name are in the store and all but one of them are expired. I exported the valid ones in the chain to create the bundle I was testing with.

[JSanford42]

I have finally been able to come back to this issue. I downloaded the most recent DEV build (2.15.0-42.0) and this issue is still present.

I decided it was time to reduce the issue down to its core. I left the wonky proxy configuration out of the equation and focused specifically on the certificate store.

I created a small .NET app that exported all certificates out of the local machine stores. The file produced by this code worked with the --root-certs-file switch. I did not get the bad certificate callback result.

class Program
{
	static void Main(string[] args)
	{
		// The locations here are CurrentUser and LocalMachine
		IEnumerable<StoreLocation> locations = (StoreLocation[])Enum.GetValues(typeof(StoreLocation));

		// The AddressBook, Disallowed, and TrustedPeople stores were explicily omitted.
		IEnumerable<StoreName> names = new StoreName[] { StoreName.AuthRoot, StoreName.CertificateAuthority, StoreName.My, StoreName.Root, StoreName.TrustedPublisher };

		List<X509Store> stores = new List<X509Store>();

		System.Console.WriteLine("Getting certificate stores.");
		foreach (StoreLocation location in locations)
		{
			foreach (StoreName name in names)
			{
				stores.Add(new X509Store(name, location));
			}
		}

		IEnumerable<X509Certificate2> storeCerts = null;

		System.Console.WriteLine("Writing certificates to valid_certs.pem");
		using (StreamWriter writer = new StreamWriter("valid_certs.pem"))
		{
			foreach (X509Store store in stores)
			{
				storeCerts = Program.GetCertificates(store);

				foreach (X509Certificate2 cert in storeCerts)
				{
					writer.WriteLine("-----BEGIN CERTIFICATE-----");
					writer.WriteLine(Convert.ToBase64String(cert.RawData, Base64FormattingOptions.InsertLineBreaks));
					writer.WriteLine("-----END CERTIFICATE-----");
					writer.WriteLine(String.Empty);
					writer.Flush();
				}

				if (store.IsOpen)
				{
					store.Close();
				}
				store.Dispose();
			}
		}

		System.Console.WriteLine("Done");
	}

	private static IEnumerable<X509Certificate2> GetCertificates(X509Store store)
	{
		IEnumerable<X509Certificate2> certs = new List<X509Certificate2>();

		store.Open(OpenFlags.ReadOnly);

		if (store.Certificates.Count > 0)
		{
			certs = store.Certificates.Cast<X509Certificate2>().Where(c => c.NotAfter > DateTime.Now);
		}

		store.Close();

		return certs;
	}
}

I then changed the GetCertificates method to omit the expiration date filter. I used the file created after the change with the --root-certs-file switch and the bad certificate callback result returned.

private static IEnumerable<X509Certificate2> GetCertificates(X509Store store)
{
	IEnumerable<X509Certificate2> certs = new List<X509Certificate2>();

	store.Open(OpenFlags.ReadOnly);

	if (store.Certificates.Count > 0)
	{
		// The call to Where was removed. This ensures all certificates are returned.
		certs = store.Certificates.Cast<X509Certificate2>();
	}

	store.Close();

	return certs;
}

The two files were also tested with pub: dart --root-certs-file=C:\certs\valid_certs.pem pub get --verbose. The first file resulted in pub running to completion with no errors. The second file resulted in a series of expired certificate errors in the pub output.

It looks like the certificate lookup is stopping on the first one found and ignoring all others.

[aam]

Okay, thank you @JSanford42 . Will you be able to give a try to dart sdk with a proposed fix https://dart-review.googlesource.com/c/sdk/+/211160 I can build for you? I don't have a clear plan on how to test this yet in ci setting.

[JSanford42]

@aam I will gladly test the change. I usually download the SDK as a zip so I can switch between builds for testing.

[aam]

Okay @JSanford42 thanks, can you ping me your email to aam@google.com so I share manually-built dart sdk google drive link with you?

[JSanford42]

@aam I received the link, downloaded the build, and tested it. I did not enounter any of the certificate issues. The bad certificate callback was not executed in the code from the original post. I also tested the build with pub by installing several packages with sizable dependency trees. I even installed some global packages (grinder and webdev) just to be sure.

I look foward to seeing this fix in a future dart build. In the mean time, I can use my manually built certificate bundle for development and package installations.