OpenSSL 3.0.0: Fix hashlib issue

The FIPS_mode() function has been deprecated and removed. It no
longer makes sense with the new provider and context system in
OpenSSL 3.0.0.

EVP_default_properties_is_fips_enabled() is good enough for our
needs in unit tests. It's an internal API, too.

Fixes: bpo-40479
From-PR: gh#python/cpython!20107
Patch: openssl-300-fix-hashlib-issue.patch
Released-in: 3.9.0
Signed-off-by: Christian Heimes <[email protected]>

Author: tiran

Misc/NEWS.d/next/Library/2020-05-15-17-38-21.bpo-40479.yamSCh.rst

@@ -0,0 +1 @@
+The :mod:`hashlib` now compiles with OpenSSL 3.0.0-alpha2.

Modules/_hashopenssl.c

@@ -965,6 +965,47 @@ generate_hash_name_list(void)
         return ret_obj; \
     }
 
+#ifndef LIBRESSL_VERSION_NUMBER
+/*[clinic input]
+_hashlib.get_fips_mode -> int
+
+Determine the OpenSSL FIPS mode of operation.
+
+For OpenSSL 3.0.0 and newer it returns the state of the default provider
+in the default OSSL context. It's not quite the same as FIPS_mode() but good
+enough for unittests.
+
+Effectively any non-zero return value indicates FIPS mode;
+values other than 1 may have additional significance.
+[clinic start generated code]*/
+
+static int
+_hashlib_get_fips_mode_impl(PyObject *module)
+/*[clinic end generated code: output=87eece1bab4d3fa9 input=2db61538c41c6fef]*/
+
+{
+    int result;
+#if OPENSSL_VERSION_NUMBER >= 0x30000000L
+    result = EVP_default_properties_is_fips_enabled(NULL);
+#else
+    ERR_clear_error();
+    result = FIPS_mode();
+    if (result == 0) {
+        // "If the library was built without support of the FIPS Object Module,
+        // then the function will return 0 with an error code of
+        // CRYPTO_R_FIPS_MODE_NOT_SUPPORTED (0x0f06d065)."
+        // But 0 is also a valid result value.
+        unsigned long errcode = ERR_peek_last_error();
+        if (errcode) {
+            _setException(PyExc_ValueError);
+            return -1;
+        }
+    }
+    return result;
+#endif
+}
+#endif  // !LIBRESSL_VERSION_NUMBER
+
 /* a PyMethodDef structure for the constructor */
 #define CONSTRUCTOR_METH_DEF(NAME)  \
     {"openssl_" #NAME, (PyCFunction)EVP_new_ ## NAME, METH_VARARGS, \