add tests against generated code

CONTRIBUTING.md

@@ -54,22 +54,36 @@ If you think that some of the added code is not testable (or testing it would ad
 2. If you're modifying the way an existing feature works, make sure an existing test generates the _old_ code in `end_to_end_tests/golden-record`. You'll use this to check for the new code once your changes are complete.
 3. If you're improving an error or adding a new error, add a [unit test](#unit-tests)
 
-#### End-to-end tests
+#### End-to-end snapshot tests
 
-This project aims to have all "happy paths" (types of code which _can_ be generated) covered by end to end tests (snapshot tests). In order to check code changes against the previous set of snapshots (called a "golden record" here), you can run `pdm e2e`. To regenerate the snapshots, run `pdm regen`.
+This project aims to have all "happy paths" (types of code which _can_ be generated) covered by end-to-end tests. There are two types of these: snapshot tests, and unit tests of generated code.
 
-There are 4 types of snapshots generated right now, you may have to update only some or all of these depending on the changes you're making. Within the `end_to_end_tets` directory:
+Snapshot tests verify that the generated code is identical to a previously-committed set of snapshots (called a "golden record" here). They are basically regression tests to catch any unintended changes in the generator output.
+
+In order to check code changes against the previous set of snapshots (called a "golden record" here), you can run `pdm e2e`. To regenerate the snapshots, run `pdm regen`.
+
+There are 4 types of snapshots generated right now, you may have to update only some or all of these depending on the changes you're making. Within the `end_to_end_tests` directory:
 
 1. `baseline_openapi_3.0.json` creates `golden-record` for testing OpenAPI 3.0 features
 2. `baseline_openapi_3.1.yaml` is checked against `golden-record` for testing OpenAPI 3.1 features (and ensuring consistency with 3.0)
 3. `test_custom_templates` are used with `baseline_openapi_3.0.json` to generate `custom-templates-golden-record` for testing custom templates
 4. `3.1_specific.openapi.yaml` is used to generate `test-3-1-golden-record` and test 3.1-specific features (things which do not have a 3.0 equivalent)
 
-#### Unit tests
+#### Unit tests of generated code
+
+These verify the runtime behavior of the generated code, without making assertions about the exact implementation of the code. For instance, they can verify that JSON data is correctly decoded into model class attributes.
+
+The tests run the generator against a small API spec (defined inline for each test class), and then import and execute the generated code. This can sometimes identify issues with validation logic, module imports, etc., that might be harder to diagnose via the snapshot tests, especially during development of a new feature.
+
+See [`end_to_end_tests/generated_code_live_tests`](./end_to_end_tests/generated_code_live_tests).
+
+#### Other unit tests
 
-> **NOTE**: Several older-style unit tests using mocks exist in this project. These should be phased out rather than updated, as the tests are brittle and difficult to maintain. Only error cases should be tests with unit tests going forward.
+These include:
 
-In some cases, we need to test things which cannot be generated—like validating that errors are caught and handled correctly. These should be tested via unit tests in the `tests` directory, using the `pytest` framework.
+* Regular unit tests of basic pieces of fairly self-contained low-level functionality, such as helper functions. These are implemented in the `tests/unit` directory, using the `pytest` framework.
+* End-to-end tests of invalid spec conditions, where we run the generator against a small spec with some problem, and expect it to print warnings/errors rather than generating code. These are implemented in `end_to_end_tests/generator_errors_and_warnings`.
+* Older-style unit tests of low-level functions like `property_from_data` that have complex behavior. These are brittle and difficult to maintain, and should not be used going forward. Instead, use either unit tests of generated code (to test happy paths), or end-to-end tests of invalid spec conditions (to test for warnings/errors), as described above.
 
 ### Creating a Pull Request
 

end_to_end_tests/__init__.py

@@ -1 +1,4 @@
 """ Generate a complete client and verify that it is correct """
+import pytest
+
+pytest.register_assert_rewrite("end_to_end_tests.end_to_end_test_helpers")

end_to_end_tests/baseline_openapi_3.0.json

@@ -2858,56 +2858,6 @@
       "ModelWithBackslashInDescription": {
         "type": "object",
         "description": "Description with special character: \\"
-      },
-      "ModelWithDiscriminatedUnion": {
-        "type": "object",
-        "properties": {
-          "discriminated_union": {
-            "allOf": [
-              {
-                "$ref": "#/components/schemas/ADiscriminatedUnion"
-              }
-            ],
-            "nullable": true
-          }
-        }
-      },
-      "ADiscriminatedUnion": {
-        "type": "object",
-        "discriminator": {
-          "propertyName": "modelType",
-          "mapping": {
-            "type1": "#/components/schemas/ADiscriminatedUnionType1",
-            "type2": "#/components/schemas/ADiscriminatedUnionType2",
-            "type2-another-value": "#/components/schemas/ADiscriminatedUnionType2"
-          }
-        },
-        "oneOf": [
-          {
-            "$ref": "#/components/schemas/ADiscriminatedUnionType1"
-          },
-          {
-            "$ref": "#/components/schemas/ADiscriminatedUnionType2"
-          }
-        ]
-      },
-      "ADiscriminatedUnionType1": {
-        "type": "object",
-        "properties": {
-          "modelType": {
-            "type": "string"
-          }
-        },
-        "required": ["modelType"]
-      },
-      "ADiscriminatedUnionType2": {
-        "type": "object",
-        "properties": {
-          "modelType": {
-            "type": "string"
-          }
-        },
-        "required": ["modelType"]
       }
     },
     "parameters": {

end_to_end_tests/baseline_openapi_3.1.yaml

@@ -2850,58 +2850,6 @@ info:
     "ModelWithBackslashInDescription": {
       "type": "object",
       "description": "Description with special character: \\"
-    },
-    "ModelWithDiscriminatedUnion": {
-      "type": "object",
-      "properties": {
-        "discriminated_union": {
-          "oneOf": [
-            {
-              "$ref": "#/components/schemas/ADiscriminatedUnion"
-            },
-            {
-              "type": "null"
-            }
-          ],
-        }
-      }
-    },
-    "ADiscriminatedUnion": {
-      "type": "object",
-      "discriminator": {
-        "propertyName": "modelType",
-        "mapping": {
-          "type1": "#/components/schemas/ADiscriminatedUnionType1",
-          "type2": "#/components/schemas/ADiscriminatedUnionType2",
-          "type2-another-value": "#/components/schemas/ADiscriminatedUnionType2"
-        }
-      },
-      "oneOf": [
-        {
-          "$ref": "#/components/schemas/ADiscriminatedUnionType1"
-        },
-        {
-          "$ref": "#/components/schemas/ADiscriminatedUnionType2"
-        }
-      ]
-    },
-    "ADiscriminatedUnionType1": {
-      "type": "object",
-      "properties": {
-        "modelType": {
-          "type": "string"
-        }
-      },
-      "required": ["modelType"]
-    },
-    "ADiscriminatedUnionType2": {
-      "type": "object",
-      "properties": {
-        "modelType": {
-          "type": "string"
-        }
-      },
-      "required": ["modelType"]
     }
   }
   "parameters": {