test: benchmark tweaks

Though it didn't result in a runnable benchmark :(

Author: nedbat

benchmark/README.rst

@@ -50,6 +50,15 @@ for the table.  There will be a row for each combination of the two dimensions.
 The `column` argument is the remaining dimension that is used to add columns to
 the table, one for each item in that dimension.
 
+To run a benchmark, create a Python file with a run_experiment call in it.
+Many are in run.py, guarded by ``if 0:`` and ``if 1:`` clauses.  In the
+benchmark directory, run your Python file.  If you haven't provided the
+``num_runs`` argument to run_experiment, put the number of runs on the command
+line::
+
+    % cd benchmark
+    % python3 run.py 3
+
 For example::
 
     run_experiment(

benchmark/benchmark.py

@@ -823,7 +823,9 @@ def __init__(
         tweaks: TweaksType = None,
         env_vars: Env_VarsType = None,
     ):
+        # Check that it really is a coverage source directory.
         directory = file_must_exist(directory_name, "coverage directory")
+        file_must_exist(str(directory / "igor.py"))
         super().__init__(
             slug=slug,
             pip_args=str(directory),

benchmark/run.py

@@ -72,7 +72,7 @@
         ],
     )
 
-if 1:
+if 0:
     # Compare N Python versions
     vers = [10, 11, 12, 13]
     run_experiment(
@@ -93,12 +93,13 @@
         ],
     )
 
-if 0:
+if 1:
     # Compare sysmon on many projects
 
     run_experiment(
         py_versions=[
-            Python(3, 12),
+            # Python(3, 12),
+            AdHocPython("/usr/local/cpython", "main"),
         ],
         cov_versions=[
             NoCoverage("nocov"),
@@ -107,14 +108,14 @@
         ],
         projects=[
             # ProjectSphinx(),  # Works, slow
-            ProjectPygments(),  # Works
+            # ProjectPygments(),  # Doesn't work on 3.14
             # ProjectRich(),  # Doesn't work
             # ProjectTornado(),  # Works, tests fail
             # ProjectDulwich(),  # Works
             # ProjectBlack(),  # Works, slow
-            # ProjectMpmath(),  # Works, slow
-            ProjectMypy(),  # Works, slow
-            # ProjectHtml5lib(),  # Works
+            ProjectMpmath(),  # Works, slow
+            # ProjectMypy(),  # Works, slow
+            # ProjectHtml5lib(),  # Doesn't work on 3.14
             # ProjectUrllib3(),  # Works
         ],
         rows=["pyver", "proj"],