|
| 1 | +|Travis Build Status| |AppVeyor Build status| |
| 2 | + |
| 3 | +About |
| 4 | +----- |
| 5 | + |
| 6 | +This is a `py.test <http://pytest.org>`__ plugin to facilitate the |
| 7 | +generation and comparison of arrays produced during tests (this is a |
| 8 | +spin-off from |
| 9 | +`pytest-arraydiff <https://github.com/astrofrog/pytest-arraydiff>`__). |
| 10 | + |
| 11 | +The basic idea is that you can write a test that generates a Numpy |
| 12 | +array. You can then either run the tests in a mode to **generate** |
| 13 | +reference files from the arrays, or you can run the tests in |
| 14 | +**comparison** mode, which will compare the results of the tests to the |
| 15 | +reference ones within some tolerance. |
| 16 | + |
| 17 | +At the moment, the supported file formats for the reference files are: |
| 18 | + |
| 19 | +- The FITS format (requires `astropy <http://www.astropy.org>`__) |
| 20 | +- A plain text-based format (baed on Numpy ``loadtxt`` output) |
| 21 | + |
| 22 | +For more information on how to write tests to do this, see the **Using** |
| 23 | +section below. |
| 24 | + |
| 25 | +Installing |
| 26 | +---------- |
| 27 | + |
| 28 | +This plugin is compatible with Python 2.7, and 3.3 and later, and |
| 29 | +requires `pytest <http://pytest.org>`__ and |
| 30 | +`numpy <http://www.numpy.org>`__ to be installed. |
| 31 | + |
| 32 | +To install, you can do: |
| 33 | + |
| 34 | +:: |
| 35 | + |
| 36 | + pip install pytest-arraydiff |
| 37 | + |
| 38 | +You can check that the plugin is registered with pytest by doing: |
| 39 | + |
| 40 | +:: |
| 41 | + |
| 42 | + py.test --version |
| 43 | + |
| 44 | +which will show a list of plugins: |
| 45 | + |
| 46 | +:: |
| 47 | + |
| 48 | + This is pytest version 2.7.1, imported from ... |
| 49 | + setuptools registered plugins: |
| 50 | + pytest-arraydiff-0.1 at ... |
| 51 | + |
| 52 | +Using |
| 53 | +----- |
| 54 | + |
| 55 | +To use, you simply need to mark the function where you want to compare |
| 56 | +images using ``@pytest.mark.array_compare``, and make sure that the |
| 57 | +function returns a plain Numpy array: |
| 58 | + |
| 59 | +:: |
| 60 | + |
| 61 | + python |
| 62 | + import pytest |
| 63 | + import numpy as np |
| 64 | + |
| 65 | + @pytest.mark.array_compare |
| 66 | + def test_succeeds(): |
| 67 | + return np.arange(3 * 5 * 4).reshape((3, 5, 4)) |
| 68 | + |
| 69 | +To generate the reference FITS files, run the tests with the |
| 70 | +``--arraydiff-generate-path`` option with the name of the directory |
| 71 | +where the generated files should be placed: |
| 72 | + |
| 73 | +:: |
| 74 | + |
| 75 | + py.test --arraydiff-generate-path=reference |
| 76 | + |
| 77 | +If the directory does not exist, it will be created. The directory will |
| 78 | +be interpreted as being relative to where you are running ``py.test``. |
| 79 | +Make sure you manually check the reference images to ensure they are |
| 80 | +correct. |
| 81 | + |
| 82 | +Once you are happy with the generated FITS files, you should move them |
| 83 | +to a sub-directory called ``reference`` relative to the test files (this |
| 84 | +name is configurable, see below). You can also generate the baseline |
| 85 | +images directly in the right directory. |
| 86 | + |
| 87 | +You can then run the tests simply with: |
| 88 | + |
| 89 | +:: |
| 90 | + |
| 91 | + py.test --arraydiff |
| 92 | + |
| 93 | +and the tests will pass if the images are the same. If you omit the |
| 94 | +``--arraydiff`` option, the tests will run but will only check that the |
| 95 | +code runs without checking the output images. |
| 96 | + |
| 97 | +Options |
| 98 | +------- |
| 99 | + |
| 100 | +The ``@pytest.mark.array_compare`` marker take an argument to specify |
| 101 | +the format to use for the reference files: |
| 102 | + |
| 103 | +.. code:: python |
| 104 | +
|
| 105 | + @pytest.mark.array_compare(file_format='text') |
| 106 | + def test_image(): |
| 107 | + ... |
| 108 | +
|
| 109 | +The default file format can also be specified using the |
| 110 | +``--arraydiff-default-format=<format>`` flag when running ``py.test``, |
| 111 | +and ``<format>`` should be either ``fits`` or ``text``. |
| 112 | + |
| 113 | +The supported formats at this time are ``text`` and ``fits``, and |
| 114 | +contributions for other formats are welcome. The default format is |
| 115 | +``text``. |
| 116 | + |
| 117 | +Another argument is the relative tolerance for floating point values |
| 118 | +(which defaults to 1e-7): |
| 119 | + |
| 120 | +.. code:: python |
| 121 | +
|
| 122 | + @pytest.mark.array_compare(rtol=20) |
| 123 | + def test_image(): |
| 124 | + ... |
| 125 | +
|
| 126 | +You can also pass keyword arguments to the writers using the |
| 127 | +``write_kwargs``. For the ``text`` format, these arguments are passed to |
| 128 | +``savetxt`` while for the ``fits`` format they are passed to Astropy's |
| 129 | +``fits.writeto`` function. |
| 130 | + |
| 131 | +.. code:: python |
| 132 | +
|
| 133 | + @pytest.mark.array_compare(file_format='fits', write_kwargs={'output_verify': 'silentfix'}) |
| 134 | + def test_image(): |
| 135 | + ... |
| 136 | +
|
| 137 | +Other options include the name of the reference directory (which |
| 138 | +defaults to ``reference`` ) and the filename for the reference file |
| 139 | +(which defaults to the name of the test with a format-dependent |
| 140 | +extension). |
| 141 | + |
| 142 | +.. code:: python |
| 143 | +
|
| 144 | + @pytest.mark.array_compare(reference_dir='baseline_images', |
| 145 | + filename='other_name.fits') |
| 146 | + def test_image(): |
| 147 | + ... |
| 148 | +
|
| 149 | +The reference directory in the decorator above will be interpreted as |
| 150 | +being relative to the test file. Note that the baseline directory can |
| 151 | +also be a URL (which should start with ``http://`` or ``https://`` and |
| 152 | +end in a slash). |
| 153 | + |
| 154 | +Finally, you can also set a custom baseline directory globally when |
| 155 | +running tests by running ``py.test`` with: |
| 156 | + |
| 157 | +:: |
| 158 | + |
| 159 | + py.test --arraydiff --arraydiff-reference-path=baseline_images |
| 160 | + |
| 161 | +This directory will be interpreted as being relative to where the tests |
| 162 | +are run. In addition, if both this option and the ``reference_dir`` |
| 163 | +option in the ``array_compare`` decorator are used, the one in the |
| 164 | +decorator takes precedence. |
| 165 | + |
| 166 | +Test failure example |
| 167 | +-------------------- |
| 168 | + |
| 169 | +If the images produced by the tests are correct, then the test will |
| 170 | +pass, but if they are not, the test will fail with a message similar to |
| 171 | +the following: |
| 172 | + |
| 173 | +:: |
| 174 | + |
| 175 | + E AssertionError: |
| 176 | + E |
| 177 | + E a: /var/folders/zy/t1l3sx310d3d6p0kyxqzlrnr0000gr/T/tmpbvjkzt_q/test_to_mask_rect-mode_subpixels-subpixels_18.txt |
| 178 | + E b: /var/folders/zy/t1l3sx310d3d6p0kyxqzlrnr0000gr/T/tmpbvjkzt_q/reference-test_to_mask_rect-mode_subpixels-subpixels_18.txt |
| 179 | + E |
| 180 | + E Not equal to tolerance rtol=1e-07, atol=0 |
| 181 | + E |
| 182 | + E (mismatch 47.22222222222222%) |
| 183 | + E x: array([[ 0. , 0. , 0. , 0. , 0.404012, 0.55 , |
| 184 | + E 0.023765, 0. , 0. ], |
| 185 | + E [ 0. , 0. , 0. , 0.112037, 1.028704, 1.1 ,... |
| 186 | + E y: array([[ 0. , 0. , 0. , 0. , 0.367284, 0.5 , |
| 187 | + E 0.021605, 0. , 0. ], |
| 188 | + E [ 0. , 0. , 0. , 0.101852, 0.935185, 1. ,... |
| 189 | + |
| 190 | +The file paths included in the exception are then available for |
| 191 | +inspection. |
| 192 | + |
| 193 | +Running the tests for pytest-arraydiff |
| 194 | +-------------------------------------- |
| 195 | + |
| 196 | +If you are contributing some changes and want to run the tests, first |
| 197 | +install the latest version of the plugin then do: |
| 198 | + |
| 199 | +:: |
| 200 | + |
| 201 | + cd tests |
| 202 | + py.test --arraydiff |
| 203 | + |
| 204 | +The reason for having to install the plugin first is to ensure that the |
| 205 | +plugin is correctly loaded as part of the test suite. |
| 206 | + |
| 207 | +.. |Travis Build Status| image:: https://travis-ci.org/astrofrog/pytest-arraydiff.svg?branch=master |
| 208 | + :target: https://travis-ci.org/astrofrog/pytest-arraydiff |
| 209 | +.. |AppVeyor Build status| image:: https://ci.appveyor.com/api/projects/status/kwbvm9u79mrq6i0w?svg=true |
| 210 | + :target: https://ci.appveyor.com/project/astrofrog/pytest-arraydiff |
0 commit comments