Remove astropy dependency

This removal breaks the circular dependency

astropy --> pytest-astropy --> pytest-arraydiff

which may be problematic when upgrading or so. It also eases the use
of pytest-arraydiff in non-astropy environments. Astropy is only
needed to compare FITS files.

Author: olebole

setup.cfg

@@ -32,6 +32,7 @@ setup_requires =
 install_requires =
     pytest>=4.6
     numpy
+test_requires =
     astropy
 
 [options.entry_points]