Unittest TextTestResult fails

[rachel3834]

I am working through the installation documentation for the pipeline (under Ubuntu 18.04.3, miniconda3 venv as described by the docs). I got to the step of running the setup.py test and I thought you'd like to know that the test for TextTestResult currently fails for v1.2.0.

The output given is:

ERROR: test_get_reference_lamp_exist (goodman_pipeline.core.tests.test_core.ReferenceDataTest)

Traceback (most recent call last):
File "/data/ROMEREA/2019/SOAR/pipeline/goodman_pipeline-1.2.0/goodman_pipeline/core/tests/test_core.py", line 640, in test_get_reference_lamp_exist
ref_lamp = self.rd.get_reference_lamp(header=self.ccd.header)
File "/data/ROMEREA/2019/SOAR/pipeline/goodman_pipeline-1.2.0/goodman_pipeline/core/core.py", line 2857, in get_reference_lamp
self._ccd = CCDData.read(full_path, unit=u.adu)
File "/data/miniconda3/envs/goodman_pipeline/lib/python3.6/site-packages/astropy/nddata/mixins/ndio.py", line 54, in call
return registry.read(self._cls, *args, **kwargs)
File "/data/miniconda3/envs/goodman_pipeline/lib/python3.6/site-packages/astropy/io/registry.py", line 508, in read
fileobj = ctx.enter()
File "/data/miniconda3/envs/goodman_pipeline/lib/python3.6/contextlib.py", line 81, in enter
return next(self.gen)
File "/data/miniconda3/envs/goodman_pipeline/lib/python3.6/site-packages/astropy/utils/data.py", line 193, in get_readable_fileobj
fileobj = io.FileIO(name_or_obj, 'r')
FileNotFoundError: [Errno 2] No such file or directory: '/data/ROMEREA/2019/SOAR/pipeline/goodman_pipeline-1.2.0/goodman_pipeline/data/ref_comp/ goodman_comp_400M2_GG455_HgArNe.fits'

======================================================================
FAIL: test__call__ (goodman_pipeline.core.tests.test_core.SpectroscopicModeTest)

Traceback (most recent call last):
File "/data/ROMEREA/2019/SOAR/pipeline/goodman_pipeline-1.2.0/goodman_pipeline/core/tests/test_core.py", line 738, in test__call__
self.assertEqual(mode_m2_header, 'm2')
AssertionError: ' m2' != 'm2'

• m2
  ? -

• m2

======================================================================
FAIL: test_get_cam_grt_targ_angle (goodman_pipeline.core.tests.test_core.SpectroscopicModeTest)

Traceback (most recent call last):
File "/data/ROMEREA/2019/SOAR/pipeline/goodman_pipeline-1.2.0/goodman_pipeline/core/tests/test_core.py", line 774, in test_get_cam_grt_targ_angle
self.assertEqual(cam_targ, '39.4')
AssertionError: ' 39.4' != '39.4'

• 39.4
  ? -

• 39.4

======================================================================
FAIL: test_get_mode (goodman_pipeline.core.tests.test_core.SpectroscopicModeTest)

Traceback (most recent call last):
File "/data/ROMEREA/2019/SOAR/pipeline/goodman_pipeline-1.2.0/goodman_pipeline/core/tests/test_core.py", line 752, in test_get_mode
self.assertEqual(mode_m2, 'm2')
AssertionError: ' m2' != 'm2'

• m2
  ? -

• m2

---

Ran 118 tests in 76.857s

FAILED (failures=3, errors=1)
Test failed: <unittest.runner.TextTestResult run=118 errors=1 failures=3>
error: Test failed: <unittest.runner.TextTestResult run=118 errors=1 failures=3>

[simontorres]

This sounds familiar. I remember there was a problem with certain version of pandas that introduces a blank space at the beginning of the string. I thought I had constrained the pandas version to be used. it was reported on pandas-dev/pandas#24980

[rachel3834]

Hmm, interesting. The venv says that I am using pandas v0.25.0.

I don't know if its related but I just ran redccd over a set of frames and got the following:
[14:18:58][W]: Ignoring all Imaging data. Assuming they are not science exposures
[14:18:58][I]: Organizing data for this configuration: Gain: 1.48, Noise: 3.89, ROI: Spectroscopic 2x2
[14:18:59][C]: There is no BIAS images for this configuration. Use --ignore-bias to proceed without BIAS.
[14:18:59][W]: The following files will be discarded:
[14:18:59][W]: 0178_OGLE-2019-BLG-0747_obs1-07-08-2019.fits
[14:18:59][W]: 0179_OGLE-2019-BLG-0747_obs1-07-08-2019_comp.fits
[14:18:59][W]: 0180_OGLE-2019-BLG-0747_obs1-07-08-2019.fits
[14:18:59][W]: 0181_OGLE-2019-BLG-0747_obs1-07-08-2019_comp.fits
[14:18:59][W]: 0182_OGLE-2019-BLG-0747_obs1-07-08-2019.fits
Traceback (most recent call last):
File "/data/miniconda3/envs/goodman_pipeline/bin/redccd", line 4, in
import('pkg_resources').run_script('goodman-pipeline==1.2.0', 'redccd')
File "/data/miniconda3/envs/goodman_pipeline/lib/python3.6/site-packages/pkg_resources/init.py", line 666, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/data/miniconda3/envs/goodman_pipeline/lib/python3.6/site-packages/pkg_resources/init.py", line 1453, in run_script
exec(code, namespace, namespace)
File "/data/miniconda3/envs/goodman_pipeline/lib/python3.6/site-packages/goodman_pipeline-1.2.0-py3.6.egg/EGG-INFO/scripts/redccd", line 12, in
GOODMAN_CCD()
File "/data/miniconda3/envs/goodman_pipeline/lib/python3.6/site-packages/goodman_pipeline-1.2.0-py3.6.egg/goodman_pipeline/images/goodman_ccd.py", line 247, in call
self.data_container.is_empty:
AttributeError: 'list' object has no attribute 'is_empty'

[rachel3834]

I re-ran the tests with pandas==0.24.0 instead, but I see the same failure

[simontorres]

I believe the right version is 0.23, In my bug report it says the latest pandas version working was 0.23.4

[simontorres]

The new error are most likely related but I can't say it this quick. The pipeline relies heavily on the keywords being correct and from what I see there might be a mix of Imaging and Spectroscopy data. Can you try to use the --debug argument to see more messages?

[rachel3834]

Ah, thanks, I misread the other issue thread. I can confirm that the tests run cleanly with pandas==0.23.4 installed.

I still see the second error however. The dataset consists of a science spectrum plus lampflat and arc exposures. Here is the output with the --debug flag set:

(goodman_pipeline) rstreet@rstreet-ubuntu:/data/ROMEREA/2019/SOAR/OGLE-2019-BLG-0747$ redccd --debug
WARNING: AstropyDeprecationWarning: astropy.extern.six will be removed in 4.0, use the six module directly if it is still needed [astropy.extern.six]
[14:35:27][I]: Starting Goodman HTS Pipeline Log [core.setup_logging:2046]
[14:35:27][I]: Local Time : 2019-08-08 14:35:27.758321 [core.setup_logging:2048]
[14:35:27][I]: Universal Time: 2019-08-08 21:35:27.758444 [core.setup_logging:2050]
[14:35:27][D]: Starting new HTTPS connection (1): api.github.com:443 [connectionpool._new_conn:815]
[14:35:28][D]: https://api.github.com:443 "GET /repos/soar-telescope/goodman/releases/latest HTTP/1.1" 301 170 [connectionpool._make_request:396]
[14:35:28][D]: https://api.github.com:443 "GET /repositories/62915938/releases/latest HTTP/1.1" 200 None [connectionpool._make_request:396]
[14:35:28][D]: Starting new HTTPS connection (1): api.github.com:443 [connectionpool._new_conn:815]
[14:35:28][D]: https://api.github.com:443 "GET /repos/soar-telescope/goodman/releases/latest HTTP/1.1" 301 170 [connectionpool._make_request:396]
[14:35:28][D]: https://api.github.com:443 "GET /repositories/62915938/releases/latest HTTP/1.1" 200 None [connectionpool._make_request:396]
[14:35:28][I]: Pipeline Version: 1.2.0 (latest) [core.setup_logging:2060]
[14:35:28][D]: Initializing DataClassifier instance [goodman_ccd.init:161]
[14:35:28][D]: Starting argument checks. [goodman_ccd._check_args:267]
[14:35:28][D]: Raw data folder "/data/ROMEREA/2019/SOAR/OGLE-2019-BLG-0747" exists. [goodman_ccd._check_args:272]
[14:35:28][I]: Found 5 fits files in /data/ROMEREA/2019/SOAR/OGLE-2019-BLG-0747. [goodman_ccd._check_args:284]
[14:35:28][I]: No special reduced data path defined. Proceeding with defaults. [goodman_ccd._check_args:293]
[14:35:28][D]: Folder for reduced data defined to: /data/ROMEREA/2019/SOAR/OGLE-2019-BLG-0747/RED [goodman_ccd._check_args:299]
[14:35:28][I]: Folder for reduced data is empty (good). [goodman_ccd._check_args:347]
[14:35:28][D]: /data/ROMEREA/2019/SOAR/OGLE-2019-BLG-0747/RED [goodman_ccd._check_args:349]
[14:35:28][D]: Calling data_classifier Instance of DataClassifier [goodman_ccd.call:199]
[14:35:28][D]: keywords in setter before pruning: . [image_collection.keywords:277]
[14:35:28][D]: keywords after pruning ['']. [image_collection.keywords:285]
[14:35:28][D]: should be building new table... [image_collection.keywords:299]
[14:35:28][D]: Raw path: /data/ROMEREA/2019/SOAR/OGLE-2019-BLG-0747 [data_classifier.call:77]
[14:35:28][D]: Detected Red camera. [data_classifier._get_instrument:124]
[14:35:28][I]: Instrument: Red Camera [data_classifier.call:81]
[14:35:28][I]: Detected Spectroscopy Data from Red Camera [data_classifier._get_obs_technique:159]
[14:35:28][I]: Observing Technique: Spectroscopy [data_classifier.call:90]
[14:35:28][D]: Initializing night organizer procedure [goodman_ccd.call:230]
[14:35:28][D]: Calling night organizer procedure [goodman_ccd.call:238]
[14:35:28][D]: keywords in setter before pruning: ['naxis', 'date', 'slit', 'date-obs', 'obstype', 'object', 'exptime', 'obsra', 'obsdec', 'grating', 'cam_targ', 'grt_targ', 'filter', 'filter2', 'gain', 'rdnoise', 'roi', 'wavmode']. [image_collection.keywords:277]
[14:35:28][D]: keywords after pruning ['date-obs', 'obsra', 'gain', 'roi', 'wavmode', 'obstype', 'obsdec', 'grating', 'date', 'naxis', 'slit', 'grt_targ', 'cam_targ', 'filter', 'filter2', 'object', 'rdnoise', 'exptime']. [image_collection.keywords:285]
[14:35:28][D]: should be building new table... [image_collection.keywords:299]
[14:35:29][W]: Ignoring all Imaging data. Assuming they are not science exposures [night_organizer.call:95]
[14:35:29][I]: Organizing data for this configuration: Gain: 1.48, Noise: 3.89, ROI: Spectroscopic 2x2 [night_organizer.call:140]
[14:35:30][D]: Sun Set 2019-08-06T22:08:50.092 [core.get_twilight_time:1199]
[14:35:30][D]: Sun Rise 2019-08-07T11:28:21.327 [core.get_twilight_time:1200]
[14:35:30][C]: There is no BIAS images for this configuration. Use --ignore-bias to proceed without BIAS. [night_organizer.spectroscopy_night:259]
[14:35:30][W]: The following files will be discarded: [night_organizer.call:169]
[14:35:30][W]: 0178_OGLE-2019-BLG-0747_obs1-07-08-2019.fits [night_organizer.call:171]
[14:35:30][W]: 0179_OGLE-2019-BLG-0747_obs1-07-08-2019_comp.fits [night_organizer.call:171]
[14:35:30][W]: 0180_OGLE-2019-BLG-0747_obs1-07-08-2019.fits [night_organizer.call:171]
[14:35:30][W]: 0181_OGLE-2019-BLG-0747_obs1-07-08-2019_comp.fits [night_organizer.call:171]
[14:35:30][W]: 0182_OGLE-2019-BLG-0747_obs1-07-08-2019.fits [night_organizer.call:171]
[14:35:30][D]: data_container is empty [night_organizer.call:172]
Traceback (most recent call last):
File "/data/miniconda3/envs/goodman_pipeline/bin/redccd", line 4, in
import('pkg_resources').run_script('goodman-pipeline==1.2.0', 'redccd')
File "/data/miniconda3/envs/goodman_pipeline/lib/python3.6/site-packages/pkg_resources/init.py", line 666, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/data/miniconda3/envs/goodman_pipeline/lib/python3.6/site-packages/pkg_resources/init.py", line 1453, in run_script
exec(code, namespace, namespace)
File "/data/miniconda3/envs/goodman_pipeline/lib/python3.6/site-packages/goodman_pipeline-1.2.0-py3.6.egg/EGG-INFO/scripts/redccd", line 12, in
GOODMAN_CCD()
File "/data/miniconda3/envs/goodman_pipeline/lib/python3.6/site-packages/goodman_pipeline-1.2.0-py3.6.egg/goodman_pipeline/images/goodman_ccd.py", line 247, in call
self.data_container.is_empty:
AttributeError: 'list' object has no attribute 'is_empty'

[simontorres]

OK. Most likely is the keywords. I have leave right now but sent you an invitation to our slack channel. I can reconnect later (2 to 3 hours) and try to figure this out. I might need to work on a cleaner exit message.

[rachel3834]

Many thanks for your quick response @simontorres. Since you've already addressed this issue, I'll close this thread and discuss the data via Slack.