BUG: IRSA: Some catalogs produce bad queries

[keflavich]

I'm trying to query some WISE catalogs, and not all work. It looks like some of the listed catalogs cannot be spatially queried.

Traceback (most recent call last):
  Cell In[11], line 2
    result = Irsa.query_region(coord, radius=1*u.arcmin, catalog=key)
  File ~/mambaforge/envs/py312/lib/python3.12/site-packages/astropy/utils/decorators.py:618 in wrapper
    return function(*args, **kwargs)
  File ~/repos/astroquery/astroquery/ipac/irsa/core.py:302 in query_region
    response = self.query_tap(query=adql, async_job=async_job)
  File ~/repos/astroquery/astroquery/ipac/irsa/core.py:85 in query_tap
    return self.tap.run_sync(query, language='ADQL', maxrec=maxrec)
  File ~/mambaforge/envs/py312/lib/python3.12/site-packages/pyvo/dal/tap.py:282 in run_sync
    **keywords).execute()
  File ~/mambaforge/envs/py312/lib/python3.12/site-packages/pyvo/dal/tap.py:1121 in execute
    return TAPResults(self.execute_votable(), url=self.queryurl, session=self._session)
  File ~/mambaforge/envs/py312/lib/python3.12/site-packages/pyvo/dal/adhoc.py:114 in __init__
    super().__init__(votable, url=url, session=session)
  File ~/mambaforge/envs/py312/lib/python3.12/site-packages/pyvo/dal/query.py:338 in __init__
    raise DALQueryError(self._status[1], self._status[0], url)
DALQueryError: UsageFault: BAD_REQUEST: ODPI call failed.    function name:: dpiStmt_execute    ACTION: execute   Message: ORA-00904: "ALLWISE_P3AM_XRF"."Z": invalid identifier

Here's a systematic search through WISE data:

for key in Irsa.list_catalogs(filter='wise'):
     try:
         result = Irsa.query_region(coord, radius=1*u.arcmin, catalog=key)
         print(key, len(result))
     except Exception as e:
         print(key, e)

allwise_p3as_psd 22
allwise_p3as_mep 437
allwise_p3as_psr 1
allwise_p3as_cdd 0
allwise_p3am_xrf UsageFault: BAD_REQUEST: ODPI call failed.    function name:: dpiStmt_execute    ACTION: execute   Message: ORA-00904: "ALLWISE_P3AM_XRF"."Z": invalid identifier
allwise_p3al_lod UsageFault: BAD_REQUEST: ODPI call failed.    function name:: dpiStmt_execute    ACTION: execute   Message: ORA-00904: "ALLWISE_P3AL_LOD"."Z": invalid identifier
allwise_p3am_cdd UsageFault: BAD_REQUEST: ODPI call failed.    function name:: dpiStmt_execute    ACTION: execute   Message: ORA-00904: "ALLWISE_P3AM_CDD"."HTM20": invalid identifier
wise.wise_allwise_p3am_cdd UsageFault: BAD_REQUEST: ERROR:  column "ra" does not exist
LINE 1: ...hy, (0.0166667)*111194.68229846345), ST_MakePoint(ra,dec)::g...
                                                             ^
wise.wise_allwise_merge_p3am_cdd UsageFault: BAD_REQUEST: ERROR:  column "ra" does not exist
LINE 1: ...hy, (0.0166667)*111194.68229846345), ST_MakePoint(ra,dec)::g...
                                                             ^
allwise_mfpos UsageFault: BAD_REQUEST: ODPI call failed.    function name:: dpiStmt_execute    ACTION: execute   Message: ORA-00904: "ALLWISE_MFPOS"."Z": invalid identifier
neowiser_p1bs_psd 4933
neowiser_p1ba_mch 25
neowiser_p1bs_frm 1
neowiser_p1bl_lod UsageFault: BAD_REQUEST: ODPI call failed.    function name:: dpiStmt_execute    ACTION: execute   Message: ORA-00904: "NEOWISER_P1BL_LOD"."Z": invalid identifier
neowiser_p1bm_frm 2
wise.neowiser_p1bm_frm UsageFault: BAD_REQUEST: ERROR:  column "ra" does not exist
LINE 1: ...hy, (0.0166667)*111194.68229846345), ST_MakePoint(ra,dec)::g...
                                                             ^
wise.neowiser_merge_p1bm_frm UsageFault: BAD_REQUEST: ERROR:  column "ra" does not exist
LINE 1: ...hy, (0.0166667)*111194.68229846345), ST_MakePoint(ra,dec)::g...
                                                             ^
wise.wise_allsky_4band_p1bm_frm UsageFault: BAD_REQUEST: ERROR:  column "ra" does not exist
LINE 1: ...hy, (0.0166667)*111194.68229846345), ST_MakePoint(ra,dec)::g...
                                                             ^
wise.wise_allsky_4band_p3am_cdd UsageFault: BAD_REQUEST: ERROR:  column "ra" does not exist
LINE 1: ...hy, (0.0166667)*111194.68229846345), ST_MakePoint(ra,dec)::g...
                                                             ^
wise.wise_allsky_3band_p1bm_frm UsageFault: BAD_REQUEST: ERROR:  column "ra" does not exist
LINE 1: ...hy, (0.0166667)*111194.68229846345), ST_MakePoint(ra,dec)::g...
                                                             ^
wise.wise_allsky_3band_p3am_cdd UsageFault: BAD_REQUEST: ERROR:  column "ra" does not exist
LINE 1: ...hy, (0.0166667)*111194.68229846345), ST_MakePoint(ra,dec)::g...
                                                             ^
wise.wise_allsky_2band_p1bm_frm UsageFault: BAD_REQUEST: ERROR:  column "ra" does not exist
LINE 1: ...hy, (0.0166667)*111194.68229846345), ST_MakePoint(ra,dec)::g...
                                                             ^
catwise_2020 11
catwise_2020_reject 68
catwise_prelim 6
catwise_prelim_reject 14
unwise_2019 95
unwise.unwise_neo3_images 0
neowisesbpropv2 UsageFault: BAD_REQUEST: ODPI call failed.    function name:: dpiStmt_execute    ACTION: execute   Message: ORA-00904: "NEOWISESBPROPV2"."Z": invalid identifier
wise.wise_prelim_2band_p1bm_frm UsageFault: BAD_REQUEST: ERROR:  column "ra" does not exist
LINE 1: ...hy, (0.0166667)*111194.68229846345), ST_MakePoint(ra,dec)::g...
                                                             ^
wise.wise_prelim_p1bm_frm UsageFault: BAD_REQUEST: ERROR:  column "ra" does not exist
LINE 1: ...hy, (0.0166667)*111194.68229846345), ST_MakePoint(ra,dec)::g...
                                                             ^
wise.wise_prelim_p3am_cdd UsageFault: BAD_REQUEST: ERROR:  column "ra" does not exist
LINE 1: ...hy, (0.0166667)*111194.68229846345), ST_MakePoint(ra,dec)::g...
                                                             ^
gaia_dr2_allwise_bnbr UsageFault: BAD_REQUEST: ODPI call failed.    function name:: dpiStmt_execute    ACTION: execute   Message: ORA-00904: "GAIA_DR2_ALLWISE_BNBR"."Z": invalid identifier
gaia_dr2_allwise_nbh UsageFault: BAD_REQUEST: ODPI call failed.    function name:: dpiStmt_execute    ACTION: execute   Message: ORA-00904: "GAIA_DR2_ALLWISE_NBH"."Z": invalid identifier
gaia_dr2_allwise_agn_gdr2_xid UsageFault: BAD_REQUEST: ODPI call failed.    function name:: dpiStmt_execute    ACTION: execute   Message: ORA-00904: "GAIA_DR2_ALLWISE_AGN_GDR2_XID"."Z": invalid identifier
gaia_allwise_best_neighbour UsageFault: BAD_REQUEST: ODPI call failed.    function name:: dpiStmt_execute    ACTION: execute   Message: ORA-00904: "GAIA_ALLWISE_BEST_NEIGHBOUR"."Z": invalid identifier
gaia_allwise_neighbourhood UsageFault: BAD_REQUEST: ODPI call failed.    function name:: dpiStmt_execute    ACTION: execute   Message: ORA-00904: "GAIA_ALLWISE_NEIGHBOURHOOD"."Z": invalid identifier
wisegalhii 7

@bsipocz I can't see a common theme here; there are successes from some that don't have irsa_spatial_idx_name, so it's not consistently just that data that's missing. Is there any way we could avoid listing the catalogs that will fail this way, or tell users that they can't do spatial searches on those?

[bsipocz]

I'll propagate this issue upstream next week, and we may need to adjust the query that lists these in the first place.

[bsipocz]

not that it should matter, but what was the coordinate you used?

[bsipocz]

To relay some of the answers: Some of these catalogs are not catalogs (and thus an astroquery issues to fix) but image metadata tables; thus there is no central coordinate but 4 coordinate pairs for the corners. It makes no sense to run query_region on them; therefore I open an issue to change list_catalogs to not include these (with an opt in include_metadata_tables=True)

[keflavich]

Sounds good. Is there a way to search for overlap with images, then? That's actually what i was trying to do - find all images in those image metadata tables that covered a point.

(I used Sgr B2, i.e., SkyCoord.from_name('Sgr B2'))

[bsipocz]

Hmm, why not just run SIA queries? Those should return the images overlapping with a coordinate you provide.

[keflavich]

i think that didn't work and/or I didn't get the right collections. Ah.... but really it's because the images I needed were on SIA v1 but I was using SIA v2? I think that's what you told me

[bsipocz]

yeap, for SIA1 you need to use pyvo (I will add a note about that into the docs), but it should still return all the images for a coordinate.