Crash inside servicex causes another crash in servicex-analysis-utils

[gordonwatts]

There is only one interesting crash here - which is the one deep inside sx due to a failed transform. The other crash (index out of range) should not occur - and can confuse people form the real source of the crash.

r21_125_15: Transform ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/1 --:--
             Download ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/1 --:--
[04/30/25 12:08:38] ERROR    Transform "r21_125_15" completed with failures: 1/1 files            [query_core.py](file:///C:/Users/gwatts/Code/iris-hep/corrections/.venv/Lib/site-packages/servicex/query_core.py):[235](file:///C:/Users/gwatts/Code/iris-hep/corrections/.venv/Lib/site-packages/servicex/query_core.py#235)
                             failed.Will not cache.                                                                
                    ERROR    A list of failed files is at [this link](https://servicex.af.uchicago.edu//transformation-request//34693da0-dfc2-4380-a47f-6ec2f4353608/results?status=failure)                               [query_core.py](file:///C:/Users/gwatts/Code/iris-hep/corrections/.venv/Lib/site-packages/servicex/query_core.py):[236](file:///C:/Users/gwatts/Code/iris-hep/corrections/.venv/Lib/site-packages/servicex/query_core.py#236)
                    ERROR    Transform Request id: 34693da0-dfc2-4380-a47f-6ec2f4353608           [query_core.py](file:///C:/Users/gwatts/Code/iris-hep/corrections/.venv/Lib/site-packages/servicex/query_core.py):[237](file:///C:/Users/gwatts/Code/iris-hep/corrections/.venv/Lib/site-packages/servicex/query_core.py#237)
                    ERROR    More information of 'r21_125_15' [HERE](https://atlas-kibana.mwt2.org:5601/s/servicex/app/dashboards?auth_provider_hint=anonymous1#/view/6d069520-f34e-11ed-a6d8-9f6a16cd6d78?embed=true&_g=(time:(from:now-30d%2Fd,to:now))&_a=(filters:!((query:(match_phrase:(requestId:'34693da0-dfc2-4380-a47f-6ec2f4353608'))),(query:(match_phrase:(level:'error')))))&show-time-filter=true)                                [query_core.py](file:///C:/Users/gwatts/Code/iris-hep/corrections/.venv/Lib/site-packages/servicex/query_core.py):[247](file:///C:/Users/gwatts/Code/iris-hep/corrections/.venv/Lib/site-packages/servicex/query_core.py#247)
Exception in callback Query.submit_and_download.<locals>.transform_complete() at 
c:\Users\gwatts\Code\iris-hep\corrections\.venv\Lib\site-packages\servicex\query_core.py:194
handle: <Handle Query.submit_and_download.<locals>.transform_complete() at 
c:\Users\gwatts\Code\iris-hep\corrections\.venv\Lib\site-packages\servicex\query_core.py:194>
Traceback (most recent call last):
  File "C:\Users\gwatts\AppData\Local\Programs\Python\Python313\Lib\asyncio\events.py", line 89, in _run
    self._context.run(self._callback, *self._args)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\gwatts\Code\iris-hep\corrections\.venv\Lib\site-packages\servicex\query_core.py", line 252, in 
transform_complete
    raise ServiceXException(errorstr)
servicex.query_core.ServiceXException: Transform "r21_125_15" completed with failures: 1/1 files failed.Will not 
cache.
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
Cell In[10], line 13
      1 jet_query_corrected = (
      2     FuncADLQueryPHYS()
      3     .Select(lambda e: e.Jets())
   (...)     10     )
     11 )
---> 13 data_corrected = to_awk(
     14     deliver(
     15         {
     16             "Sample": [
     17                 {"Name": "r21_125_15", "Dataset": ds, "Query": jet_query_corrected}
     18             ]
     19         }
     20     )
     21 )

File c:\Users\gwatts\Code\iris-hep\corrections\.venv\Lib\site-packages\servicex_analysis_utils\materialization.py:54, in to_awk(deliver_dict, dask, iterator, **kwargs)
     50 awk_arrays = {}
     52 for sample, paths in deliver_dict.items():
     53     # Check file type
---> 54     f_type = str(paths[0])
     55     if ".root" in f_type:
     56         is_root = True

File c:\Users\gwatts\Code\iris-hep\corrections\.venv\Lib\site-packages\servicex\servicex_client.py:100, in GuardList.__getitem__(self, index)
     98 else:
     99     data = cast(Sequence, self._data)
--> 100     return data[index]

IndexError: list index out of ranger21_125_15: Transform ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/1 --:--
             Download ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/1 --:--
[04/30/25 12:08:38] ERROR    Transform "r21_125_15" completed with failures: 1/1 files            [query_core.py](file:///C:/Users/gwatts/Code/iris-hep/corrections/.venv/Lib/site-packages/servicex/query_core.py):[235](file:///C:/Users/gwatts/Code/iris-hep/corrections/.venv/Lib/site-packages/servicex/query_core.py#235)
                             failed.Will not cache.                                                                
                    ERROR    A list of failed files is at [this link](https://servicex.af.uchicago.edu//transformation-request//34693da0-dfc2-4380-a47f-6ec2f4353608/results?status=failure)                               [query_core.py](file:///C:/Users/gwatts/Code/iris-hep/corrections/.venv/Lib/site-packages/servicex/query_core.py):[236](file:///C:/Users/gwatts/Code/iris-hep/corrections/.venv/Lib/site-packages/servicex/query_core.py#236)
                    ERROR    Transform Request id: 34693da0-dfc2-4380-a47f-6ec2f4353608           [query_core.py](file:///C:/Users/gwatts/Code/iris-hep/corrections/.venv/Lib/site-packages/servicex/query_core.py):[237](file:///C:/Users/gwatts/Code/iris-hep/corrections/.venv/Lib/site-packages/servicex/query_core.py#237)
                    ERROR    More information of 'r21_125_15' [HERE](https://atlas-kibana.mwt2.org:5601/s/servicex/app/dashboards?auth_provider_hint=anonymous1#/view/6d069520-f34e-11ed-a6d8-9f6a16cd6d78?embed=true&_g=(time:(from:now-30d%2Fd,to:now))&_a=(filters:!((query:(match_phrase:(requestId:'34693da0-dfc2-4380-a47f-6ec2f4353608'))),(query:(match_phrase:(level:'error')))))&show-time-filter=true)                                [query_core.py](file:///C:/Users/gwatts/Code/iris-hep/corrections/.venv/Lib/site-packages/servicex/query_core.py):[247](file:///C:/Users/gwatts/Code/iris-hep/corrections/.venv/Lib/site-packages/servicex/query_core.py#247)
Exception in callback Query.submit_and_download.<locals>.transform_complete() at 
c:\Users\gwatts\Code\iris-hep\corrections\.venv\Lib\site-packages\servicex\query_core.py:194
handle: <Handle Query.submit_and_download.<locals>.transform_complete() at 
c:\Users\gwatts\Code\iris-hep\corrections\.venv\Lib\site-packages\servicex\query_core.py:194>
Traceback (most recent call last):
  File "C:\Users\gwatts\AppData\Local\Programs\Python\Python313\Lib\asyncio\events.py", line 89, in _run
    self._context.run(self._callback, *self._args)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\gwatts\Code\iris-hep\corrections\.venv\Lib\site-packages\servicex\query_core.py", line 252, in 
transform_complete
    raise ServiceXException(errorstr)
servicex.query_core.ServiceXException: Transform "r21_125_15" completed with failures: 1/1 files failed.Will not 
cache.
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
Cell In[10], line 13
      1 jet_query_corrected = (
      2     FuncADLQueryPHYS()
      3     .Select(lambda e: e.Jets())
   (...)     10     )
     11 )
---> 13 data_corrected = to_awk(
     14     deliver(
     15         {
     16             "Sample": [
     17                 {"Name": "r21_125_15", "Dataset": ds, "Query": jet_query_corrected}
     18             ]
     19         }
     20     )
     21 )

File c:\Users\gwatts\Code\iris-hep\corrections\.venv\Lib\site-packages\servicex_analysis_utils\materialization.py:54, in to_awk(deliver_dict, dask, iterator, **kwargs)
     50 awk_arrays = {}
     52 for sample, paths in deliver_dict.items():
     53     # Check file type
---> 54     f_type = str(paths[0])
     55     if ".root" in f_type:
     56         is_root = True

File c:\Users\gwatts\Code\iris-hep\corrections\.venv\Lib\site-packages\servicex\servicex_client.py:100, in GuardList.__getitem__(self, index)
     98 else:
     99     data = cast(Sequence, self._data)
--> 100     return data[index]

IndexError: list index out of range

[ArturU043]

Hi,

Although deliver failed, it must have returned a dictionary that was passed to to_awk. This seems to be a FE issue...

In any case, I added a raise ValueError to to_awk if the input is invalid and pointing to that in the comments.