Elastic Search 7.16.1 Superset 1.4 missing Tables for default schema

[vincentytran]

In Superset 1.4.0 I'm connecting to Elastic Search 7.16.1 and I'm having trouble seeing any Tables show up in the list for the default schema. Is this because elasticsearch-dbapi doesn't support this latest release yet because pip says that elasticsearch-dbapi requires elasticsearch release < 7.14.0.

I went through many of the issues here and I even created a test index test_index_000001 with a single test message. That doesn't appear to show up in superset too.

These newer releases of Elastic Search fix the log4j exploit which is why we upgraded.

Superset logs seems to show it run SHOW TABLES without errors.

10.15.10.175 - - [21/Jan/2022:09:58:00 +0000] "GET /superset/tables/3/default/undefined/false/ HTTP/1.1" 200 33 "http://myserverhere:8088/tablemodelview/list/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:96.0) Gecko/20100101 Firefox/96.0",
2022-01-21 09:58:00,129:DEBUG:elasticsearch:> {"query":"SHOW TABLES","fetch_size":10000},
2022-01-21 09:58:00,129:INFO:elasticsearch:POST http://myserverhere:9200/_sql/ [status:200 request:0.011s]
2022-01-21 09:58:00,129:DEBUG:elasticsearch:< {"columns":[{"name":"catalog","type":"keyword"},{"name":"name","type":"keyword"},{"name":"type","type":"keyword"},{"name":"kind","type":"keyword"}],"rows":[["docker-cluster",".apm-agent-configuration","TABLE","INDEX"],["docker-cluster",".apm-custom-link","TABLE","INDEX"],["docker-cluster",".async-search","TABLE","INDEX"],["docker-cluster",".data-frame-internal-3","VIEW","ALIAS"],["docker-cluster",".kibana","VIEW","ALIAS"],["docker-cluster",".kibana-event-log-7.10.1","VIEW","ALIAS"],["docker-cluster",".kibana_1","TABLE","INDEX"],["docker-cluster",".kibana_7.16.1","VIEW","ALIAS"],["docker-cluster",".kibana_7.16.1_001","TABLE","INDEX"],["docker-cluster",".kibana_task_manager","VIEW","ALIAS"],["docker-cluster",".kibana_task_manager_1","TABLE","INDEX"],["docker-cluster",".kibana_task_manager_7.16.1","VIEW","ALIAS"],["docker-cluster",".kibana_task_manager_7.16.1_001","TABLE","INDEX"],["docker-cluster",".monitoring-es-7-2022.01.15","TABLE","INDEX"],["docker-cluster",".monitoring-es-7-2022.01.16","TABLE","INDEX"],["docker-cluster",".monitoring-es-7-2022.01.17","TABLE","INDEX"],["docker-cluster",".monitoring-es-7-2022.01.18","TABLE","INDEX"],["docker-cluster",".monitoring-es-7-2022.01.19","TABLE","INDEX"],["docker-cluster",".monitoring-es-7-2022.01.20","TABLE","INDEX"],["docker-cluster",".monitoring-es-7-2022.01.21","TABLE","INDEX"],["docker-cluster",".monitoring-kibana-7-2022.01.15","TABLE","INDEX"],["docker-cluster",".monitoring-kibana-7-2022.01.16","TABLE","INDEX"],["docker-cluster",".monitoring-kibana-7-2022.01.17","TABLE","INDEX"],["docker-cluster",".monitoring-kibana-7-2022.01.18","TABLE","INDEX"],["docker-cluster",".monitoring-kibana-7-2022.01.19","TABLE","INDEX"],["docker-cluster",".monitoring-kibana-7-2022.01.20","TABLE","INDEX"],["docker-cluster",".monitoring-kibana-7-2022.01.21","TABLE","INDEX"],["docker-cluster",".monitoring-logstash-7-2022.01.15","TABLE","INDEX"],["docker-cluster",".monitoring-logstash-7-2022.01.16","TABLE","INDEX"],["docker-cluster",".monitoring-logstash-7-2022.01.17","TABLE","INDEX"],["docker-cluster",".monitoring-logstash-7-2022.01.18","TABLE","INDEX"],["docker-cluster",".monitoring-logstash-7-2022.01.19","TABLE","INDEX"],["docker-cluster",".monitoring-logstash-7-2022.01.20","TABLE","INDEX"],["docker-cluster",".monitoring-logstash-7-2022.01.21","TABLE","INDEX"],["docker-cluster",".security","VIEW","ALIAS"],["docker-cluster",".security-7","TABLE","INDEX"],["docker-cluster",".siem-signals-default","VIEW","ALIAS"],["docker-cluster",".siem-signals-default-000001","TABLE","INDEX"],["docker-cluster",".siem-signals-default-000002","TABLE","INDEX"],["docker-cluster",".siem-signals-default-000003","TABLE","INDEX"],["docker-cluster",".siem-signals-default-000004","TABLE","INDEX"],["docker-cluster",".siem-signals-default-000005","TABLE","INDEX"],["docker-cluster",".siem-signals-default-000006","TABLE","INDEX"],["docker-cluster",".siem-signals-default-000007","TABLE","INDEX"],["docker-cluster",".siem-signals-default-000008","TABLE","INDEX"],["docker-cluster",".siem-signals-default-000009","TABLE","INDEX"],["docker-cluster",".siem-signals-default-000010","TABLE","INDEX"],["docker-cluster",".siem-signals-default-000011","TABLE","INDEX"],["docker-cluster",".siem-signals-default-000012","TABLE","INDEX"],["docker-cluster",".siem-signals-default-000013","TABLE","INDEX"],["docker-cluster",".tasks","TABLE","INDEX"],["docker-cluster",".transform-internal-005","TABLE","INDEX"],["docker-cluster",".transform-internal-007","TABLE","INDEX"],["docker-cluster","ilm-history-3","VIEW","ALIAS"],["docker-cluster","ilm-history-3-000013","TABLE","INDEX"],["docker-cluster","internal","TABLE","INDEX"],["docker-cluster","logstash","TABLE","INDEX"],["docker-cluster","metricbeat-7.10.1","VIEW","ALIAS"],["docker-cluster","metricbeat-7.10.1-2021.12.03-000018","TABLE","INDEX"],["docker-cluster","metricbeat-7.10.1-2022.01.02-000019","TABLE","INDEX"],["docker-cluster","metricbeat-7.16.1","VIEW","ALIAS"],["docker-cluster","metricbeat-7.16.1-2021.12.15-000001","TABLE","INDEX"],["docker-cluster","metricbeat-7.16.1-2021.12.26-000002","TABLE","INDEX"],["docker-cluster","metricbeat-7.16.1-2022.01.06-000003","TABLE","INDEX"],["docker-cluster","metricbeat-7.16.1-2022.01.16-000004","TABLE","INDEX"],["docker-cluster","metrics-endpoint.metadata_current_default","TABLE","INDEX"],["docker-cluster","test_index_000001","TABLE","INDEX"]]}

[frbenoit]

I have exactly the same problem with elasticsearch 7.16.2 and superset 1.3.1 and 1.3.2

[dpgaspar]

Sorry for the delay, I'll troubleshoot this week

[dpgaspar]

We have pinned elasticsearch-py here: #71
Because on 7.14 Elastic broke compatibility with AWS elasticsearch now called Opensearch. Can you confirm this the type of Elasticsearch your using?

[vincentytran]

We have pinned elasticsearch-py here: #71
Because on 7.14 Elastic broke compatibility with AWS elasticsearch now called Opensearch. Can you confirm this the type of Elasticsearch your using?

The issue is with the official elastic search distribution.

[dpgaspar]

@vincentytran I was able to replicate the problem, I'll work on a fix has soon has possible

[dpgaspar]

@vincentytran Just merged a fix for this issue, would be great if you could test it. I'll make a new release soon

[dpgaspar]

released

[frbenoit]

Hello, sorry for the delay, in my company we just have try this morning. We have always the issue we cannot discover the index of elasticsearch from superset. We have try to use the pluging version 0.2.8 and 0.2.9 with elasticsearch 7.16.2 and 8.1.0 but it's the same kind of issue.

[paularmand]

Just tried this too with preset.io (unsure what plugin version that is) against a cloud managed elastic (elastic.co) v7.17.1 but tables list remains empty on default schema.

[frbenoit]

The error that i described it's exactly the same than you Paul. In my side we are using the normal elastic not the elastic managed.

[paularmand]

@dpgaspar possible to reopen this one?

[vincent-imt]

FYI - the fix that was released did fix the problem for me.

[frbenoit]

Wich version of the pluging are you used ? Wich version of elastic are you used ?