Introduce eql search status API

[mayya-sharipova]

Introduce eql search status API,
that reports the status of eql stored or async search.

GET _eql/search/status/

The API is restricted to the monitoring_user role.

For a running eql search, a response has the following format:

{
  "id" : <id>,
  "is_running" : true,
  "is_partial" : true,
  "start_time_in_millis" : 1611690235000,
  "expiration_time_in_millis" : 1611690295000
}

For a completed eql search, a response has the following format:

{
  "id" : <id>,
  "is_running" : false,
  "is_partial" : false,
  "expiration_time_in_millis" : 1611690295000,
  "completion_status" : 200
}

Closes #66955

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[elasticmachine]

Pinging @elastic/es-ql (Team:QL)

[mayya-sharipova]

@lizozom Unfortunately for eql searches, we could not the same response format as for async searches, as we don't store shards progress for eql searches. I hope the format presented above works for you.

[imotov]

Looks good to me in general. Left a couple of comments. I think it would be great if @costin could also take a look.

[imotov]

There is quite a bit of code repetition between AsyncStatusResponse and EqlStatusResponse, and between TransportEqlAsyncGetStatusAction and TransportGetAsyncStatusAction, any reason for not moving more shared things into SearchStatusResponse and AsyncTaskIndexService respectively?

[mayya-sharipova]

@imotov Thank for the feedback, addressed in 4321efb. Can you please continue with the review when you have time.

I was not able to combine AsyncStatusResponse and EqlStatusResponse together because it would complicate the wire serialization/deserialization code. As we already have AsyncStatusResponse form 7.11, I did not want to change that part of the code.

[mayya-sharipova]

@imotov @astefan Thanks for reviewing the PR. I will study and address your comments.
One questions I have is that the Kibana team is interested to show the progress of EQL async search task, but it looks like currently EQL doesn't store partial results. Is there a plan to store partial results in a similar way how async search stores them?

[costin]

LGTM to me from my end though I'm not familiar with the async code so I'm non-binding :)
Regarding the EQL results, and this should be a separate PR, how can the partial information be exposed? The completed sequences (or in case of events query searches) are kept based on their doc ids and their sources retrieved at the end (see

elasticsearch/x-pack/plugin/eql/src/main/java/org/elasticsearch/xpack/eql/execution/sequence/SequenceMatcher.java

Line 250 in ef26b53

List<Sequence> completed() {

).
However it should be trivial to signal a new match during the process.

[mayya-sharipova]

@costin Thank you for checking.

Regarding the EQL results, and this should be a separate PR, how can the partial information be exposed?

For a partial information for a status report, we don't need partial results, Kibana just needs progress indication about the number of the completed shards so far and total shards.
But overall, I was wondering if the EQL team has plans to store partial results or is that something that doesn't make sense for EQL and is not planned?

[mark-vieira]

@elasticmachine update branch

[imotov]

Thanks for the modifications! Left a small comment about a potential race condition in the integration test. Otherwise LGTM.

[imotov]

These two checks will probably fail occasionally due to a race condition with task completion.

[mayya-sharipova]

@imotov Thanks! Good feedback! Indeed, we are not certain if the task is completed or not by this time! I changed the test in 4eca9a0 to test for less.