Support execution stats

[joshliberty]

We should allow task plug-ins to generate an object containing execution stats upon completion. This will allow saving information such as timings, IDs, references, and other information which is specific to the app server that is executed the task.

The exact content of the execution stats object will depend on the specific plug-in. For example, the Argo plug-in should at the very least include the Argo workflow ID so that users can utilise the Argo API to retrieve stats such as memory and CPU used by the task.

While we allow model developers to generate meta data which is stored with the task upon completion, here we are talking about data defined by the plug-in itself, rather than by developers.

This is essentially the inverse of the task_plugin_arguments object which is part of the task definition in the workflow spec.

Argo plug-in execution stats

The following information should be included in the execution stats object generated by the Argo plug-in upon successful completion of an Argo task.

Must haves:

• Argo Workflow Id.
• Argo Execution Time.

Nice to haves:

• Resource usage.
• Pod IDs

Acceptance criteria

1. GIVEN a task completes WHEN a task plug-in generates an execution stats object THEN it should be stored in the database along with the task object
2. GIVEN a task completed AND was saved with an execution stats object WHEN the task completion is logged THEN the execution stats object should be included in the log (Log task completions #170 )

Implementation notes

• The execution stats object needs to be added to the task status update event
• It then needs to be saved to MongoDB to the execution object of the completed task

[lillie-dae]

just some notes for myself and future developers if they find this...

The Argo.WorkflowStatus object is produced from Workflow.Status and has EstimatedDuration for time, also has resourceDuration which is a dict so we can get CPU usage and Memory usage and GPU usage from that.

Also there is then further metrics from Workflow.Spec.Prometheus that can product complex analytics and real time data.

See here for further info on that
https://argoproj.github.io/argo-workflows/metrics/
thats accessed by referencing the Workflow.Spec

[lillie-dae]

I've added

"workflowId" : string
"duration" : int (if its -1 means it came back null on argo side)
"resourceDuration": dict<string, long> ( https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ for further info)
"nodeInfo": dict<string, NodeStatus> key is id of the pod value is NodeStatus(see "io.argoproj.workflow.v1alpha1.NodeStatus" on https://raw.githubusercontent.com/argoproj/argo-workflows/master/api/openapi-spec/swagger.json for further info)
"startedAt": datetimeoffset
"finishedAt": datetimeoffset

[neildsouth]

These are added now