Add support for tracking missing results

[gctucker]

When a test is not run or the result couldn't be sent to the API, for any reason, it would be very helpful to be able to report a warning. There are many reasons why this could happen, for example:

• The test has been removed from the test suite
• The test timed out
• The test crashed the system or the main program
• The system or the main program crashed before starting the test
• The results were in an invalid format
• The infrastructure failed (network, power, storage...)

Some of these cases are normal and expected, some are probably due to real kernel issues, some to test issues, some to infrastructure issues... Maybe with additional information we could refine the result, but by default this could be reported as a "warning" to the users.

Here's a proposed implementation:

• Before submitting a job, send nodes to the API for each expected test result (either loaded from some config or read back from the previous test run) with the result set to null
• When the job runs, nodes get updated with the results from each test as pass/fail/skip
• If any test wasn't run then its result will still be null
• Email reports, web dashboards etc. can interpret a node in the done state with a null result as a missing test case

Right now, the pipeline scheduler creates just one top-level node for each job and the job then sends a hierarchy of nodes for the results. The proposed implementation would mean creating a hierarchy with no results in the scheduler and updating the existing nodes in the job rather than creating new ones (which would lead to duplicates otherwise). The helper method and API endpoints to submit a hierarchy of nodes would need to be tweaked to look for existing nodes first and update any that has a matching path or create new ones otherwise. So the job logic itself wouldn't need to be changed.

[nuclearcat]

Thats one of features i am dreaming about.
Maybe we can implement also limited retry mechanism, if certain result is missing after timeout.