`task` status is confusing and not working properly

[DavidGOrtega]

We have some race conditions aside not very clear definition in the status field that make insecure to use it to check if a task is done (despite its resources are not yet being disposed).

This is specially a bad issue for CML

[DavidGOrtega]

related to #357

[0x2b3bfa0]

A task is unequivocally “done” when status.succeeded is equal to the parallelism value. At least, when parallelism equals 1.

Can you add more information with regard to race conditions and other issues you've detected?

[0x2b3bfa0]

Still not sure whether this issue is going to be solved with code, documentation or both, but it's definitely worth the effort.

[0x2b3bfa0]

Off–topic: see https://airflow.apache.org/docs/apache-airflow/stable/concepts/tasks.html#task-instances for inspiration on additional statuses.

[DavidGOrtega]

I like apache airflow statuses, is what we defined originally.

Look at this three examples of task being refreshed after task completion

    "status":{"running":0} -> should be "status":{"running":1} ?
    "status":{"running":0,"succeeded":2} -> succeeded 2?

    "status":{"running":1}
    "status":{"running":1,"succeeded":1}
    "status":{"running":0,"succeeded":2} -> why again 2?

    "status":{"running":1}
    "status":{"running":1,"succeeded":1}

I have even seen "status":{"running":1,"succeeded":6}

Its succeeded related to the time the task is refreshed?
Running is attached to the disposal of the instance and its misleading. To me running should be reflecting the state of the task as a bash script and not tied to the hardware.
Also succeeded looks like if the task itself had succeeded
As you say and what I wanted to say is from trial 1 to 2 both tasks were finished with valid results but the output is difficult to assimilate.

[0x2b3bfa0]

Look at this three examples of task being refreshed after task completion [...] I have even seen "status":{"running":1,"succeeded":6}

That's what I wanted to know, thank you very much! 🙏🏼 If this happened with parallelism set to 1, it's a bug. Can you please provide a minimal example to reproduce this condition?

Its succeeded related to the time the task is refreshed?

No, it shouldn't. 🙃

Running is attached to the disposal of the instance and its misleading. To me running should be reflecting the state of the task as a bash script and not tied to the hardware.

In that case, what should happen amidst two spot instances; waiting / suspended?

Currently, running means that there is a cloud instance running, and, implicitly, that an script has been launched on it. If the script fails or succeeds, the status will transition accordingly to failed or succeeded.

Also succeeded looks like if the task itself had succeeded

Yes, it means exactly that. If the current behavior is different, it's a bug.

As you say and what I wanted to say is from trial 1 to 2 both tasks were finished with valid results but the output is difficult to assimilate.

Can you please elaborate on the meaning of “trial 1 to 2”?

[DavidGOrtega]

That's what I wanted to know, thank you very much! 🙏🏼 If this happened with parallelism set to 1, it's a bug. Can you please provide a minimal example to reproduce this condition?

Sure! no parallelism being used. This is the basic example that allowed me to reach that condition.

resource "iterative_task" "basic" {
    provider = iterative

    cloud     = "az"
    region    = "us-east"
    machine   = "Standard_D2S_v3"

    workdir {
        input = "./"
    }

    script = <<-END
    #!/bin/bash
    echo 'hello tpi' > data.txt
    printenv >> data.txt
    END
}

[DavidGOrtega]

Yes, it means exactly that. If the current behavior is different, it's a bug.

But in the context of the script or just the internal execution of the task? If my script fails the task succeeded will still be true right?