Failures handling best practice

[bguban]

Hi,
I have Ruby on Rails app with Resque as a background jobs engine. I need to notify developers when some of taskinator flows failed.
As I can see taskinator uses ActiveSupport::Notifications. Does it mean that I should subscribe to failure events in an initializer? If I have several taskinator processes how to determine which process failed? I have neither process class nor file name in the arguments passed into ActiveSupport::Notifications. Also would be good to know a task/job name.
Is it possible to continue execution from failed task/job after the problem was fixed or retry the failed task/job automatically?
Is it possible to set up on_failure hooks?

[virtualstaticvoid]

Hi @bguban

Yeah, this is an area where taskinator isn't fully fledged. I've only ever used the notifications to ActiveSupport::Notifications to wire up statsd metrics, and have a grafana dashboard which shows if there were any errors. I then manually retry the failed resque job in the resque-web interface, which effectively retries the relevant failed process.

Originally I planned to build support for inspecting processes and their tasks via Taskinator::Api::Processes but never got around to more than just providing the list of processes. I guess this could be extended to provide filters for failed processes and provide a retry method for a process.

I'll look into your suggestions to have a built-in retry mechanism and on_failure hooks.

[virtualstaticvoid]

And in response to your specific questions:

Does it mean that I should subscribe to failure events in an initializer?

Yes, typically the wire up for ActiveSupport::Notifications is done in an initializer.

If I have several taskinator processes how to determine which process failed?
I have neither process class nor file name in the arguments passed into ActiveSupport::Notifications.

Each event carries a payload, which includes the process class, UUID and other metadata.
See instrumentation in the README and lib/taskinator/instrumentation.rb#L58-L71 for what's included. It doesn't include the resque/delayed_job/sidekiq specific job though.

[bguban]

Hi @virtualstaticvoid.
Thanks for the reply. I don't see the exact class name in the options. I see only :type => Taskinator::Process::Sequential but I can't determine which flow failed from it. The only way I found is to add custom options like klass: 'MyClassName' and then check process_options parameter if it contains the keyword.

[bguban]

One more question. Is there any best practice for testing flows. I stubbed all my jobs perform method and run

Resque.inline = true
process = MyProcess.create_process
process.enqueue!

but I see that there is a 3 seconds delay between the execution of the jobs. So the test takes a very long time.

One more problem I faced is that I can't stub tasks. I would like to be able to stub this task to check that flow calls it and then write a separate test for each of the jobs. As I can see in the source code that taskinator includes my flow into instances of Taskinator::Executor.

Thanks