Skip to content

Tutorial Creating a Task

Jump to bottom
dm03514 edited this page Apr 24, 2017 · 23 revisions

Creating a task

In this tutorial, we will create, develop and execute a new task.

The task we create will be used as a functional test for func-y task engine itself. Creating a task consists of:

  • Install python requirments
  • Creating a yaml file to house our task
  • Start the services the task depends on
  • Defining metadata associated with the task
  • Defining the states of our task
  • Incrementally developing task
  • Debugging Task
  • Running tasks as a unit test

Install python requirements

Func-y is written in python using gevent. Unfortunately there aren't enough abstractions to insulate you from it yet. To get started install all the python requirements using pip install -r requirements.txt.

Creating a Yaml file

For this tutorial, we are creating a func-y task to test func-y task engine itself. Func-y tasks can live anywhere in your project but we'll be putting ours in:

func-y-task-engine/tests/funcy/ We are going to be cross-cutting a number of features:

  • env var preprocessing
  • UUID preprocessing
  • nsq plugin
  • postgres plugin

Because of this we'll name our test envvar-uuid-nsq-postgres.yml

Start Services

Our task will be interacting with nsq and postgres through their public interfaces. docker-compose is used to manage test dependencies. To bring up local services to test against, navigate to the func-y-task-engine root directory and run:

docker-compose up

This will start nsqd and postgres.

Define the task metadata

Metadata is stored as top level attributes in the task definition. For our task we'll specify a max_timeout for this task. A version (not used) and a task name:

---
max_timeout: 10
name: ENVVAR_UUID_nsq_postgres_assertions
version: "1"
events:

When max_timeout is reached the task engine will stop any running states and return a failure to the user.

Defining task states/ Incremental Development

Next we'll have to define each state of our task. Since we are testing nsq and postgres we'll:

  • Create a nsq test topic if it doesn't exit Each item in the events yaml represents a single state in a linear state progression. The first state will always be executed and complete, before the second state; the second state will come before the third, etc.
---
max_timeout: 10
name: ENVVAR_UUID_nsq_postgres_assertions
version: "1"
events:
  - name: create_nsq_topic
    initiator:
      method: post
      type: http.HTTPInitiator
      url: "http://localhost:4151/topic/create?topic=funcy_task"

Incrementally validating each state during testing may help keep your full head of hair in tact. func-y task engine provides a command line utility for executing a single test or a collection of tests, and reporting on the test execution.

(funcytestengine) vagrant@vagrant-ubuntu-trusty-64:/vagrant_data/func-y-task-engine$ python bin/funcy-task-engine.py run -t tests/funcy/envvar-uuid-nsq-postgres.yml
2017-04-24 18:30:06,956 - funcytaskengine.engine - DEBUG - {'message': 'sending_first_state', 'first_state': 'create_nsq_topic'}
2017-04-24 18:30:06,956 - funcytaskengine.engine - DEBUG - {'message': 'state_change_requested'}
2017-04-24 18:30:06,956 - transitions.core - DEBUG - Initiating transition from state pending to state create_nsq_topic...
2017-04-24 18:30:06,957 - transitions.core - DEBUG - Exiting state pending. Processing callbacks...
2017-04-24 18:30:06,957 - transitions.core - INFO - Exited state pending
2017-04-24 18:30:06,957 - transitions.core - DEBUG - Entering state create_nsq_topic. Processing callbacks...
2017-04-24 18:30:06,957 - transitions.core - INFO - Entered state create_nsq_topic
2017-04-24 18:30:06,965 - requests.packages.urllib3.connectionpool - INFO - Starting new HTTP connection (1): localhost
2017-04-24 18:30:06,976 - requests.packages.urllib3.connectionpool - DEBUG - "POST /topic/create?topic=funcy_task HTTP/1.1" 200 0
2017-04-24 18:30:06,980 - funcytaskengine.engine - DEBUG - {'message': 'state_change_requested'}
2017-04-24 18:30:06,980 - transitions.core - DEBUG - Initiating transition from state create_nsq_topic to state finished...
2017-04-24 18:30:06,980 - transitions.core - DEBUG - Exiting state create_nsq_topic. Processing callbacks...
2017-04-24 18:30:06,980 - transitions.core - INFO - Exited state create_nsq_topic
2017-04-24 18:30:06,980 - transitions.core - DEBUG - Entering state finished. Processing callbacks...
2017-04-24 18:30:06,981 - transitions.core - INFO - Entered state finished
2017-04-24 18:30:06,981 - funcytaskengine.engine - DEBUG - {'message': 'task_execution_finished'}
{'max_timeout': 10, 'version': '1', 'name': 'ENVVAR_UUID_nsq_postgres_assertions', 'events': [{'initiator': {'url': 'http://localhost:4151/topic/create?topic=funcy_task', 'type': 'http.HTTPInitiator', 'method': 'post'}, 'name': 'create_nsq_topic'}]}
  • Send a message to the test topic
  • Subscribe to the test test topic
  • Wait until we receive a message from the test topic
  • Insert a record into postgres
  • Retrieve a record from postgres
Clone this wiki locally