Skip to content

Tutorial Creating a Task

Jump to bottom
dm03514 edited this page Apr 24, 2017 · 23 revisions

Creating a task

In this tutorial, we will create, develop and execute a new task.

The task we create will be used as a functional test for func-y task engine itself. Creating a task consists of:

  • Install python requirments
  • Creating a yaml file to house our task
  • Start the services the task depends on
  • Defining metadata associated with the task
  • Defining the states of our task
  • Incrementally developing task
  • Debugging Task
  • Running tasks as a unit test

Install python requirements

Func-y is written in python using gevent. Unfortunately there aren't enough abstractions to insulate you from it yet. To get started install all the python requirements using pip install -r requirements.txt.

Creating a Yaml file

For this tutorial, we are creating a func-y task to test func-y task engine itself. Func-y tasks can live anywhere in your project but we'll be putting ours in:

func-y-task-engine/tests/funcy/ We are going to be cross-cutting a number of features:

  • env var preprocessing
  • UUID preprocessing
  • nsq plugin
  • postgres plugin

Because of this we'll name our test envvar-uuid-nsq-postgres.yml

Start Services

Our task will be interacting with nsq and postgres through their public interfaces. docker-compose is used to manage test dependencies. To bring up local services to test against, navigate to the func-y-task-engine root directory and run:

docker-compose up

This will start nsqd and postgres.

Define the task metadata

Metadata is stored as top level attributes in the task definition. For our task we'll specify a max_timeout for this task. A version (not used) and a task name:

---
max_timeout: 10
name: ENVVAR_UUID_nsq_postgres_assertions
version: "1"
events:

When max_timeout is reached the task engine will stop any running states and return a failure to the user.

Defining task states/ Incremental Development

Next we'll have to define each state of our task. Since we are testing nsq and postgres we'll:

  • Create a nsq test topic if it doesn't exist

Each item in the events yaml represents a single state in a linear state progression. The first state will always be executed and complete, before the second state; the second state will come before the third, etc.

---
max_timeout: 10
name: ENVVAR_UUID_nsq_postgres_assertions
version: "1"
events:
  - name: create_nsq_topic
    initiator:
      method: post
      type: http.HTTPInitiator
      url: "http://localhost:4151/topic/create?topic=funcy_task"

Incrementally validating each state during testing may help keep your full head of hair in tact. func-y task engine provides a command line utility for executing a single test or a collection of tests, and reporting on the test execution.

(funcytestengine) vagrant@vagrant-ubuntu-trusty-64:/vagrant_data/func-y-task-engine$ python bin/funcy-task-engine.py run -t tests/funcy/envvar-uuid-nsq-postgres.yml
2017-04-24 18:30:06,956 - funcytaskengine.engine - DEBUG - {'message': 'sending_first_state', 'first_state': 'create_nsq_topic'}
2017-04-24 18:30:06,956 - funcytaskengine.engine - DEBUG - {'message': 'state_change_requested'}
2017-04-24 18:30:06,956 - transitions.core - DEBUG - Initiating transition from state pending to state create_nsq_topic...
2017-04-24 18:30:06,957 - transitions.core - DEBUG - Exiting state pending. Processing callbacks...
2017-04-24 18:30:06,957 - transitions.core - INFO - Exited state pending
2017-04-24 18:30:06,957 - transitions.core - DEBUG - Entering state create_nsq_topic. Processing callbacks...
2017-04-24 18:30:06,957 - transitions.core - INFO - Entered state create_nsq_topic
2017-04-24 18:30:06,965 - requests.packages.urllib3.connectionpool - INFO - Starting new HTTP connection (1): localhost
2017-04-24 18:30:06,976 - requests.packages.urllib3.connectionpool - DEBUG - "POST /topic/create?topic=funcy_task HTTP/1.1" 200 0
2017-04-24 18:30:06,980 - funcytaskengine.engine - DEBUG - {'message': 'state_change_requested'}
2017-04-24 18:30:06,980 - transitions.core - DEBUG - Initiating transition from state create_nsq_topic to state finished...
2017-04-24 18:30:06,980 - transitions.core - DEBUG - Exiting state create_nsq_topic. Processing callbacks...
2017-04-24 18:30:06,980 - transitions.core - INFO - Exited state create_nsq_topic
2017-04-24 18:30:06,980 - transitions.core - DEBUG - Entering state finished. Processing callbacks...
2017-04-24 18:30:06,981 - transitions.core - INFO - Entered state finished
2017-04-24 18:30:06,981 - funcytaskengine.engine - DEBUG - {'message': 'task_execution_finished'}
{'max_timeout': 10, 'version': '1', 'name': 'ENVVAR_UUID_nsq_postgres_assertions', 'events': [{'initiator': {'url': 'http://localhost:4151/topic/create?topic=funcy_task', 'type': 'http.HTTPInitiator', 'method': 'post'}, 'name': 'create_nsq_topic'}]}

Above shows the output of our very first test execution!!! Output is very verbose currently. Above lists all the state transitions that the task went through. In the middle we can see we made a POST request to our local nsqd over its http port, and it resulted in a 200!

We can confirm that the test topic was created by querying local nsqd:

$ curl http://localhost:4151/stats
nsqd v1.0.0-compat (built w/go1.8)
start_time 2017-04-24T18:13:07Z
uptime 23m37.736277826s

Health: OK

   [funcy_task     ] depth: 0     be-depth: 0     msgs: 0        e2e%:
  • Send a message to the test topic
  • Subscribe to the test test topic
  • Wait until we receive a message from the test topic
  • Insert a record into postgres
  • Retrieve a record from postgres
Clone this wiki locally