Create OpenAPI spec for CloudLaunch

[afgane]

CloudLaunch exposes a REST API for manipulating IaaS infrastructure and CloudLaunch-specific appliance management functionality that would be great to have language-specific bindings for. OpenAPI spec, probably using swagger.io tools, seems like the best way to go about generating the bindings.

[machristie]

I'm currently working on this.

[machristie]

TODO:

• Add a schema view
• Create an OpenAPI/Swagger renderer and update schema view to use this renderer
• Does authentication need any special handling?
• Generate code with AutoRest
• Evaluate drf_openapi as a possible alternative to django-rest-swagger
• customize the drf_openapi generated schema: allow anyone to view and customize the name
• try running AutoRest on the drf_openapi generated schema
• Customize the schema rendering so that instances are Documents and can be acted upon?
• try swagger-codegen to generate an api client
• enable coreapi schema
• use coreapi python client to develop a simple command line interface to the CloudLaunch REST API
• how to specify a config file as an input (launch config) - prototype by launching a deployment
• how to get an authentication token when logging in with OpenID Connect? See issue Support getting an auth token when logging in through OpenID Connect #125
• save the authentication token in a config file for use with the command line client

[bgruening]

@machristie this is so cool! Can you please document what you are doing. We might want to do the same for whole Galaxy and you experience can be priceless!

[nuwang]

Moved from: #95
With Django Rest Framework Swagger, we should be able to generate an OpenAPI spec for the current API. We can then use that spec to generate client bindings for multiple languages.

• Generate OpenAPI spec from Django Rest Framework, possibly using DRF Swagger.
  References:
  a. http://www.django-rest-framework.org/api-guide/schemas/#schemas
  b. https://github.com/marcgibbons/django-rest-swagger
• Generate a python client library using available code generators. Compare and choose a suitable generator:
  References:
  a. https://github.com/Azure/autorest
  b. https://github.com/swagger-api/swagger-codegen
• Make OpenAPI spec and client library available for download
• Stretch: Create a commandline client for cloudlaunch

Additional references:
https://apievangelist.com/2015/06/06/comparison-of-automatic-api-code-generation-tools-for-swagger/

[machristie]

@bgruening will do!

@nuwang thanks for these resources! Looks like I don't need to create a Swagger renderer after all.

[machristie]

Using the simple openapi client

Following the OpenAPI/Swagger instructions on this page.

Also Django REST Framework docs

Create a new virtualenv for the client

BL-UITS-RTLT021:cloudlaunch machrist$ python3 -m venv ../coreapi-cli-env
BL-UITS-RTLT021:cloudlaunch machrist$ source ../coreapi-cli-env/bin/ac
activate       activate.csh   activate.fish
BL-UITS-RTLT021:cloudlaunch machrist$ source ../coreapi-cli-env/bin/activate
(coreapi-cli-env) BL-UITS-RTLT021:cloudlaunch machrist$ pip install coreapi-cli
Collecting coreapi-cli
  Downloading coreapi-cli-1.0.6.tar.gz
Collecting coreapi>=1.32.0 (from coreapi-cli)
  Using cached coreapi-2.3.3-py2.py3-none-any.whl
Collecting click>=6.0 (from coreapi-cli)
  Using cached click-6.7-py2.py3-none-any.whl
Collecting coreschema (from coreapi>=1.32.0->coreapi-cli)

Collecting uritemplate (from coreapi>=1.32.0->coreapi-cli)
  Using cached uritemplate-3.0.0-py2.py3-none-any.whl
Collecting requests (from coreapi>=1.32.0->coreapi-cli)
  Using cached requests-2.18.4-py2.py3-none-any.whl
Collecting itypes (from coreapi>=1.32.0->coreapi-cli)
Collecting jinja2 (from coreschema->coreapi>=1.32.0->coreapi-cli)
  Using cached Jinja2-2.9.6-py2.py3-none-any.whl
Collecting idna<2.7,>=2.5 (from requests->coreapi>=1.32.0->coreapi-cli)
  Using cached idna-2.6-py2.py3-none-any.whl
Collecting urllib3<1.23,>=1.21.1 (from requests->coreapi>=1.32.0->coreapi-cli)
  Using cached urllib3-1.22-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests->coreapi>=1.32.0->coreapi-cli)
  Using cached chardet-3.0.4-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests->coreapi>=1.32.0->coreapi-cli)
  Using cached certifi-2017.7.27.1-py2.py3-none-any.whl
Collecting MarkupSafe>=0.23 (from jinja2->coreschema->coreapi>=1.32.0->coreapi-cli)
  Using cached MarkupSafe-1.0.tar.gz
Installing collected packages: MarkupSafe, jinja2, coreschema, uritemplate, idna, urllib3, chardet, certifi, requests, itypes, coreapi, click, coreapi-cli
  Running setup.py install for MarkupSafe ... done
  Running setup.py install for coreapi-cli ... done
Successfully installed MarkupSafe-1.0 certifi-2017.7.27.1 chardet-3.0.4 click-6.7 coreapi-2.3.3 coreapi-cli-1.0.6 coreschema-0.0.4 idna-2.6 itypes-1.1.0 jinja2-2.9.6 requests-2.18.4 uritemplate-3.0.0 urlli
b3-1.22
(coreapi-cli-env) BL-UITS-RTLT021:cloudlaunch machrist$ pip install openapi-codec
Collecting openapi-codec
Requirement already satisfied: coreapi>=2.2.0 in /Users/machrist/SGCI/cloudlaunch/launcher/coreapi-cli-env/lib/python3.6/site-packages (from openapi-codec)
Requirement already satisfied: coreschema in /Users/machrist/SGCI/cloudlaunch/launcher/coreapi-cli-env/lib/python3.6/site-packages (from coreapi>=2.2.0->openapi-codec)
Requirement already satisfied: uritemplate in /Users/machrist/SGCI/cloudlaunch/launcher/coreapi-cli-env/lib/python3.6/site-packages (from coreapi>=2.2.0->openapi-codec)
Requirement already satisfied: requests in /Users/machrist/SGCI/cloudlaunch/launcher/coreapi-cli-env/lib/python3.6/site-packages (from coreapi>=2.2.0->openapi-codec)
Requirement already satisfied: itypes in /Users/machrist/SGCI/cloudlaunch/launcher/coreapi-cli-env/lib/python3.6/site-packages (from coreapi>=2.2.0->openapi-codec)
Requirement already satisfied: jinja2 in /Users/machrist/SGCI/cloudlaunch/launcher/coreapi-cli-env/lib/python3.6/site-packages (from coreschema->coreapi>=2.2.0->openapi-codec)
Requirement already satisfied: certifi>=2017.4.17 in /Users/machrist/SGCI/cloudlaunch/launcher/coreapi-cli-env/lib/python3.6/site-packages (from requests->coreapi>=2.2.0->openapi-codec)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /Users/machrist/SGCI/cloudlaunch/launcher/coreapi-cli-env/lib/python3.6/site-packages (from requests->coreapi>=2.2.0->openapi-codec)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in /Users/machrist/SGCI/cloudlaunch/launcher/coreapi-cli-env/lib/python3.6/site-packages (from requests->coreapi>=2.2.0->openapi-codec)
Requirement already satisfied: idna<2.7,>=2.5 in /Users/machrist/SGCI/cloudlaunch/launcher/coreapi-cli-env/lib/python3.6/site-packages (from requests->coreapi>=2.2.0->openapi-codec)
Requirement already satisfied: MarkupSafe>=0.23 in /Users/machrist/SGCI/cloudlaunch/launcher/coreapi-cli-env/lib/python3.6/site-packages (from jinja2->coreschema->coreapi>=2.2.0->openapi-codec)
Installing collected packages: openapi-codec
Successfully installed openapi-codec-1.3.2

Getting an application list

(coreapi-cli-env) BL-UITS-RTLT021:cloudlaunch machrist$ coreapi get http://localhost:8000/api/v1/schema?format=openapi
<CloudLaunch API "http://localhost:8000/api/v1/schema/?format=openapi">
    applications: {
        list([page], [page_size], [search])
        create(name, [status], [summary], [maintainer], [description], [info_url], [icon_url])
        read(slug)
        update(slug, name, [status], [summary], [maintainer], [description], [info_url], [icon_url])
        partial_update(slug, [name], [status], [summary], [maintainer], [description], [info_url], [icon_url])
        delete(slug)
    }
    auth: {
        list()

        login_create(password, [username], [email])
        logout_list()
        logout_create()
        password_reset_create(email)
        password_reset_confirm_create(new_password1, new_password2, uid, token)
        registration_create(username, password1, password2, [email])
        registrationverify-email_create(key)
        user_credentials_aws_list([page])
        user_credentials_aws_create(cloud_id, name, access_key, [default], [secret_key])
        user_credentials_aws_read(id)
        user_credentials_aws_update(id, cloud_id, name, access_key, [default], [secret_key])
        user_credentials_aws_partial_update(id, [cloud_id], [name], [default], [access_key], [secret_key])
        user_credentials_aws_delete(id)
        user_credentials_azure_list([page])
        user_credentials_azure_create(cloud_id, name, subscription_id, client_id, [default], [secret], [tenant])
        user_credentials_azure_read(id)
        user_credentials_azure_update(id, cloud_id, name, subscription_id, client_id, [default], [secret], [tenant])
        user_credentials_azure_partial_update(id, [cloud_id], [name], [default], [subscription_id], [client_id], [secret], [tenant])
        user_credentials_azure_delete(id)
        user_credentials_gce_list([page])
        user_credentials_gce_create(credentials, cloud_id, name, [default])
        user_credentials_gce_read(id)
        user_credentials_gce_update(id, credentials, cloud_id, name, [default])
        user_credentials_gce_partial_update(id, [credentials], [cloud_id], [name], [default])
        user_credentials_gce_delete(id)
        user_credentials_openstack_list([page])
        user_credentials_openstack_create(cloud_id, name, username, project_name, [default], [password], [project_domain_name], [user_domain_name])
        user_credentials_openstack_read(id)
        user_credentials_openstack_update(id, cloud_id, name, username, project_name, [default], [password], [project_domain_name], [user_domain_name])
        user_credentials_openstack_partial_update(id, [cloud_id], [name], [default], [username], [password], [project_name], [project_domain_name], [user_domain_name])
        user_credentials_openstack_delete(id)
    }
    cors_proxy: {
        list()
    }
...

(coreapi-cli-env) BL-UITS-RTLT021:cloudlaunch machrist$ coreapi action applications list
{
    "count": 1,
    "next": null,
    "previous": null,
    "results": [
        {
            "url": "http://localhost:8000/api/v1/applications/ubuntu/",
            "slug": "ubuntu",
            "versions": [

                {
                    "version": "16.04",
                    "cloud_config": [
                        {
                            "cloud": {
                                "slug": "amazon-us-east-n-virginia",
                                "compute": "http://localhost:8000/api/v1/infrastructure/clouds/amazon-us-east-n-virginia/compute/",
                                "security": "http://localhost:8000/api/v1/infrastructure/clouds/amazon-us-east-n-virginia/security/",
                                "storage": "http://localhost:8000/api/v1/infrastructure/clouds/amazon-us-east-n-virginia/storage/",
                                "networks": "http://localhost:8000/api/v1/infrastructure/clouds/amazon-us-east-n-virginia/networks/",
                                "static_ips": "http://localhost:8000/api/v1/infrastructure/clouds/amazon-us-east-n-virginia/static_ips/",
                                "region_name": "us-east-1",
                                "cloud_type": "aws",
                                "extra_data": {
                                    "ec2_region_name": "us-east-1",
                                    "ec2_region_endpoint": "ec2.amazonaws.com",
                                    "ec2_conn_path": "/",
                                    "ec2_port": null,
                                    "ec2_is_secure": false,
                                    "s3_host": "s3.amazonaws.com",
                                    "s3_conn_path": "/",
                                    "s3_port": null,
                                    "s3_is_secure": true
                                },
                                "added": "2016-06-26T03:08:59.048000Z",
                                "updated": "2017-04-21T14:30:15.694000Z",
                                "name": "Amazon US East - N. Virginia",
                                "access_instructions_url": "https://aws-portal.amazon.com/gp/aws/developer/account/index.html?action=access-key"
...

Issues:

• deployments isn't listed although it is in the swagger UI page
• not sure how to read an application. This coreapi action applications read --params slug=ubuntu doesn't work.
• Looking at coreapi, the auto generated schema seems really simple. We might need to customize it or go full manual. For example, there is just a single "Document" that is returned.
• haven't figured out how to authenticate yet. This coreapi credentials add --auth basic localhost admin:PASSWORD didn't work

[nuwang]

@machristie This is looking great! Did you have to annotate things manually using CoreAPI, or did it pick up most of the required data from DRF automatically?

Also, is there any advantage in using django-rest-swagger or does that generate the Swagger web UI only?

[machristie]

@nuwang As a first step I didn't do any manual annotating or anything, just to see what is available out of the box. This is all automatically generated at this time. You can see in this commit that there's almost nothing to it. However, as I mentioned above, this is probably not sufficient.

django-rest-swagger generates the Swagger web UI but also the schema document. So that seems like a good advantage to using it, otherwise we'd need to implement it.

[nuwang]

Ok, that's good to hear.

basic auth was explicitly disabled, as I recall it was either because it was not particularly secure when not using https, or because it kept popping up the dialogue or maybe both.
https://github.com/galaxyproject/cloudlaunch/blob/dev/django-cloudlaunch/cloudlaunchserver/settings.py#L217

However, it looks like token auth should work? coreapi credentials add localhost "Token 9944b09199c62bcf9418ad846dd0e4bbdfc6ee4b

[machristie]

There is a bug preventing the setting of the Authorization header in coreapi-cli. To workaround this I downgraded coreapi to version 2.2.4

pip install coreapi==2.2.4

Here are the steps to authenticate with the command line client:

1. Log in with username and password using the form at http://localhost:8000/api/v1/auth/login/
2. Take the token and run the following command:

coreapi credentials add localhost "Token 846dd0e4bbdf...."

3. Now I can make calls and get back a user's saved credentials

$ coreapi action -d auth user_credentials_aws_list
> GET /api/v1/auth/user/credentials/aws/ HTTP/1.1
> Accept-Encoding: gzip, deflate
> Connection: keep-alive
> Accept: application/coreapi+json, application/vnd.coreapi+json, */*
> Authorization: Token 846dd0e4bbdf....
> Host: localhost

> User-Agent: coreapi
< 200 OK
< Allow: GET, POST, HEAD, OPTIONS
< Content-Length: 1269
< Content-Type: application/json
< Date: Mon, 06 Nov 2017 14:24:56 GMT
< Server: WSGIServer/0.2 CPython/3.6.0
< Vary: Accept, Cookie
< X-Frame-Options: SAMEORIGIN
<
< {"count":1,"next":null,"previous":null,"results":[{"url":...

Note: the -d flag shows the HTTP headers in the request and response. I used this to confirm that the Authorization header is being sent.

Also: with auth configured I'm now seeing deployments in the returned schema document.

[machristie]

There's a typo in the coreapi docs: there is no --params argument, instead it is --param or -p. I can use the following to read a specific deployment

coreapi action deployments read -p id=1