Merge pull request apache#9 from mesosphere/btm-demo-branch

Fix python env, marathon definition, app labels

Author: ssk2

README.md

@@ -1,16 +1,22 @@
 # cd-demo
 A continuous delivery demo using Jenkins on DCOS.
 
-This demo is a Python script that performs the following sequence of actions when run with the `install` command:
+This demo is a Python script that, when run with the `install` command, will:
 
 1. Installs Jenkins if it isn't already available.
+
+Running the `pipeline` command will:
+
 2. Sets up a series of build jobs, the necessary credentials and a [Build Pipeline](https://wiki.jenkins-ci.org/display/JENKINS/Build+Pipeline+Plugin) view to demonstrate a basic continuous delivery pipeline. Jenkins will:
     + Spin up a new Jenkins slave using the Mesos plugin. This slave runs inside a Docker container on one of our DCOS agents.
     + Clone the git repository
     + Build a Docker container based off the [Jekyll Docker image](https://hub.docker.com/r/jekyll/jekyll/) that includes the content stored in [/site](/site) and push it to DockerHub.
     + Run the newly created container and a [Linkchecker container](https://github.com/mesosphere/docker-containers/blob/master/utils/linkchecker/Dockerfile) that runs a basic integration test against the container, checking that the webserver comes up correctly and that all links being served are valid (i.e. no 404s).
     + Manually trigger a Marathon deployment of the newly created container to the DCOS base Marathon instance. If the application already exists, Marathon will simply upgrade it.
-    + Make the application available on a public slave at port 80.
+    + Make the application available on a public slave at port 80 using Marathon-lb.
+
+When run with the `dynamic-slaves` command, it will:
+
 3. Creates 50 build jobs that take a random amount of time between 1 and 2 minutes. These jobs will randomly fail.
     + The Mesos plugin will spin up build slaves on demand for these jobs, using as much capacity as your cluster has available.
     + When these jobs are finished, the Jenkins tasks will terminate and the resources will be relinquished back to other users of your cluster.
@@ -31,38 +37,64 @@ When run with the `uninstall` command, it will:
     ```
     git clone https://github.com/mesosphere/cd-demo.git
     ```
+2. Create a branch, this is mandatory:
+
+    ```
+    git checkout -b my-demo-branch
+    git push origin my-demo-branch
+    ```
 2. [Set up the DCOS CLI](https://docs.mesosphere.com/administration/introcli/cli/) locally.
-3. Ensure you have a DCOS cluster available. 1 node will work but more than 1 node is preferable to demonstrate build parallelism. If you already had the CLI installed, make sure you set the new cluster URL and authenticate against it:
+
+3. Ensure you have a DCOS cluster available. 1 node will work but more than 1 node is preferable to demonstrate build parallelism. If you already had the CLI installed, make you sure you set the new cluster URL and authenticate against it (else the script will complain):
 
     ```
     dcos config set core.dcos_url http://my.dcos.cluster/
     dcos auth login
     ```
+4. Export the password to an environment variable (ideally put it in your ~/.bashrc). You will need to replace the password here with the password for the `cddemo` user with permission to push to `mesosphere/cd-demo-app` (or your own repo, if you override the `--org` and `--username` flags later):
+
+    ```
+    export PASSWORD=mypass123
+    ```
 
 ### Running Demo
 
-1. Run the demo script. You will need to replace the password here with the password for the `cddemo` user with permission to push to `mesosphere/cd-demo-app`:
+1. Run the install command. This is mainly a wrapper for the `dcos package install` command but will also check to see if you're authenticated.
+
+    ```
+    python bin/demo.py install http://my.dcos.cluster/
+    ```
+
+    NOTE: You must use the domain name for your cluster; the IP address will fail.
+
+2. Check that the Jenkins UI is running before proceeding. You can now run either the pipeline demo or the dynamic slaves demo. To run the pipeline demo, grab the ELB address (`Public Slave`), and make sure to specify the branch to run against:
 
     ```
-    bin/demo.py install --branch=my-demo-branch --password=mypass123 http://my.dcos.cluster/
+    python bin/demo.py pipeline --branch=my-demo-branch --password=$PASSWORD http://my.elb/ http://my.dcos.cluster/
     ```
-    NOTE: Depending on your environment you may need to prepend the above command with `python` Also you must use the domain name for your cluster; the IP address will fail.
-    
-2. The script will install Jenkins and pause. Check that the Jenkins UI is running before hitting enter to proceed.
-3. The script will now use the Jenkins HTTP API to install jobs, necessary credentials and a view. It will automatically trigger the initial build before pausing.
-4. Navigate to the Jenkins UI to see the builds in progress. After a few seconds, you should see a build executor spinning up on Mesos. If you navigate to the configured view, you'll see the pipeline in progress.
-5. Once the tests have completed successfully, you will need to manually deploy the build using the button in the bottom right of the "deploy" box on the view.
-![deploy](/img/manual-deploy.png)
-6. The deploy will happen almost instantaneously. After a few seconds, you should be able to load the application by navigating to your public slave's IP address in your browser.
+
+3. The script will first install Marathon-lb if it looks like it isn't available. It will also update the `marathon.json` in the branch you specified to include the ELB hostname so that Marathon-lb can route to it.
+4. The script will then use the Jenkins HTTP API to install jobs, necessary credentials and a view. It will automatically trigger the initial build before finishing.
+5. Navigate to the Jenkins UI to see the builds in progress. After a few seconds, you should see a build executor spinning up on Mesos. If you navigate to the configured view, you'll see the pipeline in progress.
+6. The deploy will happen almost instantaneously. After a few seconds, you should be able to load the application by navigating to the ELB hostname you provided earlier in your browser.
 ![deployed-app](/img/deployed-jekyll-app.png)
-7. Hit Enter to proceed to the next step of the demo. It will create 50 jobs that will randomly fail.
+7. Now let's run the dynamic slaves demo. It will create 50 jobs that will randomly fail.
+
+    ```
+    python bin/demo.py dynamic-slaves http://my.dcos.cluster/
+    ```
+
 8. Navigate back to the Jenkins and/or DCOS UI to show build slaves spinning up manually.
-9. Hit enter to complete the demo.
 
 ### Uninstalling
 
 1. Simply run the uninstall command to remove any persisted configuration and to uninstall the DCOS service itself. This will allow you to run multiple demos on the same cluster but you should recycle clusters if the version of the Jenkins package has changed (to ensure plugins are upgraded):
-    bin/demo.py uninstall http://my.dcos.cluster/
+
+    ```
+    python bin/demo.py uninstall http://my.dcos.cluster/
+    ```
+
+Alternatively, run the cleanup command instead to just remove jobs and to avoid having to re-install Jenkins.
 
 ## Advanced Usage
 
@@ -72,27 +104,20 @@ By default, this script assumes you will be pushing to the [mesosphere/cd-demo-a
 
 1. Create a public repo called `cd-demo-app` under your example organisation.
 2. Create a Docker Hub user that has credentials to push to this repository.
-3. Run the demo script, passing in the credentials:
+3. Run the pipeline demo, passing in the credentials:
 
     ```
-    bin/demo.py install --branch=my-demo-branch --org=myorg --username=myuser --password=mypass123 http://my.dcos.cluster/
+    python bin/demo.py pipeline --branch=my-demo-branch --org=myorg --username=myuser --password=$PASSWORD http://my.elb/ http://my.dcos.cluster/
     ```
 
 ### Build on Commit
 
-If you'd like to demonstrate the build running automatically with every commit:
-
-1. By default this build operates off the `demo` branch. However, it's recommended that you create your own branches for demo purposes to avoid collisions. Create a new branch in this repository and push it up to origin:
+1. Run the demo to completion with the `--branch` parameter to monitor your branch. The pipeline will continue to monitor your branch after the script finishes:
 
     ```
-    git checkout -b my-demo-branch
-    git push origin my-demo-branch
+    python bin/demo.py pipeline --branch=my-demo-branch --password=$PASSWORD http://my.elb/ http://my.dcos.cluster/
     ```
-2. Run the demo to completion with the `--branch` parameter to monitor your branch. The pipeline will continue to monitor your branch after the script finishes:
 
-    ```
-    bin/demo.py install --branch=my-demo-branch --password=mypass123 http://my.dcos.cluster/
-    ```
 3. Create a new blog post with today's date, open it up in your text editor and make whatever changes you'd like to:
 
     ```
@@ -115,12 +140,12 @@ To demonstrate how you can install multiple Jenkins instances side by side on DC
 1. Create one instance:
 
     ```
-    bin/demo.py install --name=jenkins-1 --password=mypass123 http://my.dcos.cluster/
+    bin/demo.py install --name=jenkins-1 http://my.dcos.cluster/
     ```
 2. Open a new terminal tab and create a second instance and so on:
 
     ```
-    bin/demo.py install --name=jenkins-2 --password=mypass123 http://my.dcos.cluster/
+    bin/demo.py install --name=jenkins-2 http://my.dcos.cluster/
     ```
 3. You can uninstall these in the same way:
 
@@ -129,10 +154,6 @@ To demonstrate how you can install multiple Jenkins instances side by side on DC
     bin/demo.py uninstall --name=jenkins-2 http://my.dcos.cluster/
     ```
 
-### Skipping Demos
-
-Only want to run one of the demos? Simply specify `--no-pipeline` to skip the continuous delivery demo, or `--no-dynamic-slaves` to skip the dynamic slaves demo.
-
 ## TODO
 
 + This script is currently untested on Windows.

bin/demo.py

@@ -1,45 +1,45 @@
-#!/usr/bin/python
+#!/usr/bin/env python
 """demo.py
 
 Usage:
-    demo.py install [--name=<name>] (--no-pipeline | [--branch=<branch>] [--org=<org>] [--username=<user>] --password=<pass>) [--no-dynamic-slaves] [--builds=<n>] <dcos_url>
+    demo.py install [--name=<name>] <dcos_url>
+    demo.py pipeline  [--org=<org>] [--username=<user>] --branch=<branch> --password=<pass> <elb_url> <dcos_url>
+    demo.py dynamic-slaves [--builds=<n>] <dcos_url>
     demo.py cleanup [--name=<name>] [--builds=<n>] <dcos_url>
     demo.py uninstall [--name=<name>] [--builds=<n>] <dcos_url>
 
 Options:
     --name=<name>          Jenkins instance name to use [default: jenkins].
-    --no-pipeline          Don't run continuous delivery demo.
-    --branch=<branch>      Git branch for continuous delivery demo [default: demo].
+    --branch=<branch>      Git branch for continuous delivery demo.
     --org=<org>            Docker Hub organisation where repo lives [default: mesosphere].
     --username=<user>      Docker Hub username to push image with [default: cddemo].
     --password=<pass>      Docker Hub password to push image with.
-    --no-dynamic-slaves    Don't run dynamic slaves demo.
     --builds=<n>           Number of builds to create [default: 50].
 
 This script is used to demonstrate various features of Jenkins on the DCOS.
 
 Pre-requisites:
 + A DCOS CLI is configured and available on the PATH of the host system
-+ A running DCOS cluster greater than version 1.4. It currently does not
-work with clusters that have authentication enabled.
++ A running DCOS cluster greater than version 1.7.0.
 + Python dependencies are installed (pip install -r requirements.txt)
 
 The continuous delivery demo will create a build pipeline that will deploy a Docker
 container to the DCOS Marathon.
 
 The dynamic slaves demo will create 50 (by default) "freestyle" Jenkins jobs.
 Each of these jobs will appear as a separate Jenkins build, and will randomly
-pass or fail. The duration of each job will be between 120 and 240 seconds.
+password or fail. The duration of each job will be between 120 and 240 seconds.
 """
 
 import json
 import os
 import random
+import requests
 import shutil
-from subprocess import call, CalledProcessError, check_output
 
-import requests
 from docopt import docopt
+from subprocess import call, CalledProcessError, check_output
+from urlparse import urlparse
 
 def log(message):
     print "[demo] {}".format(message)
@@ -85,24 +85,58 @@ def rename(path, jenkins_name):
         json.dump(config, f, indent=4)
 
 def install(dcos_url, jenkins_name, jenkins_url):
-    log ("Installing Jenkins with name {}".format(jenkins_name))
+    log ("Installing Jenkins with name {}.".format(jenkins_name))
     shutil.copyfile("conf/jenkins.json", "tmp/jenkins.json")
     rename("tmp/jenkins.json", jenkins_name)
     command = "dcos package install --yes --options=tmp/jenkins.json jenkins"
     print ("\n> " + command)
     if call (['dcos', 'package', 'install', '--yes', '--options=tmp/jenkins.json', 'jenkins']) != 0:
         log_and_exit ("Failed to install Jenkins.")
-    print("\n[demo] Jenkins has been installed! Wait for it to come up before proceeding at: {}".format(jenkins_url))
-    raw_input("[demo] Press [Enter] to continue, or ^C to cancel...")
+    log("Jenkins has been installed! Wait for it to come up before proceeding at: {}".format(jenkins_url))
 
 def verify(jenkins_url):
     r = requests.get(jenkins_url, headers=auth_func({}))
     if r.status_code != 200:
-        log ("Couldn't find a Jenkins instance running at {}".format(jenkins_url))
+        log ("Couldn't find a Jenkins instance running at {}.".format(jenkins_url))
         return False
     log ("Jenkins is up and running! Got Jenkins version {}".format(r.headers['x-jenkins']))
     return True
 
+def install_marathon_lb(elb_url):
+    log("Checking to see if Marathon-lb is installed.")
+    r = requests.get(elb_url)
+    if r.status_code == 503:
+        log ("Couldn't find a Marathon-lb instance running at {}.".format(elb_url))
+        log ("Installing Marathon-lb.")
+        command = "dcos package install --yes marathon-lb"
+        print ("\n> " + command)
+        if call (['dcos', 'package', 'install', '--yes', 'marathon-lb']) != 0:
+            log ("Failed to install Marathon-lb.")
+        else:
+            log("Marathon-lb has been installed!")
+    else:
+        log ("Marathon-lb already seems to be running at {}. Not installing Marathon-lb.".format(elb_url))
+
+def strip_to_hostname(url):
+    parsed_url = urlparse(url)
+    return parsed_url.netloc
+
+def update_marathon_json(elb_url):
+    with open('marathon.json', 'r+') as f:
+        marathon = json.load(f)
+        marathon['labels']['HAPROXY_0_VHOST'] = elb_url
+        f.seek(0)
+        json.dump(marathon, f, indent=4, sort_keys=True, separators=(',', ': '))
+
+def update_and_push_marathon_json(elb_url, branch):
+    elb_hostname = strip_to_hostname(elb_url)
+    update_marathon_json(elb_hostname)
+    if call (['git', 'commit', '-a', '-m', 'Update marathon.json with ELB URL']) != 0:
+        log_and_exit ("Failed to commit updated marathon.json.")
+    if call (['git', 'push', 'origin', branch]) != 0:
+        log_and_exit ("Failed to push updated marathon.json.")
+    log ("Updated marathon.json with ELB hostname {}.".format(elb_hostname))
+
 def create_credentials(jenkins_url, id, username, password):
     credential = { 'credentials' : { 'scope' : 'GLOBAL', 'id' : id, 'username' : username, 'password' : password, 'description' : id, '$class' : 'com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl'} }
     data = {'json' : json.dumps(credential) }
@@ -116,8 +150,9 @@ def create_job(jenkins_url, job_name, job_config):
     post_url = "{}/createItem?name={}".format(jenkins_url, job_name)
     r = requests.post(post_url, headers=headers, data=job_config)
     if r.status_code != 200:
-        log_and_exit ("Failed to create job {} at {}".format(job_name, jenkins_url))
-    log ("Job {} created successfully.".format(job_name))
+        log ("Failed to create job {} at {}.".format(job_name, jenkins_url))
+        r.raise_for_status()
+    log ("Job {} created successfully".format(job_name))
 
 def create_view(jenkins_url, view_name, view_config):
     log ("Creating view")
@@ -151,7 +186,7 @@ def delete_view(jenkins_url, view_name):
 def remove_temp_dir():
     shutil.rmtree("tmp", ignore_errors=True)
 
-def demo_pipeline(jenkins_url, dcos_url, name, branch, org, username, password):
+def demo_pipeline(jenkins_url, dcos_url, elb_url, name, branch, org, username, password):
     log ("Creating demo pipeline.")
     create_credentials(jenkins_url, 'docker-hub-credentials', username, password)
     with open("jobs/build-cd-demo/config.xml") as build_job:
@@ -171,7 +206,7 @@ def demo_pipeline(jenkins_url, dcos_url, name, branch, org, username, password):
         create_view(jenkins_url, "cd-demo-pipeline", view_config)
     trigger_build(jenkins_url, "build-cd-demo")
     log ("Created demo pipeline.")
-    raw_input("[demo] Press [Enter] to continue, or ^C to cancel...")
+    log("Once deployed, your application should be available at {}.".format(elb_url))
 
 def demo_dynamic_slaves(jenkins_url, builds):
     log ("Creating {} freestyle Jenkins jobs.".format(builds))
@@ -187,7 +222,6 @@ def demo_dynamic_slaves(jenkins_url, builds):
             trigger_build(jenkins_url, job_name, parameter_string)
             log ("Job {} created successfully. Duration: {}. Result: {}. Triggering build.".format(job_name, duration, result))
     log ("Created {} freestyle Jenkins jobs.".format(builds))
-    raw_input("[demo] Press [Enter] to continue, or ^C to cancel...")
 
 def cleanup_pipeline_jobs (jenkins_url):
     log ("Cleaning up demo pipeline.")
@@ -223,26 +257,30 @@ def uninstall(dcos_url, jenkins_name):
     jenkins_name = arguments['--name'].lower()
     builds = int(arguments['--builds'])
     dcos_url = arguments['<dcos_url>']
+    elb_url = arguments['<elb_url>'] #TODO: FIX ME
     jenkins_url = '{}service/{}/'.format(dcos_url, jenkins_name)
 
-    check_and_set_token(jenkins_url)
     config_dcos_cli(dcos_url)
+    check_and_set_token(jenkins_url)
+
     try:
         if arguments['install']:
             make_temp_dir()
             if not verify(jenkins_url):
                 install(dcos_url, jenkins_name, jenkins_url)
-            if not arguments['--no-pipeline']:
-                branch = arguments['--branch'].lower()
-                if branch == 'master':
-                    log_and_exit ("Cannot run demo against the master branch.")
-                org = arguments['--org']
-                username = arguments['--username']
-                password = arguments['--password']
-                demo_pipeline(jenkins_url, dcos_url, jenkins_name, branch, org, username, password)
-            if not arguments['--no-dynamic-slaves']:
-                demo_dynamic_slaves(jenkins_url, builds)
             remove_temp_dir()
+        elif arguments['pipeline']:
+            branch = arguments['--branch'].lower()
+            if branch == 'master':
+                log_and_exit ("Cannot run demo against the master branch.")
+            org = arguments['--org']
+            username = arguments['--username']
+            password = arguments['--password']
+            install_marathon_lb(elb_url)
+            update_and_push_marathon_json(elb_url, branch)
+            demo_pipeline(jenkins_url, dcos_url, elb_url, jenkins_name, branch, org, username, password)
+        elif arguments['dynamic-slaves']:
+            demo_dynamic_slaves(jenkins_url, builds)
         elif arguments['cleanup']:
             cleanup(jenkins_url, builds)
         elif arguments['uninstall']: