diff --git a/docs/epidata_development.md b/docs/epidata_development.md index c8c35e11f..024f1b4dc 100644 --- a/docs/epidata_development.md +++ b/docs/epidata_development.md @@ -49,346 +49,125 @@ $ [sudo] make test pdb=1 $ [sudo] make test test=repos/delphi/delphi-epidata/integrations/acquisition ``` -## Long version +You can read the commands executed by the Makefile [here](../dev/local/Makefile). -**Prerequisite:** this guide assumes that you have read the -[frontend development guide](https://github.com/cmu-delphi/operations/blob/main/docs/frontend_development.md). +## Rapid Iteration and Bind Mounts -This guide describes how to write and test code for the Epidata API. For -preliminary steps, -[install docker and create a virtual network](https://github.com/cmu-delphi/operations/blob/main/docs/frontend_development.md#setup). +To reduce friction, we +[bind-mount](https://docs.docker.com/storage/bind-mounts/) local source files to +the containers, which replaces the corresponding files from the image and allows +your code changes to be reflected immediately, without needing to rebuild. This +approach comes with some drawbacks you should be aware of: -After reading this guide, you may want to visit -[the `fluview_meta` tutorial](new_endpoint_tutorial.md) for an example of how -to add a new endpoint to the API. +- the container will be able read and write to your local filesystem (which may + be a security concern, especially if you are running the containers as root) +- there may be also be strange behaviors with file permissions, especially if + you are running the containers as root +- bind mounts do not interact well with `selinux` on some systems, leading to + various access denials at runtime. As a workaround, you may have to use the + [dangerous "Z" + flag](https://docs.docker.com/storage/bind-mounts/#configure-the-selinux-label) + or temporarily disable selinux -- neither of which is advised. +- for more see the [Epicast development + guide](https://github.com/cmu-delphi/www-epicast/blob/main/docs/epicast_development.md#develop). -# setup +## Manual Installation -For working on the Epidata API, you'll need the following two Delphi -repositories: +We recommend using the quickstart above. If you need to customize the install, +please inspect the installation script `install.sh` above and look in the +`Makefile` to find the Docker commands. -- [operations](https://github.com/cmu-delphi/operations) -- [delphi-epidata](https://github.com/cmu-delphi/delphi-epidata) - -You likely won't need to modify the `operations` repo, so cloning directly from -`cmu-delphi` is usually sufficient. However, since you _are_ going to be -modifying `delphi-epidata` sources, you'll first need to fork the repository -and then clone your personal fork. For more details, see the Delphi-specific -[discussion on forking and branching](https://github.com/cmu-delphi/operations/blob/main/docs/backend_development.md#everyone). - -Here's an example of how to set up your local workspace. Note that you will need -to use your own GitHub username where indicated. - -```bash -# collect everything in a directory called "repos" -mkdir repos && cd repos - -# delphi python (sub)packages -mkdir delphi && cd delphi -git clone https://github.com/cmu-delphi/operations -git clone https://github.com/Your-GitHub-Username/delphi-epidata -cd .. - -# go back up to the workspace root -cd .. -``` - -Your workspace should now look like this: - -```bash -tree -L 3 . -``` - -``` -. -└── repos - └── delphi - ├── delphi-epidata - └── operations -``` - -# build images - -We now need images for the Epidata API web server and the `epidata` database. -These are both based on core Delphi images as defined in the -[`operations` repo](https://github.com/cmu-delphi/operations) which you cloned -above. The base images are built first, followed by the derived -`epidata`-specific images. - -- The [`delphi_web_epidata` image](https://github.com/cmu-delphi/delphi-epidata/blob/main/dev/docker/web/epidata/README.md) adds - the Epidata API to the `delphi_web_epidata` image. -- The - [`delphi_database_epidata` image](https://github.com/cmu-delphi/delphi-epidata/blob/main/dev/docker/database/epidata/README.md) - adds user accounts, `epidata` & other appropriate databases, and relevant tables - (initially empty) to a Percona database image. - -From the root of your workspace, all of the images can be built as follows: - -```bash -docker build -t delphi_web_epidata\ - -f ./devops/Dockerfile .;\ - -docker build -t delphi_database_epidata \ - -f repos/delphi/delphi-epidata/dev/docker/database/epidata/Dockerfile . -``` - -# test - -At this point, you're ready to bring the stack online. - -First, make sure you have the docker network set up so that the containers can -communicate: - -``` -docker network create --driver bridge delphi-net -``` - -Next, start containers for the epidata-specific web and database images. As an aside, the -output from these commands (especially the webserver) can be very helpful for -debugging. For example: - -```bash -# launch the database -docker run --rm -p 127.0.0.1:13306:3306 \ - --network delphi-net --name delphi_database_epidata \ - delphi_database_epidata - -# launch the web server -docker run --rm -p 127.0.0.1:10080:80 \ - --network delphi-net --name delphi_web_epidata \ - delphi_web_epidata -``` - -## unit tests - -Unit tests are self-contained and do not depend on external services like -databases or web servers. You can run unit tests at any time according to the -instructions in the -[backend development guide](https://github.com/cmu-delphi/operations/blob/main/docs/backend_development.md). - -First, [build the `delphi_python` image](https://github.com/cmu-delphi/operations/blob/main/docs/backend_development.md#creating-an-image). -Your test sources will live in, and be executed from within, this image. - -Then run the tests in a container based on that image: - -```bash -docker run --rm delphi_python \ - python3 -m undefx.py3tester.py3tester --color \ - repos/delphi/delphi-epidata/tests -``` - -The final line of output should be similar to the following: - -``` -All 48 tests passed! 68% (490/711) coverage. -``` - -You can also run tests using pytest like this: -``` -docker run --rm delphi_python pytest repos/delphi/delphi-epidata/tests/ -``` -and with pdb enabled like this: -``` -docker run -it --rm delphi_python pytest repos/delphi/delphi-epidata/tests/ --pdb -``` - -## manual tests +## Manual Tests You can test your changes manually by: 1. inserting test data into the relevant table(s) 2. querying the API using your client of choice (`curl` is handy for sanity - checks) + checks) -Here's a full example based on the `fluview` endpoint: +What follows is a worked demonstration based on the `fluview` endpoint. Before +starting, make sure that you have the `delphi_database_epidata`, +`delphi_web_epidata`, and `delphi_redis` containers running; if you don't, see +the Makefile instructions above. -1. Populate the database (particularly the `fluview` table) with some fake - data. For example: +First, let's insert some fake data into the `fluview` table: - ```bash - echo 'insert into fluview values \ - (0, "2020-04-07", 202021, 202020, "nat", 1, 2, 3, 4, 3.14159, 1.41421, \ - 10, 11, 12, 13, 14, 15)' | \ +```bash +# If you have the mysql client installed locally: +echo 'insert into fluview values \ + (0, "2020-04-07", 202021, 202020, "nat", 1, 2, 3, 4, 3.14159, 1.41421, \ + 10, 11, 12, 13, 14, 15)' | \ mysql --user=user --password=pass \ - --port 13306 --host 127.0.0.1 epidata - ``` + --port 13306 --host 127.0.0.1 epidata - Note that the host and port given above are "external" values, which are - locally visible. You'll need the `mysql` client installed locally to run the - above command. - - In case you don't have the `mysql` client installed on your machine and - don't want to install it, you can simply use the binary that's bundled with - the `mariadb` docker image, which you should already have from building the - `delphi_database` image. In that case, use the "internal" values, which are - visible to containers on the same virtual network. For example: - - ```bash - echo 'insert into fluview values \ +# If you do not have mysql locally, you can use a Docker image that has it: +echo 'insert into fluview values \ (0, "2020-04-07", 202021, 202020, "nat", 1, 2, 3, 4, 3.14159, 1.41421, \ 10, 11, 12, 13, 14, 15)' | \ - docker run --rm -i --network delphi-net mariadb \ - mysql --user=user --password=pass \ + docker run --rm -i --network delphi-net percona:ps-8 \ + mysql --user=user --password=pass \ --port 3306 --host delphi_database_epidata epidata - ``` - - Note that for these inserts, absence of command-line output is a sign of - success. On the other hand, output after the insertion likely indicates - failure (like, for example, attempting to insert a duplicate unique key). - -2. Query the API directly: - - ```bash - curl -s \ - 'http://localhost:10080/epidata/api.php?source=fluview&epiweeks=202020®ions=nat' | \ - python3 -m json.tool - ``` - - The pipe to python's built-in JSON formatter is optional, but handy. You - should expect to see the following response from the server: - - ```json - { - "result": 1, - "epidata": [ - { - "release_date": "2020-04-07", - "region": "nat", - "issue": 202021, - "epiweek": 202020, - "lag": 1, - "num_ili": 2, - "num_patients": 3, - "num_providers": 4, - "num_age_0": 10, - "num_age_1": 11, - "num_age_2": 12, - "num_age_3": 13, - "num_age_4": 14, - "num_age_5": 15, - "wili": 3.14159, - "ili": 1.41421 - } - ], - "message": "success" - } - ``` - - Alternatively, you could query the API using one of the available client - libraries. However, this would require you to modify the base URL within the - client's code, and there is some additional amount of boilerplate involved in - calling the client and displaying the result. For these reasons, client - libraries are better candidates for automated integration tests (and unit - tests, in the case of the python client) than one-off manual tests. - -## integration tests - -Writing an integration test is outside of the scope of this document. However, -a number of existing integration tests exist and can be used as a good starting -point for additional tests. For example, see the tests for the -[`fluview` API endpoint](https://github.com/cmu-delphi/delphi-epidata/blob/main/integrations/server/test_fluview.py) and the -[`covidcast` ingestion pipeline](https://github.com/cmu-delphi/delphi-epidata/blob/main/integrations/acquisition/covidcast). - -To run the existing tests and any new tests that you write, you must -follow the -[backend development guide](https://github.com/cmu-delphi/operations/blob/main/docs/backend_development.md) -_within the same workspace_, so that the `delphi_python` image is created with -any changes you have made (e.g., adding new integration tests). That image will -contain the test driver and the source code of your integration tests. Then, -run the tests inside a container based on that image. Note that the container -of tests will need to be attached to the virtual network `delphi-net` -to see and communicate with the web and database servers. - -More concretely, you can run Epidata API integration tests like this: - -1. Build server images as described in the [building section](#build-images) - above. - -2. Launch the server containers as described in the [test section](#test) - above. - -3. Build the `delphi_python` image per the - [backend development guide](https://github.com/cmu-delphi/operations/blob/main/docs/backend_development.md#creating-an-image). - Your test sources will live in, and be executed from within, this image. - -4. Run integration tests in a container based on the `delphi_python` image: - - ```bash - docker run --rm --network delphi-net delphi_python \ - python3 -m undefx.py3tester.py3tester --color \ - repos/delphi/delphi-epidata/integrations - ``` - - You should see output similar to the following (edited for brevity): - - ``` - test_privacy_filtering (repos.delphi.delphi-epidata.integrations.test_covid_survey_hrr_daily.CovidSurveyHrrDailyTests - Don't return rows with too small of a denominator. ... ok - test_round_trip (repos.delphi.delphi-epidata.integrations.test_covid_survey_hrr_daily.CovidSurveyHrrDailyTests) - Make a simple round-trip with some sample data. ... ok - - test_round_trip (repos.delphi.delphi-epidata.integrations.test_fluview.FluviewTests) - Make a simple round-trip with some sample data. ... ok - - ✔ All 3 tests passed! [coverage unavailable] - ``` - - You can also run tests using pytest like this: - ``` - docker run --network delphi-net --rm delphi_python pytest repos/delphi/delphi-epidata/integrations/ - ``` - and with pdb enabled like this: - ``` - docker run --network delphi-net -it --rm delphi_python pytest repos/delphi/delphi-epidata/integrations/ --pdb - ``` - -5. Bring down the servers, for example with the `docker stop` command. - -# rapid iteration - -The workflow described above requires containers to be stopped, rebuilt, and -restarted each time code (including tests) is changed, which can be tedious. To -reduce friction, it's possible to -[bind-mount](https://docs.docker.com/storage/bind-mounts/) your local source -files into a container, which replaces the corresponding files from the image. -This allows your code changes to be reflected immediately, without needing to -rebuild containers. - -There are some drawbacks, however, as discussed in the -[Epicast development guide](https://github.com/cmu-delphi/www-epicast/blob/main/docs/epicast_development.md#develop). -For example: - -- Code running in the container can read (and possibly also write) your local filesystem. -- The command-line specification of bind-mounts is quite tedious. -- Bind mounts do not interact well with `selinux` on some systems, leading to -various access denials at runtime. As a workaround, you may have to use the -[dangerous "Z" flag](https://docs.docker.com/storage/bind-mounts/#configure-the-selinux-label) -or temporarily disable `selinux` -- neither of which is advised. - -## bind-mounting - -### non-server code - -Python sources (e.g. data acquisition, API clients, and tests), can be -bind-mounted into a `delphi_python` container as follows: - -```bash -docker run --rm --network delphi-net \ - --mount type=bind,source="$(pwd)"/repos/delphi/delphi-epidata,target=/usr/src/app/repos/delphi/delphi-epidata,readonly \ - --mount type=bind,source="$(pwd)"/repos/delphi/delphi-epidata/src,target=/usr/src/app/delphi/epidata,readonly \ - delphi_python \ -python3 -m undefx.py3tester.py3tester --color \ - repos/delphi/delphi-epidata/integrations ``` -The command above maps two local directories into the container: +(The host and port given in the first command are "external" values, which are +locally visible. In the second command, we use the Docker "internal" values, +which are visible to containers on the same virtual network. Port 3306 on the +outside of the container is mapped to 13360, which can be seen in the Makefile.) + +For the inserts above, absence of command-line output is a sign of success. On +the other hand, output after the insertion likely indicates failure (like, for +example, attempting to insert a duplicate unique key). -- `/repos/delphi/delphi-epidata`: The entire repo, notably including unit and - integration test sources. -- `/repos/delphi/delphi-epidata/src`: Just the source code, which forms the - container's `delphi.epidata` python package. +Next, you can query the API directly (and parse with Python's JSON tool): -## instrumentation with Sentry +```bash +curl -s \ + 'http://localhost:10080/epidata/api.php?source=fluview&epiweeks=202020®ions=nat' | \ +python3 -m json.tool +``` + +You should expect to see the following response from the server, which is the +data you inserted earlier: + +```json +{ + "epidata": [ + { + "release_date": "2020-04-07", + "region": "nat", + "issue": 202021, + "epiweek": 202020, + "lag": 1, + "num_ili": 2, + "num_patients": 3, + "num_providers": 4, + "num_age_0": 10, + "num_age_1": 11, + "num_age_2": 12, + "num_age_3": 13, + "num_age_4": 14, + "num_age_5": 15, + "wili": 3.14159, + "ili": 1.41421 + } + ], + "result": 1, + "message": "success" +} +``` + +Alternatively, you could query the API using one of the available client +libraries. However, this would require you to modify the base URL within the +client's code, and there is some additional amount of boilerplate involved in +calling the client and displaying the result. For these reasons, client +libraries are better candidates for automated integration tests (and unit tests, +in the case of the python client) than one-off manual tests. + +Our API integration tests use this same Docker image and network setup, but +truncate the database tables before running tests, so any manual changes to the +database will be lost after running integration tests. + +## Instrumentation with Sentry Delphi uses [Sentry](https://sentry.io/welcome/) in production for debugging, APM, and other observability purposes. You can instrument your local environment if you want to take advantage of Sentry's features during the development process. In most cases this option is available to internal Delphi team members only.