Skip to content

Commit 7e00e24

Browse files
committed
docs(README): update for more precise description
1 parent 928c83b commit 7e00e24

File tree

1 file changed

+12
-9
lines changed

1 file changed

+12
-9
lines changed

README.md

Lines changed: 12 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -42,18 +42,21 @@ Using Airflow to implement our ETL pipelines
4242
> Please don't use the .env for local development, or it might screw up the production tables.
4343
4444
1. Build docker image:
45-
1. First, build a prod image (for prod): `docker build -t davidtnfsh/pycon_etl:prod --cache-from davidtnfsh/pycon_etl:prod -f Dockerfile .`
46-
2. Build dev/test image (for dev/test): `docker build -t davidtnfsh/pycon_etl:test --cache-from davidtnfsh/pycon_etl:prod -f Dockerfile.test .`
45+
* Build a production image (for production): `docker build -t davidtnfsh/pycon_etl:prod --cache-from davidtnfsh/pycon_etl:prod -f Dockerfile .`
46+
If you want to build dev/test image, you also need to build this docker image first because dev/test image is on top of this production image. See below.
47+
* Build dev/test image (for dev/test): `docker build -t davidtnfsh/pycon_etl:test --cache-from davidtnfsh/pycon_etl:prod -f Dockerfile.test .`
4748
2. Fill in some secrets:
48-
1. `cp .env.template .env.staging`
49-
2. Follow the instruction in `.env.staging` and fill in your secrets
49+
1. `cp .env.template .env.staging` for dev/test. `cp .env.template .env.production` instead if you are going to start a production instance.
50+
2. Follow the instruction in `.env.<staging|production>` and fill in your secrets.
51+
If you are just running the staging instance for development as a sandbox, and not going to access any specific thrid-party service, leave the `.env.staging` as-is should be fine.
5052
3. Start the Airflow server:
51-
* dev/test: `docker run --rm -p 80:8080 --name airflow -v $(pwd)/dags:/usr/local/airflow/dags -v $(pwd)/service-account.json:/usr/local/airflow/service-account.json --env-file=./.env davidtnfsh/pycon_etl:test webserver`
52-
* prod: `docker run --rm -p 80:8080 --name airflow -v $(pwd)/dags:/usr/local/airflow/dags -v $(pwd)/service-account.json:/usr/local/airflow/service-account.json --env-file=./.env davidtnfsh/pycon_etl:prod webserver`
53+
* production: `docker run --rm -p 80:8080 --name airflow -v $(pwd)/dags:/usr/local/airflow/dags -v $(pwd)/service-account.json:/usr/local/airflow/service-account.json --env-file=./.env.production davidtnfsh/pycon_etl:prod webserver`
54+
* dev/test: `docker run --rm -p 80:8080 --name airflow -v $(pwd)/dags:/usr/local/airflow/dags -v $(pwd)/service-account.json:/usr/local/airflow/service-account.json --env-file=./.env.staging davidtnfsh/pycon_etl:test webserver`
55+
* Note the difference are just the env file name and the image cache.
5356
#### BigQuery (Optional)
5457
1. Setup the Authentication of GCP: <https://googleapis.dev/python/google-api-core/latest/auth.html>
55-
* After invoking `gcloud auth application-default login`, you'll get a credentials.json resides in `/Users/<xxx>/.config/gcloud/application_default_credentials.json`. Invoke `export GOOGLE_APPLICATION_CREDENTIALS="/path/to/keyfile.json"` if you have it.
56-
* service-account.json: Please contact @david30907d using email, telegram or discord.
58+
* After invoking `gcloud auth application-default login`, you'll get a credentials.json resides in `$HOME/.config/gcloud/application_default_credentials.json`. Invoke `export GOOGLE_APPLICATION_CREDENTIALS="/path/to/keyfile.json"` if you have it.
59+
* service-account.json: Please contact @david30907d using email, telegram or discord. No worry about this json if you are just running the sandbox staging instance for development.
5760
2. Give [Toy-Examples](#Toy-Examples) a try
5861

5962
## Deployment
@@ -83,4 +86,4 @@ rows = query_job.result() # Waits for query to finish
8386

8487
for row in rows:
8588
print(row.diet)
86-
```
89+
```

0 commit comments

Comments
 (0)