Skip to content

docs: remove references to SQS #18268

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

docs: remove references to SQS #18268

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
442e8fb
docs: remove references to SQS
miketheman Jun 3, 2025
98de9c5
Merge branch 'main' into miketheman/docs
miketheman Jun 17, 2025
325ae2c
Merge branch 'main' into miketheman/docs
miketheman Jun 17, 2025
fa7a7e6
Merge branch 'main' into miketheman/docs
miketheman Jun 20, 2025
76c647e
Merge branch 'main' into miketheman/docs
miketheman Jun 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 5 additions & 12 deletions docs/dev/architecture.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,15 +90,13 @@ C4Container
Container_Boundary(c1, "Warehouse & Supporting Systems") {
Container(camo, "Camo", "image proxy")
Container(web_app, "Web", "Python (Pyramid, SQLAlchemy)", "Delivers HTML and API content")
SystemQueue(sqs, "AWS SQS", "task broker")
SystemDb(opensearch, "OpenSearch", "Index of projects, packages, metadata")
SystemDb(db, "Postgres Database", "Store project, package metadata, user details")
SystemDb(redis, "Redis", "Store short-term cache data")

Rel(web_app, sqs, "queue tasks")
Rel(web_app, opensearch, "search for projects")
Rel(web_app, db, "store/retrieve most data")
Rel(web_app, redis, "cache data")
Rel(web_app, redis, "cache data & task queue")
}

Rel(endUser, camo, "load images from project descriptions", "HTTPS")
Expand Down Expand Up @@ -126,10 +124,7 @@ C4Container
System(s3, "AWS S3", "Object store (archive)")
Rel(web_app, s3, "stores package files")

SystemQueue(sqs, "AWS SQS", "task broker")
Rel(web_app, sqs, "queue sync to cache task")

SystemDb(redis, "Redis", "Store short-term cache data")
SystemDb(redis, "Redis", "Store short-term cache data, task broker")
Rel(web_app, redis, "get/set rate limits and cache data")
}

Expand All @@ -142,7 +137,7 @@ Our workers use Celery to run tasks.
We run a single worker type, feeding off multiple queues.
We also use Celery Beat to schedule tasks.

We currently use AWS SQS as the queue,
We currently use Redis as the queue,
and Redis as the result backend and schedule storage.

```{mermaid}
Expand All @@ -152,21 +147,19 @@ C4Container

Container_Boundary(c1, "Supporting Systems") {
SystemDb(redis, "Redis", "Store short-term cache data")
SystemQueue(sqs, "AWS SQS", "task broker")
SystemDb(opensearch, "OpenSearch", "Index of projects, packages, metadata")
SystemDb(db, "Postgres Database", "Store project, package metadata, user details")
System(ses, "AWS SES", "Simple Email Service")
}

System_Ext(fastly, "Fastly", "Content Delivery Network")

BiRel(worker, sqs, "get next task/ack")
BiRel(worker, redis, "store task results")
BiRel(worker, redis, "get next task/ack")
BiRel(worker, db, "interact with models")
BiRel(worker, opensearch, "update search index")
Rel(worker, fastly, "purge URLs")
Rel(worker, ses, "send emails")

BiRel(worker_beat, redis, "fetch/store task schedules")
Rel(worker_beat, sqs, "schedule tasks")
Rel(worker_beat, redis, "schedule tasks")
```
Loading