Ensure celery tasks execute once

[shanbady]

What are the relevant tickets?

Closes https://github.com/mitodl/hq/issues/7276

Description (What does it do?)

Since we moved to EKS, there are potentially multiple celery containers running - each with their own celery beat. By default there is no locking mechanism to ensure that only a single instance of a scheduled task is run. This PR integrates "celery redbeat" as a celery beat scheduler which resolves that issue and ensures only a single task is run regardless of the number of celery beat instances.

How can this be tested?

1. checkout main
2. in docker-compose.yml - copy the celery service but rename it to “celery2”

  celery2:
    extends:
      file: docker-compose.apps.yml
      service: celery
    env_file:
      - path: env/shared.env
      - path: env/shared.local.env
        required: false
      - path: env/backend.env
      - path: env/backend.local.env
        required: false
      - path: .env
        required: false

3. Add a test task that prints the current minute in vector_search/tasks.py

@app.task
def test_task():
    import datetime
    now = datetime.datetime.now()
    print(f"test task called - {now.minute}")

4. in main/settings_celery.py - add the test task to CELERY_BEAT_SCHEDULE so that it executes every 30 seconds:
   "test-task-every-30s": {"task": "vector_search.tasks.test_task", "schedule": 30}
5. restart celery and bring up the duplicate celery server docker compose down celery redis then docker compose up -d
6. ensure you have a "celery" and "celery2" app running via docker compose ps
7. check your logs and note that every 30 seconds you get 2 tasks outputting test task called - {current minute}
8. checkout this branch shanbady/integrate-celery-redbeat
9. rebuild your containers to pull in the new celery-redbeat package docker compose build web celery
10. re-init your celery and web containers. - make sure you still have the test task in place as well as the celery2 service definition in docker compose from the first set of steps docker compose down celery celery2 web and docker compose up -d
11. note that only a single instance of the task is outputted

[github-actions]

OpenAPI Changes

Show/hide No detectable change.

Unexpected changes? Ensure your branch is up-to-date with main (consider rebasing).

[mbertrand]

works great! 👍