Skip to content

Make GeoIp downloader multi-project aware #128282

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 50 commits into from
Jun 23, 2025

Conversation

samxbr
Copy link
Contributor

@samxbr samxbr commented May 22, 2025

This change makes the GeoIp persistent task executor/downloader multi-project aware.

  • the database downloader persistent task will be at the project level, meaning there will be a downloader instance per project
  • persistent task id is prefixed with project id, namely <project-id>/geoip-downloader for cluster in MP mode

To keep the size of PR review friendly, this PR only focus on the downloading part of GeoIP database, there will be more changes coming in separate PRs to make GeoIP multi-project aware in general.

@samxbr samxbr requested a review from Copilot June 8, 2025 09:38
Copilot

This comment was marked as outdated.

@samxbr samxbr requested a review from nielsbauman June 17, 2025 16:10
GeoIpDownloader currentDownloader = getCurrentTask();
if (currentDownloader != null) {
currentDownloader.requestReschedule();
for (var projectMetadataEntry : event.state().metadata().projects().entrySet()) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer that we put the annotation here as well. That way we're sure it won't be forgotten.

@samxbr samxbr requested a review from nielsbauman June 19, 2025 14:44
@samxbr samxbr requested a review from nielsbauman June 20, 2025 04:48
Copy link
Contributor

@nielsbauman nielsbauman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for the iterations, Sam!

@samxbr samxbr merged commit e3838a4 into elastic:main Jun 23, 2025
27 checks passed
kderusso pushed a commit to kderusso/elasticsearch that referenced this pull request Jun 23, 2025
This change makes the GeoIp persistent task executor/downloader multi-project aware. 
- the database downloader persistent task will be at the project level, meaning there will be a downloader instance per project
- persistent task id is prefixed with project id, namely `<project-id>/geoip-downloader` for cluster in MP mode
julian-elastic pushed a commit to julian-elastic/elasticsearch that referenced this pull request Jun 24, 2025
This change makes the GeoIp persistent task executor/downloader multi-project aware. 
- the database downloader persistent task will be at the project level, meaning there will be a downloader instance per project
- persistent task id is prefixed with project id, namely `<project-id>/geoip-downloader` for cluster in MP mode
mridula-s109 pushed a commit to mridula-s109/elasticsearch that referenced this pull request Jun 25, 2025
This change makes the GeoIp persistent task executor/downloader multi-project aware. 
- the database downloader persistent task will be at the project level, meaning there will be a downloader instance per project
- persistent task id is prefixed with project id, namely `<project-id>/geoip-downloader` for cluster in MP mode
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP >non-issue Team:Data Management Meta label for data/management team v9.1.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants