Skip to content

Adds the signals sync script here #171

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Jul 19, 2024

Conversation

korlaxxalrok
Copy link
Collaborator

This PR makes it so we can build this script into the app container image and ultimately call it from there via some remote mechanism.

Reasons:

  • The container has access to a full runtime environment (config, secrets, etc.)
  • Hosting it in the container instead of our job system reduces the necessity to extend the Python environment of the job system (it is a shared system and potentially more easily corruptible)
  • Simplifies the wrapper command we need to configure in the job system

Changes here:

  • Puts sync-signals-with-metadata.py in a place where it will be built into the web application container image
  • Adds a couple of dependencies necessary to run the script

There has been some discussion in a different repo on a few changes we might want to make. Gonna put a link to those here: https://github.com/cmu-delphi/delphi-admin/pull/205

@@ -12,7 +12,7 @@

SQLALCHEMY_DATABASE_URI = os.environ.get(
"SQLALCHEMY_DATABASE_URI",
"mysql+mysqlconnector://root:Njcnth_1234@localhost:3306/mysql_database",
"mysql+mysqlconnector://root:ROOT_PASSWORD@localhost:3306/mysql_database",
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dmytrotsko You can switch this back if you want, but maybe we want to get this from the local .env during development?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, good call. Will change this to get credentials from .env file.

@korlaxxalrok korlaxxalrok requested a review from dmytrotsko July 18, 2024 23:28
@korlaxxalrok
Copy link
Collaborator Author

@dmytrotsko I've tested most of this in staging and it seems to work. The script doesn't seem to output anything, but I was able to observe the associated DB query. I've also tested running it from Cronicle.

Let me know if you think we should go another way, but this seems like it will work, and we are more likely to keep the script updated in this repo vs. delphi-admin.

(Sorry about adding the package sort to this PR! I only added sqlalchemy and mysql-connector-python.)

@korlaxxalrok korlaxxalrok requested review from dmytrotsko and removed request for dmytrotsko July 19, 2024 12:17
@korlaxxalrok
Copy link
Collaborator Author

@melange396 Tagging you here because you had some potential changes you wanted made in https://github.com/cmu-delphi/delphi-admin/pull/205.

@korlaxxalrok korlaxxalrok requested a review from melange396 July 19, 2024 12:33
@melange396
Copy link
Collaborator

cool, ill copy my comments from the other PR into here

Copy link
Collaborator

@melange396 melange396 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

other than those comments, LGTM!

race_breakdown = sqlalchemy.Column(sqlalchemy.Integer)
reporting_cadence = sqlalchemy.Column(sqlalchemy.String)
restrictions = sqlalchemy.Column(sqlalchemy.Text)
severenity_pyramid_rungs = sqlalchemy.Column(sqlalchemy.String)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
severenity_pyramid_rungs = sqlalchemy.Column(sqlalchemy.String)
severity_pyramid_rungs = sqlalchemy.Column(sqlalchemy.String)

you were talking about fixing this... is this something we can do now, or should we wait so we can change the spelling in other places at the same time?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@melange396 Oh no that happened again. Sorry for that 😢
BUT, we may not want to apply this change right now because I've not merged PR with fix in the signal-documentation repo, so this field actually exists in signal-documentation's database 🤡

Comment on lines +118 to +119
elif len(date) == 8:
formated_date = datetime.strptime(date, self.year_month_day_date_format)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe put an else: Exception() here? maybe just log an error?

Copy link
Collaborator

@dmytrotsko dmytrotsko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@@ -12,7 +12,7 @@

SQLALCHEMY_DATABASE_URI = os.environ.get(
"SQLALCHEMY_DATABASE_URI",
"mysql+mysqlconnector://root:Njcnth_1234@localhost:3306/mysql_database",
"mysql+mysqlconnector://root:ROOT_PASSWORD@localhost:3306/mysql_database",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, good call. Will change this to get credentials from .env file.

@korlaxxalrok korlaxxalrok merged commit 56e3b8e into development Jul 19, 2024
1 of 3 checks passed
@korlaxxalrok korlaxxalrok deleted the integrate-db-update-script branch July 19, 2024 15:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants