Skip to content

V2 refactor #22

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 158 commits into from
Jul 11, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
158 commits
Select commit Hold shift + click to select a range
c3c395e
Manual code changes complete.
Jul 6, 2025
eaf86fd
Config testing complete
Jul 6, 2025
caf4f0d
utils unit testing complete
Jul 6, 2025
3858ec9
route testing mostly done
Jul 6, 2025
7eb4844
Routes unit tested
Jul 6, 2025
ab3b2bb
mongo testing updated
Jul 6, 2025
80f9677
All routes now use consistent error handling
Jul 6, 2025
fd12dd7
enumerator service tested
Jul 6, 2025
b8464e9
testing updates for error handing updates
Jul 6, 2025
0fa352a
Removing deprecated code
Jul 6, 2025
af4a9a9
type_services update for testing
Jul 6, 2025
cada073
test_cases folder renames
Jul 6, 2025
ac54930
Type unit test improvements
Jul 6, 2025
a65392b
template types standardized
Jul 6, 2025
128e19b
template types finalized
Jul 6, 2025
2a4ff80
clean feature take1
Jul 6, 2025
b06a544
clean feature finalized
Jul 6, 2025
c0a40f8
testing for clean endpoints added
Jul 6, 2025
e1086f1
All service and route files have now been refactored to use the new, …
Jul 6, 2025
ba8551e
Improved exception handling
Jul 6, 2025
6492251
utils unit test fixes
Jul 6, 2025
879f9c7
route tests passing
Jul 6, 2025
a30894d
updated enumerator testing complete
Jul 6, 2025
8cb5458
implemented standard types from test_cases/template_sample
Jul 6, 2025
c0a3378
arranging type unit test data
Jul 6, 2025
5ca6653
LLM Generated expected bson/json renders for Type tests
Jul 6, 2025
4c59150
Updated to match schema of existing test data
Jul 6, 2025
8e25efd
test data refactor #1
Jul 6, 2025
8717bf7
Type updates
Jul 6, 2025
7d5bd42
type documentation updates
Jul 6, 2025
09ba373
documentation updates
Jul 6, 2025
a26754b
Documentation updates
Jul 6, 2025
2ae119c
Render updates to include description
Jul 6, 2025
19f073f
operations working
Jul 6, 2025
5dc197e
dictionary operations working!
Jul 6, 2025
328576a
configuration testing underway
Jul 6, 2025
35b525a
configuration service renders and operations now tested
Jul 6, 2025
2d8f6a9
circular reference safety feature is now fully implemented and tested
Jul 6, 2025
3ba2c24
simplification and housekeeping updates
Jul 7, 2025
a063492
large sample testing part1
Jul 7, 2025
4a14874
new create collection endpoint, and fixed all endpoints to be trailin…
Jul 7, 2025
6d7b6cf
recovered large_sample from /main
Jul 7, 2025
5160506
getting standard types
Jul 7, 2025
412341c
moved testing data
Jul 7, 2025
55a8114
rename $ref to ref
Jul 7, 2025
f53dab8
Implemented one_of feature that was missed previously
Jul 7, 2025
89ed301
unit testing complete except processing.
Jul 7, 2025
dc23abe
unit testing complete - required small VersionNumber fix.
Jul 7, 2025
074f117
small_sample and large_sample integration testing complete
Jul 7, 2025
3109c79
integration test multiple input files
Jul 7, 2025
877b124
Fixing version number problems
Jul 7, 2025
e3332d8
Processing debug updates
Jul 7, 2025
aa57836
enumerator processing added
Jul 8, 2025
5d0152f
mongo_io testing updates
Jul 8, 2025
6eb6ef8
small sample processing and rendering passing
Jul 8, 2025
51559b6
Enumerator upsert fix, and versionManager use of VersionNumber fixed.
Jul 8, 2025
06de3d8
unit test data added
Jul 8, 2025
e7f8bd8
improved event logging
Jul 8, 2025
6bb5d27
Feature: Moved migrations into their own json files so we can handled…
Jul 8, 2025
75165ae
Migrations moved, integration tests all passing!
Jul 8, 2025
9ac53a9
readme update
Jul 8, 2025
843524e
Code Review Refactor:
Jul 8, 2025
2af8386
Added mongo extended json support to json handling.
Jul 8, 2025
21db7ba
openapi sync to implementation
Jul 8, 2025
44fe451
documentation and sample updates
Jul 8, 2025
3b860ba
README updates complete
Jul 8, 2025
74f9996
improved event tracking, debuging processing
Jul 9, 2025
664b78c
Integration tests passing, ready for data and code reviews.
Jul 9, 2025
0ddbdf4
Fix testing concurrency problem
Jul 9, 2025
04891ae
housekeeping
Jul 9, 2025
105fa1b
housekeeping
Jul 9, 2025
48b732d
Setting up template configurations
Jul 9, 2025
6295c76
openapi review
Jul 9, 2025
65ab841
snake_case consistency refactor
Jul 9, 2025
0309417
aligned openapi with implementation
Jul 9, 2025
88b871f
Housekeeping verbose comments
Jul 9, 2025
1cf20d7
Housekeeping debug notes
Jul 9, 2025
699392c
housekeeping overly verbose doc strings
Jul 9, 2025
78d6f39
refactor to pass enumerations
Jul 9, 2025
9bbd63f
Doc updates
Jul 9, 2025
e960eae
not publishing a package so we don't need this file
Jul 9, 2025
c4649fb
script updates
Jul 9, 2025
e0ae255
Refactor file_io and configuration_routes for better error handling. …
Jul 9, 2025
af65b30
wrapper updated to allow endpoint to return data, not an event wrappi…
Jul 9, 2025
bcb1d7d
fixed typo - voice-to-text induced error
Jul 9, 2025
6b760d4
routes updated to serialize returns with to_dict()
Jul 9, 2025
38a5abf
improved processing event trace
Jul 10, 2025
c88e820
Improve migration event structure and error handling
Jul 10, 2025
ff96f9d
Audit config.py and collection usage: ensure ENUMERATORS_COLLECTION_N…
Jul 10, 2025
4c75fe0
Simplify clean configurations feature and remove complex event tracking
Jul 10, 2025
bcddec0
Fix clean configurations endpoint to serialize File objects with to_d…
Jul 10, 2025
10b3f5c
Simplify dictionary clean endpoint and Dictionary.save()
Jul 10, 2025
981a92e
Simplify type clean endpoint and Type.save()\n\n- Type.save() now ret…
Jul 10, 2025
598a130
Simplify enumerators clean endpoint and Enumerators.save()
Jul 10, 2025
5eda831
Update test_data PUT endpoint to return file.to_dict() and add migrat…
Jul 10, 2025
4bb126a
Migration routes now use FileIO and return 500 if file not found; upd…
Jul 10, 2025
a36cd7f
Remove backward compatibility for old version format; require collect…
Jul 10, 2025
232d5f1
Remove legacy file name parsing support; simplify test file format ha…
Jul 10, 2025
f92f10d
Add static file serving for /docs and update OpenAPI spec with API Ex…
Jul 10, 2025
65d8a90
Refactor static folder configuration to use separate statements\n\n- …
Jul 10, 2025
25380af
serve openapi spec
Jul 10, 2025
7c963b9
description update
Jul 10, 2025
086a6e4
API now serves openapi spec at localhost:8081/docs/index.html
Jul 10, 2025
edeb00d
API now serves openapi spec, and delete database has been tested.
Jul 10, 2025
ed74ac6
CI Testing structure established
Jul 10, 2025
72ac4aa
separating CRUD from processing
Jul 10, 2025
9c023ea
Created playground input folder for CI testing
Jul 10, 2025
75ed7d6
Fix database route test to properly mock drop_database return value a…
Jul 10, 2025
b2580a3
Refactor API route serialization and test mocks for consistent JSON r…
Jul 10, 2025
7a569f2
housekeeping
Jul 10, 2025
49d3844
Playground can drop database
Jul 10, 2025
b8e350a
StepCI progress
Jul 10, 2025
770e2e2
Fix lock/unlock configuration endpoint to return File on success, Con…
Jul 10, 2025
1d8d789
Complete delete functionality refactor for all routes - return Config…
Jul 10, 2025
4512311
delete action refactor
Jul 10, 2025
2a9f597
Consistent name property for Configuration and Dictionary classes; up…
Jul 10, 2025
9ed25b7
Configuration routes testing
Jul 10, 2025
5d0215e
Remove lock/unlock functionality and read_only property, implement _l…
Jul 10, 2025
33b5144
Move _locked property to Enumerations level, fix enum value format an…
Jul 10, 2025
28d3419
Refactor lock_all endpoints and simplify save() behavior
Jul 10, 2025
2120cc4
Fix stepCI configuration tests and improve lock_all functionality
Jul 10, 2025
dffe7a0
Remove legacy support from Dictionary class - clean implementation wi…
Jul 10, 2025
ed3957f
Simplify Dictionary class - clean implementation with Property wrapper
Jul 10, 2025
90627c2
Fix Dictionary class to use file_name instead of name - clean impleme…
Jul 10, 2025
15b9e8e
Fix Dictionary class to truncate file_name and update tests
Jul 10, 2025
37a981f
Refactor Dictionary, Type, and Configuration classes to use file_name…
Jul 10, 2025
104bff3
stepCI: Dictionary Update operation now passing
Jul 10, 2025
21d59b9
stepCI: Dictionary Lock operation now passing
Jul 10, 2025
4d17f7e
stepCI: Delete a locked dictionary returns 500 as expected
Jul 10, 2025
7b2822f
stepCI: Unlock operation now passing
Jul 10, 2025
0412988
stepCI: All Dictionary endpoints now passing
Jul 10, 2025
2112f9c
housekeeping
Jul 10, 2025
d08c52e
Add lock all tests to dictionaries and configurations stepCI tests
Jul 10, 2025
5357d4b
Fix lock_all method to process all files instead of returning after f…
Jul 11, 2025
fd6249e
Remove all legacy 'fields' support and ensure Property class only sup…
Jul 11, 2025
d118aab
Update stepCI dictionaries test to expect current API behavior (no ve…
Jul 11, 2025
c6aef76
Cleanup: remove debug print from Type.save(), update type service ope…
Jul 11, 2025
dd81c51
cleaning up test data
Jul 11, 2025
3c528ba
Fix stepCI types test: add unlock step before cleanup delete to ensur…
Jul 11, 2025
077d127
Fix stepCI configurations test: add unlock step before cleanup delete…
Jul 11, 2025
5c153ef
All stepCI tests now passing! Fixed cleanup delete issues by adding u…
Jul 11, 2025
3b19065
Fix configuration routes to return File objects instead of Configurat…
Jul 11, 2025
210eeb4
Fix lock_all methods to use consistent error handling with Configurat…
Jul 11, 2025
e2c8703
Fix record_success method calls in lock_all methods to not pass data …
Jul 11, 2025
0ec764f
Fix lock_all functionality: Add record_success() calls and update ste…
Jul 11, 2025
253b76b
Implement comprehensive migrations stepCI test
Jul 11, 2025
683e58e
Fix workflow.yaml: correct migrations test reference to #/tests/migra…
Jul 11, 2025
3b7c833
Add working GET /api/enumerators stepCI test (checks known enumerator…
Jul 11, 2025
5dbb5ed
Add working PUT and GET-after-PUT /api/enumerators stepCI tests (over…
Jul 11, 2025
8b0be93
Update OpenAPI spec for enumerators and migrations endpoints to match…
Jul 11, 2025
0843343
housekeeping
Jul 11, 2025
1a83e69
All stepci tests are now passing!
Jul 11, 2025
7782009
Add rendering tests for sample.1.0.0.1 JSON and BSON schemas
Jul 11, 2025
afe675d
Fix enumerator tests to preserve original state and prevent data corr…
Jul 11, 2025
bf58c9b
Refactor Enumerators.to_dict() to return array, update tests for new …
Jul 11, 2025
fa39867
testing enumerators and rendering
Jul 11, 2025
d91ed75
Fix enumerators stepCI tests to preserve existing content and expect …
Jul 11, 2025
ce25a42
Containerization working!
Jul 11, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
1 change: 0 additions & 1 deletion .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,6 @@ Thumbs.db

# Project specific
tests/
docs/
*.egg-info/
.pytest_cache/
.coverage
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/docker-push.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,4 +35,4 @@ jobs:
context: .
platforms: linux/amd64,linux/arm64
push: true
tags: ghcr.io/agile-learning-institute/stage0_mongodb_api:latest
tags: ghcr.io/agile-learning-institute/mongodb_configurator_api:latest
199 changes: 0 additions & 199 deletions CONTRIBUTING.md

This file was deleted.

57 changes: 32 additions & 25 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,38 +1,45 @@
# Stage 1: Build stage
FROM python:3.12-slim as build
# Use Python 3.12 slim image
FROM python:3.12-slim

# Set the working directory in the container
WORKDIR /app
# Set the working directory
WORKDIR /opt/mongo_configurator

# Copy the entire context to the container
COPY . .
# Install system dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
gcc \
&& rm -rf /var/lib/apt/lists/*

# Get the current Git branch and build time
RUN DATE=$(date +'%Y%m%d-%H%M%S') && \
echo "${DATE}" > /app/BUILT_AT
# Copy dependency files first for better caching
COPY Pipfile Pipfile.lock ./

# Stage 2: Production stage
FROM python:3.12-slim
# Install pipenv
RUN pip install pipenv

# Set the working directory in the container
WORKDIR /opt/stage0_mongodb_api
# Copy application code
COPY . .

# Copy the entire source code and the BUILT_AT file from the build stage
COPY --from=build /app/ /opt/stage0_mongodb_api/
# Install dependencies
RUN pipenv install --deploy --system

# Install pipenv and dependencies
COPY Pipfile Pipfile.lock /opt/stage0_mongodb_api/
RUN pip install pipenv && pipenv install --deploy --system
# Create build timestamp
RUN echo $(date +'%Y%m%d-%H%M%S') > /opt/mongo_configurator/BUILT_AT

# Install Gunicorn for running the Flask app in production
# Install Gunicorn for production
RUN pip install gunicorn

# Expose the port the app will run on
# Create non-root user for security
RUN useradd --create-home --shell /bin/bash app && \
chown -R app:app /opt/mongo_configurator

# Switch to non-root user
USER app

# Expose the port
EXPOSE 8081

# Set Environment Variables
ENV PYTHONPATH=/opt/stage0_mongodb_api/stage0_mongodb_api
ENV MONGODB_API_PORT=8081
# Set environment variables
ENV PYTHONPATH=/opt/mongo_configurator/configurator
ENV API_PORT=8081

# Command to run the application using Gunicorn
CMD exec gunicorn --bind 0.0.0.0:${MONGODB_API_PORT} --timeout 120 --preload stage0_mongodb_api.server:app
# Command to run the application
CMD ["gunicorn", "--bind", "0.0.0.0:8081", "--timeout", "10", "--preload", "configurator.server:app"]
28 changes: 11 additions & 17 deletions Pipfile
Original file line number Diff line number Diff line change
Expand Up @@ -4,33 +4,27 @@ verify_ssl = true
name = "pypi"

[scripts]
local = "sh -c 'PYTHONPATH=$(pwd)/stage0_mongodb_api python3 -m server'"
debug = "sh -c 'PYTHONPATH=$(pwd)/stage0_mongodb_api LOGGING_LEVEL=DEBUG python3 -m server'"
batch = "sh -c 'PYTHONPATH=$(pwd)/stage0_mongodb_api AUTO_PROCESS=True EXIT_AFTER_PROCESSING=True LOAD_TEST_DATA=True python3 -m server'"
test = "sh -c 'PYTHONPATH=$(pwd)/stage0_mongodb_api LOGGING_LEVEL=CRITICAL coverage run -m unittest discover -s ./tests -p \"test_*.py\" -v && echo \"\\n=== COVERAGE REPORT ===\" && coverage report && coverage html'"
stepci-observability = "stepci run ./tests/stepci/observability.yaml"
stepci-small = "stepci run ./tests/stepci/small_sample.yaml"
stepci-large = "stepci run ./tests/stepci/large_sample.yaml"
db-drop = "python tests/db_util.py drop"
db-drop-silent = "python tests/db_util.py drop --passphrase DROP_DROWSSAP_YEK"
db-compare = "python tests/db_util.py compare"
db-harvest = "python tests/db_util.py harvest"
build = "docker build --tag ghcr.io/agile-learning-institute/stage0_mongodb_api:latest ."
service = "sh -c 'pipenv run down && docker compose --profile mongodb up --detach && echo `Visit http://localhost:8082/`'"
database = "sh -c 'pipenv run down && docker compose --profile mongodb-only up --detach'"
down = "docker compose down mongodb_spa mongodb_api mongodb"
local = "sh -c 'PYTHONPATH=$(pwd)/configurator BUILT_AT=Local LOAD_TEST_DATA=True python3 -m server'"
debug = "sh -c 'PYTHONPATH=$(pwd)/configurator BUILT_AT=Local LOAD_TEST_DATA=True LOGGING_LEVEL=DEBUG python3 -m server'"
batch = "sh -c 'PYTHONPATH=$(pwd)/configurator AUTO_PROCESS=True EXIT_AFTER_PROCESSING=True LOAD_TEST_DATA=True python3 -m server'"
test = "sh -c 'PYTHONPATH=$(pwd)/configurator LOGGING_LEVEL=CRITICAL coverage run -m unittest discover -s ./tests -p \"test_*.py\" -v && echo \"\\n=== COVERAGE REPORT ===\" && coverage report && coverage html'"
stepci = "stepci run ./tests/stepci/workflow.yaml"
container = "docker build --tag ghcr.io/agile-learning-institute/mongodb_configurator_api:latest ."
database = "sh -c 'pipenv run down && docker compose --profile mongodb up --detach'"
api = "sh -c 'pipenv run down && docker compose --profile configurator-api up --detach'"
service = "sh -c 'pipenv run down && docker compose --profile configurator up --detach && echo `Visit http://localhost:8082/`'"
down = "docker compose down mongodb configurator_api configurator_spa"

[packages]
python-dotenv = "*"
flask = "*"
prometheus-flask-exporter = "*"
pymongo = "*"
stage0-py-utils = "*"
pyyaml = ">=6.0.1"
stage0-mongodb-api = {file = ".", editable = true}

[dev-packages]
coverage = "*"
pytest = "*"

[requires]
python_version = "3.12"
Loading
Loading