-
Notifications
You must be signed in to change notification settings - Fork 8
Partition Manager Review #4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from 65 commits
Commits
Show all changes
66 commits
Select commit
Hold shift + click to select a range
1b96dc9
Initial functionality
jcjones 5d06188
Fix cli test to expect the current date
jcjones 58375fe
Add more CircleCI tests - flake8, pylint
jcjones 8ffc807
Initial PEP249 MySQL connector support
jcjones 3bf8eb1
Remove db option
jcjones 1ef13f6
Use structures instead of line parsing for all DB queries
jcjones 4504c0d
Functioning via DB connect for single-value partitioned tables
jcjones 93e424b
Test for duplicates, add a few tests of reorganize_partition
jcjones 6897140
Catch truncated xml results from a subprocess
jcjones 883383e
Dwarf tables
jcjones eac56fa
Rename to sequential partition manager, and change CLI tool to partit…
jcjones 72ad177
Typo fix in README
jcjones c6184fb
Support multiple-column partitions
jcjones 7b74012
Confirm partitioned status before operating on any supplied table
jcjones 8abb8f2
XML parser improved tests and assertions
jcjones b400ce6
Move partition definitions from tuples to explicit classes
jcjones 69ee351
Show query debugging
jcjones 78d188f
Add basic YAML configuration processing.
jcjones 8568cf7
Rename add_partition to just add
jcjones 2f489a1
Add a lifespan configuration value and only partition when needed
jcjones bce1527
Have full time resolution for partition decisions. Some logging impro…
jcjones 6dd5c2e
Add basic statistcs command.
jcjones c681cb7
Print table compatibility issues for all tables before exiting.
jcjones 5b64d19
Bugfix: Tables with multiple columns use BY RANGE COLUMNS
jcjones 09f1a1d
Export Prometheus-style statistics for stats command, if configured
jcjones 3f8240b
Improve tests for the prometheus stats
jcjones bce1224
Add a time_since_oldest gauge, and rename the gauges a bit
jcjones b46b703
Emit stats on an 'add' command too
jcjones ae7e133
Fix prometheus quoting of the labels
jcjones f3c03b4
Remove the optional timestamp, as our version of Prometheus doesn't l…
jcjones d9df699
Per-table partition durations
jcjones 4371ecd
v0.1.0: Minimal features complete
jcjones a830c33
v0.1.1, Bugfix: yaml dburls weren't preprocessed
jcjones dc1289d
Emit a statistic for table alteration time
jcjones 5a8b3a3
Pre-commit: Make Black run before other tools
jcjones e2c263e
Spelling fix: partition_name_now
jcjones b014b7a
Rename partition_duration to partition_period
jcjones 96bf2ad
Predictive partitioning (#3)
jcjones 476aedf
Update README, bugfix that --table and --in/--out were all mutually e…
jcjones c7c1664
More documentation for the Types
jcjones 5e09ae2
Update per reviews as of 10am PT 21 April, except DocString revisions.
jcjones 6ee10ac
Reorder conditionals in generate_sql_reorganize_partition_commands
jcjones ef1c3c8
PR#4 Docstring + Readability Updates
jcjones a9b0f93
Make Partition and PlannedPartition be internal types
jcjones 9313228
Make everything in the Partition classes internal
jcjones d7adb39
ChangePlannedPartition's old should be internal
jcjones d1e2d49
Make all attributes of _PlannedPartition be internal
jcjones bd983ec
Doc updates
jcjones 1a1155e
Make all _internal where possible.
jcjones 891d7d0
Move away from "from ... import" syntax
jcjones 42dcdfa
Rename "add" algorithm and mode to "maintain"
jcjones 43922b5
Refactor a get_pending_sql_reorganize_partition_commands entry for th…
jcjones 1e88ee8
Initial fixes for tgeoghegan's review comments
jcjones 462b5d4
Fix an order of operations issue for large math
jcjones e09ba8a
Handle mispredictions cleanly.
jcjones d62b8ef
Don't allow unexpected equality comparisions between Partitions
jcjones f0fdc6f
Why are you so important anyway?
jcjones fa99d75
Update partitionmanager/table_append_partition.py
jcjones f0c7262
Apply README suggestions from code review
jcjones 6305e5f
Update partitionmanager/cli.py
jcjones faf90a2
Documentation update for _plan_paritions_for_time_offsets
jcjones 91d795d
Remove blank newlines before all return statements
jcjones c523947
rename retention_from_dict to timedelta_from_dict
jcjones 73d8374
Update partitionmanager/cli.py
jcjones e48ec1a
Updates per Tim
jcjones fcbb489
Update partitionmanager/types.py
jcjones File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,46 @@ | ||
version: 2.1 | ||
|
||
orbs: | ||
python: circleci/[email protected] | ||
|
||
jobs: | ||
python-build-and-test: | ||
executor: python/default | ||
steps: | ||
- checkout | ||
- run: pip install --editable . | ||
- run: | ||
name: make test-reesults dir | ||
command: | | ||
mkdir test-results | ||
|
||
- run: | ||
name: Check format with Black | ||
command: | | ||
pip install "black==19.3b0" | ||
python -m black --check . | ||
- run: | ||
name: run tests | ||
command: | | ||
pip install pytest | ||
pytest --junitxml=test-results/junit.xml | ||
- run: | ||
name: Check for linting errors | ||
command: | | ||
pip install "pylint==2.6.0" | ||
python -m pylint -E partitionmanager | ||
- run: | ||
name: Check for flake8 errors | ||
command: | | ||
pip install "flake8==3.8.4" | ||
python -m flake8 | ||
|
||
- store_test_results: | ||
path: test-results | ||
- store_artifacts: | ||
path: test-results | ||
|
||
workflows: | ||
main: | ||
jobs: | ||
- python-build-and-test | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
[flake8] | ||
# See http://pep8.readthedocs.io/en/latest/intro.html#configuration | ||
ignore = E121, E123, E126, E129, E133, E203, E226, E241, E242, E704, W503, E402, E741 | ||
max-line-length = 99 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,131 @@ | ||
# Byte-compiled / optimized / DLL files | ||
__pycache__/ | ||
*.py[cod] | ||
*$py.class | ||
|
||
# C extensions | ||
*.so | ||
|
||
# Distribution / packaging | ||
.Python | ||
build/ | ||
develop-eggs/ | ||
dist/ | ||
downloads/ | ||
eggs/ | ||
.eggs/ | ||
lib/ | ||
lib64/ | ||
parts/ | ||
sdist/ | ||
var/ | ||
wheels/ | ||
pip-wheel-metadata/ | ||
share/python-wheels/ | ||
*.egg-info/ | ||
.installed.cfg | ||
*.egg | ||
MANIFEST | ||
|
||
# PyInstaller | ||
# Usually these files are written by a python script from a template | ||
# before PyInstaller builds the exe, so as to inject date/other infos into it. | ||
*.manifest | ||
*.spec | ||
|
||
# Installer logs | ||
pip-log.txt | ||
pip-delete-this-directory.txt | ||
|
||
# Unit test / coverage reports | ||
htmlcov/ | ||
.tox/ | ||
.nox/ | ||
.coverage | ||
.coverage.* | ||
.cache | ||
nosetests.xml | ||
coverage.xml | ||
*.cover | ||
*.py,cover | ||
.hypothesis/ | ||
.pytest_cache/ | ||
|
||
# Translations | ||
*.mo | ||
*.pot | ||
|
||
# Django stuff: | ||
*.log | ||
local_settings.py | ||
db.sqlite3 | ||
db.sqlite3-journal | ||
|
||
# Flask stuff: | ||
instance/ | ||
.webassets-cache | ||
|
||
# Scrapy stuff: | ||
.scrapy | ||
|
||
# Sphinx documentation | ||
docs/_build/ | ||
|
||
# PyBuilder | ||
target/ | ||
|
||
# Jupyter Notebook | ||
.ipynb_checkpoints | ||
|
||
# IPython | ||
profile_default/ | ||
ipython_config.py | ||
|
||
# pyenv | ||
.python-version | ||
|
||
# pipenv | ||
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. | ||
# However, in case of collaboration, if having platform-specific dependencies or dependencies | ||
# having no cross-platform support, pipenv may install dependencies that don't work, or not | ||
# install all needed dependencies. | ||
#Pipfile.lock | ||
|
||
# PEP 582; used by e.g. github.com/David-OConnor/pyflow | ||
__pypackages__/ | ||
|
||
# Celery stuff | ||
celerybeat-schedule | ||
celerybeat.pid | ||
|
||
# SageMath parsed files | ||
*.sage.py | ||
|
||
# Environments | ||
.env | ||
.venv | ||
env/ | ||
venv/ | ||
ENV/ | ||
env.bak/ | ||
venv.bak/ | ||
|
||
# Spyder project settings | ||
.spyderproject | ||
.spyproject | ||
|
||
# Rope project settings | ||
.ropeproject | ||
|
||
# mkdocs documentation | ||
/site | ||
|
||
# mypy | ||
.mypy_cache/ | ||
.dmypy.json | ||
dmypy.json | ||
|
||
.venv | ||
|
||
# Pyre type checker | ||
.pyre/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,35 @@ | ||
repos: | ||
- repo: https://github.com/pre-commit/pre-commit-hooks | ||
rev: v2.3.0 | ||
hooks: | ||
- id: check-ast | ||
- id: detect-private-key | ||
- id: check-merge-conflict | ||
- id: end-of-file-fixer | ||
- id: requirements-txt-fixer | ||
- id: trailing-whitespace | ||
- repo: https://github.com/psf/black | ||
rev: 19.3b0 | ||
hooks: | ||
- id: black | ||
- repo: https://gitlab.com/pycqa/flake8 | ||
rev: 3.8.4 | ||
hooks: | ||
- id: flake8 | ||
- repo: https://github.com/PyCQA/pylint | ||
rev: pylint-2.6.0 | ||
hooks: | ||
- id: pylint | ||
args: | ||
- -E | ||
additional_dependencies: | ||
- PyMySQL | ||
- pyyaml | ||
- repo: local | ||
hooks: | ||
- id: pytest | ||
name: Python Tests | ||
language: system | ||
entry: python3 -m pytest | ||
pass_filenames: false | ||
files: '.py$' |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.