Skip to content
Closed
Show file tree
Hide file tree
Changes from 65 commits
Commits
Show all changes
66 commits
Select commit Hold shift + click to select a range
1b96dc9
Initial functionality
jcjones Jan 22, 2021
5d06188
Fix cli test to expect the current date
jcjones Jan 23, 2021
58375fe
Add more CircleCI tests - flake8, pylint
jcjones Jan 25, 2021
8ffc807
Initial PEP249 MySQL connector support
jcjones Feb 17, 2021
3bf8eb1
Remove db option
jcjones Feb 17, 2021
1ef13f6
Use structures instead of line parsing for all DB queries
jcjones Feb 17, 2021
4504c0d
Functioning via DB connect for single-value partitioned tables
jcjones Feb 17, 2021
93e424b
Test for duplicates, add a few tests of reorganize_partition
jcjones Feb 17, 2021
6897140
Catch truncated xml results from a subprocess
jcjones Feb 17, 2021
883383e
Dwarf tables
jcjones Feb 18, 2021
eac56fa
Rename to sequential partition manager, and change CLI tool to partit…
jcjones Feb 18, 2021
72ad177
Typo fix in README
jcjones Feb 18, 2021
c6184fb
Support multiple-column partitions
jcjones Feb 18, 2021
7b74012
Confirm partitioned status before operating on any supplied table
jcjones Feb 18, 2021
8abb8f2
XML parser improved tests and assertions
jcjones Feb 18, 2021
b400ce6
Move partition definitions from tuples to explicit classes
jcjones Feb 19, 2021
69ee351
Show query debugging
jcjones Feb 24, 2021
78d188f
Add basic YAML configuration processing.
jcjones Mar 3, 2021
8568cf7
Rename add_partition to just add
jcjones Mar 3, 2021
2f489a1
Add a lifespan configuration value and only partition when needed
jcjones Mar 4, 2021
bce1527
Have full time resolution for partition decisions. Some logging impro…
jcjones Mar 4, 2021
6dd5c2e
Add basic statistcs command.
jcjones Mar 5, 2021
c681cb7
Print table compatibility issues for all tables before exiting.
jcjones Mar 9, 2021
5b64d19
Bugfix: Tables with multiple columns use BY RANGE COLUMNS
jcjones Mar 9, 2021
09f1a1d
Export Prometheus-style statistics for stats command, if configured
jcjones Mar 9, 2021
3f8240b
Improve tests for the prometheus stats
jcjones Mar 9, 2021
bce1224
Add a time_since_oldest gauge, and rename the gauges a bit
jcjones Mar 9, 2021
b46b703
Emit stats on an 'add' command too
jcjones Mar 9, 2021
ae7e133
Fix prometheus quoting of the labels
jcjones Mar 9, 2021
f3c03b4
Remove the optional timestamp, as our version of Prometheus doesn't l…
jcjones Mar 9, 2021
d9df699
Per-table partition durations
jcjones Mar 10, 2021
4371ecd
v0.1.0: Minimal features complete
jcjones Mar 10, 2021
a830c33
v0.1.1, Bugfix: yaml dburls weren't preprocessed
jcjones Mar 10, 2021
dc1289d
Emit a statistic for table alteration time
jcjones Mar 11, 2021
5a8b3a3
Pre-commit: Make Black run before other tools
jcjones Mar 11, 2021
e2c263e
Spelling fix: partition_name_now
jcjones Mar 11, 2021
b014b7a
Rename partition_duration to partition_period
jcjones Mar 11, 2021
96bf2ad
Predictive partitioning (#3)
jcjones Apr 15, 2021
476aedf
Update README, bugfix that --table and --in/--out were all mutually e…
jcjones Apr 16, 2021
c7c1664
More documentation for the Types
jcjones Apr 20, 2021
5e09ae2
Update per reviews as of 10am PT 21 April, except DocString revisions.
jcjones Apr 22, 2021
6ee10ac
Reorder conditionals in generate_sql_reorganize_partition_commands
jcjones Apr 23, 2021
ef1c3c8
PR#4 Docstring + Readability Updates
jcjones May 3, 2021
a9b0f93
Make Partition and PlannedPartition be internal types
jcjones May 14, 2021
9313228
Make everything in the Partition classes internal
jcjones May 14, 2021
d7adb39
ChangePlannedPartition's old should be internal
jcjones May 14, 2021
d1e2d49
Make all attributes of _PlannedPartition be internal
jcjones May 14, 2021
bd983ec
Doc updates
jcjones May 14, 2021
1a1155e
Make all _internal where possible.
jcjones May 14, 2021
891d7d0
Move away from "from ... import" syntax
jcjones May 14, 2021
42dcdfa
Rename "add" algorithm and mode to "maintain"
jcjones May 17, 2021
43922b5
Refactor a get_pending_sql_reorganize_partition_commands entry for th…
jcjones May 17, 2021
1e88ee8
Initial fixes for tgeoghegan's review comments
jcjones Aug 18, 2021
462b5d4
Fix an order of operations issue for large math
jcjones Aug 18, 2021
e09ba8a
Handle mispredictions cleanly.
jcjones Jun 8, 2021
d62b8ef
Don't allow unexpected equality comparisions between Partitions
jcjones Aug 18, 2021
f0fdc6f
Why are you so important anyway?
jcjones Aug 18, 2021
fa99d75
Update partitionmanager/table_append_partition.py
jcjones Aug 18, 2021
f0c7262
Apply README suggestions from code review
jcjones Aug 18, 2021
6305e5f
Update partitionmanager/cli.py
jcjones Aug 18, 2021
faf90a2
Documentation update for _plan_paritions_for_time_offsets
jcjones Aug 18, 2021
91d795d
Remove blank newlines before all return statements
jcjones Aug 18, 2021
c523947
rename retention_from_dict to timedelta_from_dict
jcjones Aug 18, 2021
73d8374
Update partitionmanager/cli.py
jcjones Aug 18, 2021
e48ec1a
Updates per Tim
jcjones Aug 18, 2021
fcbb489
Update partitionmanager/types.py
jcjones Sep 10, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 46 additions & 0 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
version: 2.1

orbs:
python: circleci/[email protected]

jobs:
python-build-and-test:
executor: python/default
steps:
- checkout
- run: pip install --editable .
- run:
name: make test-reesults dir
command: |
mkdir test-results

- run:
name: Check format with Black
command: |
pip install "black==19.3b0"
python -m black --check .
- run:
name: run tests
command: |
pip install pytest
pytest --junitxml=test-results/junit.xml
- run:
name: Check for linting errors
command: |
pip install "pylint==2.6.0"
python -m pylint -E partitionmanager
- run:
name: Check for flake8 errors
command: |
pip install "flake8==3.8.4"
python -m flake8

- store_test_results:
path: test-results
- store_artifacts:
path: test-results

workflows:
main:
jobs:
- python-build-and-test
4 changes: 4 additions & 0 deletions .flake8
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
[flake8]
# See http://pep8.readthedocs.io/en/latest/intro.html#configuration
ignore = E121, E123, E126, E129, E133, E203, E226, E241, E242, E704, W503, E402, E741
max-line-length = 99
131 changes: 131 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
pip-wheel-metadata/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
target/

# Jupyter Notebook
.ipynb_checkpoints

# IPython
profile_default/
ipython_config.py

# pyenv
.python-version

# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock

# PEP 582; used by e.g. github.com/David-OConnor/pyflow
__pypackages__/

# Celery stuff
celerybeat-schedule
celerybeat.pid

# SageMath parsed files
*.sage.py

# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/
.dmypy.json
dmypy.json

.venv

# Pyre type checker
.pyre/
35 changes: 35 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v2.3.0
hooks:
- id: check-ast
- id: detect-private-key
- id: check-merge-conflict
- id: end-of-file-fixer
- id: requirements-txt-fixer
- id: trailing-whitespace
- repo: https://github.com/psf/black
rev: 19.3b0
hooks:
- id: black
- repo: https://gitlab.com/pycqa/flake8
rev: 3.8.4
hooks:
- id: flake8
- repo: https://github.com/PyCQA/pylint
rev: pylint-2.6.0
hooks:
- id: pylint
args:
- -E
additional_dependencies:
- PyMySQL
- pyyaml
- repo: local
hooks:
- id: pytest
name: Python Tests
language: system
entry: python3 -m pytest
pass_filenames: false
files: '.py$'
Loading