Let bootstrap work on tables that aren't partitioned yet. #13
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- This moves from the -E to -X XML-based CLI and parses it
This removes any dependnecy on the auto-increment value, as AI can only tell us a single column's position, and for multi-column partitions we'll need more than that. This change does not totally remove the get_autoincrement method, as we'll still want to confirm the table has the partitioned feature, which we'll do in a next commit.
Removes the auto_incremennt mechanisms.
This adds the necessary retention options for later command processing, which is not included in this commit.
There are other BY RANGE options too, but let's just start with COLUMNS
* First pass algorithm
* Add ability to compare partition positions
* Add a split method for dividing partition lists
* More tests
* Add a position rate function
* Add methods to determine a weighted rate of increase
* Add docs to the new table_append_partition methods
* Use the Partition timestamp() method
* plan_partition_changes algorithm
* More partition planning tests
* Predictive partitiong algorithm functioning in tests
* Rework the CLI to use the new partition planning algorithm
* Passing integration tests
* Handle short and bespoke partition names.
* Improve logging
* Remove spurious strip
* Moving to 0.2.0
* Logging cleanups
* Fix a host of pylint issues
$ pylint --ignore-patterns=.*_test.py partitionmanager/ --disable W1203 --disable invalid-name --disable bad-continuation
************* Module partitionmanager.tools
partitionmanager/tools.py:22:11: R1708: Do not raise StopIteration in generator, use return statement instead (stop-iteration-return)
************* Module partitionmanager.sql
partitionmanager/sql.py:36:0: R0903: Too few public methods (1/2) (too-few-public-methods)
************* Module partitionmanager.stats
partitionmanager/stats.py:12:0: R0903: Too few public methods (0/2) (too-few-public-methods)
partitionmanager/stats.py:65:0: R0912: Too many branches (14/12) (too-many-branches)
************* Module partitionmanager.table_append_partition
partitionmanager/table_append_partition.py:98:0: R0914: Too many local variables (16/15) (too-many-locals)
partitionmanager/table_append_partition.py:306:0: R0914: Too many local variables (23/15) (too-many-locals)
------------------------------------------------------------------
Your code has been rated at 9.92/10 (previous run: 9.91/10, +0.01)
* Better logging on partition
* Never adjust the active_partition
MariaDB has a limitation on editing the active partition, particularly:
`ERROR 1520 (HY000): Reorganize of range partitions cannot change total ranges
except for last partition where it can extend the range`
so we can't edit the active partition, either.
* Never edit positions on empty partitions
Like the previous commit, MariaDB has a limitation on editing any partition's
offset:
`ERROR 1520 (HY000): Reorganize of range partitions cannot change total ranges
except for last partition where it can extend the range`
So the positions field should never be edited for existing partitions, only
their names.
* Consolidate logic to use partition names as start-of-fill dates
* stderr is not so useful from the Subprocess Database Command, let's dump it
* Bugfix: get_current_positions needs to query the latest of each column
Before, get_current_positions returned each column for the entry with the
largest ID from the first column, while for partitioning purposes we
actually want to always be strictly increasing.
This does make such tables less space-efficient, but that's a matter for
partition design.
* Add "bootstrap" methods to prepare partitioned tables
Tables whose partitions don't contain datestamps of the p_YYYYMMDD form don't
provide partman enough info to derive rates of change, so these bootstrap
routines will save a YAML file somewhere with point-in-time data that can be
reloaded to derive a rate-of-change. This is only intended to be used for the
initial partitioning of a table, or when a table has no empty partitions.
In a subsequent commit I'll tie this into cli.py, ensuring to add alerts that
these ALTERs cannot be expected to complete quickly, that likely the database
will hold locks for substantial amounts of time for each of the ALTER commands,
and the tool will simply be printing potential ALTER commands to console for
an operator to analyze and run in the manner they find best.
* Wire up Bootstrap to the CLI
* Rework CLI to print yaml-like but stringified output
tgeoghegan
reviewed
Aug 5, 2021
There was a problem hiding this comment.
So I see where inserts and updates to the old table will be replicated to the new table with triggers. However I don't understand how the rows from the old table get copied to the new table. Or is that intentionally not done, the idea being that only the new partitions are created on the new table?
Right. For our purposes we don't backfill the data, instead we rely on ongoing copies until we atomically shift traffic over. Doing data backfills takes too much CPU time. |
jcjones
force-pushed
the
bootstrap_tablecopy
branch
from
411e288
to
ee018d7
Compare
Commands: pushd /tmp; git clone https://github.com/letsencrypt/mariadb-sequential-partition-manager-py.git; popd pushd /tmp/mariadb-sequential-partition-manager-py; git checkout -b pr-branch origin/pr-branch; popd git checkout main; git fetch origin; git reset --hard main cp -a /tmp/mariadb-sequential-partition-manager-py/* . git commit -a
Add a --assume-partitioned-on flag to bootstrap, to facilitate operation on tables that do not yet have a partition map. One needs to supply `--assume-partitioned-on COLUMN_NAME` as many times as needed to identify all the columns which will be part of the partition expression. The 'bootstrap' command now emits table-copy instructions The original plan for 'bootstrap' was to do live alterations, that they should only lock what they needed, however InnoDB likes to lock everything. So instead we need to always assume that "bootstrapping" will be a live table clone, and at their conclusion the team should perform an atomic rename. Fix unit test to not depend on time of day (oops) Catch an arithmetic error Emit output lines with MySQL comment characters
This corrects a bug in MaxValuePartition where its value string wasn't paren-surrounded if there are actually multiple columns.
jcjones
force-pushed
the
bootstrap_tablecopy
branch
from
ee018d7
to
afc8400
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add a --assume-partitioned-on flag to bootstrap, to facilitate operation on
tables that do not yet have a partition map. One needs to supply
--assume-partitioned-on COLUMN_NAMEas many times as needed to identify allthe columns which will be part of the partition expression.
The 'bootstrap' command now emits table-copy instructions
The original plan for 'bootstrap' was to do live alterations, that they should
only lock what they needed, however InnoDB likes to lock everything. So instead
we need to always assume that "bootstrapping" will be a live table clone,
and at their conclusion the team should perform an atomic rename.