Merge PR#4 into main as a single commit

Commands: pushd /tmp; git clone https://github.com/letsencrypt/mariadb-sequential-partition-manager-py.git; popd pushd /tmp/mariadb-sequential-partition-manager-py; git checkout -b pr-branch origin/pr-branch; popd git checkout main; git fetch origin; git reset --hard main cp -a /tmp/mariadb-sequential-partition-manager-py/* . git commit -a
letsencrypt · Sep 20, 2021 · 51e10be · 51e10be
1 parent 9b707e1
commit 51e10be
Show file tree

Hide file tree

Showing 13 changed files with 1,057 additions and 710 deletions.
diff --git a/README.md b/README.md
@@ -1,6 +1,8 @@
 [![Build Status](https://circleci.com/gh/letsencrypt/mariadb-sequential-partition-manager-py.svg?style=shield)](https://circleci.com/gh/letsencrypt/mariadb-sequential-partition-manager-py)
 ![Maturity Level: Beta](https://img.shields.io/badge/maturity-beta-blue.svg)
 
+# Partman
+
 This tool partitions and manages MariaDB tables by sequential IDs.
 
 This is primarily a mechanism for dropping large numbers of rows of data without using `DELETE` statements.
@@ -11,24 +13,60 @@ Similar tools:
 * https://github.com/davidburger/gomypartition, intended for tables with date-based partitions
 * https://github.com/yahoo/mysql_partition_manager, which is archived and in pure SQL
 
-# Usage
+## Usage
+
+```sh
+ → git clone https://github.com/letsencrypt/mariadb-sequential-partition-manager-py.git
+ → cd mariadb-sequential-partition-manager-py
+ → python3 -m venv .venv
+ → . .venv/bin/activate
+ → python3 -m pip install .
+ → tee /tmp/partman.conf.yml <<EOF
+ partitionmanager:
+  num_empty: 2
+  partition_period:
+      days: 90
+  dburl: "sql://user:password@localhost3306:/test_db"
+  tables:
+    cats: {}
+    dogs:
+      partition_period:
+        days: 30
+  prometheus_stats: "/tmp/prometheus-textcollect-partition-manager.prom"
+EOF
+ → partition-manager --config /tmp/partman.conf.yml maintain --noop
+INFO:root:No-op mode
+INFO:partition:Evaluating Table dogs (duration=30 days, 0:00:00) (pos={'id': 150})
+INFO:partition:Table dogs planned SQL: ALTER TABLE `dogs` REORGANIZE PARTITION `p_20201204` INTO (PARTITION `p_20210422` VALUES LESS THAN (221), PARTITION `p_20210522` VALUES LESS THAN MAXVALUE);
+
+dogs:
+ sql: ALTER TABLE `dogs` REORGANIZE PARTITION `p_20201204` INTO (PARTITION `p_20210422` VALUES LESS THAN (221), PARTITION `p_20210522` VALUES LESS THAN MAXVALUE);
+ noop: True
+```
+
+### Running `partman` in your development environment
 
 ```sh
- → pip install --editable .
+ → git clone https://github.com/letsencrypt/mariadb-sequential-partition-manager-py.git
+ → cd mariadb-sequential-partition-manager-py
+ → python3 -m venv .venv
+ → . .venv/bin/activate
+ → python3 -m pip install --editable .
  → partition-manager --log-level=debug  \
     --mariadb test_tools/fake_mariadb.sh \
-    add --noop --table tablename
+    maintain --noop --table tablename
 DEBUG:root:Auto_Increment column identified as id
 DEBUG:root:Partition range column identified as id
 DEBUG:root:Found partition before = (100)
 DEBUG:root:Found tail partition named p_20201204
 INFO:root:No-op mode
 
 ALTER TABLE `dbname`.`tablename` REORGANIZE PARTITION `p_20201204` INTO (PARTITION `p_20201204` VALUES LESS THAN (3101009), PARTITION `p_20210122` VALUES LESS THAN MAXVALUE);
-
 ```
 
-You can also use a yaml configuration file with the `--config` parameter of the form:
+## Configuration
+You can use a yaml configuration file with the `--config` parameter of the form:
+
 ```yaml
 partitionmanager:
   dburl: sql://user:password@localhost/db-name
@@ -48,6 +86,7 @@ partitionmanager:
     table3:
       retention:
         days: 14
+    table4: {}
 ```
 
 For tables which are either partitioned but not yet using this tool's schema, or which have no empty partitions, the `bootstrap` command can be useful for proposing alterations to run manually. Note that `bootstrap` proposes commands that are likely to require partial copies of each table, so likely they will require a maintenance period.
@@ -63,12 +102,20 @@ INFO:calculate_sql_alters:Reading prior state information
 INFO:calculate_sql_alters:Table orders, 24.0 hours, [9236] - [29236], [20000] pos_change, [832.706363653845]/hour
 orders:
  - ALTER TABLE `orders` REORGANIZE PARTITION `p_20210405` INTO (PARTITION `p_20210416` VALUES LESS THAN (30901), PARTITION `p_20210516` VALUES LESS THAN (630449), PARTITION `p_20210615` VALUES LESS THAN MAXVALUE);
-
 ```
 
-# Algorithm
+## Getting started
+
+### Configuring `partman`
+
+- At start, if any configuration file specified as a CLI argument, read that configuration file to set all other values.
+- Then, process all remaining command line arguments, overriding values loaded from the configuration file in case of conflicts.
+- From those command-line arguments, determine whether to collect statistics `stats`, determine an initial partition layout `bootstrap`, or operate in the normal `maintain` mode.
+- Use the configuration information as inputs to the required algorithm.
 
-The core algorithm is implemented in a method `plan_partition_changes` in `table_append_partition.py`. That algorithm is:
+### How does `partman` determine when an additional partition is needed?
+
+The core algorithm is implemented in a method `get_pending_sql_reorganize_partition_commands` in `table_append_partition.py`. That algorithm is:
 
 For a given table and that table's intended partition period, desired end-state is to have:
 - All the existing partitions containing data,
@@ -105,9 +152,17 @@ Procedure:
   - Append the new partition to the intended empty partition list.
 - Return the lists of non-empty partitions, the current empty partitions, and the post-algorithm intended empty partitions.
 
-# TODOs
+#### How do I run `partman` in `noop` mode?
+
+The results of the algorithm are converted into `ALTER` statements; if the user configured `--noop` they're emitted to console and the logs for each table. If not set to `--noop`, the application will execute the ALTERs at the database server and emit the results, including execution time as prometheus statistics if so configured.
+
+#### "Bootstrap" algorithm
+
+The bootstrap mode is a limited form of the "Maintain" Algorithm, using a temporary state file to determine rates-of-change. The bootstrap mode also does not limit itself to only affecting empty partitions, it can and will request changes that will prompt row copies, in order to prepare a table for future use of the "Maintain" algorithm.
+
+## TODOs
 
 Lots:
-[X] Support for tables with partitions across multiple columns.
-[ ] A drop mechanism, for one. Initially it should take a retention period and log proposed `DROP` statements, not perform them.
-[ ] Yet more tests, particularly live integration tests with a test DB.
+- [x] Support for tables with partitions across multiple columns.
+- [ ] A drop mechanism, for one. Initially it should take a retention period and log proposed `DROP` statements, not perform them.
+- [ ] Yet more tests, particularly live integration tests with a test DB.
diff --git a/partitionmanager/bootstrap.py b/partitionmanager/bootstrap.py
@@ -8,18 +8,9 @@
 import operator
 import yaml
 
-from partitionmanager.types import (
-    ChangePlannedPartition,
-    MaxValuePartition,
-    NewPlannedPartition,
-)
-from partitionmanager.table_append_partition import (
-    table_is_compatible,
-    get_current_positions,
-    get_partition_map,
-    generate_sql_reorganize_partition_commands,
-)
-from .tools import iter_show_end
+import partitionmanager.table_append_partition as pm_tap
+import partitionmanager.tools
+import partitionmanager.types
 
 RATE_UNIT = timedelta(hours=1)
 MINIMUM_FUTURE_DELTA = timedelta(hours=2)
@@ -35,12 +26,14 @@ def write_state_info(conf, out_fp):
     log.info("Writing current state information")
     state_info = {"time": conf.curtime, "tables": dict()}
     for table in conf.tables:
-        problem = table_is_compatible(conf.dbcmd, table)
-        if problem:
-            raise Exception(problem)
+        problems = pm_tap.get_table_compatibility_problems(conf.dbcmd, table)
+        if problems:
+            raise Exception("; ".join(problems))
 
-        map_data = get_partition_map(conf.dbcmd, table)
-        positions = get_current_positions(conf.dbcmd, table, map_data["range_cols"])
+        map_data = pm_tap.get_partition_map(conf.dbcmd, table)
+        positions = pm_tap.get_current_positions(
+            conf.dbcmd, table, map_data["range_cols"]
+        )
 
         log.info(f'(Table("{table.name}"): {positions}),')
         state_info["tables"][str(table.name)] = positions
@@ -60,21 +53,26 @@ def _get_time_offsets(num_entries, first_delta, subseq_delta):
     while len(time_units) < num_entries:
         prev = time_units[-1]
         time_units.append(prev + subseq_delta)
-
     return time_units
 
 
 def _plan_partitions_for_time_offsets(
     now_time, time_offsets, rate_of_change, ordered_current_pos, max_val_part
 ):
     """
-    Return a list of PlannedPartitions, starting from now, corresponding to
-    each supplied offset that will represent the positions then from the
-    supplied current positions and the rate of change. The first planned
-    partition will be altered out of the supplied MaxValue partition.
+    Return a list of PlannedPartitions whose positions are predicted to
+    lie upon the supplied time_offsets, given the initial conditions supplied
+    in the other parameters.
+
+    types:
+        time_offsets: an ordered list of timedeltas to plan to reach
+
+        rate_of_change: an ordered list of positions per RATE_UNIT.
     """
     changes = list()
-    for (i, offset), is_final in iter_show_end(enumerate(time_offsets)):
+    for (i, offset), is_final in partitionmanager.tools.iter_show_end(
+        enumerate(time_offsets)
+    ):
         increase = [x * offset / RATE_UNIT for x in rate_of_change]
         predicted_positions = [
             int(p + i) for p, i in zip(ordered_current_pos, increase)
@@ -84,13 +82,15 @@ def _plan_partitions_for_time_offsets(
         part = None
         if i == 0:
             part = (
-                ChangePlannedPartition(max_val_part)
+                partitionmanager.types.ChangePlannedPartition(max_val_part)
                 .set_position(predicted_positions)
                 .set_timestamp(predicted_time)
             )
 
         else:
-            part = NewPlannedPartition().set_timestamp(predicted_time)
+            part = partitionmanager.types.NewPlannedPartition().set_timestamp(
+                predicted_time
+            )
 
             if is_final:
                 part.set_columns(len(predicted_positions))
@@ -130,12 +130,12 @@ def calculate_sql_alters_from_state_info(conf, in_fp):
             log.info(f"Skipping {table_name} as it is not in the current config")
             continue
 
-        problem = table_is_compatible(conf.dbcmd, table)
+        problem = pm_tap.get_table_compatibility_problems(conf.dbcmd, table)
         if problem:
             raise Exception(problem)
 
-        map_data = get_partition_map(conf.dbcmd, table)
-        current_positions = get_current_positions(
+        map_data = pm_tap.get_partition_map(conf.dbcmd, table)
+        current_positions = pm_tap.get_current_positions(
             conf.dbcmd, table, map_data["range_cols"]
         )
 
@@ -150,7 +150,7 @@ def calculate_sql_alters_from_state_info(conf, in_fp):
         rate_of_change = list(map(lambda pos: pos / time_delta, delta_positions))
 
         max_val_part = map_data["partitions"][-1]
-        if not isinstance(max_val_part, MaxValuePartition):
+        if not isinstance(max_val_part, partitionmanager.types.MaxValuePartition):
             log.error(f"Expected a MaxValue partition, got {max_val_part}")
             raise Exception("Unexpected part?")
 
@@ -163,6 +163,9 @@ def calculate_sql_alters_from_state_info(conf, in_fp):
         if table.partition_period:
             part_duration = table.partition_period
 
+        # Choose the times for each partition that we are configured to
+        # construct, beginning in the near future (see MINIMUM_FUTURE_DELTA),
+        # to provide a quick changeover into the new partition schema.
         time_offsets = _get_time_offsets(
             1 + conf.num_empty, MINIMUM_FUTURE_DELTA, part_duration
         )
@@ -176,7 +179,6 @@ def calculate_sql_alters_from_state_info(conf, in_fp):
         )
 
         commands[table.name] = list(
-            generate_sql_reorganize_partition_commands(table, changes)
+            pm_tap.generate_sql_reorganize_partition_commands(table, changes)
         )
-
     return commands
diff --git a/partitionmanager/bootstrap_test.py b/partitionmanager/bootstrap_test.py
@@ -37,7 +37,6 @@ def run(self, cmd):
 
         if "SELECT" in cmd:
             return [{"id": 150}]
-
         return self.response
 
     def db_name(self):