Perform post upgrade checks/verification using a Pulp Smash script

[omaciel]

Our existing upgrade process relies on a shell script that calls pulp-admin to loop through some expected content populated by a prior task. However, there is no reliable way to use pulp-admin to assert that the content populated prior to the upgrade is actually still available and migrated to the newer version.

It would be great to write a Python script which can import Pulp Smash to perform these post upgrade checks and therefore increase our trust in the upgrade jobs.

[kdelee]

@omaciel @Ichimonji10

As I mentioned in issue #520 I am working on this, here are the tests of the populated content

If you can take look I'm sure I could benefit from some code review.

[Ichimonji10]

I have a few suggestions:

1. De-indent this call to delete orphans. When orphans are deleted, they are removed globally from the entire Pulp application. De-indenting that line of code lets there be just one call to delete orphans, instead of one call for each for loop iteration.
2. Run the linters over that code. I noticed a few nit-picky things that could be fixed, like unused imports. The linters will tell you about those issues, and they might find some more serious issues to boot.
3. Add a docstring in the __init__.py module explaining how to use these scripts. Alternately, link from that module to some other place that explains how to use the scripts.

Nice work.

[kdelee]

Great, thanks for the feedback. I'll ping you when I have made some more progress!

[kdelee]

@kersommoura see https://github.com/kdelee/pulp-smash/blob/backup-restore-tests/pulp_migrate/test-restore.py

[Ichimonji10]

This is done. Pulp Migrate contains a Python code that populates Pulp, plus Python code that verifies that Pulp has been correctly populated, both pre- and post-upgrade. Also, pulp_packaging contains an Ansible playbook that uses these scripts, and a JJB definition that calls the Ansible playbook.

We need to fully switch over to using the new code.