Provide NixOS module option to enable the paperless exporter.

[ctheune]

Description of changes

Integrate the paperless document exporter as a backup feature into the module.

Also fixes a configuration (quoting) issue.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 23.11 Release Notes (or backporting 23.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

[erikarvstedt]

Here are fixups for your first commit, which should definitely be merged.
Maybe add a dedicated PR because these changes are orthogonal to backups and entirely uncontroversial.

[erikarvstedt]

The term backup is misleading because this only exports the documents but not the database (containing all the metadata).
Is a doc-only export generally useful for users?

Also, the export command used by the backup service creates redundant copies of each doc in multiple formats, which is also not suitable for backups.

[ctheune]

The term backup is misleading because this only exports the documents but not the database (containing all the metadata). Is a doc-only export generally useful for users?

Well it is what is recommended as the official backup strategy according to the manual. It also seems comprehensive with the metadata. I can see all the table content like tags, correspondents, users, even saved filters and UI settings ...

Also, the export command used by the backup service creates redundant copies of each doc in multiple formats, which is also not suitable for backups.

That is configurable using the command parameters that I've made adjustable.

[ctheune]

Here are fixups for your first commit, which should definitely be merged.
Maybe add a dedicated PR because these changes are orthogonal to backups and entirely uncontroversial.

Thanks for those. I'll double check (and I guess I should add a test) that the quoting works both for systemd and the manage command now.

[erikarvstedt]

It also seems comprehensive with the metadata. I can see all the table content like tags, correspondents, users, even saved filters and UI settings ...

Ah right, the metadata is exported to manifest.json.

That is configurable using the command parameters that I've made adjustable.

These should be fixed so that no redundant content is exported by default.

I'm still not convinced that paperless needs a dedicated backup service. It can be generically backed up like many other database-based NixOS services: Either by (1) snappshotting and transfering the whole of /var/lib or by (2) using services.postgresqlBackup and just rsyncing /var/lib/paperless/.
Note that like (2) the document exporter method used in this PR is not atomic/consistent. Quote from the manual: "Before making backups, make sure that paperless is not running."
Let's delegate the decision to other paperless users. @mweinelt, @Flakebi, @lukegb, @leona-ya, what do you think?

[erikarvstedt]

Thanks for those. I'll double check (and I guess I should add a test) that the quoting works both for systemd and the manage command now.

There's no quoting involved when setting up the systemd env, so this has always worked correctly.
As for the manage script, escapeShellArg is guaranteed to work. It's a common pattern in NixOS:

nixpkgs/lib/strings.nix

Line 482 in 189a069

"${name}=${escapeShellArg value}"

[mweinelt]

(2) using services.postgresqlBackup and just rsyncing /var/lib/paperless/.

That is what we're doing, but with ZFS snapshots of the data dir.

[leona-ya]

I think there are two different use-cases for backups:

1. Restore to paperless, after a host failure
2. Clean backup independent from the software

1 can probably just be handled by backuping postgresql + the paperless state dir.
2 is also an idea that I like very much. It respects the value of the documents (in the metadata, ASNs are relevant, for example).

---

Probably i would still, even if this feature was available, just make a backup in the style of 1. But I can also understand why people (probably @ctheune) may think different.
If we want to include this I would definitely rename the option from backup to documentExporter. I'm not 100% sure how to decide in the conflict between 'don't overcomplicate module' (is this change something a user should do in their own config) and 'allow users to easily use the features they want for a DMS'.

[erikarvstedt]

The export service just boils down to this simple snippet, which we could simply add to the NixOS manual:

{ config, ... }:
let
  paperless = config.services.paperless;
in
  systemd.services.paperless-export = {
    startAt = "daily";
    serviceConfig = {
      User = paperless.user;
      ExecStart = ''
        ${paperless.dataDir}/paperless-manage document_exporter <export_dir> --no-progress-bar --no-color --compare-checksums --delete
      '';
    };
  };

[erikarvstedt]

@Atemu, maybe let's first decide if or in what form we want to include this.

[ctheune]

Here are fixups for your first commit, which should definitely be merged.
Maybe add a dedicated PR because these changes are orthogonal to backups and entirely uncontroversial.

Thanks. I've created #243084 for the orthogonal changes.

[ctheune]

Alright. I'm happy to follow pretty much most of the suggestions, but as the PR itself is still under question, I'll postpone that.

Having to take paperless offline was something I overlooked and I guess the exporter could arrange for that.

Whether to include it, I see the following parts that need discussion:

1. Not using the name "backup" – sure, I can perfectly live with calling it documentExporter as @leona-ya suggested.
2. I'm running it with the built-in sqlite thing and I don't have snapshots as simple solution available in my environment, so the document exporter seems the most attractive route to me but I'd prefer it to be part of the tested feature set. Apparently it did not run out of the box due to the quoting issue and I'd love if we can avoid regressions here that everyone has to fix themselves.
3. Complication of the module ... well ... not sure what the metric here is in value per complication ;) ... as a user I would have loved this being already in there as keeping my data safe is kind of the job of a DMS and we see similar complications in other database modules.

[Atemu]

Migrated my config over to this and it was a breeze.

Atemu/nixos-config@c34c9e9

[erikarvstedt]

The module and the test are broken. Fixups.
Fetch with:

git fetch https://github.com/erikarvstedt/nixpkgs paperless-backup-master-ea-4
git log ..FETCH_HEAD

Further remarks. Medium prio, so ignore if you want to get this merged fast:
I agree with @Atemu that pre/postScript is redudant. There are many ways to achieve this without extra options, most notably so:

systemd.services.paperless-exporter = {
  preStart = "...";
  postStart = "...";
}

Also, like @Atemu mentioned before, needless log output is annoying and doesn't fit the style of other NixOS services.
The exporter prints Running pre/post script even when no pre/post script is run. We should remove this kind of output entirely.

When using pre/postStart = "...";, systemd always shows which stage is currently running and the log shows the offending stage on failures. So diagnosability is fine.

[ctheune]

Ugh, dang. Apparently I didn't run the tests after my last changes - sorry, that was bad form on my side. Although I did run the test when twiddling the v.default stuff and that did fulfill the test ... if it did that by accident, then I'll need to revisit that test. I'll pick up your changes and double check that.

I'll remove the pre/post options.

[Edit]: Note that I'm personally still on the fence with the pre/post stuff. I guess I dislike splitting up those units of execution in an IMO clobbered way over multiple scripts. I know we rely endlessly on systemd, but it still feels clunky, especially if you want to run things manually for development/debugging.

[Edit]: I'm still removing the pre/post script stuff, but please note that it doesn't provide the diagnosability things as I mentioned: it does show the stage when it breaks. It doesn't show the stage if it's silent and gets stuck, so there's also no info how long steps take ... but I guess that's something everyone can put in their pre/post scripts if they want to see this. (Except being able to see the moment when the actual export starts because the unit start log entry will reflect when the unit is started, not when the main process starts)

[erikarvstedt]

LGTM. Thanks for your patience!

I did run the test when twiddling the v.default stuff and that did fulfill the test

Weird, v.default will always error when evaluating the module, regardless of what config options are used.

[leona-ya]

Thanks again! LGTM

And sorry for not testing correctly @erikarvstedt

[SuperSandro2000]

So many maintainers will probably just diverge responsibility and in the end no one feels really responsible.

[SuperSandro2000]

#373472