Shell commands test suite

[Taitava]

I have been building an Obsidian vault that contains instructions and a template file for performing tests to see if all the features of this plugin work as intended. The tests are performed manually, so there is no automation at this point, and no unit tests at least yet. I have been using this test suite before releasing new versions since version 0.4.0, and partly in version 0.3.0. The test suite has been working great and provides a solid workflow for testing the plugin.

This post is not so useful for normal users who are not so interested in the development of this plugin. I'm targeting this post more towards:

• People, who contribute Macintosh tests for this plugin. (I'm doing Windows & Linux tests myself).
• People, who are interested to see at detail level how this plugin is tested before each release.
• People, who suspect that they have encoutered a possible bug and would like to use some time to carry out some tests before creating an actual issue or a discussion about their findings. Of course, I do not expect that bug reporters would use a lot of time going through a big suite of tests, so using this test suite is absolutely optional!

A test suite cannot cover all possible problems, but at least it helps to reveal previously known issues. It includes tests for the following kind of things:

• Each {{variable}} has at least one shell command, that is used to check that the particular variable works ok both when previewed in command palette, and when executed.
• User interface is tested, which mainly consists of going through the settings panel and creating a new, temporary shell command, assigning a hotkey for it and finally deleting it.
• All settings are tested, e.g. Working directory and Preview variables in command palette. Toggleable settings are tested when they are on and off. Freeform settings (such as Working directory) are tested with some common sets of values.
• Most of the shell commands use echo and redirecting output to a file using >> operator. So commands tend to be really simple.
• Almost all test commands provide output to a file that is open in Obsidian. The test's author then reads that file and checks that the result is correct.
• The test shell commands are designed to try to avoid writing stuff outside of the vault. However, a test author should be careful and pay attention to for example issue Variable values should be escaped #11 which is still not resolved, and to possible bugs that might cause unexpected behavior of shell commands. It's recommended that a test author reads through all shell commands before executing them. When the test author has learned the test suite well, risks are quite small.
• All test items have checkboxes. When a test item has been performed, the checkbox will be ticked if the test completed successfully. If something went wrong, the checkbox is left unchecked, and a comment is written near the item using ==syntax==.

Other features of the test vault:

• Designed to be lightweight: the only community plugin that the vault uses is Shell comands. Only a really good reason could bring more community plugins onboard, because the aim is to make the test suite easy to install and as easy as possible to learn and understand. Core plugins are mostly disabled, but some useful ones (such as Daily notes) are enabled. This way the UI mostly contains buttons that are really needed.
• Testing is started by creating a "daily note", which creates a report file in a correct folder with a correct name, and uses a template that contains all details related to testing. An easy start.
• Test report files are submitted via Git, so be prepared to learn Git commands if you want to perform testing. A separate repository is used, not this one that is used by the actual plugin.

The test suite is not yet public. Shell commands still needs a bit of development until I'm ready to publish the test suite. Mainly it's about #38: I want to be able to make Mac specific versions of certain commands, so that it's easy to distinguish between what commands are meant to be executed on Mac, and what not. Currently I have some commands that are only for Windows and some only for Linux. I can handle them myself, but it's a mess, because there are currently almost 30 commands in total. Luckily, most of them are not OS specific.

I just wanted to post this to let you know that something like this is coming, and also to make it easier for me to discuss about certain features that are needed to be developed mainly for this test suite (#29 and possibly more in the future).

A general preview of the vault. Left: general instructions. Middle: the actual test report file that a test author will use/fill. Right: will contain test commands' output, which is combined to the report file at the end of the test round.

[FelipeRearden]

Hello @Taitava !

This is absolutely fantastic. As far as I understood, this a "framework" that could be applied to any plugin developer before a public release.

You are always bring things to a whole new level!

I just wanted to post this to let you know that something like this is coming, and also to make it easier for me to discuss about certain features that are needed to be developed mainly for this test suite (#29 and possibly more in the future).

You are using Shell Commands to test new features to Shell Commands

I know that I don't have the background, but I think this is amazing and the right way to test plug-ins.

Thanks for everything @Taitava !

[Taitava]

As far as I understood, this a "framework" that could be applied to any plugin developer before a public release.

It's dedicated to Shell commands plugin, but other plugin developers can take ideas from this, for sure! The actual vault itself is so much tied around Shell commands related content, that it's not so feasible to modify that vault for testing other plugins.

This is absolutely fantastic.

Thanks! 🙂 Have a nice day!

[Taitava]

I'll just comment here that the test suite is now public: https://github.com/Taitava/obsidian-shellcommands-test-suite

I'm still improving it, mainly concerning installation instructions and Mac support.