Make Snippets great again (Docs snippet management and qa improvements)

[sh-rp]

Feature description

Main Goal

Improve our snippets and code examples management

Why?

• Creating a snippet is too complicated
• Create an extra py file for every doc page with the snippets is too cumbersome
• Snippet code lives in two places because of the inlining script in the docs, this is confusing
• The way to create the longer examples also is too hard, also we have too much text around those examples
• There are still many snippets in the docs that are not extracted into this snippet system, mainly because there is too much manual work involved
• Even with this snippet system, snippets do not come with metadata, e.g. description and tags
• Snippets are not searchable and do not really appear in an accessible way in any other page that nested on the doc page where they were embedded.

What we can do

• start finding a super simple way to create snippets that really are just the code, default asssertions and imports can easily be removed automatically for insertion. I will do a PR with some examples that we can discuss. They need to be linted, ast-parseable and if there are assertions in there, the snippet will be run. Ideally they will have a short description and tags taken from a pre-defined set (which can be extended if needed)
• Migrate existing snippets and examples to this new format.
• Experiment with a way to extract all python blocks from the docs automatically, I think this could work for a certain amount of snippets.
• Change the docs deployment in such way, that the snippets (and other generated content such as the additional footer links) get inserted at build time but are not persisted to the repo and confuse everyone (including myself if i am completely honest)
• Think about completely auto-generated pages based on a certain snippet tag or maybe have a searchable page for all the snippets. If the snippets are all tagged and described, we can always change this and play around with it without the need to actually change any .md pages.
• Parse and lint all inline snippets for now

Open concerns

• How to maintain many snippets? Will we get into the situation where changing the dlt core will force us to update many snippets?
• How to make contributing from the users easier?
• What is a good balance of docs text vs actual code on a regular doc page?
• We need to make sure that dhelp ingests the generated docs with all content inserted.
• How to not require python or a crazy amount of python dependencies when working on the docs? We want users to be able to contribute without installing stuff of having a working python env.

Are you a dlt user?

Yes, I use it for fun.

Use case

No response

Proposed solution

No response

Related issues

No response