Add support for auto-indented blocks

[tomas-mazak]

Fixes #178

Blocks now support a new syntax {%* ... %} that aligns the indentation of
multiline string with the block statement itself. This is especially
useful when templating YAML or other languages where indentation matters. Example:

labels.j2:

tla: webtool
env: {{ env }}

deployment.yaml.j2:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    {% include 'labels.j2' %}
  name: webtool
spec:
  selector:
    matchLabels:
      {% include 'labels.j2' %}
  strategy:
    type: Recreate

...renders to broken YAML:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    tla: webtool
env: qa
  name: webtool
spec:
  selector:
    matchLabels:
      tla: webtool
env: qa
  strategy:
    type: Recreate

deployment_new_syntax.yaml.j2:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    {%* include 'labels.j2' %}
  name: webtool
spec:
  selector:
    matchLabels:
      {%* include 'labels.j2' %}
  strategy:
    type: Recreate

...renders correctly:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    tla: webtool
    env: qa
  name: webtool
spec:
  selector:
    matchLabels:
      tla: webtool
      env: qa
  strategy:
    type: Recreate

[tomas-mazak]

This actually only supports blocks now, but if the idea is generally accepted, I am happy to extend support to variables as well

[tomas-mazak]

I've updated the PR with variable support, so now {{* ... }} syntax is supported as well

[Ashish-Bansal]

@davidism Any update on this ? I would love to have this feature merged.

[davidism]

There's now no way to trim leading newlines, unless I'm missing that you can combine * and -.

[tomas-mazak]

@davidism yeah, I only implemented the feature I immediately needed for now... I am happy to look into extensions if required to get this merged

[tomas-mazak]

@davidism Please let me know if you require any amendments to this PR, I am happy to do some extra work on it, but would really like to see this merged.

[stan1y]

Does it support extends syntax as well?

[tomas-mazak]

@stan1y I haven't tested it with extend, but it should work with any kind of block (as it wraps the whole block in a indent filter under the hood)

[caidanw]

I would love to have this feature as well, looks great!

[stan1y]

I can confirm that it worked with extends as expected. Very useful.

[thirtytwobits]

I pulled this into a fork I'm using to generate source code. It's an essential feature in some regards unless someone else knows a better way to do this?

[153957]

👍 I like this new functionality!
However, this PR is missing new tests..

[tomas-mazak]

@153957 Care to provide more details? I've closed/reopened the PR and the tests are failing now with Python nightly but that doesn't seem to be related to this PR's code ...

[153957]

This PR adds new functionality but there are no changes to any files in /tests/, should new functionality not be tested?

/edit
Something similar to this existing test.

jinja/tests/test_lexnparse.py

Lines 70 to 74 in fb7e12c

	def test_raw3(self, env):
	# The second newline after baz exists because it is AFTER the {% raw %} and is ignored.
	env = Environment(lstrip_blocks=True, trim_blocks=True)
	tmpl = env.from_string("bar\n{% raw %}\n {{baz}}2 spaces\n{% endraw %}\nfoo")
	assert tmpl.render(baz='test') == "bar\n\n {{baz}}2 spaces\nfoo"

[tomas-mazak]

@153957 ah, thanks for heads up Arne, I will look into that

[KlavsKlavsen]

This would be a really really nice feature to have.. We are currently using jinja to template yaml files.. and have hit this exact issue with no solution.. (it works in helm - which have a jinja-like templating format)..

[davidism]

Due to #858, which fixed a bug that caused catastrophic backtracking while lexing whitespace, this no longer merges, and it's not trivial for me to adapt this to the new code.

Comments:

• Implementing this by automatically producing a filter call during parsing seems incorrect. Why can't the whitespace be preserved during lexing, as other whitespace handling is done?

• Needs tests. It's possible to apply this to a lot of tokens ({% raw %}, {%, {{, line statements) but it's not clear if all the results will make sense. Fix lexer being extremely slow on large amounts of whitespace. Fixes #857 #858 had to modify the TOKEN_RAW_BEGIN rule as well, is that applicable here?

• It would be nice to make this work with whitespace control.

• Maybe it would make more sense to implement this as another environment option similar to trim_blocks and lstrip_blocks. Maybe block_indent? That way you're opting in to the indent behavior everywhere, and you don't need to add new syntax.

[davidism]

Another thing to consider here is that we also need to consider dedent as well as indent. It's common to indent both within the markup you're generating as well as the Jinja blocks, resulting in extra long indentation once the Jinja blocks are gone.

def validate(data):
    {% for name in required %}
        if "{{ name }}" not in data:
            raise ValidationError("{{ name }} is required")
    {% endfor %}

    return True

This is a contrived example, but the use of indentation to visually distinguish blocks becomes more useful as the template becomes more complex. So what you really want is to ensure that the first non-whitespace character in the contents starts at the same position as the start tag, and then apply that to the rest of the contents.

[davidism]

Thanks for working on this! I'm closing it now based on my comments above. Need to find a better solution for this.