From 92071ea4c745158caffa30f212196776b5041d51 Mon Sep 17 00:00:00 2001 From: Sam Weaver Date: Thu, 24 Oct 2024 11:07:11 -0400 Subject: [PATCH] Update the docs for reconstruction --- README.md | 6 +- tree-to-hcl2-reconstruction.md | 124 ++++++++++++++++++++++++++++++++- 2 files changed, 124 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 549ac341..aa4591c3 100644 --- a/README.md +++ b/README.md @@ -44,11 +44,11 @@ with open('foo.tf', 'r') as file: ### Parse Tree to HCL2 reconstruction -With version 5.0.0 the possibility of HCL2 reconstruction from Lark Parse Tree was introduced. +With version 5.x the possibility of HCL2 reconstruction from the Lark Parse Tree and Python dictionaries directly was introduced. -Example of manipulating Lark Parse Tree and reconstructing it back into valid HCL2 can be found in [tree-to-hcl2-reconstruction.md](https://github.com/amplify-education/python-hcl2/blob/main/tree-to-hcl2-reconstruction.md) file. +Documentation and an example of manipulating Lark Parse Tree and reconstructing it back into valid HCL2 can be found in [tree-to-hcl2-reconstruction.md](https://github.com/amplify-education/python-hcl2/blob/main/tree-to-hcl2-reconstruction.md) file. -More details about reconstruction implementation can be found in this [PR](https://github.com/amplify-education/python-hcl2/pull/169). +More details about reconstruction implementation can be found in PRs #169 and #177. ## Building From Source diff --git a/tree-to-hcl2-reconstruction.md b/tree-to-hcl2-reconstruction.md index c403be5e..5089164a 100644 --- a/tree-to-hcl2-reconstruction.md +++ b/tree-to-hcl2-reconstruction.md @@ -1,13 +1,111 @@ -Given `example.tf` file with following content +# Writing HCL2 from Python + +Version 5 of this library supports reconstructing HCL files directly from +Python. This guide details how the reconstruction process takes place. See +also: [Limitations](#limitations) + +There are three major phases: + +- [Building a Python Dictionary](#building-a-python-dictionary) +- [Building an AST](#building-an-ast) +- [Reconstructing the file from the AST](#reconstructing-the-file-from-the-ast) + +## Example + +To create the `example.tf` file with the following content: ```terraform resource "aws_s3_bucket" "bucket" { bucket = "bucket_id" - force_destroy = true + force_destroy = true +} +``` + +You can use the `hcl2.Builder` class like so: + +```python +import hcl2 + +example = hcl2.Builder() + +example.block( + "resource", + ["aws_s3_bucket", "bucket"], + bucket="bucket_id", + force_destroy=True, +) + +example_dict = example.build() +example_ast = hcl2.reverse_transform(example_dict) +example_file = hcl2.writes(example_ast) + +print(example_file) +# resource "aws_s3_bucket" "bucket" { +# bucket = "bucket_id" +# force_destroy = true +# } +# +``` + +This demonstrates a couple of different phases of the process worth mentioning. + +### Building a Python dictionary + +The `hcl2.Builder` class produces a dictionary that should be identical to the +output of `hcl2.load(example_file, with_meta=True)`. The `with_meta` keyword +argument is important here. HCL "blocks" in the Python dictionary are +identified by the presence of `__start_line__` and `__end_line__` metadata +within them. The `Builder` class handles adding that metadata. If that metadata +is missing, the `hcl2.reconstructor.HCLReverseTransformer` class fails to +identify what is a block and what is just an attribute with an object value. +Without that metadata, this dictionary: + +```python +{ + "resource": [ + { + "aws_s3_bucket": { + "bucket": { + "bucket": "bucket_id", + "force_destroy": True, + # "__start_line__": -1, + # "__end_line__": -1, + } + } + } + ] } ``` -below code will add a `tags` object to the S3 bucket definition. The code can also be used to print out readable representation of **any** Parse Tree (any valid HCL2 file), which can be useful when working on your own logic for arbitrary Parse Tree manipulation. +Would produce this HCL output: + +```terraform +resource = [{ + aws_s3_bucket = { + bucket = { + bucket = "bucket_id" + force_destroy = true + } + } +}] +``` + +(This output parses to the same datastructure, but isn't formatted in blocks +as desired by the user. Therefore, using the `Builder` class is recommended.) + +### Building an AST + +The `hcl2.reconstructor.HCLReconstructor` class operates on an "abstract +syntax tree" (`hcl2.AST` or `Lark.Tree`, they're the same.) To produce this AST +from scratch in Python, use `hcl2.reverse_transform(hcl_dict)`, and to produce +this AST from an existing HCL file, use `hcl2.parse(hcl_file)`. + +You can also build these ASTs manually, if you want more control over the +generated HCL output. If you do this, though, make sure the AST you generate is +valid within the `hcl2.lark` grammar. + +Here's an example, which would add a "tags" element to that `example.tf` file +mentioned above. ```python from copy import deepcopy @@ -128,3 +226,23 @@ if __name__ == "__main__": main() ``` + +### Reconstructing the file from the AST + +Once the AST has been generated, you can convert it back to valid HCL using +`hcl2.writes(ast)`. In the above example, that conversion is done in the +`main()` function. + +## Limitations + +- Some formatting choices are impossible to specify via `hcl2.Builder()` and + require manual intervention of the AST produced after the `reverse_transform` + step. + + - Most notably, this means it's not possible to generate files containing + comments (both inline and block comments) + +- Even when parsing a file directly and writing it back out, some formatting + information may be lost due to Terminals discarded during the parsing process. + The reconstructed output should still parse to the same dictionary at the end + of the day though.