Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support full reconstruction of HCL from output dictionary #177

Open
wants to merge 16 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,11 +44,11 @@ with open('foo.tf', 'r') as file:

### Parse Tree to HCL2 reconstruction

With version 5.0.0 the possibility of HCL2 reconstruction from Lark Parse Tree was introduced.
With version 5.x the possibility of HCL2 reconstruction from the Lark Parse Tree and Python dictionaries directly was introduced.

Example of manipulating Lark Parse Tree and reconstructing it back into valid HCL2 can be found in [tree-to-hcl2-reconstruction.md](https://github.com/amplify-education/python-hcl2/blob/main/tree-to-hcl2-reconstruction.md) file.
Documentation and an example of manipulating Lark Parse Tree and reconstructing it back into valid HCL2 can be found in [tree-to-hcl2-reconstruction.md](https://github.com/amplify-education/python-hcl2/blob/main/tree-to-hcl2-reconstruction.md) file.

More details about reconstruction implementation can be found in this [PR](https://github.com/amplify-education/python-hcl2/pull/169).
More details about reconstruction implementation can be found in PRs #169 and #177.

## Building From Source

Expand Down
12 changes: 11 additions & 1 deletion hcl2/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,14 @@
except ImportError:
__version__ = "unknown"

from .api import load, loads, parse, parses, transform, writes, AST
from .api import (
load,
loads,
parse,
parses,
transform,
reverse_transform,
writes,
AST,
Builder,
)
14 changes: 14 additions & 0 deletions hcl2/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
from lark.tree import Tree as AST
from hcl2.parser import hcl2
from hcl2.transformer import DictTransformer
from hcl2.builder import Builder


def load(file: TextIO, with_meta=False) -> dict:
Expand Down Expand Up @@ -56,6 +57,19 @@ def transform(ast: AST, with_meta=False) -> dict:
return DictTransformer(with_meta=with_meta).transform(ast)


def reverse_transform(hcl2_dict: dict) -> AST:
"""Convert a dictionary to an HCL2 AST.
:param dict: a dictionary produced by `load` or `transform`
"""
# defer this import until this method is called, due to the performance hit
# of rebuilding the grammar without cache
from hcl2.reconstructor import ( # pylint: disable=import-outside-toplevel
hcl2_reverse_transformer,
)

return hcl2_reverse_transformer.transform(hcl2_dict)


def writes(ast: AST) -> str:
"""Convert an HCL2 syntax tree to a string.
:param ast: HCL2 syntax tree, output from `parse` or `parses`
Expand Down
53 changes: 53 additions & 0 deletions hcl2/builder.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
"""A utility class for constructing HCL documents from Python code."""

from typing import List
from typing_extensions import Self


class Builder:
def __init__(self, attributes: dict = {}):
self.blocks = {}
self.attributes = attributes

def block(
self, block_type: str, labels: List[str] = [], **attributes: dict
) -> Self:
"""Create a block within this HCL document."""
block = Builder(attributes)

# initialize a holder for blocks of that type
if block_type not in self.blocks:
self.blocks[block_type] = []

# store the block in the document
self.blocks[block_type].append((labels.copy(), block))

return block

def build(self):
"""Return the Python dictionary for this HCL document."""
body = {
"__start_line__": -1,
"__end_line__": -1,
**self.attributes,
}

for block_type, blocks in self.blocks.items():

# initialize a holder for blocks of that type
if block_type not in body:
body[block_type] = []

for labels, block_builder in blocks:
# build the sub-block
block = block_builder.build()

# apply any labels
labels.reverse()
for label in labels:
block = {label: block}

# store it in the body
body[block_type].append(block)

return body
30 changes: 22 additions & 8 deletions hcl2/hcl2.lark
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
start : body
body : (new_line_or_comment? (attribute | block))* new_line_or_comment?
attribute : identifier EQ expression
block : identifier (identifier | STRING_LIT)* new_line_or_comment? "{" body "}"
block : identifier (identifier | STRING_LIT | string_with_interpolation)* new_line_or_comment? "{" body "}"
new_line_and_or_comma: new_line_or_comment | "," | "," new_line_or_comment
new_line_or_comment: ( NL_OR_COMMENT )+
NL_OR_COMMENT: /\n[ \t]*/ | /#.*\n/ | /\/\/.*\n/ | /\/\*(.|\n)*?(\*\/)/
Expand All @@ -22,12 +22,26 @@ conditional : expression "?" new_line_or_comment? expression new_line_or_comment
binary_op : expression binary_term new_line_or_comment?
!binary_operator : BINARY_OP
binary_term : binary_operator new_line_or_comment? expression
BINARY_OP : "==" | "!=" | "<" | ">" | "<=" | ">=" | "-" | "*" | "/" | "%" | "&&" | "||" | "+"
BINARY_OP : DOUBLE_EQ | NEQ | LT | GT | LEQ | GEQ | MINUS | ASTERISK | SLASH | PERCENT | DOUBLE_AMP | DOUBLE_PIPE | PLUS
DOUBLE_EQ : "=="
NEQ : "!="
LT : "<"
GT : ">"
LEQ : "<="
GEQ : ">="
MINUS : "-"
ASTERISK : "*"
SLASH : "/"
PERCENT : "%"
DOUBLE_AMP : "&&"
DOUBLE_PIPE : "||"
PLUS : "+"

expr_term : "(" new_line_or_comment? expression new_line_or_comment? ")"
| float_lit
| int_lit
| STRING_LIT
| string_with_interpolation
| tuple
| object
| function_call
Expand All @@ -42,11 +56,10 @@ expr_term : "(" new_line_or_comment? expression new_line_or_comment? ")"
| for_tuple_expr
| for_object_expr


STRING_LIT : "\"" (STRING_CHARS | INTERPOLATION)* "\""
STRING_CHARS : /(?:(?!\${)([^"\\]|\\.))+/+ // any character except '"" unless inside a interpolation string
NESTED_INTERPOLATION : "${" /[^}]+/ "}"
INTERPOLATION : "${" (/(?:(?!\${)([^}]))+/ | NESTED_INTERPOLATION)+ "}"
STRING_LIT : "\"" (STRING_CHARS)* "\""
STRING_CHARS : /(?:(?!\${)([^"\\]|\\.))+/+ // any character except '"' unless inside a interpolation string
Comment on lines +59 to +60
Copy link

@Nfsaavedra Nfsaavedra Nov 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
STRING_LIT : "\"" (STRING_CHARS)* "\""
STRING_CHARS : /(?:(?!\${)([^"\\]|\\.))+/+ // any character except '"' unless inside a interpolation string
STRING_LIT : "\"" STRING_CHARS? "\""
STRING_CHARS : /(?:(?!\${)([^"\\]|\\.))+/ // any character except '"'

This seems to fix the problem, but I did not run the tests to check it. It does parse this file correctly tho.

string_with_interpolation: "\"" (STRING_CHARS)* interpolation_maybe_nested (STRING_CHARS | interpolation_maybe_nested)* "\""
interpolation_maybe_nested: "${" expression "}"

int_lit : DECIMAL+
!float_lit: DECIMAL+ "." DECIMAL+ (EXP_MARK DECIMAL+)?
Expand Down Expand Up @@ -77,8 +90,9 @@ get_attr : "." identifier
attr_splat : ".*" get_attr*
full_splat : "[*]" (get_attr | index)*

FOR_OBJECT_ARROW : "=>"
!for_tuple_expr : "[" new_line_or_comment? for_intro new_line_or_comment? expression new_line_or_comment? for_cond? new_line_or_comment? "]"
!for_object_expr : "{" new_line_or_comment? for_intro new_line_or_comment? expression "=>" new_line_or_comment? expression "..."? new_line_or_comment? for_cond? new_line_or_comment? "}"
!for_object_expr : "{" new_line_or_comment? for_intro new_line_or_comment? expression FOR_OBJECT_ARROW new_line_or_comment? expression "..."? new_line_or_comment? for_cond? new_line_or_comment? "}"
!for_intro : "for" new_line_or_comment? identifier ("," identifier new_line_or_comment?)? new_line_or_comment? "in" new_line_or_comment? expression new_line_or_comment? ":" new_line_or_comment?
!for_cond : "if" new_line_or_comment? expression

Expand Down
Loading
Loading