Skip to content

Commit

Permalink
Release 2.1.0
Browse files Browse the repository at this point in the history
Semantic Analysis Framework:

- #22: Introduced the new `Slot` object. This object is similar to `Attr`, except that its value is edited directly by the API user, and it does not have an associated computable function. This allows users to inject external metadata into the semantic model.
- #22: Implemented the common semantic feature. This feature enables the user to specify Analyzer-wide semantic graph nodes (attributes and slots) that are shared across all compilation units. These enhancements address the lack of global state within the Analyzer's semantic analysis framework and introduce a conventional method to organize cross-document relationships.

Lexer:

- #16: Introduced new `$xid_start` and `$xid_continue` Lexer Regex classes.
- #16: Added support for classes with combined Unicode properties: `${alpha | num}`.
- #16: Adjusted the `$alpha` class in accordance with UCD specifications.
- #16: Introduced the `lexis::Char` and `lexis::CharProperties` types in the main crate, enabling users to test characters for Unicode properties.
- #20: Added a new Token operator `i("abc")` that expands to case-insensitive matching.
- Fixed an edge-case bug in the Document (MutableUnit). The Mutable Document's lexer sometimes misinterpreted trailing token bounds when the user rewrote the end of the text.
- Fixed a minor bug in the `#[constructor]` attribute of the Token macro.

Syntax Parser:

- #19: Fixed a minor bug in the conflict resolutions of the Node macro's root rule.

Breaking Changes:

- #22: The `analysis::Feature` and `analysis::AbstractFeature` traits have received new members.
  • Loading branch information
Eliah-Lakhin committed Sep 3, 2024
1 parent e6a27d9 commit 13dccee
Show file tree
Hide file tree
Showing 33 changed files with 188 additions and 188 deletions.
4 changes: 2 additions & 2 deletions work/book/src/code-formatters/code-formatters.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,11 +47,11 @@ their respective lines.
Lady Deirdre offers two tools to aid in implementing the code formatter:

1. The
[ParseTree](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/syntax/struct.ParseTree.html)
[ParseTree](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/syntax/struct.ParseTree.html)
builder constructs a concrete parsing tree. Unlike abstract syntax trees, it
intentionally preserves all original source code whitespaces and comments.
2. The
[PrettyPrinter](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/format/struct.PrettyPrinter.html)
[PrettyPrinter](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/format/struct.PrettyPrinter.html)
tool automatically decides on splitting the text into lines and determines
line indentation to ensure that the final lines are aligned according to the
predefined maximum line length.
Expand Down
22 changes: 11 additions & 11 deletions work/book/src/code-formatters/pretty-printer.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ following lines in accordance with the nesting of the groups.

## Printing Words

The [PrettyPrinter::word](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/format/struct.PrettyPrinter.html#method.word)
The [PrettyPrinter::word](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/format/struct.PrettyPrinter.html#method.word)
function inputs a string word token into the printer, which will be printed on
the current line of the output, thus increasing the line length.

Expand All @@ -66,19 +66,19 @@ forcibly break lines. In such cases, you should use the *hardbreak* function.
To separate the words in the output of the printer, you utilize one of the
pretty printer's "blank" token functions:

- [PrettyPrinter::blank](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/format/struct.PrettyPrinter.html#method.blank)
- [PrettyPrinter::blank](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/format/struct.PrettyPrinter.html#method.blank)
is the default blank token, interpreted either as a single whitespace or a
line break. You call this function when the next word should be separated from
the previous one, possibly with a line break, depending on the printing
algorithm's decision.

- [PrettyPrinter::hardbreak](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/format/struct.PrettyPrinter.html#method.hardbreak)
- [PrettyPrinter::hardbreak](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/format/struct.PrettyPrinter.html#method.hardbreak)
is a blank token that enforces the printer to always interpret it as a line
break. You call this function, for instance, when printing the inner content
of multi-line comments, as the structure of the comment's text typically needs
to be preserved.

- [PrettyPrinter::softbreak](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/format/struct.PrettyPrinter.html#method.softbreak)
- [PrettyPrinter::softbreak](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/format/struct.PrettyPrinter.html#method.softbreak)
is similar to the *blank* function, but if the printer's algorithm decides to
preserve the next token on the line, it does not insert whitespace. This
function is useful, for example, to delimit the `.` dot tokens in a
Expand Down Expand Up @@ -115,13 +115,13 @@ more suitable.
```

Content breaking occurs within word groups. To initiate a new group, you use
either [PrettyPrinter::cbox](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/format/struct.PrettyPrinter.html#method.cbox)
or [PrettyPrinter::ibox](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/format/struct.PrettyPrinter.html#method.ibox).
either [PrettyPrinter::cbox](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/format/struct.PrettyPrinter.html#method.cbox)
or [PrettyPrinter::ibox](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/format/struct.PrettyPrinter.html#method.ibox).
The former
begins a consistent word group, while the latter starts an inconsistent group.

Each group must be closed by
calling [PrettyPrinter::end](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/format/struct.PrettyPrinter.html#method.end).
calling [PrettyPrinter::end](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/format/struct.PrettyPrinter.html#method.end).

Both *cbox* and *ibox* functions accept indentation level shifting for the
group, represented by a signed integer. Positive values increase the inner
Expand All @@ -132,7 +132,7 @@ with whitespace according to the current indentation level.
## Overriding Indentations

You can manually adjust line indentation by calling
the [PrettyPrinter::indent](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/format/struct.PrettyPrinter.html#method.indent)
the [PrettyPrinter::indent](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/format/struct.PrettyPrinter.html#method.indent)
function **immediately after** submitting any of the blank tokens. If the
algorithm interprets the submitted token as a line break, the next line, as well
as all subsequent lines, will be shifted accordingly.
Expand All @@ -153,7 +153,7 @@ are situations where it's preferable to keep the parental content aligned in
line and splitting of the nested groups instead.

In such cases, you can utilize
the [PrettyPrinter::neverbreak](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/format/struct.PrettyPrinter.html#method.neverbreak)
the [PrettyPrinter::neverbreak](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/format/struct.PrettyPrinter.html#method.neverbreak)
function, which instructs the printer to reset the current line length counter
to zero. Consequently, the algorithm assumes that the previously submitted text
fits on the line, and begins splitting from the subsequent nested submissions.
Expand All @@ -173,8 +173,8 @@ single line.

This formatting rule depends on whether the algorithm decides to insert a line
break at the blank token. To address this, you can use
the [PrettyPrinter::pre_break](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/format/struct.PrettyPrinter.html#method.pre_break)
and [PrettyPrinter::pre_space](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/format/struct.PrettyPrinter.html#method.pre_space)
the [PrettyPrinter::pre_break](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/format/struct.PrettyPrinter.html#method.pre_break)
and [PrettyPrinter::pre_space](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/format/struct.PrettyPrinter.html#method.pre_space)
functions to configure the preceding blank token.

Both functions must be called **immediately after** submitting the blank token.
Expand Down
26 changes: 13 additions & 13 deletions work/book/src/documents.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,8 @@ compilation unit, along with its lexical and syntax structures. It ensures all
three components remain synchronized.

This object has two
constructors: [Document::new_mutable](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/units/enum.Document.html#method.new_mutable)
and [Document::new_immutable](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/units/enum.Document.html#method.new_immutable).
constructors: [Document::new_mutable](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/units/enum.Document.html#method.new_mutable)
and [Document::new_immutable](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/units/enum.Document.html#method.new_immutable).

Both constructors take the source code text as the initial input for the
Document. The first constructor creates a Document that supports write
Expand All @@ -49,7 +49,7 @@ document's text. The second constructor creates a Document that does not support
write operations but is slightly faster during the document's creation.

To edit a mutable Document, you use
the [Document::write](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/units/enum.Document.html#method.write)
the [Document::write](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/units/enum.Document.html#method.write)
function. Thisfunction takes an arbitrary span of the source code text that you
wish to rewrite and the text you want to insert in place of the specified span.
It rescans the tokens of the affected source code fragment (localized to the
Expand Down Expand Up @@ -79,7 +79,7 @@ documents depending on the current mode of the program.

When the content of a file is being transferred in parts, for example, through a
network or by loading the file from disk in chunks, you can create
a [TokenBuffer](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/lexis/struct.TokenBuffer.html)
a [TokenBuffer](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/lexis/struct.TokenBuffer.html)
and continuously append these chunks into the buffer.

Once the file loading is complete, you can use this token buffer as input for
Expand Down Expand Up @@ -116,7 +116,7 @@ example, a mutable Document can be used as a simple storage of strings with
random read/write access.

In this case, you can use
the [VoidSyntax](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/syntax/struct.VoidSyntax.html)
the [VoidSyntax](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/syntax/struct.VoidSyntax.html)
helper object to enforce the Document to bypass syntax analysis.

```rust,noplayground
Expand All @@ -130,9 +130,9 @@ assert_eq!(doc.substring(..), r#"{ "foo": 456 }"#);
```

The above document has full capabilities of
the [SourceCode](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/lexis/trait.SourceCode.html)
the [SourceCode](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/lexis/trait.SourceCode.html)
trait, but
the [SyntaxTree](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/syntax/trait.SyntaxTree.html)
the [SyntaxTree](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/syntax/trait.SyntaxTree.html)
implementation represents a dummy syntax tree with just a single root node that
covers empty text.

Expand All @@ -141,15 +141,15 @@ covers empty text.
Each instance of the Document (and similar source code storage objects such as
the TokenBuffer) has a globally unique identifier within the current process.

The [Document::id](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/units/enum.Document.html#method.id)
The [Document::id](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/units/enum.Document.html#method.id)
function returns an object of
type [Id](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/arena/struct.Id.html).
type [Id](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/arena/struct.Id.html).
This object is Copy, Eq, and Hash, ensuring that two distinct instances of
documents have distinct identifiers.

Related objects of a Document, such
as [NodeRef](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/syntax/struct.NodeRef.html),
[TokenRef](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/lexis/struct.TokenRef.html),
as [NodeRef](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/syntax/struct.NodeRef.html),
[TokenRef](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/lexis/struct.TokenRef.html),
and others, store the identifier of the document to which they belong.

For example, from a NodeRef referential object, you can determine the identifier
Expand All @@ -176,8 +176,8 @@ You can assign a possibly non-unique string name to the document to simplify
document identification during debugging. For instance, you can use a file name
as a document's name.

The [Id::set_name](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/arena/struct.Id.html#method.set_name)
and [Id::name](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/arena/struct.Id.html#method.name)
The [Id::set_name](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/arena/struct.Id.html#method.set_name)
and [Id::name](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/arena/struct.Id.html#method.name)
functions set and retrieve the current document name, respectively.
Additionally, the crate API uses the document's name in various debugging
functions. For example, the Display implementation of the Document object prints
Expand Down
28 changes: 14 additions & 14 deletions work/book/src/lexis/code-inspection.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,23 +39,23 @@
In Lady Deirdre, the minimal unit for indexing the source code text is the
Unicode character.

A [Site](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/lexis/type.Site.html)
A [Site](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/lexis/type.Site.html)
is a numeric type (an alias of `usize`) representing the absolute Unicode
character index in a string.

[SiteSpan](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/lexis/type.SiteSpan.html)
[SiteSpan](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/lexis/type.SiteSpan.html)
is an alias type of `Range<Site>` that denotes a fragment (or *span*) of the
source code.

Most API functions within the crate conveniently
accept [impl ToSite](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/lexis/trait.ToSite.html)
or [impl ToSpan](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/lexis/trait.ToSpan.html)
accept [impl ToSite](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/lexis/trait.ToSite.html)
or [impl ToSpan](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/lexis/trait.ToSpan.html)
objects when users need to address specific source code
characters or spans of characters. These traits facilitate automatic conversion
between different representations of source code indices.

One example of a source code index type is
the [Position](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/lexis/struct.Position.html)
the [Position](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/lexis/struct.Position.html)
object, which references code in terms of the line and column within the
line[^position]. It implements the *ToSite* trait.

Expand All @@ -69,13 +69,13 @@ However, a particular span instance could be invalid; for instance, `20..10` is
invalid because its lower bound is greater than its upper bound.

Certain API functions in the crate (e.g.,
[Document::write](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/units/enum.Document.html#method.write))
[Document::write](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/units/enum.Document.html#method.write))
require that the specified span must be valid; otherwise, the function would
panic. This behavior aligns with Rust's behavior when indexing arrays with
invalid ranges.

You can check the validity of a range upfront using
the [ToSpan::is_valid_span](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/lexis/trait.ToSpan.html#tymethod.is_valid_span)
the [ToSpan::is_valid_span](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/lexis/trait.ToSpan.html#tymethod.is_valid_span)
function.

The RangeFull `..` object always represents the entire content and is always
Expand All @@ -94,14 +94,14 @@ assert!((Position::new(1, 2)..Position::new(3, 1)).is_valid_span(&buf));
```

[^position]: Please note that the line and column numbers in
the [Position](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/lexis/struct.Position.html)
the [Position](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/lexis/struct.Position.html)
object are one-based: 1 denotes the first line, 2 denotes the second line, and
so forth.

## Text Inspection

The
following [SourceCode](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/lexis/trait.SourceCode.html)'
following [SourceCode](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/lexis/trait.SourceCode.html)'
s functions enable you to query various metadata of the compilation unit's text.

```rust,noplayground
Expand Down Expand Up @@ -131,16 +131,16 @@ assert_eq!(buf.lines().lines_count(), 1);
```

From `buf.lines()`, you receive
a [LineIndex](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/lexis/struct.LineIndex.html)
a [LineIndex](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/lexis/struct.LineIndex.html)
object that provides additional functions for querying metadata about the source
code lines. For example, you can fetch the length of a particular line using
this object.

## Tokens Iteration

The [SourceCode::cursor](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/lexis/trait.SourceCode.html#tymethod.cursor)
The [SourceCode::cursor](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/lexis/trait.SourceCode.html#tymethod.cursor)
and its simplified
version [chunks](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/lexis/trait.SourceCode.html#method.chunks)
version [chunks](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/lexis/trait.SourceCode.html#method.chunks)
allow you to iterate through the tokens of the source code.

Both functions accept a span of the source code text and yield tokens that
Expand All @@ -158,7 +158,7 @@ are in any way related to the specified span.

The *chunks* function simply returns a standard iterator over the token
metadata. Each
[Chunk](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/lexis/struct.Chunk.html)
[Chunk](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/lexis/struct.Chunk.html)
object contains the token instance, a reference to its string, the absolute Site
of the beginning of the token, and the substring length in Unicode
characters[^chunk].
Expand All @@ -184,7 +184,7 @@ for Chunk {
```

The *cursor* function returns a more
complex [TokenCursor](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/lexis/trait.TokenCursor.html)
complex [TokenCursor](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/lexis/trait.TokenCursor.html)
object that implements a cursor-like API with built-in lookahead capabilities
and manual control over the iteration process. This object is particularly
useful for syntax parsing and will be discussed in more detail in the subsequent
Expand Down
4 changes: 2 additions & 2 deletions work/book/src/lexis/lexical-grammar.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@
# Lexical Grammar

The lexical grammar is defined using
the [Token derive macro](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/lexis/derive.Token.html)
the [Token derive macro](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/lexis/derive.Token.html)
on an arbitrary enum type, which represents the type of the token.

Each enum variant represents a token variant. To specify the scanning rule for
Expand Down Expand Up @@ -90,7 +90,7 @@ The type must be Copy, Eq, and `#[repr(u8)]` enum type with variants without a
body.

The macro will implement
the [Token](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/lexis/trait.Token.html)
the [Token](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/lexis/trait.Token.html)
trait on the applicable object, providing not only the scan function itself but
also additional metadata about the lexical grammar.

Expand Down
12 changes: 6 additions & 6 deletions work/book/src/lexis/site-references.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,14 +34,14 @@

# Site References

The [Site](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/lexis/type.Site.html)
The [Site](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/lexis/type.Site.html)
index, which represents the absolute offset of a Unicode character in the source
code text, cannot reliably address a token's absolute offset after source code
edits. This is because the token could be shifted left or right, or it could
disappear during incremental rescanning, depending on the bounds of the edit.

In contrast,
[TokenRef::site](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/lexis/struct.TokenRef.html#method.site)
[TokenRef::site](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/lexis/struct.TokenRef.html#method.site)
returns the absolute offset of the beginning of the token's string fragment at
the time of the call. In other words, this function returns an updated absolute
offset of the token after an edit operation, provided the incremental rescanner
Expand All @@ -50,12 +50,12 @@ did not remove the token during rescanning.
This allows for addressing a token's character bounds relative to changes in the
source code.

The [SiteRef](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/lexis/struct.SiteRef.html)
The [SiteRef](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/lexis/struct.SiteRef.html)
helper object (backed by the TokenRef under the hood) addresses token bounds.
Specifically, this object addresses either the beginning of the token or the end
of the source code.

[ToSite](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/lexis/trait.ToSite.html)
[ToSite](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/lexis/trait.ToSite.html)
implements the ToSite trait, so it can be used as a valid bound of a range span.

```rust,noplayground
Expand All @@ -79,7 +79,7 @@ assert_eq!(doc.substring(brackets_start..brackets_end), "[12345]");
```

Similar to TokenRef, the SiteRef interface has a
special [nil](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/lexis/struct.SiteRef.html#method.nil)
special [nil](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/lexis/struct.SiteRef.html#method.nil)
value and
the [is_nil](https://docs.rs/lady-deirdre/2.0.1/lady_deirdre/lexis/struct.SiteRef.html#method.is_nil)
the [is_nil](https://docs.rs/lady-deirdre/2.1.0/lady_deirdre/lexis/struct.SiteRef.html#method.is_nil)
test function.
Loading

0 comments on commit 13dccee

Please sign in to comment.