Skip to content

Attach trailing trivia to tokens#102

Merged
aatxe merged 1 commit intoluau-lang:primaryfrom
JohnnyMorganz:cst-trailing-trivia
Apr 22, 2025
Merged

Attach trailing trivia to tokens#102
aatxe merged 1 commit intoluau-lang:primaryfrom
JohnnyMorganz:cst-trailing-trivia

Conversation

@JohnnyMorganz
Copy link
Copy Markdown
Collaborator

As a follow up to #71, this PR starts adding trailing trivia to tokens.

Trivia on a token is split up into leading and trailing trivia. Leading / trailing split is based on the Roslyn Compiler trivia rules: trailing trivia consists of everything up to and including the first \n newline character, and all further trivia is leading trivia of the next token. The idea being that local x = ...\n can then be represented as a single line in trivia, to allow easier replacement (rather than the \n being part of the next statement).

This is done by:

  • Storing a reference to the previously serialized token
  • When serializing a new token, split the extracted trivia into trailing trivia of the previous token, and leading trivia of the next token
  • Attach trailing trivia onto the previous token (via the stored reference lookup)

@JohnnyMorganz JohnnyMorganz marked this pull request as ready for review April 22, 2025 10:30
@aatxe aatxe merged commit fe56f84 into luau-lang:primary Apr 22, 2025
5 checks passed
@JohnnyMorganz JohnnyMorganz deleted the cst-trailing-trivia branch April 22, 2025 17:09
green-real pushed a commit to green-real/lute that referenced this pull request May 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants