Add static comment helpers

Author: jg-rp

CHANGES.md

@@ -3,6 +3,7 @@
 ## Version 2.2.0 (unreleased)
 
 - Added the `squish` filter. `{{ x | squish }}` is equivalent to `{{ x | strip | split | join }}`. See [#195](https://github.com/jg-rp/liquid/pull/195).
+- Added `BoundTemplate.comments()` and `BoundTemplate.docs()` for statically retrieving `{% comment %}`, `{% # inline comment %}` and `{% doc %}` nodes.
 - Added an **experimental** `{% snippet %}` tag. Shopify/liquid released then quickly removed `{% snippet %}`. We're calling it "experimental" and keeping it disabled by default, pending more activity from Shopify/liquid. See [#191](https://github.com/jg-rp/liquid/pull/191) and [#193](https://github.com/jg-rp/liquid/pull/193).
 - Improved static analysis of partial templates. Previously we would visit a partial template only once, regardless of how many times it is rendered with `{% render %}`. Now we visit partial templates once for each distinct set of arguments passed to `{% render %}`, potentially reporting "global" variables that we'd previously missed.
 - Changed the `string_filter` decorator to coerce `None` to an empty string instead of `"None"`. This is what Shopify/liquid does with `nil.to_s`.

docs/known_issues.md

@@ -32,7 +32,7 @@ The built-in [`date`](filter_reference.md#date) filter uses [dateutil](https://d
 
 Python Liquid might not handle syntax or type errors in the same way as the reference implementation. We might fail earlier or later, and will almost certainly produce a different error message.
 
-Python Liquid does not have a "lax" parser, like Ruby Liquid. If the lexer encounters unknown symbols, a `LiquidSyntaxError` is raised. Upon finding an error in [lax mode](/introduction/strictness), the parser simply discards the current block and continues to to the next block, if one is available. Also, Python Liquid will never inject error messages into an output document.
+Python Liquid does not have a "lax" parser, like Ruby Liquid. If the lexer encounters unknown symbols, a `LiquidSyntaxError` is raised. Upon finding an error in [lax mode](environment.md#tolerance), the parser simply discards the current block and continues to to the next block, if one is available. Also, Python Liquid will never inject error messages into an output document.
 
 ## Orphaned `{% break %}` and `{% continue %}`
 

docs/static_analysis.md

@@ -140,3 +140,38 @@ print(template.tag_names())
 ## Variable, tag and filter locations
 
 [`analyze()`](api/template.md#liquid.BoundTemplate.analyze) and [`analyze_async()`](api/template.md#liquid.BoundTemplate.analyze_async) return an instance of [`TemplateAnalysis`](api/template.md#liquid.static_analysis.TemplateAnalysis). It contains all of the information provided by the methods described above, but includes the location of each variable, tag and filter, each of which can appear many times across many templates.
+
+## Comment and doc nodes
+
+**_New in version 2.2.0_**
+
+[`comments()`](api/template.md#liquid.BoundTemplate.comments) and [`docs()`](api/template.md#liquid.BoundTemplate.docs) return a list of `CommentNode` and `DocNode` instances, respectively. Both have `token` and `text` properties.
+
+```python
+from liquid import parse
+
+source = """\
+{% doc %}
+    some doc comment
+{% enddoc %}
+
+Hello!
+
+{% comment %}
+    some comment
+{% endcomment %}
+
+{% if false %}
+    {% # an inline comment %}
+{% endif %}
+"""
+
+template = parse(source)
+print([node.text for node in template.comments()])
+print([node.text for node in template.docs()])
+```
+
+```plain title="output"
+['\n    some comment\n', 'an inline comment']
+['\n    some doc comment\n']
+```

docs/tag_reference.md

@@ -46,6 +46,16 @@ Inside [liquid tags](#liquid), any line starting with a hash will be considered
 %}
 ```
 
+### Doc comments
+
+**_New in version 2.0.0_**
+
+```liquid2
+{% doc %} ... {% enddoc %}
+```
+
+Like [block comments](#block-comments), doc comments are not rendered. Doc comments are designed to be read by tooling to improve Liquid template development.
+
 ## Output
 
 ```

liquid/builtin/tags/comment_tag.py

@@ -37,7 +37,7 @@ def __init__(self, token: Token, text: Optional[str] = None):
     def __str__(self) -> str:
         return f"{{% comment %}}{self.text}{{% endcomment %}}"
 
-    def render_to_output(self, _: RenderContext, __: TextIO) -> int:
+    def render_to_output(self, context: RenderContext, buffer: TextIO) -> int:  # noqa: ARG002
         """Render the node to the output buffer."""
         return 0
 
@@ -61,7 +61,7 @@ def parse(self, stream: TokenStream) -> CommentNode:
 
         # A block `{% comment %}`
         token = stream.eat(TOKEN_TAG)
-        text = []
+        text: list[str] = []
 
         while True:
             if stream.current.kind == TOKEN_EOF:

liquid/static_analysis.py

@@ -127,7 +127,11 @@ class TemplateAnalysis:
     tags: dict[str, list[Span]]
 
 
-def _analyze(template: BoundTemplate, *, include_partials: bool) -> TemplateAnalysis:
+def analyze(template: BoundTemplate, *, include_partials: bool) -> TemplateAnalysis:
+    """Report variable, filters and tags used in the given template.
+
+    This is used by `BoundTemplate.analyze`.
+    """
     variables = _VariableMap()
     globals = _VariableMap()  # noqa: A001
     locals = _VariableMap()  # noqa: A001
@@ -246,9 +250,13 @@ def _visit(
     )
 
 
-async def _analyze_async(
+async def analyze_async(
     template: BoundTemplate, *, include_partials: bool
 ) -> TemplateAnalysis:
+    """Report variable, filters and tags used in the given template.
+
+    This is used by `BoundTemplate.analyze_async`.
+    """
     variables = _VariableMap()
     globals = _VariableMap()  # noqa: A001
     locals = _VariableMap()  # noqa: A001

liquid/template.py

@@ -13,17 +13,20 @@
 from typing import Optional
 from typing import TextIO
 from typing import Type
+from typing import TypedDict
 from typing import Union
 
+from .builtin.tags.comment_tag import CommentNode
+from .builtin.tags.doc_tag import DocNode
 from .context import FutureContext
 from .context import RenderContext
 from .exceptions import LiquidError
 from .exceptions import LiquidInterrupt
 from .exceptions import LiquidSyntaxError
 from .exceptions import StopRender
 from .output import LimitedStringIO
-from .static_analysis import _analyze
-from .static_analysis import _analyze_async
+from .static_analysis import analyze
+from .static_analysis import analyze_async
 from .utils import ReadOnlyChainMap
 
 if TYPE_CHECKING:
@@ -262,11 +265,11 @@ def analyze(self, *, include_partials: bool = True) -> TemplateAnalysis:
             include_partials: If `True`, we will try to load partial templates and
                 analyze those templates too.
         """
-        return _analyze(self, include_partials=include_partials)
+        return analyze(self, include_partials=include_partials)
 
     async def analyze_async(self, *, include_partials: bool = True) -> TemplateAnalysis:
         """An async version of `analyze`."""
-        return await _analyze_async(self, include_partials=include_partials)
+        return await analyze_async(self, include_partials=include_partials)
 
     def variables(self, *, include_partials: bool = True) -> list[str]:
         """Return a list of variables used in this template without path segments.
@@ -520,6 +523,52 @@ async def tag_names_async(self, *, include_partials: bool = True) -> list[str]:
         """Return a list of tag names used in this template."""
         return list((await self.analyze_async(include_partials=include_partials)).tags)
 
+    def comments(self) -> list[CommentNode]:
+        """Return a list of comment tag nodes found in this template.
+
+        Instances of `CommentNode` and `InlineCommentNode` have `token` and
+        `text` properties. Use `[node.text for node in template.comments()]` to
+        get a list of comment strings.
+
+        Note that this method does not try to load included or rendered
+        templates.
+        """
+        context = RenderContext(self)
+        nodes: list[CommentNode] = []
+
+        def visit(node: Node) -> None:
+            if isinstance(node, CommentNode):
+                nodes.append(node)
+
+            for child in node.children(context, include_partials=False):
+                visit(child)
+
+        for child in self.nodes:
+            visit(child)
+
+        return nodes
+
+    def docs(self) -> list[DocNode]:
+        """Return a list of doc tag nodes found in this template.
+
+        Instances of `DocNode` have `token` and `text` properties. Use
+        `[node.text for node in template.docs()]` to get a list of doc strings.
+        """
+        context = RenderContext(self)
+        nodes: list[DocNode] = []
+
+        def visit(node: Node) -> None:
+            if isinstance(node, DocNode):
+                nodes.append(node)
+
+            for child in node.children(context, include_partials=False):
+                visit(child)
+
+        for child in self.nodes:
+            visit(child)
+
+        return nodes
+
 
 class AwareBoundTemplate(BoundTemplate):
     """A `BoundTemplate` subclass with a `TemplateDrop` in the global namespace."""
@@ -559,6 +608,12 @@ class FutureAwareBoundTemplate(AwareBoundTemplate):
     context_class = FutureContext
 
 
+class _TemplateDropItems(TypedDict):
+    directory: str
+    name: str
+    suffix: str | None
+
+
 class TemplateDrop(Mapping[str, Optional[str]]):
     """Template meta data mapping."""
 
@@ -575,7 +630,7 @@ def __init__(self, name: str, path: Optional[Union[str, Path]]):
         if "." in self.stem:
             self.suffix = self.stem.split(".")[-1]
 
-        self._items = {
+        self._items: _TemplateDropItems = {
             "directory": self.path.parent.name,
             "name": self.path.name.split(".")[0],
             "suffix": self.suffix,
@@ -594,7 +649,7 @@ def __contains__(self, item: object) -> bool:
         return item in self._items
 
     def __getitem__(self, key: object) -> Optional[str]:
-        return self._items[str(key)]
+        return self._items[str(key)]  # type: ignore
 
     def __len__(self) -> int:
         return len(self._items)

tests/test_static_analysis_helpers.py

@@ -148,3 +148,36 @@ async def coro() -> list[str]:
         return await TEMPLATE.tag_names_async()
 
     assert sorted(asyncio.run(coro())) == sorted(["assign", "for"])
+
+
+def test_get_comment_nodes() -> None:
+    source = """\
+A template with comments.
+{% comment %}a{% endcomment %}
+{% comment %}b{% endcomment %}
+{% # c %}
+{% if false %}
+    {% comment %}d{% endcomment %}
+    {% # e %}
+{% endif %}
+"""
+
+    template = parse(source)
+    nodes = template.comments()
+    comment_strings = [node.text for node in nodes]
+    assert comment_strings == ["a", "b", "c", "d", "e"]
+
+
+def test_get_doc_nodes() -> None:
+    source = """\
+{% doc %}
+    Some docs
+{% enddoc %}
+
+{% doc %}More docs{% enddoc %}
+"""
+
+    template = parse(source)
+    nodes = template.docs()
+    doc_strings = [node.text for node in nodes]
+    assert doc_strings == ["\n    Some docs\n", "More docs"]