Support for Adding MarkdownIt Plugins in `conf.py`

[adam-grant-hendry]

note by @chrisjsewell

This is not something that will be added: the fact that myst-parser uses markdown-it-py is really an implementation detail that is not exposed to the user:

1. This implementation could change in the future
2. It makes myst-parser "responsible" for changes in the markdown-it extension API
3. Its rare that markdown-it extensions can simply be added, without complimentary changes/addition to the base docutils renderer
4. this all adds maintenance burden, for limited gain
5. MyST also has a clear specification, allowing for arbitrary change to the parser means it is no longer myst that is being parsed

---

Context

Originally asked in Discussion #515, it would be nice to add support for using custom MarkdownIt plugins by specifying them in conf.py. One simple use case is discussed in Issue #565 where short code emoji syntax could be utilized with the mdit-py-emoji plugin.

Unfortunately, the myst-parser parsers validate against a fixed set of syntax extensions (see myst_parser/config/main.py), preventing customization.

Proposal

Similar to how sphinx supports built-in extensions and 3rd-party extensions via sphinxcontrib (and others), myst-parser would support built-in extensions and custom extensions via markdown-it-py (and/or others).

In sphinx:

1. The application reads the configuration file and loads built-in and custom extensions. It does so by
   (a) attempting to import the extension as a module with importlib.import_module(),
   (b) run its setup.py, and
   (c) add it to the list of app extensions

Per the docs:

When sphinx-build is executed, Sphinx will attempt to import each module that is listed, and execute yourmodule.setup(app). This function is used to prepare the extension (e.g., by executing Python code), linking resources that Sphinx uses in the build process (like CSS or HTML files), and notifying Sphinx of everything the extension offers (such as directive or role definitions). The app argument is an instance of Sphinx and gives you control over most aspects of the Sphinx build.

In myst-parser, the situation would be much simpler:
(NOTE: Users must install the extension they wish to use so it can be imported.)

1. Add a string parameter extname to create_md_parser
2. Try to import the extension with importlib.import_module() and issue a warning if the extension cannot be loaded (also helpful in the event a user mispells the name of the extension in conf.py)
3. Enable it with MarkdownIt.use()

Tasks and updates

Update myst_parser/parsers/mdit.py::create_md_parser() with the following:

• Add string parameter for extension module name
• Write try...except logic to import the module
• For importable extensions, enable them with MarkdownIt.use()

Modify myst_parser/config/main.py::check_extensions() to:

• Separate built-in extensions from custom extensions rather than raise a ValueError

[welcome]

Thanks for opening your first issue here! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out EBP's Code of Conduct. Also, please try to follow the issue template as it helps other community members to contribute more effectively.

If your issue is a feature request, others may react to it, to raise its prominence (see Feature Voting).

Welcome to the EBP community! 🎉

[jessicah]

I tried adding support for this, it's about as far as loading the plugin I think, but when generating HTML output, I get the following warning:

WARNING: No render method for: emoji [myst.render]

I'd have assumed that https://github.com/BlueGlassBlock/mdit-py-emoji/blob/master/mdit_py_emoji/__init__.py#L49 takes care of that, but it seems not? To be fair, I'm not really a Python dev, so a lot of shooting in the dark.

My current patch:

diff --git a/myst_parser/config/main.py b/myst_parser/config/main.py
index a134ea7..1dc34fb 100644
--- a/myst_parser/config/main.py
+++ b/myst_parser/config/main.py
@@ -49,6 +49,10 @@ def check_extensions(_, __, value):
     if diff:
         raise ValueError(f"'enable_extensions' items not recognised: {diff}")

+# should probably see if we can load the extension?
+def check_loadable_extensions(_, __, value):
+    if not isinstance(value, Iterable):
+        raise TypeError(f"'load_extensions' not iterable: {value}")

 def check_sub_delimiters(_, __, value):
     if (not isinstance(value, (tuple, list))) or len(value) != 2:
@@ -196,6 +200,13 @@ class MdParserConfig:
         },
     )

+    load_extensions: Sequence[str] = dc.field(
+        default_factory=list,
+        metadata={
+            "validator": check_loadable_extensions,
+            "help": "Load additional extensions"},
+    )
+
     # Extension specific

     substitutions: Dict[str, Union[str, int, float]] = dc.field(
diff --git a/myst_parser/parsers/mdit.py b/myst_parser/parsers/mdit.py
index 8476495..2077314 100644
--- a/myst_parser/parsers/mdit.py
+++ b/myst_parser/parsers/mdit.py
@@ -3,6 +3,8 @@ which creates a parser from the config.
 """
 from __future__ import annotations

+from importlib import import_module
+
 from typing import Callable

 from markdown_it import MarkdownIt
@@ -112,6 +114,11 @@ def create_md_parser(
     for name in config.disable_syntax:
         md.disable(name, True)

+    for name in config.load_extensions:
+        module, plugin = name.split('/', 1)
+        mod = import_module(module)
+        md.use(getattr(mod, plugin, None))
+
     md.options.update(
         {
             "typographer": typographer,

My conf.py looks like:

extensions = [
    "myst_parser",
]

# emoji depends on linkify
myst_enable_extensions = [
    "linkify"
]

# newly added config option
myst_load_extensions = [
    "mdit_py_emoji/emoji_plugin"
]

[chrisjsewell]

This is not something I'm really willing to add: the fact that myst-parser uses markdown-it-py is really an implementation detail that is not exposed to the user:

1. This implementation could change in the future
2. It makes myst-parser "responsible" for changes in the markdown-it extension API
3. Its rare that markdown-it extensions can simply be added, without complimentary changes/addition to the base docutils renderer (see e.g. Plugin output is not being rendered #702 (comment))
4. this all adds maintenance burden, for limited gain

[chrisjsewell]

Obviously, if you have ideas for new/improved syntaxes, then I welcome issues here, and also in https://github.com/executablebooks/myst-spec / https://github.com/executablebooks/myst-enhancement-proposals

[strefli3]

In other words, is it currently not possible to use custom extensions with MyST-Parser outside of a Sphinx build?

I was working following the single page builds at https://myst-parser.readthedocs.io/en/latest/docutils.html#single-page-builds, specifically the code

from docutils.core import publish_string
from myst_parser.docutils_ import Parser

source = "hallo world\n: Definition"
output = publish_string(
    source=source,
    writer_name="html5",
    settings_overrides={
        "myst_enable_extensions": ["deflist","MyCustomExt"],
        "embed_stylesheet": False,
    },
    parser=Parser(),
)

which raises (ERROR/3) Global myst configuration invalid: 'enable_extensions' items not recognised: {'MyCustomExt'}.

Based on the discussion in this thread I understand that this is not supported at this time. Is my understanding correct?

[jessicah]

@strefli3 that's correct, as it needs a docutils sub-tree, not an html one.

[Alexey-NM]

@chrisjsewell

1. This implementation could change in the future
2. It makes myst-parser "responsible" for changes in the markdown-it extension API

If we consider the number of plugins this project currently have It will probably never happen.

3. Its rare that markdown-it extensions can simply be added, without complimentary changes/addition to the base docutils renderer (see e.g. Plugin output is not being rendered #702 (comment))

It is a very good argument to allow to adding plugins in conf.py because it will never be added as standard code

IMHO: there are at lease one case for which this issue should be implemented.
It is the ability to add metadata to content.
Of cause there is attrs_block extension.
But attr block require manual editing and it does not allow any automation.
For example currently we need manualy write {#id} in the cases where it can be generated automaticaly.

Will the pull request for this issue be merged, If somebody implements it?

[chrisjsewell]

Will the pull request for this issue be merged, If somebody implements it?

I'm afraid not, for the reasons already outlined