Hook to replace rst parsing in autodoc

[sbliven]

Is your feature request related to a problem? Please describe.

I would like to use an alternate markup syntax for my docstrings (specifically MyST). Sphinx plugins that provide alternate syntaxes for documentation files (e.g recommonmark, MyST-parser, etc) seem to produce docutil AST objects, but the autodoc modules store and manipulate rST strings instead.

Describe the solution you'd like

It would be great if there was an interface for replacing rST parsing in autodoc via plugins, analogous to how plugins can provide parsers for specific file extensions.

Describe alternatives you've considered

The closest I could find was the autodoc-process-docstring hook, which does allow rewriting the docstrings prior to parsing. However, it expects rST output and most parsers output AST. I suppose it would be possible to create a AST to rST writer, but this seems backwards since the rST will eventually have to be parsed back to AST.

Additional context

• MyST issue "How to use autodoc with myst?"

[tk0miya]

It would be great if there was an interface for replacing rST parsing in autodoc via plugins, analogous to how plugins can provide parsers for specific file extensions.

Basically, autodoc does not parse docstrings. It only generates an internal reST document and passes it to upstream's parser.

def foo(x, y):
    """blah blah blah"""

For example, autodoc generates the following internal reST document from above code:

.. py:function:: foo(x, y)

   blah blah blah

I agree that autodoc is strongly coupled with reST. So it would be nice if we can use it from other parsers. But is hard to inject a parser because it does not parse docstring at all.

[sbliven]

Thanks, @tk0miya. So in principle could I overwrite the various Documenter classes to emit directives in my syntax of choice, and then switch the parser upstream?

[tk0miya]

Everything is okay if you'll implement it. What you said is rewriting almost of autodoc extension. It's not an easy way. But autodoc is mere python code, not magic. So you can do it.

FYI: Here is a place where autodoc calls upstream parser after generation of internal reST document. You also need to hack here to allow switching parsers.

sphinx/sphinx/ext/autodoc/directive.py

Line 157 in 376d4fe

result = parse_generated_content(self.state, params.result, documenter)

[tk0miya]

Now I'm closing this because it is hard to realize this request (autodoc does not parse at all!).