Allow for autodoc to parse Markdown docstrings

[chrisjsewell]

Originally posted by @asmeurer in #163 (comment)

This issue will be of relevance here: sphinx-doc/sphinx#8018

[asmeurer]

There's also the question of numpydoc, which defines its own syntax for some things like parameters. Should myst use the same syntax, but just using Markdown markup in the text? Or should it use something more markdownic?

[chrisjsewell]

I have just added definition list syntax rendering 😄 : see https://myst-parser.readthedocs.io/en/latest/using/syntax-optional.html#definition-lists

I think this could come in handy for an autodoc extension. Something like:

# Parameters

param1
: Description of param1

param2
: Description of param2

Thats maybe more markdownic?

[asmeurer]

I don't know. There's also the Google docstring style, which is a little different (and preferred by many people). It would probably be a good idea to get broader community feedback on these things.

[chrisjsewell]

It would probably be a good idea to get broader community feedback on these things.

Yep absolutely

But note, numpydoc and Google formats are both built around rST syntax.
A markdown extension would use markdown-it-py to initially parse the docstring, and so any format has to be compatible with it in some fashion: utilising existing syntax plugins, or writing new ones.

[Carreau]

If it matters I'm working on decoupling parsing from rendering of docstring in IPython/Jupyter ; basically saying the if you can write a parser that goes from __doc__ to some well defined data structure with the right fields/info, then IPython (and by extension Jupyter) will know how to render it properly/nicely. (This could also pull some informations out of __signature__).

So, if the raw rendering to user In IPython/Jupyter is bothering you and influencing the syntax you are choosing, this will likely become less of an issue for users.

[chrisjsewell]

Thanks @Carreau, I'll bear that in mind 😄

While you're here; I just added https://myst-parser.readthedocs.io/en/latest/using/syntax-optional.html#auto-generated-header-anchors, so that you can write e.g. [](path/to/doc.md#heading-anchor) and it will work correctly both directly on GitHub and building via sphinx.

These anchor slugs, I've found, are a bit changeable in their implementation across renderers, but generally they are converging to the GitHub "specification".

Jupyter Notebook/Lab seems to be a bit outdated in this respect (or at least the versions I tested)?
They don't lower-case or remove punctuation, etc.

I'm surprised by this, because I thought they were both generally built around markedjs at the moment (please move to markdown-it 😉), which does implement this behaviour: https://github.com/styfle/marked/blob/a41d8f9aa69a4095aedae93c6e6ee5522a588217/lib/marked.js#L1991

[john-hen]

I'm very much interested in this feature as I've been using Markdown doc-strings for a while and would like to move from recommonmark to MyST.

By the way, it took me quite a while to get to this GitHub issue here. It would have helped if the section in the docs regarding the autodoc extension clearly stated that Markdown is not supported in doc-strings.

[choldgraf]

That's a great point @John-Hennig - any interest in adding a PR to add a ```{warning} block there that also links to this issue in case folks want to give feedback?

[dmwyatt]

Originally posted by @asmeurer in #163 (comment)

This issue will be of relevance here: sphinx-doc/sphinx#8018

From the feedback autodoc issue it sounds like it might just be better to write a replacement for autodoc rather than trying to extend it?

[oricou]

Here is a trick to have Markdown docstring with commonmark. I guess it could be done with myst_parser.

https://stackoverflow.com/questions/56062402/force-sphinx-to-interpret-markdown-in-python-docstrings-instead-of-restructuredt

Sphinx's Autodoc extension emits an event named autodoc-process-docstring every time it processes a doc-string. You can hook into that mechanism to convert the syntax from Markdown to reStructuredText.

import commonmark

def docstring(app, what, name, obj, options, lines):
    md  = '\n'.join(lines)
    ast = commonmark.Parser().parse(md)
    rst = commonmark.ReStructuredTextRenderer().render(ast)
    lines.clear()
    lines += rst.splitlines()

def setup(app):
    app.connect('autodoc-process-docstring', docstring)

[dmwyatt]

It's funny that you posted that as I made this comment on that a few hours ago.