How does `xdoctest` work with the Sphinx doctest extension?

[Zac-HD]

sphinx.ext.doctest lets you write doctests in your prose documentation (e.g. .rst files), which is really useful for ensuring that usage examples in those docs do in fact still work.

If it's possible to use xdoctest instead of the standard library doctest, that would be lovely - and could do with documentation. If not, a comparison section of the readme would be nice 🙂

[Erotemic]

A sphinx extension for xdoctest is something that I'd really like as well. I've looked into this in the past, but didn't get very far.

Parsing code out of .. code:: python blocks in RST or ```python``` blocks in markdown should be relatively straight forward. This might be best accomplished by modifying xdoctest.core.package_calldefs to accept rst/md files and handle them appropriately. Although I might want to change the name of the function to reflect that it can now extract "calldefs" from more than just packages. Maybe "extract_calldefs"?

It also might be a good idea to even treat RST blocks inside docstrings specially. The docstr in xdoctest.__init__ is currently hacked so the docstring in an RST block doesn't run (which would fail because it lacks the context of the other code in the RST block). Parsing RST blocks explicitly would prevent embedded doctests from being detected as standalone entities.

This also has some overlap with #49

[tony]

You can count me as a markdown sphinx user (MyST-Parser, gh) that'd be interested.

Looks like I may the only one. If this requires an independent issue feel free to create!

Someone made a myst + doctest gist 2 years ago: https://gist.github.com/tonyfast/cfb55f41f5452ef33ec6fbb4e0bda991

[Erotemic]

I'd accept any PR that targets either RST or markdown.

[Erotemic]

FYI: I've started playing around with some hacky stuff that executes xdoctest on sphinx docstrings.

I have the start of this in my xcookie module:
https://github.com/Erotemic/xcookie/blob/a1af10d953302eb04583bf56f6a4da76ff0b8773/docs/source/conf.py#L585

What I'm doing here is I'm using xdoctest to execute any test that creates a matplotlib figure and then it munges the sphinx output to include that image in the final docs. Such an approach can be used to simply execute them all instead.

The basic idea is: use the sphinx.ext.autodoc plugin, and then register a custom callback via the autodoc-process-docstring hook. Then you can write a function that will be passed each docstring.

A hook like this should work pretty well:

def custom_sphinx_run_doctests(app, obj, name, lines):
    import xdoctest
    import sys
    import types
    if isinstance(obj, types.ModuleType):
        module = obj
    else:
        module = sys.modules[obj.__module__]
    modpath = module.__file__
    docstr = '\n'.join(lines)

    import re
    split_parts = re.split('({}\\s*\n)'.format(re.escape('.. rubric:: Example')), docstr)

    # It would be better if xdoctest just has an RST parser. 
    doctests = list(xdoctest.core.parse_docstr_examples(
        part, modpath=modpath, callname=name,
        # style='google'
    ))

    # Not sure why this is so complex. It really shouldn't be.
    try:
        import pytest  # NOQA
    except ImportError:
        pass
    try:
        from xdoctest.exceptions import Skipped
    except ImportError:  # nocover
        # Define dummy skipped exception if pytest is not available
        class Skipped(Exception):
            pass

    for doctest in doctests:
        try:
            doctest.mode = 'native'
            doctest.run(verbose=1, on_error='raise')
        except Skipped:
            print(f'Skip doctest={doctest}')
        except Exception as ex:
            print(f'ex={ex}')
            

I could absolutely see something like this being made into a proper sphinx extension fairly easily. You could even include the real output of each doctest in the generated docs (I think the standard doctest plugin can do that too).

[tony]

Good to hear this

I went from wanting a sphinx extension, to finding it was an impediment. The dream now is pytest / pure doctest can nibble on, so it can scoop up tests in markdown and reStructuredText.

I am going to be studying doctest.py more closely over the weeks