Missing file extension metadata in notebooks

[juhuebner]

Using specially jupytext-paired notebook content raises the error

 WARNING: Notebook code has no file extension metadata, defaulting to `.txt`

during conversion by sphinx.

Below I first explain what I am up to - which partly works like a charm - and then outline the error upon a modification of the juptext-pairing. Either there happens an error somewhere during nbconvert in myst-nb or I am missing a configuration option, or it's a jupytext issue(?) but I don't see where.

In my sphinx docs folder I have some jupyter notebooks which I pair via

jupytext --set-formats ipynb,.py:percent --sync *.ipynb  

The *.ipynb are ignored by git and in my conf.py I set:

exclude_patterns = ["conf.py"]
# ...
nb_custom_formats = {".py": ["jupytext.reads", {"fmt": "py"}]}

and reverted the registration of the .ipynb files via https://gist.github.com/juhuebner/2d032ebe74a3b57876996487a54ef589

If I now run the sphinx-build it works out nice and in the folder jupyter_execute the notebooks metadata have entries like

"metadata": {
  "jupytext": {
   "encoding": "# -*- coding: utf-8 -*-",
   "formats": "ipynb,.nb.py:percent",
   "text_representation": {
    "extension": ".py",
    "format_name": "percent",
    "format_version": "1.3",
    "jupytext_version": "1.11.2"
   }
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}

However, now in my git workflow I want to treat the notebook python equivalent specially and use a little change to the suffix .nb before the extension .py according to
https://jupytext.readthedocs.io/en/latest/config.html

jupytext --set-formats ipynb,.nb.py:percent --sync *.ipynb  

However, now this error occurs

 WARNING: Notebook code has no file extension metadata, defaulting to `.txt`

and the notebook metadata is missing the language data:

"metadata": {
  "jupytext": {
   "formats": "ipynb,.nb.py:percent"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}

However, the *.nb.py files have some information on the extension

# ---
# jupyter:
#   jupytext:
#     formats: ipynb,.nb.py:percent
#     text_representation:
#       extension: .py
#       format_name: percent
#       format_version: '1.3'
#       jupytext_version: 1.11.2
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

Can anyone help me out with this? Possibly related to #81 #325 ?
Environment

• Python Version 3.8.8:
• nbformat 5.0.8
• myst-nb 0.12.0
• Operating System: Windows 10

[welcome]

Thanks for opening your first issue here! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out EBP's Code of Conduct. Also, please try to follow the issue template as it helps other community members to contribute more effectively.

If your issue is a feature request, others may react to it, to raise its prominence (see Feature Voting).

Welcome to the EBP community! 🎉

[chrisjsewell]

The warning actually arises fro jupyter_sphinx, and should not have been fixed. What version of jupyter_sphinx do you have installed(with the latest myst-nb 0.12.3 you should now be able to have the latest jupyter_sphinx)

[juhuebner]

Thanks for the quick response. The error is still there, but different s.b.

• myst-nb 0.12.3 still has the requirement of myst-parser~=0.13.5 which itself is then inconsistent with jupytext etc.
  • so I went directly to pip install myst-nb==0.13.0a1

My environment now has the following versions:

$> pip show jupyter-sphinx myst-nb markdown-it-py myst-parser
Name: jupyter-sphinx
Version: 0.3.2
Requires: nbconvert, Sphinx, IPython, ipywidgets, nbformat
Required-by: myst-nb
---
Name: myst-nb
Version: 0.13.0a1
Requires: sphinx, importlib-metadata, docutils, nbformat, ipython, ipywidgets, pyyaml, sphinx-togglebutton, jupyter-cache, nbconvert, myst-parser, jupyter-sphinx
Required-by:
---
Name: markdown-it-py
Version: 1.1.0
Requires: attrs
Required-by: myst-parser, mdit-py-plugins, jupytext
---
Name: myst-parser
Version: 0.14.0a2
Requires: markdown-it-py, docutils, jinja2, mdit-py-plugins, sphinx, pyyaml
Required-by: myst-nb

Now the WARNING: Notebook code has no file extension metadata... is gone, however with the exact above procedure the metadata in the formatted notebooks are the same for both cases as shown above, which itself might be a bit misleading.

I noted if first do jupytext --set-formats ipynb,.nb.py:percent --sync *.ipynb and change

nb_custom_formats = {".py": ["jupytext.reads", {"fmt": "py"}]}

the notebooks are formatted correctly and executed. Of course now just sphinx complains about the missing files since they are listed in, e.g., index.rst as notebook and not notebook.nb. I could just the renaming here, but this undermines the flexibility in switching between the notebook source files.

[juhuebner]

The hick-up in my special case is actually happening here

MyST-NB/myst_nb/execution.py

Line 228 in 6be37cd

extension = os.path.splitext(doc_path)[1]

A possible solution would be using the more mighty capabilities of regexes instead of L228/9:

...
    matches = tuple(
        re.search(re.escape(suffix) + "$", doc_path)
        for suffix in env.nb_allowed_exec_suffixes
    )
    if not any(matches):
   ...