sphinx make mode fails: `SyntaxError: invalid syntax`

[aherrmann]

Issue description

The latest version update of Sphinx changed the behaviour of
sphinx-quickstart to create a Makefile that calls sphinx-build
in make mode (-M flag) by default.

This fails in a Nix shell.

The previous non-make mode still works.

CC: @FRidh

Steps to reproduce

• Create the following file

  shell.nix:

  let
    nixpkgs = (import <nixpkgs> {}).fetchFromGitHub {
      owner = "NixOS";
      repo = "nixpkgs";
      rev = "3ccc139b3db8d9417006eb9deeb75038b2fc33fa";
      sha256 = "1fzbkqlni45hsx313c6cnh4apwzsqasv3vmgp7ih20lhmxv9w5sj";
    };
  in
  
  with (import nixpkgs {});
  
  let
    py = python35.withPackages (ps: with ps; [ sphinx ]);
  in
  pkgs.stdenv.mkDerivation {
    name = "sphinx-env";
    buildInputs = [ py ];
  }

• Enter a Nix shell

  $ nix-shell
  

• Run sphinx-quickstart and enter the default options.

  sphinx-quickstart session

  $ sphinx-quickstart
  Welcome to the Sphinx 1.5.1 quickstart utility.
  
  Please enter values for the following settings (just press Enter to
  accept a default value, if one is given in brackets).
  
  Enter the root path for documentation.
  > Root path for the documentation [.]:
  
  You have two options for placing the build directory for Sphinx output.
  Either, you use a directory "_build" within the root path, or you separate
  "source" and "build" directories within the root path.
  > Separate source and build directories (y/n) [n]:
  
  Inside the root directory, two more directories will be created; "_templates"
  for custom HTML templates and "_static" for custom stylesheets and other static
  files. You can enter another prefix (such as ".") to replace the underscore.
  > Name prefix for templates and static dir [_]:
  
  The project name will occur in several places in the built documentation.
  > Project name: test
  > Author name(s): test
  
  Sphinx has the notion of a "version" and a "release" for the
  software. Each version can have multiple releases. For example, for
  Python the version is something like 2.5 or 3.0, while the release is
  something like 2.5.1 or 3.0a1.  If you don't need this dual structure,
  just set both to the same value.
  > Project version []:
  > Project release []:
  
  If the documents are to be written in a language other than English,
  you can select a language here by its language code. Sphinx will then
  translate text that it generates into that language.
  
  For a list of supported codes, see
  http://sphinx-doc.org/config.html#confval-language.
  > Project language [en]:
  
  The file name suffix for source files. Commonly, this is either ".txt"
  or ".rst".  Only files with this suffix are considered documents.
  > Source file suffix [.rst]:
  
  One document is special in that it is considered the top node of the
  "contents tree", that is, it is the root of the hierarchical structure
  of the documents. Normally, this is "index", but if your "index"
  document is a custom template, you can also set this to another filename.
  > Name of your master document (without suffix) [index]:
  
  Sphinx can also add configuration for epub output:
  > Do you want to use the epub builder (y/n) [n]:
  
  Please indicate if you want to use one of the following Sphinx extensions:
  > autodoc: automatically insert docstrings from modules (y/n) [n]:
  > doctest: automatically test code snippets in doctest blocks (y/n) [n]:
  ) [n]: sphinx: link between Sphinx documentation of different projects (y/n)
  : todo: write "todo" entries that can be shown or hidden on build (y/n) [n]:
  > coverage: checks for documentation coverage (y/n) [n]:
  > imgmath: include math, rendered as PNG or SVG images (y/n) [n]:
  > mathjax: include math, rendered in the browser by MathJax (y/n) [n]:
  n]: config: conditional inclusion of content based on config values (y/n) [n
  y/n) [n]: : include links to the source code of documented Python objects (y
  s (y/n) [n]: : create .nojekyll file to publish the document on GitHub pages
  
  A Makefile and a Windows command file can be generated for you so that you
  only have to run e.g. `make html' instead of invoking sphinx-build
  directly.
  > Create Makefile? (y/n) [y]:
  > Create Windows command file? (y/n) [y]: n
  
  Creating file ./conf.py.
  Creating file ./index.rst.
  Creating file ./Makefile.
  
  Finished: An initial directory structure has been created.
  
  You should now populate your master file ./index.rst and create other documentation
  source files. Use the Makefile to build the docs, like so:
     make builder
  where "builder" is one of the supported builders, e.g. html, latex or linkcheck.
  

• Run make html and observe the error:

  $ make html
    File "/nix/store/r9p2wdbaz1ll67cqvr97zvhih7yvyhn2-python3.5-Sphinx-1.5.1/bin/sphinx-build", line 2
      export PATH=/nix/store/8qn95swyn4higc63c1rlaacvwkz6iaw7-python3-3.5.2/bin:/nix/store/r9p2wdbaz1ll67cqvr97zvhih7yvyhn2-python3.5-Sphinx-1.5.1/bin:/nix/store/rmhvpa9m94m4iniamnldy9y9l3kgx921-python3.5-docutils-0.13.1/bin:/nix/store/xrlbgkpz9ksvbhj11iwvv1844ryfy4i5-python3.5-setuptools-30.2.0/bin:/nix/store/f8ijxyzx7ilzxpbkpvydnqyxrxb13gx8-python3.5-Pygments-2.1.3/bin:/nix/store/i1zrk7ia2qac7mp3a39p9gmily04bxpv-python3.5-Babel-2.3.4/bin${PATH:+:}$PATH
                ^
  SyntaxError: invalid syntax
  make: *** [Makefile:20: html] Error 1
  

Workaround

• Run sphinx-quickstart --no-use-make-mode instead of sphinx-quickstart:

  sphinx-quickstart session

  $ sphinx-quickstart --no-use-make-mode
  Welcome to the Sphinx 1.5.1 quickstart utility.
  
  Please enter values for the following settings (just press Enter to
  accept a default value, if one is given in brackets).
  
  Enter the root path for documentation.
  > Root path for the documentation [.]:
  
  You have two options for placing the build directory for Sphinx output.
  Either, you use a directory "_build" within the root path, or you separate
  "source" and "build" directories within the root path.
  > Separate source and build directories (y/n) [n]:
  
  Inside the root directory, two more directories will be created; "_templates"
  for custom HTML templates and "_static" for custom stylesheets and other static
  files. You can enter another prefix (such as ".") to replace the underscore.
  > Name prefix for templates and static dir [_]:
  
  The project name will occur in several places in the built documentation.
  > Project name: test
  > Author name(s): test
  
  Sphinx has the notion of a "version" and a "release" for the
  software. Each version can have multiple releases. For example, for
  Python the version is something like 2.5 or 3.0, while the release is
  something like 2.5.1 or 3.0a1.  If you don't need this dual structure,
  just set both to the same value.
  > Project version []:
  > Project release []:
  
  If the documents are to be written in a language other than English,
  you can select a language here by its language code. Sphinx will then
  translate text that it generates into that language.
  
  For a list of supported codes, see
  http://sphinx-doc.org/config.html#confval-language.
  > Project language [en]:
  
  The file name suffix for source files. Commonly, this is either ".txt"
  or ".rst".  Only files with this suffix are considered documents.
  > Source file suffix [.rst]:
  
  One document is special in that it is considered the top node of the
  "contents tree", that is, it is the root of the hierarchical structure
  of the documents. Normally, this is "index", but if your "index"
  document is a custom template, you can also set this to another filename.
  > Name of your master document (without suffix) [index]:
  
  Sphinx can also add configuration for epub output:
  > Do you want to use the epub builder (y/n) [n]:
  
  Please indicate if you want to use one of the following Sphinx extensions:
  > autodoc: automatically insert docstrings from modules (y/n) [n]:
  > doctest: automatically test code snippets in doctest blocks (y/n) [n]:
  ) [n]: sphinx: link between Sphinx documentation of different projects (y/n)
  : todo: write "todo" entries that can be shown or hidden on build (y/n) [n]:
  > coverage: checks for documentation coverage (y/n) [n]:
  > imgmath: include math, rendered as PNG or SVG images (y/n) [n]:
  > mathjax: include math, rendered in the browser by MathJax (y/n) [n]:
  n]: config: conditional inclusion of content based on config values (y/n) [n
  y/n) [n]: : include links to the source code of documented Python objects (y
  s (y/n) [n]: : create .nojekyll file to publish the document on GitHub pages
  
  A Makefile and a Windows command file can be generated for you so that you
  only have to run e.g. `make html' instead of invoking sphinx-build
  directly.
  > Create Makefile? (y/n) [y]:
  > Create Windows command file? (y/n) [y]: n
  
  Creating file ./conf.py.
  Creating file ./index.rst.
  Creating file ./Makefile.
  
  Finished: An initial directory structure has been created.
  
  You should now populate your master file ./index.rst and create other documentation
  source files. Use the Makefile to build the docs, like so:
     make builder
  where "builder" is one of the supported builders, e.g. html, latex or linkcheck.
  

• Run make html. This time it works:

  $ make html
  sphinx-build -b html -d _build/doctrees   . _build/html
  Running Sphinx v1.5.1
  making output directory...
  loading pickled environment... not yet created
  building [mo]: targets for 0 po files that are out of date
  building [html]: targets for 1 source files that are out of date
  updating environment: 1 added, 0 changed, 0 removed
  reading sources... [100%] index
  looking for now-outdated files... none found
  pickling environment... done
  checking consistency... done
  preparing documents... done
  writing output... [100%] index
  generating indices... genindex
  writing additional pages... search
  copying static files... done
  copying extra files... done
  dumping search index in English (code: en) ... done
  dumping object inventory... done
  build succeeded.
  
  Build finished. The HTML pages are in _build/html.
  

Technical details

• System:

  $ lsb_release -a
  LSB Version:    core-2.0-noarch:core-3.2-noarch:core-4.0-noarch:core-2.0-x86_64:core-3.2-x86_64:core-4.0-x86_64:desktop-4.0-amd64:desktop-4.0-noarch:graphics-2.0-amd64:graphics-2.0-noarch:graphics-3.2-amd64:graphics-3.2-noarch:graphics-4.0-amd64:graphics-4.0-noarch
  Distributor ID: openSUSE project
  Description:    openSUSE 13.2 (Harlequin) (x86_64)
  Release:        13.2
  Codename:       Harlequin
  

• Nix version:

  $ nix-env --version
  nix-env (Nix) 1.11.2
  

• Nixpkgs version:

  revision: 3ccc139b3db8d9417006eb9deeb75038b2fc33fa

[FRidh]

Thanks for reporting this.

The Makefile that is created with sphinx-quickstart contains

# Minimal makefile for Sphinx documentation
#

# You can set these variables from the command line.
SPHINXOPTS    =
SPHINXBUILD   = sphinx-build
SPHINXPROJ    = a
SOURCEDIR     = .
BUILDDIR      = _build

# Put it first so that "make" without argument is like "make help".
help:
        @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

.PHONY: help Makefile

# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
        @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

Al unknown targets are passed to sphinx-build, which results in a syntax error as reported by @aherrmann.

@vcunat , would you know how to refer to sphinx-build ($(SPHINXBUILD)) in such a way that the wrapper of sphinx-build works?

[aherrmann]

I should add, if you execute sphinx-build -M html . _build in the nix shell, that will also fail with the same error. However, if you execute sphinx-build -b html . _build then it will succeed.

[FRidh]

So in make-mode it seems to think the wrapper is Python code. It tries to execute it in a subprocess with call
https://github.com/sphinx-doc/sphinx/blob/1.5.2/sphinx/make_mode.py#L265
and uses sys.executable to obtain the name
https://github.com/sphinx-doc/sphinx/blob/1.5.2/sphinx/make_mode.py#L263

Edit: sys.executable is of course the name of the Python interpreter. Therefore, what it does is a python sphinx-build ... and that obviously fails since the wrapper is called first, which isn't Python. Therefore, we should patch sphinx to not invoke sphinx-build with Python, but to invoke it directly, e.g. by using sys.argv[0].

[FRidh]

Possibly related issue #21568

[aherrmann]

@FRidh Thanks a lot. I can confirm that it works now.