Ensuring compatibility between CLI and theme + theme packaging and versioning

[parmentelat]

wording

theme and template are 2 different things and the naming can be confusing; in this document we use

• theme to refer to the myst-theme repo that contains the source for the app that consumes content
• template to refer to the myst-templates repo that contains essentially the API that the CLI uses to locate and download themes

context

story 1. over summer, at some point freshly built sites with the book theme would have their left-hand pane slightly broken (current section would collapse, making navigation awkward); not a huge deal, but annoying; it took a few months to get it back to working correctly

story 2. current rollout on the new AST for outputs seems to have broken many people's builds

this is no sustainable situation; too many moving pieces; on the longer run we cannot afford breaking users builds

current drawbacks

• mystmd cli pulls latest theme by default
  so an old CLI will pull a recent theme, possibly incompatible
  theme/template caching makes things even more awkward and unpredictable

• theme released through awkward scheme involving "ghost" repos
  a forked theme needs to reproduce this, barriers to entry for creating custom themes

• the API does not provide versioning; the CLI can only resolve 'book-theme' but not 'book-theme v0.5.0'

• minor point, but possible cleanup: there is one extra level of indirection in the case of html themes: the API currently answers a fake .git URL that is later resolved by the CLI; not quite clear why this indirection is needed, if at all
  again this is second order

proposal

theme packaging

through GH releases

• using GH releases to package themes (one PR already ready to go for this)
• allowed for forked themes to be used very easily
• questions
  • what triggers a release ? current PR goes with one release per tag, must be improved given the tagging scheme in place on that repo (ie: multiple subtags per release)
  • be wary about the way actions work wrt tags, esp. when several tags are pushed at the same time

using package managers

Anton suggested to publish on npm/pypi instead of GH releases, and to use the package managers to resolve versions

This track might lead to deeper changes than the gh release track

API changes

regardless, it feels like the following changes would improve the situation

• add versioning (optional) parameters when resolving a theme
• this way the CLI will have a way to request a specific version of a theme
• either because the CLI itself has a default version to use for a given theme (eg: the mystmd CLI v1.2.0 uses book-theme v0.5.0 by default)
• or because the user has specified it in their myst.yml - like Steve's suggestion to say

  theme: book-theme@1.8.0

  or maybe more like

  site:
    theme:
      name: book-theme
      version: 0.5.0
      kind: release

  or even, like we support today

  site:
    theme:
      url: https://custom.dev/my/own/theme.zip
      # or at least something that lets people craft and publish their own theme
      # if we go for pypi/npm then there is a need to be able to express that

questions:

• the theme-version mapping could then be baked in the CLI itself
• or the API could know about that mapping, meaning the CLI would need to pass its own version when requesting a theme

removals

with this scheme, there is no longer a need for using these fake '.git' URLs are they are being used now, in a rather confusing way (ie: the API returns a fake git URL that the CLI then resolves to a GH commit download URL)

migration path

the npm/pypi path might involve more steps, but assuming the gh-release path for now, one possible migration path would be

• adopt the GH releases packaging scheme for themes; at this point both packaging schemes (old commit-based and new release-based) would co-exist
• update the API to support versioned theme requests
  • because older CLI instances will be around for some time we need to, at least temporarily, maintain backward compatibility
  • however in this release of the API, we would stop returning .git URLs, and instead return .zip URLs that do not need further resolution by the CLI (no change needed in the CLI at this point)
  • so this means that the API starts returning pre-resolved URLs for the html themes
  • honestly I'm not exposed to other kinds of themes, so essentially no change for these at all at this point
  • but crucially add support for optional parameters to specify theme version - see above
• the CLI can then be altered to support:
  • user-specified theme versions
  • default theme versions per CLI version
• once all this is in place the CLI code can be simplified to remove the hack around resolving .git URLs

references

this matter has been touched on in the following threads
mostly apart from the last one that could be kept as-is if we go for gh-releases,
they can be ignored, and deemed as having contributed to the above, just mentioned here for the record

• Theme versioning improvements! mystmd#854 - a bit old
• using gh release to publish themes / beneficial for custom themes ? mystmd#2317 - obsolete
• support theme URLs that end in .release mystmd#2344 - obsolete
• publish themes through GH releases myst-theme#659 - still valid for gh-releases
• Consider improving theme packaging mystmd#2553 - about using package managers

[parmentelat]

here's a list of people who to my knowledge have engaged at least once on this matter

• @fwkoch
• @rowanc1
• @agoose77
• @choldgraf
• @akhmerov
• @stevejpurves

please feel free to comment and elaborate on this point of view

ideally and if time permits, it would be nice to summarize and merge in here what's still valid in the various pieces quoted above, and close them afterwards, as a cleanup measure

[stefanv]

For simplicity, I quite like the GH releases pathway for now. Also because it better democratizes user theme publishing than npm or pypi does.

[akhmerov]

Let us also consider the security perspective. pypi/npm/conda-forge guarantee immutable releases, while in github releases it is so far an opt-in. Furthermore, package managers/installers will also have hash validation and compatibility checking, which means a couple less things to implement.

I think it is probably OK to let the user to point to a file online, but I think there should be a more secure option as well.

[akhmerov]

Would an option of a local installation work too? For example the user could supply the myst.yml with the command required to start the theme server or the name of the executable.

This would allow to reuse package managers without interacting with them too closely.

[choldgraf]

To me, GitHub releases is the simplest improvement over what we have right now, and given that we are severely bandwidth limited id recommend that in the name of iteration.

We could add the @M.m.p pattern to our config so users start pinning versions, and then the question of whether the themes are on GitHub or pypi or whatever is an implementation thing that we could change when we have the time.

[agoose77]

Let's put some time into narrowing the scope of this proposal! There are a few ideas at play here, or recently mentioned:

1. Deprecating the API server machinery in favour of something simpler: Overhaul template handling mystmd#2041
2. Defining theme compatibility constraints
3. Reducing the vulnerability of mystmd to supply chain attacks: Improve supply chain security of mystmd and myst-theme mystmd#2490
4. Improving ability for packaging ecosystems to vendor / manage plugins, themes, etc.

I'm passionate about (4), but we can handle that separately. I think we can tackle (1) and (2) to begin with.

At the core, mystmd and myst-theme have a contract, partially defined by the myst spec, implemented by the contents server of mystmd. This means that every mystmd binary has at least one theme that should be supported. For most users, we need to be able to answer the question: "I'm running myst X.Y.Z, which versions of book-theme, or agoose77/my-theme are supported?

This is a deceptively nuanced problem.

The naive way to do this that leans in to (1) is to e.g. define this in template.yml, and mystmd can search through releases to find a compatible release. Near term, we can just add a compatibility table to template.yml and specify the version(s) of mystmd that we support. We can take inspiration from PyPI and publish GitHub releases that have both theme.zip and template.yml artifacts to make it easy to quickly check compatibility.

However, we're not actually worried about the version of mystmd, we're worried about the version of "the contract satisfied by the content server". This inherently means "the contract satisfied by the content of _build/site". Jupyter Book is another tool that can populate this directory. This is also not the MyST spec, which is probably more constrained to just the AST schema + frontmatter schema. But do we need more version numbers? Probably not. I'll keep thinking about this, but we can meanwhile start making progress on standardising publishing via GH releases.

[akhmerov]

With jupyter-book/mystmd#2566 in place I understand the situation much better, and I find it quite tricky.

To break down the complexity let me share how I view the requirement of not breaking stuff that works.

I can think of the following categories of users:

• old mystmd < 1.7.0 (didn't yet upgrade): myst-theme is broken, needs manual pinning
• current mystmd == 1.7.0 (freshly upgraded): themes/templates that weren't yet updated are broken
• future mystmd >= <after_this_mep>: needs to find what is compatible and what isn't

The main motivation to improve over the status quo is that the recent upgrade broke user's code without them taking any action, and this should just never happen. The reason why things broke was mutable external state. Allowing immutable state is the central goal of this MEP. Still, dealing with the current situation is also important, and among other things I would really hope that freezing or diminishing the current mutability is one of the goals.

I would hope that moving forward, myst wouldn't break anything for the users of current myst, and if possible, fix things for old myst. I think that the main tools that can help with that are:

• Changing the API server by e.g. pinning its responses and introducing new endpoints, e.g. /v2/.
• Introducting patch releases to old and/or current myst and jupyter book.

[akhmerov]

Another consideration: usually unpinned version means "get the latest". Because current API doesn't implement version pinning at all, keeping the current semantics will necessarily mean that the themes break when latest becomes incompatible.

For that reason, I believe that theme discovery needs a new path.

[parmentelat]

updated references

here's an updated version of the last paragraph

this matter has been touched on in the following threads
mostly apart from the last one that could be kept as-is if we go for gh-releases,
they can be ignored, and deemed as having contributed to the above, just mentioned here for the record

• Theme versioning improvements! mystmd#854 - a bit old
• using gh release to publish themes / beneficial for custom themes ? mystmd#2317 - obsolete
• support theme URLs that end in .release mystmd#2344 - obsolete
• publish themes through GH releases myst-theme#659 - still valid for gh-releases
• Consider improving theme packaging mystmd#2553 - about using package managers
• Overhaul template handling mystmd#2041 - remove the API altogether ?
• Improve supply chain security of mystmd and myst-theme mystmd#2490 - security concerns wrt supply-chain attacks
• Add a description of the theme server contract mystmd#2566 - improves doc on howto develop/publish a theme

feel free to close any of these if the gist of it has been captured in the present discussion, or if it became obsolete in light of the current discussion

[parmentelat]

thank you guys for all the answers

here's a couple additional comments inspired by them

npm & security

for starters, it seems to me that the discussion about being prone to supply-chain attacks, no matter how important that is obviously, is almost entirely orthogonal to the current discussion, and for the sake of focus at least, I'm going to assume we can ignore that dimension in the current discussion; at this point we're mostly concerned with "how to find a compatible theme", not so mush with "what's in the theme or how does it work"
please speak up if you disagree with that

another consideration on this track is, when specifying a theme, there is now, and I believe we want to preserve this:

• an "abstract" way to say, I want the "book-theme"
• a concrete way, to say lets' use the zip whose URL is ...

my point is, when people go for the latter, it seems fair to say it's their responsibility to pick something safe, and compatible; we cannot be expected to vouch for, say, a locally built URL referred to as a file:///...

with the former on the other hand, it's the tool responsability to provide something safe; and same if we introduce a pinned scheme like book-theme@1.18.0

2 teams

there are clearly two trends in the answers

• a pragmatic view is to say, we need to get something simple to implement, and that can shortly avoid awkward broken builds
• a more ambitious, longer-term goal, that would ideally involve some sort of dynamic constraint resolution

honestly at this point I'm personally more sympathetic to the pragmatic view; however I believe it's possible to move in a way that can achieve the former while not jeopardising the latter

in this regard it would help to have some sense of how exactly we could represent what @agoose77 calls a "contract"; it feels like versioning the Abstract Syntax would play a crucial role, is that the only thing that describes such contract ?
talking of which, is the Abstract Syntax currently versioned ?

API or not API

in jupyter-book/mystmd#2041, there has been a suggestion that was not at all captured in the first draft above, about removing the API altogether

clearly having to run and maintain the API is an additional burden - not even sure who does that - as well as another, single, point of failure btw

on the other hand, the alternative (getting rid of the API) means to bake theme/template resolution right in the CLI; I suspect that, short of achieving a smart constraint resolution thing, we're not yet in a position to realistically remove the API and at the same time achieve the initial goal of ensuring smooth operation of older myst CLI instances

I think the presence of the API right now is an asset; in a situation like last week, having a better-designed API would have allowed us to fix the problem by simply patching the API server; now if we ever reach the point where the API server code is just a pure constraint resolution algo - as opposed to a compatibility matrix - then at that point maybe we can consider removing the API server ?

what now

so that's all the comments that your answers inspired to me
please iterate one more time with your opinions on this way to describe the state of affairs

[parmentelat]

However, we're not actually worried about the version of mystmd, we're worried about the version of "the contract satisfied by the content server". This inherently means "the contract satisfied by the content of _build/site". Jupyter Book is another tool that can populate this directory. This is also not the MyST spec, which is probably more constrained to just the AST schema + frontmatter schema. But do we need more version numbers? Probably not. I'll keep thinking about this, but we can meanwhile start making progress on standardising publishing via GH releases.

at first sight I'd lean to take it this "contract" boils down to the Abstract Syntax version that this content server delivers; are there other dimensions to take into account to properly and accurately describe such contract ?