Pre-proposal: Incorporate `jupyter-book` as a Jupyter sub-project

[choldgraf]

Context

For the last several years, Jupyter Book has been developed as part of the executablebooks/ project. It has grown considerably over that time in usage, and the technical stack behind it has evolved considerably.

In addition, in the last two years there's been an effort within executablebooks to create a document engine that aligns more directly with Jupyter's infrastructure, and that could serve as a back-end for Jupyter Book, rather than Sphinx. This work is at https://mystmd.org and has made significant progress.

The executablebooks/ grant is winding down, and we are thinking about the best "home" for various technical pieces that we've created as part of the project. Some will stay within the executablebooks/ org, and others may be moved to other orgs. This brings me to our proposal:

Proposal summary

We'd like to move several repositories within executablebooks/ into a new GitHub organization called jupyter-book, and incorporate this as a sub-project within Jupyter. See the linked document below for the full proposal, feel free to comment and/or suggest edits.

Proposal text

There are a few documents that can help others understand this proposal, and we invite feedback on the first in particular.

1. Proposal to incorporate a jupyter-book subproject. This is structured as a JEP, and provides the information needed to help the community make a decision. It includes a list of the repositories we intend to incorporate under jupyter-book/, and which will remain where they are.
2. A blog post describing our intent to use MyST as Jupyter Book's document engine. Our hope is to bring the MyST document engine to feature parity with Sphinx, and to then offer it as a replacement back-end for Jupyter Book. This will align Jupyter Book's technical stack with MyST, and offer the unified technical product that drives much of the jupyter-book/ sub-project.
3. List of repositories we propose incorporating under jupyter-book. These are the subset of repositories in executablebooks/ that we believe are within the strategic scope of the proposed jupyter-book/ sub-project.

Proposed authors

• @choldgraf
• @gregcaporaso
• @jstac
• @rowanc1

To do

• Create initial draft
• Iterate with current executablebooks/ steering council -> v0.1
• Create GitHub issue and e-mail the Jupyter EC -> broad-level feedback
• Iterate per feedback -> v1.0alpha
• Create a JEP and post our draft in the Jupyter "enhancement proposals" repository.
• Iterate with EC/SSC feedback -> v1.0
• Vote on the proposal

[choldgraf]

cc @gregcaporaso, and @jstac @rowanc1, who are the other Steering Council members of executablebooks/ and co-proposers here.

[fperez]

This is exciting, many thanks team!!

We had a chance to briefly discuss this during today's EC team meeting. Basic feedback is "please go ahead and move forward", aka 🚀 :)

Specifically, we encourage you to clarify in the proposal the current state of the JupyterBook community/team/maintainers, so it's clear this is a net benefit to the project not just in terms of code/technology, but also of bandwidth to work as part of the overall Project Jupyter extended team.

I am personally very happy to see this development, so full steam ahead with a full proposal! As a reminder, once ready it will need a vote from both the EC and the SSC, which is required for all new project incorporations.

[tonyfast]

cool to see the progress in jupyter books. i can imagine this proposal maturing jupyter book to effectively, or eventually, become another jupyter front end with thebe in the mix. this will add another front end for the accessibility group to consider, and where there are accessibility concerns there are security concerns. i looked through the proposal for both of those topics, but couldn't find any information. do y'all have any plans accessibility or security plans that could be added to the proposal? i think this would help the @jupyter/accessibility-council understand the impact of this decision.

[rowanc1]

Thanks @fperez and @tonyfast, excited to hear the enthusiasm from the EC -- I will aim to add a section to the document in the next few days on accessibility/security, and we will also document the current team+contributors+community.

[rpwagner]

From a security perspective, the first thing I would like to ask is that you consider creating the new repositories in an existing Jupyter GitHub organization. It’s very challenging to manage security policies, roles, and permissions across multiple GitHub organizations. Whenever possible, please to consolidate repositories under existing organizations.

Thanks

[fperez]

@rpwagner - thanks for that input, though I want to make you aware of something that just happened this morning :) GitHub apparently has a new feature called "Enterprises", and today we upgraded Jupyter's org to be an enterprise.

There's still details being worked out, and I didn't follow all the steps - @jasongrout and @blink1073 were closer to the process in case you have questions.

But without in any way contradicting you, I wanted to mention it because it's precisely supposed to help complex projects that are spread out across many orgs. We planned to reach out shortly to various teams to help with the implications of this, I have already seen various subprojects accepting the invitation to join.

And obviously, you and the rest of the security team will have first say on managing security matters under this new setup!

[rpwagner]

Thanks, @fperez! I just saw the email from @jasongrout about the Enterprise upgrade. That will help immensely with applying uniform policies across the orgs. And it reduces my concern about a new org being created.

[fperez]

Yup, we crossed messages - I hadn't seen Jason's email when I replied above. I'm very happy with this development, I suspect it's going to be super useful for us overall.

[choldgraf]

Does that mean we no longer need to address the "separate organization or not" question in the document? I'll admit that I have a strong preference to have a standalone GitHub organization, in order to allow the Jupyter Book community to define its own norms and identity as a project, have more flexibility over the repositories in the org, and be able to give users permissions without worrying they'll spill over into neighboring project repositories. That said, I also don't want to expose Jupyter to unnecessary security risks and I'm happy to discuss pros / cons with the security team if needed.

[krassowski]

thanks for that input, though I want to make you aware of something that just happened this morning :) GitHub apparently has a new feature called "Enterprises", and today we upgraded Jupyter's org to be an enterprise.

Hooray, that's amazing! I was highlighting that going Enterprise would ease organising security managers roles for a long time (jupyter-governance/ec-team-compass#25 (comment), jupyter-governance/ec-team-compass#12 (comment)), pushing back against the security team here.

[Carreau]

any link about this new "enterprise" feature ? googling for it return the classic "github enterprise" cloud hosting. Or is it the same but on github.com ?

Thanks for doing this.

it also would be great to have 1) a channel for such announce 2) a top level issue that explain this is happening, as it's hard to find this in a subthread

[tonyfast]

seems like there are two threads going on here. might make sense to open another issue about github enterprise or rename this one? otherwise, this pre-proposal won't get the attention it needs.