ShellAC protocol specification

[roberth]

Hi,
First of all, thank you for writing ShellAC, which seems to be exactly what the command line shell ecosystem needs.
I'm looking into supporting it in Haskell's de-facto standard library for option parsing, but I have some questions that aren't answered by the ShellAC pages on the wiki (https://github.com/oilshell/oil/wiki/Shellac-Protocol-Proposal-V2 and its links).
Judging from those pages, I'd have to conclude that it doesn't satisfy my needs, but I hope that that's a premature conclusion.
So I'd like to know, are those pages the only documentation of the ShellAC protocol?
Does the ShellAC protocol have forward compatibility when it comes to extensions?
If not, do you think it is feasible to change the protocol? (How widely has it been implemented?)

For completeness, here's my initial evaluation. I'd be happy to update it. pcapriotti/optparse-applicative#414 (comment)

[HuwCampbell]

A little background on what we do in optparse-applicative might help here.

All optparse-applicative programs contain context sensitive completions automatically. This works by equiping them with a few hidden extra options which enable completion mode.

Using ghcup (which is an installer like rustup for haskell) as an example, after installing the common zsh completion script, we get this without any extra effort by the builder of the ghcup tool.

On the command line, we can see how this works by running:

# Simple completion of the first argument to ghcup, having typed `ghcup li`
> ghcup --bash-completion-index 1 --bash-completion-word ghcup --bash-completion-word li --
list

# Rich completion of the first argument to ghcup, having typed `ghcup li`
> ghcup --bash-completion-index 1 --bash-completion-word ghcup --bash-completion-word li --bash-completion-enriched
list	Show available GHCs and other tools

# Rich completion of the arguments to list, having typed `ghcup list `
> ghcup --bash-completion-index 2 --bash-completion-word ghcup --bash-completion-word list --bash-completion-enriched
--tool	Tool to list versions for. Default is...
-t	Tool to list versions for. Default is...
--show-criteria	Show only installed or set tool versions
-c	Show only installed or set tool versions
--raw-format	More machine-parsable format
-r	More machine-parsable format
-h	Show this help text
--help	Show this help text

How this works is relatively orthogonal, but it actually runs the program and its command line parser proper, but interpreting the parser with a different monad. The benefits are that the options presented are always the ones available (that is, it's always correct) and that we can write custom actions like looking up which versions of GHC are already installed in haskell as well, sharing code with the application itself.

Our protocol is essentially:

• call the application you want completions for, but instead of "bare" options and arguments:
• use --bash-completion-index to say what word you're editing
• use --bash-completion-word for each word the user has written so far (but clip these to the cursor position).
• if you want to include descriptions, use --bash-completion-enriched, and (optionally --bash-completion-option-desc-length and --bash-completion-command-desc-length).

We will then send back:

• a completion per line
• if it's enriched, we include a tab character then a description if we have one.

We then have bash, zsh, and fish completion scripts which are able to communicate in this protocol. These are autogenerated with --bash-completion-scipt, or their zsh and fish equivalents. All programs get the same scripts (changing only the function names and the program to invoke with the magic arguments).

Now aside from the fact that the names of these options include the word bash; this works very, very well. @roberth is however interested in adding to the protocol to iron out a few niggles; and I'm keen to see more languages and libraries adopt a system like this, and any improvements would be great.

• the first issue is that we don't tell the shells if something is actually a file or directory after we complete it. So if there happens to be a directory with the same name as a command for instance, the command will get a trailing slash via the shell even though it shouldn't.
• the next is how we should handle trailing spaces. Currently, they're always added if the shell completes the completion word, which is ok, but some extra flexibility can help in places where the completion is more complex.
• one other thing we're not addressing right now is that we usually shell out to bash to complete files and directories by invoking compgen -A file -- thing_to_complete, but it doesn't use the user's bash preferences, so we sometimes show hidden files where we shouldn't. ShellAC's delegation back to the shell for file completions is probably a bit better here.

So this is an invitation for collaboration. I'm comfortable with optparse-applicative to continue to go it alone; but it would be nice for a unified protocol to emerge.

[andychu]

Hi, thanks for the message and I'm glad to see work on this.

So Shellac was basically a discussion, not something ever implemented. Most of the discussions are here:

https://oilshell.zulipchat.com/#narrow/stream/146045-shell-autocompletion

And actually the most significant work was done separately under Redox OS (there's a Zulip thread, and links at the end of https://github.com/oilshell/oil/wiki/Shellac-Protocol-Proposal-V2)

So Shellac was basically "cut" from the Oil project sometime in 2020. I think there was a tension between designing a "good" protocol (for correctness, generality), and designing one that could actually be implemented in bash, fish, and zsh! So I'm curious to see how you've done that.

---

Oil emulates bash, which gives it a lot of completions "for free". On the other hand, supporting a brand new protocol that wasn't implemented by shells wouldn't be all that useful.

I think there has to be an upgrade path from bash or zsh completions to general shellac completions -- basically some wrappers. A binary with a flag parser can act as a shellac plugin / "server", but a shell running a program should also be a valid plugin / server.

So it's great that you have actually implemented something in bash zsh and fish! That's the kind of thing I was looking for. (I know all about bash completion, but little about zsh or fish completion.)

---

Comments:

• One thing that would be interesting is to run optparse-applicative bash completion scripts under Oil. Oil runs git-completion.bash and many other bash completion scripts.
• On trailing slashes, I remember that the separate bash-completion project also has this issue. It will add trailing slashes to subcommands, etc.
• On trailing spaces -- I think I encountered this issue too. I would like Shellac to be "correct" in the sense that a file with a trailing space should be quoted, and the plugin can control whether the word is "done" or not (trailing space or not). I think this was difficult to do given the constraints of existing shells.

More:

• completion per line would imply that the filenames and descriptions can't contain newlines ? I think that might be OK but I was trying to be more general.
• ditto for the tab separator

Oil uses QSN but I think it's too hard to implement in other shells: http://www.oilshell.org/release/latest/doc/qsn.html

i.e. you could have a completion format like:

'completion \n with \n newlines' 'description \n for \n it'

[andychu]

@AdminXVII may be interested in this, having done prior work on Ion / Redox

And refreshing my memory from a couple years ago

• https://github.com/oilshell/oil/wiki/Projects-Already-Doing-Something-Like-Shellac
• npm, clang, and git have some completion logic built into their binaries, but there's no protocol. They just parse that output with bash or zsh and then use the bash/zsh completion APIs (https://github.com/oilshell/oil/wiki/Shell-Autocompletion)