[Bug]: highlighting broken after lsp format

[raineszm]

Neovim version

v0.10.0-dev-510+g3c4890d1e

Language affected

clojure

Query

No response

Strategy

No response

Description

In a clojure file, using the clojure-lsp formatting, the rainbow delimiters become out of sync until the file is edited again.

This can be seen in the below asciicast (which I hopefully formatted correctly).

I can reproduce the issue using the following clojure file, deps.edn, and minimal config by opening the file with nvim -u init.lua minimal.clj and running :lua vim.lsp.buf.format()

init.lua

local lazypath = "/tmp/rainbow/lazy/lazy.nvim"
if not vim.loop.fs_stat(lazypath) then
	vim.fn.system({
		"git",
		"clone",
		"--filter=blob:none",
		"https://github.com/folke/lazy.nvim.git",
		"--branch=stable", -- latest stable release
		lazypath,
	})
end
vim.opt.rtp:prepend(lazypath)
vim.cmd.colorscheme("retrobox")

require("lazy").setup({
	{
		"nvim-treesitter/nvim-treesitter",
		build = ":TSUpdate",
		event = { "BufReadPost", "BufNewFile" },
		opts = {
			ensure_installed = {
				"clojure",
			},
			highlight = {
				enable = true,
			},
			rainbow = {
				enable = true,
			},
		},
		config = function(_, opts)
			if type(opts.ensure_installed) == "table" then
				---@type table<string, boolean>
				local added = {}
				opts.ensure_installed = vim.tbl_filter(function(lang)
					if added[lang] then
						return false
					end
					added[lang] = true
					return true
				end, opts.ensure_installed)
			end
			require("nvim-treesitter.configs").setup(opts)
		end,
	},
	"HiPhish/nvim-ts-rainbow2",
	{
		"neovim/nvim-lspconfig",
		event = { "BufReadPre", "BufNewFile" },
		dependencies = {
			"mason.nvim",
			"williamboman/mason-lspconfig.nvim",
		},
		autoformat = true,
		config = function(_, opts)
			require("lspconfig").clojure_lsp.setup(opts)
		end,
	},
	{
		"williamboman/mason.nvim",
		cmd = "Mason",
		config = true,
	},
	{
		"williamboman/mason-lspconfig.nvim",
		opts = {
			ensure_installed = { "clojure_lsp" },
		},
	},
})

minimal.clj

(defn a-func [x y]
  (let [a 1]
  (let [b 2]
  (+ a b x y))))

(a-func 2 3)

deps.edn

{
 :paths ["."]
 }

It's possible this may be the same bug as this in which case the linked fixes might help?

[raineszm]

Here's a minimal repro zipped up. minimal.tgz

[HiPhish]

I can replicate the issue in Fennel with fnlfmt as the formatter. It looks like the extmarks are are not being moved because the old highlighting stays in place, painting the wrong characters. You should be able to make it go away by executing :edit on the buffer. I will have to investigate what is going on.

[raineszm]

Cool. Thanks for the workaround. I'll give it a shot. Let me know if there's anything else I can do to help narrow it down.

Also fyi the issue also occurs with the local strategy sometimes leading to some pretty dramatic highlighting issues.

[HiPhish]

It looks like a Neovim issue with extmarks. Here is a test which does not involve Tree-sitter, rainbows or LSP at all. Disable the rainbow plugin, then take this Snippet of Fennel code:

(fn foo []
  (if (and (= :a :a)
        (not= :a :b))
      (print "It just works!")))

Now let's create a new namespace and add highlighting to the second to last closing parenthesis on the third line.

let g:nsid = nvim_create_namespace('derp')
call nvim_buf_add_highlight(0, nsid, 'WarningMsg', 2, 19, 20)

This will highlight the parenthesis (yellow in my case). Now shell out to fnlfmt to format the entire buffer. This assumes we have the compiled fnlfmt program in the same directory.

!./fnlfmt --fix %

This is the result:

(fn foo []
  (if (and (= :a :a) (not= :a :b))
      (print "It just works!")))

Now the s in just should be highlighted yellow because it is where the parenthesis used to be.

I don't know if this is even a bug or if it works as intended. Extmarks move with their text, but this is not really a case of text moving. Should the extmark have been deleted?

It raises questions for this plugin too. Should I delete extmarks in the changed region? What if one delimiter of the pair is inside the formatted region and one is outside? I don't know, I will have to think about this.

[HiPhish]

Quick heads-up: it looks like the parser is parsing the old buffer from before it was formatted. Or at least it is applying the highlighting to the old positions. This then puts extmarks in weird places, causing the highlighting to be wrong. I have not yet been able to figure out why this is happening.

There is another issue: callbacks don't get cleared for a parser and multiple callbacks can start building up if the buffer is edited multiple times. Maybe this is part of the problem? I'll try and see if I can fix this problem upstream in Neovim.

[HiPhish]

I have mixed news, both good and bad.

Let's start with the bad: I don't think this issue can be fixed in this plugin at all. This plugin is implemented as a module for nvim-treesitter and it relies on nvim-treesitter to register (attach) and unregister (detach) autocommands for buffers. Unfortunately the nvim-treesitter way seems to be what is causing this problem and a number of other problems as well.

The mixed news is that nvim-treesitter has deprecated the module system and will remove it in the future (source). This means that sooner or later I will have to fork this plugin yet again and make a new standalone plugin anyway.

As for the good news, I have had my first success. The new version does not suffer from lingering extmarks, but it still does not insert new ones. There is still some fine tuning to be done to get the autocommands correct and make the configuration work without nvim-treesitter. I will be able to reuse most of my existing code, so the amount of effort needed will be reasonable.

The configuration will have to be done differently, that's why I want to make it a new standalone plugin and leave this one here for legacy. The configuration format will be the same, but it will have to be specified somewhere else now.

[raineszm]

Wow. Thanks for the updates. If you need beta testers for the new version I'm happy to give it a spin.

I appreciate all your work. It would be a sad world without a good rainbow parens plugin.

[HiPhish]

The new rainbow plugin is here: rainbow-delimiters.nvim (GitHub mirror). Please file an issue over there if the problem still persists.