Bug: tags command gets a lot of time indexing

[RaulCote]

RTFM Checklist

• I have searched exisiting issues / discussions
• I have read the Wiki including the Advanced section
• I have read man fzf / I am well versed in shell fzf

Operating system

Mac

Shell

zsh

Neovim version (nvim --version)

0.12.0

Fzf version (fzf --version)

0.70.0 (Homebrew)

Output of :lua print(os.getenv('FZF_DEFAULT_OPTS'))

mostly color customisations

Is the problem reproducible with mini.sh?

• My issue is reproducible with mini.sh
• My issue IS NOT reproducible with mini.sh
• I have not tested with mini.sh (not relevant, requires LSP, Windows, etc)

Fzf-lua configuration

Describe the bug / steps to reproduce

I noticed that the tags command gets a lot of time to index the tags from my repository. Sometimes I search for a tag and is not matched until later because it's still indexing. A long ago I opened an issue as well and I remember rg_glob = false solved it for grep command. But it seems the tags command doesn't have this option, still I'm not even sure if it would be the same issue. On fzf.vim it takes not even 2 seconds to load all the tags, and fzf-lua it takes more than 10 seconds. The number of tags seems to be 1535991, in both of them shows the same quantity.

[phanen]

maybe try to disable these option,

          fn_preprocess = false,
          fn_postprocess = false,
          fn_transform = false,

[RaulCote]

You are right the culprint seems to be fn_transform

[ibhagwan]

You are right the culprint seems to be fn_transform

I’ll check, transforming a tag entry shouldn’t be any more taxing than a regular grep entry with multiprocess, can you run :FzfLua tags debug=1 and post the screenshot/output of the first debug line (I want to see if it’s running single threaded or multi process)?

[phanen]

transforming a tag entry shouldn’t be any more taxing than a regular grep entry with multiprocess

I think this number is too huge.

The number of tags seems to be 1535991

It's can be slow even with multiprocess. To get this huge entries, I use neovim repo with :FzfLua grep search="" debug=v, take me ~10s to get 1644822 items even with multiprocess. Disable fn_transform take me 1.5s.

I don't use tags but maybe :FzfLua tags_live_grep can be faster.

[ibhagwan]

I don't use tags but maybe :FzfLua tags_live_grep can be faster.

IMHO that’s the way to work with lots of tags, start with live grep, narrow down the search and then switch to fuzzy with ctrl-g.

[RaulCote]

You are right the culprint seems to be fn_transform

I’ll check, transforming a tag entry shouldn’t be any more taxing than a regular grep entry with multiprocess, can you run :FzfLua tags debug=1 and post the screenshot/output of the first debug line (I want to see if it’s running single threaded or multi process)?

[DEBUG] [mt] rg --no-heading --color-always --smart-case -v "^!_TAG_" '/Users/raul/path/to/project/tags'

The problem of live_grep likes is that you need to have a better sense of what you are looking for, so it matches exactly. So as long as I can skip using them, I do, the UX of fuzzy search is much better depending on your needs. Still, it's a possible way out that's for sure.

As context, here is a vid of fzf.vim processing:

fzf-vim.mov

And another of fzf-lua (it doesn't matter if I put max-perf profile, it doesn't really improve)

fzf-lua.mov

And also as well for reference, fzf-lua.grep({ search = "" }) with even more info, it indexes everything really fast:

fzf-lua-rg.mov

No matter the output of this request as I guess is a niche one, thank you both just for responding, fzf-lua is very good overall

[phanen]

It's still possible to speed up the stream parsing with more workers. I see some unfinished parts in code.

Some thing can be optimized:

• reduce string copy with string.buffer (lua5.1 can use a wrapper).
• run fn_transform in a few workers, but each worker have separate lua_state
• rewrite functions like utils.strip_ansi_coloring

[ibhagwan]

And also as well for reference, fzf-lua.grep({ search = "" }) with even more info, it indexes everything really fast:

I haven’t looked into tags in a while but this is a good lead, why is tag processing so much slower? It’s basically “file” with some additional code (namely pattern matching):

fzf-lua/lua/fzf-lua/make_entry.lua

Lines 705 to 732 in a16c537

	M.tag = function(x, opts)
	local name, file, text = x:match("([^\t]+)\t([^\t]+)\t(.*)")
	if not file or not name or not text then return x end
	text = text:match([[(.*);"]]) or text -- remove ctag comments
	-- unescape ctags special chars
	-- '\/' -> '/'
	-- '\\' -> '\'
	for _, s in ipairs({ "/", "\\" }) do
	text = text:gsub([[\]] .. s, s)
	end
	-- different alignment fmt if string contains ansi coloring
	-- from rg/grep output when using `tags_grep_xxx`
	local align = utils.has_ansi_coloring(name) and 47 or 30
	local line, tag = text:match("(%d-);?(/.*/)")
	if not tag then
	-- lines with a tag located solely by line number contain nothing but the
	-- number at this point (e.g. using "ctags -R --excmd=number")
	line = text:match("%d+")
	end
	line = line and #line > 0 and tonumber(line)
	return string.format("%-" .. tostring(align) .. "s%s%s%s: %s",
	name,
	utils.nbsp,
	M.file(file, opts),
	not line and "" or ":" .. utils.ansi_codes.green(tostring(line)),
	utils.ansi_codes.blue(tag)
	), line
	end

Lua’s pattern matching is notoriously slow, if this code is indeed where we spend most time we can easily speed that significantly.

@phanen, regardless of worker thread processing we should make this more efficient.

[RaulCote]

As I mentioned it in the first comment, it's exactly fzf-lua.grep({ search = "" }) along with rg_glob = false that I have in the setup config for grep. without rg_glob = false is very slow as well. Just in case it was forgotten.

[ibhagwan]

As I mentioned it in the first comment, it's exactly fzf-lua.grep({ search = "" }) along with rg_glob = false that I have in the setup config for grep. without rg_glob = false is very slow as well. Just in case it was forgotten.

Are you sure about the difference rg_glob=false makes here? it only applies once when the command is constructed to parse and extrat the globs (if there are any) after the -- separator, it should have zero effect on the processing, if it is that would be one interesting bug :-)

[ibhagwan]

Just tried on the Linux kernel source, make entry code optimization is what we need to focus on, there’s a lot to be improved and we can at least match fzf.vim performance.

[ibhagwan]

I was able to make this much faster with this diff (obviously dropping a lot of features but matching fzf):

diff --git a/lua/fzf-lua/make_entry.lua b/lua/fzf-lua/make_entry.lua
index 1cecd15a..473dd2eb 100644
--- a/lua/fzf-lua/make_entry.lua
+++ b/lua/fzf-lua/make_entry.lua
@@ -703,6 +703,9 @@ M.file = function(x, opts)
 end

 M.tag = function(x, opts)
+  if true then
+    return (x:gsub("([^\t]+)\t([^\t]+)\t(.*)", "%1\t\t" .. utils.nbsp .. "%2: %3"))
+  end

However it's still not as fast as fzf.vim which uses the below simple perl script which I knew about beforehand but wanted to avoid the perl depdency:

#!/usr/bin/env perl

use strict;

my $prefix = shift @ARGV;

foreach my $file (@ARGV) {
  my $lines;
  if ($prefix eq "") {
    open $lines, $file;
  } else {
    # https://perldoc.perl.org/perlopentut#Expressing-the-command-as-a-list
    open $lines, '-|', 'readtags', '-t', $file, '-e', '-p', '-', $prefix;
  }
  while (<$lines>) {
    unless (/^\!/) {
      s/^[^\t]*/sprintf("%-24s", $&)/e;
      s/$/\t$file/;
      print;
    }
  }
  close $lines;
}

[RaulCote]

As I mentioned it in the first comment, it's exactly fzf-lua.grep({ search = "" }) along with rg_glob = false that I have in the setup config for grep. without rg_glob = false is very slow as well. Just in case it was forgotten.

Are you sure about the difference rg_glob=false makes here? it only applies once when the command is constructed to parse and extrat the globs (if there are any) after the -- separator, it should have zero effect on the processing, if it is that would be one interesting bug :-)

Totally sure, here is the proof, just recorded now:

Without rg_glob set up as false:

without.rg.glob.false.mov

With rg_glob set up as false:

with.rg.glob.false.mov

And for reference my fzf-lua config:

local fzfActions = require("fzf-lua").actions

require("fzf-lua").setup({
  winopts = {
    treesitter = {
      enabled = false,
    },
    backdrop = 100,
    width = 0.95,
    height = 0.8,
    preview = { scrollbar = false },
    icons_enabled = false,
  },
  fzf_opts = { ["--layout"] = "default" },
  fzf_colors = {
    ["border"] = { "fg", { "FloatBorder", "Normal", "Function" } },
    ["gutter"] = { "bg", { "CursorLine", "NormalFloat" } },
    ["preview-bg"] = { "bg", { "Normal" } },
    ["header-bg"] = { "bg", { "Normal" } },
    ["list-bg"] = { "bg", { "Normal" } },
    ["bg+"] = { "bg", { "CursorLine", "NormalFloat" } },
    ["bg"] = { "bg", { "Normal" } },
    ["fg+"] = { "fg", { "CursorLineNr", "Search" } },
    ["hl+"] = { "fg", { "CursorLineNr", "Search" } },
    ["fg"] = { "fg", { "Pmenu", "CmpItemAbbr", "Comment" } },
    ["hl"] = { "fg", { "PmenuSelMatch", "CmpItemAbbrMatch", "CursorLineNr", "Search" } },
    ["pointer"] = { "fg", { "Constant", "Operator" } },
    ["marker"] = { "fg", { "Constant", "Operator" } },
    ["info"] = { "fg", { "Constant", "Operator" } },
    ["prompt"] = { "fg", { "Constant", "Operator" } },
    ["preview-fg"] = { "fg", { "Type", "Search" } },
  },
  previewers = {
    git_diff = { pager = "delta --width=$FZF_PREVIEW_COLUMNS" },
  },
  defaults = { git_icons = false, file_icons = false, color_icons = false },
  grep = {
    rg_opts = "--column --line-number --hidden --no-heading --color=always --smart-case --max-columns=4096 -e",
    rg_glob = false,
  },
  oldfiles = {
    include_current_session = true,
    sort_mru = true,
  },
  buffers = {
    sort_lastused = true,
    sort_mru = true,
  },
  git = {
    status = { preview_pager = "delta --width=$FZF_PREVIEW_COLUMNS" },
    commits = { preview_pager = "delta --width=$FZF_PREVIEW_COLUMNS" },
    bcommits = {
      cmd = "git log --color --follow --pretty=format:'%C(yellow)%h%Creset %Cgreen(%><(12)%cr%><|(12))%Creset %s %C(blue)<%an>%Creset' <file>",
      preview_pager = "delta --width=$FZF_PREVIEW_COLUMNS",
    },
  },
  marks = {
    marks = "[A-Z]",
  },
  actions = {
    files = {
      ["ctrl-y"] = fzfActions.file_edit,
      ["enter"] = fzfActions.file_edit_or_qf,
      ["ctrl-s"] = fzfActions.file_split,
      ["ctrl-v"] = fzfActions.file_vsplit,
      ["ctrl-t"] = fzfActions.file_tabedit,
      ["alt-q"] = fzfActions.file_sel_to_qf,
      ["alt-Q"] = fzfActions.file_sel_to_ll,
    },
  },
})

require("fzf-lua").setup_fzfvim_cmds()

require("fzf-lua").register_ui_select(function(_, items)
  local min_h, max_h = 0.15, 0.70
  local h = (#items + 4) / vim.o.lines
  if h < min_h then
    h = min_h
  elseif h > max_h then
    h = max_h
  end
  return { winopts = { height = h, width = 0.60, row = 0.40 } }
end)