Speed up gettext: linear template merge, single POT read, plural-info dedup, and runtime interpolation fast paths

[oliver-kriska]

Five small, output-preserving optimizations across the extraction/compile path and the runtime interpolation path, stacked on top of the --from-attributes work (extract-from-attributes). Each change is output-equivalent: the extraction changes (1-3) were verified to produce byte-identical PO/POT output against a production app's catalog (~1,875 files, 11 domains × 16 locales, ~4,700 msgids), and the runtime changes (4-5) are covered by the existing interpolation test suite.

This PR targets the fork's extract-from-attributes branch so the diff shows only the new commits. Opened to review the diff and dogfood it in CI/dev before considering an upstream PR.

Commits

1. Make POT template merge linear instead of O(n*m) - merge_template/3 matched messages with Expo.Messages.find/2 (a linear scan) once per existing message and once per new message, so it was O(N*M) per domain on every extract (and again in prune_unmerged/2). It now indexes both sides by Expo.Message.key/1 and looks matches up in a map / MapSet. Message.key/1 is exactly what Message.same?/2 (hence Messages.find/2) compares on, and Map.put_new/3 preserves Enum.find/2's first-match-wins, so output is unchanged.

2. Avoid reading each existing POT file twice - read_contents_and_parse/1 did File.read!/1 and then PO.parse_file!/2, which reads the same file again. PO.parse_file!(path, file: path) is defined as File.read + parse_string(contents, file: path), so feeding the bytes we already hold to PO.parse_string!/2 is identical; File.read!/1 still guards a missing file.

3. Compute plural info once per PO file - compile_po_file/5 called Plural.plural_info/3 twice (in nplurals/3 and compile_plural_forms/4). That function runs Code.ensure_compiled!/1 and parses the Plural-Forms header. It is computed once now; the generated code is identical.

4. Stop recompiling the interpolation patterns on every call - to_interpolatable/1 rebuilt :binary.compile_pattern/1 for "%{" and "}" on every invocation. This runs on every runtime interpolation, including the common case where the current locale has no translation for a msgid and falls back to interpolating the msgid itself. It now matches on the literal "%{" / "}" patterns directly in :binary.split/2; the split result is identical, and these two-byte patterns gain nothing from precompilation (and a compiled pattern can't be hoisted to a module attribute - it's a reference).

5. Skip String.Chars dispatch for binary bindings - the runtime interpolation path always ran binding values through to_string/1, dispatching the String.Chars protocol even when the value is already a binary (the common case). An is_binary/1 clause now uses the value directly; to_string/1 on a binary returns it unchanged, so the result is identical.

Benchmark (commit 1)

The merge runs in both the stock and --from-attributes paths, so this win applies regardless of the flag. This script isolates the matching cost (the merge step is reduced to identity so both implementations do the same number of calls) and asserts identical result counts. Run it with mix run, optionally setting BENCH_POT to a real .pot/.po path to add a real-catalog case:

alias Expo.{Message, Messages, PO}

defmodule OldMerge do
  def run(existing, new) do
    old_and_merged =
      Enum.flat_map(existing.messages, fn message ->
        cond do
          same = Messages.find(new, message) -> [keep(message, same)]
          true -> [message]
        end
      end)

    old_and_merged ++ Enum.reject(new.messages, &Messages.find(existing, &1))
  end

  defp keep(old, _new), do: old
end

defmodule NewMerge do
  def run(existing, new) do
    new_by_key =
      Enum.reduce(new.messages, %{}, fn message, acc ->
        Map.put_new(acc, Message.key(message), message)
      end)

    existing_keys = MapSet.new(existing.messages, &Message.key/1)

    old_and_merged =
      Enum.flat_map(existing.messages, fn message ->
        cond do
          same = Map.get(new_by_key, Message.key(message)) -> [keep(message, same)]
          true -> [message]
        end
      end)

    old_and_merged ++
      Enum.reject(new.messages, &MapSet.member?(existing_keys, Message.key(&1)))
  end

  defp keep(old, _new), do: old
end

time = fn iters, fun ->
  fun.()
  {us, _} = :timer.tc(fn -> Enum.each(1..iters, fn _ -> fun.() end) end)
  us / iters / 1000.0
end

synthetic = fn n ->
  %Messages{messages: for(i <- 1..n, do: %Message.Singular{msgid: ["msgid number #{i}"], msgstr: [""]})}
end

bench = fn label, existing, new, iters ->
  old_ms = time.(iters, fn -> OldMerge.run(existing, new) end)
  new_ms = time.(iters, fn -> NewMerge.run(existing, new) end)
  equal? = length(OldMerge.run(existing, new)) == length(NewMerge.run(existing, new))
  IO.puts("#{String.pad_trailing(label, 22)} old=#{Float.round(old_ms, 2)} ms  " <>
            "new=#{Float.round(new_ms, 3)} ms  equal_count=#{equal?}")
end

case System.get_env("BENCH_POT") do
  nil -> :ok
  path ->
    pot = PO.parse_file!(Path.expand(path))
    bench.("real (all match)", pot, pot, 20)
    fresh = %Messages{messages: for(m <- pot.messages, do: %{m | msgid: ["NEW " | m.msgid]})}
    bench.("real (all new)", pot, fresh, 20)
end

for n <- [500, 1000, 2000, 4000, 8000], do: bench.("n=#{n}", synthetic.(n), synthetic.(n), max(3, div(40_000, n)))

Results on a real default.pot (~4,200 msgids), 3 fresh-VM runs, warmup + 20-iteration average, equal_count true throughout:

input	old	new
real catalog, all match (steady-state re-extract)	~1.0 s	~3 ms
real catalog, all new (worst case)	~3.7 s	~3 ms
synthetic n=500 / 1k / 2k / 4k / 8k	14 / 51 / 230 / 800 / 3235 ms	0.23 / 0.46 / 1.0 / 2.1 / 4.3 ms

The old time quadruples per doubling (O(n²)); the new one only doubles (O(n)). End-to-end, the stock (no-flag) extract is dominated by the force-recompile, so the merge saving is small there in absolute terms but grows with catalog size; on the recompile-free --from-attributes path it is proportionally visible.

Validation

• Full mix test is green (the only failures are the pre-existing order-dependent gettext.extract_test.exs cases under Elixir 1.20+, present on the base branch as well).
• Against a large production app: it recompiles cleanly through the changed codegen, and mix gettext.extract + mix gettext.merge leave the committed priv/gettext tree byte-identical and idempotent.

[stage-review]

Ready to review this PR? Stage has broken it down into 5 individual chapters for you:

	Title
1	Optimize POT template merge to linear time
2	Avoid redundant file reads during extraction
3	Deduplicate plural info computation during compilation
4	Optimize runtime interpolation pattern matching
5	Fast-path binary values in interpolation

_{Chapters generated by Stage for commit 574dcba on Jun 16, 2026 3:47pm UTC.}