Add recompile-free extraction via persisted module attributes

[oliver-kriska]

Claude Code generated text:

Motivation

mix gettext.extract discovers messages by force-recompiling the whole project, because extraction happens during gettext macro expansion and the results only live in the extractor agent for that single compilation. On large
projects this dominates extraction time, and because --check-up-to-date runs in CI on every change, the cost is paid constantly.

This implements the recompile-free design @maennchen proposed in #373 (#373 (comment)). That issue was closed once the immediate duplicate-reference / module redefinition bug was fixed in v1.0.2, but the extraction redesign itself was never implemented.

Approach

Two halves:

1. Write path — during normal compilation, when Mix.env() is in the new:extraction_environments config (default [:dev]), each gettext macro persists its message into an accumulated, persist: true module attribute
   @__gettext_messages__ on the calling module. Backends additionally persist a @__gettext_backend_module__ marker.
2. Read path — mix gettext.extract --from-attributes runs a normal incremental compile, then reads those attributes back from the compiled BEAM files with :beam_lib.chunks(beam, [:attributes]) (without loading the modules), feeds them into the existing Gettext.ExtractorAgent, and reuses the unchanged Gettext.Extractor.pot_files/2 merge logic.

The force-recompile path is untouched and remains the default.

Why it stays correct

• Staleness needs no bookkeeping. The BEAM files are the cache: a changed source file is recompiled by the incremental compile and gets fresh attributes; a deleted file's BEAM is pruned by the compiler. No manifest, no mtime/hash logic. --check-up-to-date composes with --from-attributes and gets the same speedup.
• Releases are unaffected. Environments outside :extraction_environments (notably :prod) persist nothing — no attributes, no marker. Unit-tested (a fixture compiled in :test carries no gettext attributes on any BEAM).
• No silent data loss. If the scan finds zero messages and zero backends (e.g. the first run after upgrading, with stale attribute-less BEAMs), the task raises with instructions to mix compile --force rather than silently pruning every autogenerated message or silently falling back to a surprise full recompile.
• Equivalence. Old-path and new-path POT output is byte-identical; covered by tests and validated against a large production app.

Benchmarks

Large production Phoenix app (1,875 files, 11 domains × 16 locales, ~4,700 msgids; Elixir 1.20 / OTP 29):

	force-recompile (current)	--from-attributes
gettext.extract --merge	49.7s	7.0s
gettext.extract --check-up-to-date (CI)	~60s	4.6s

In that app's CI the extraction step dropped from ~66–94s to ~4–11s and has run on every merge for several days with identical POT output. Dev BEAM size grew ~2.7% (≈ +2.3 MiB for ~4,700 msgids); prod is unaffected.

Scope / limitations

• Opt-in. Default behavior is unchanged; the fast path is active only when you pass --from-attributes and set :extraction_environments.
• Modern Gettext.Macros backends only. The deprecated use Gettext, otp_app: ... backends generate their own *_noop macros and don't persist attributes; the zero-found error catches a legacy-only project that opts in.
• extraction_apps (cross-app dependency extraction) deferred. gettext.extract is @recursive, so each umbrella child already scans its own compile path (covered by a fixture test); extracting messages that live
  in dependencies is new functionality for a follow-up.

Design notes for review

• Persisted attributes + :beam_lib, rather than the @before_compile __translations__/0 function from the original sketch: same data, no function injected into user modules, and reading attributes doesn't load the modules
  (so scanning 1,000+ BEAMs stays cheap and side-effect-free). Happy to switch to the generated-function approach if you prefer it.
• The new path feeds the existing ExtractorAgent rather than refactoring pot_files/2, so duplicate-message merging, plural-mismatch warnings and comment de-duplication are reused verbatim — equivalence is structural, not re-implemented.
• Happy to split this into smaller commits (attribute writing / BEAM reading / task wiring) if that is easier to review.

Tests

New test/mix/tasks/gettext.extract_from_attributes_test.exs covers: old-path / new-path byte-equivalence (fresh + idempotent), macro-generated messages (gettext calls injected into modules that don't textually contain them),
_with_backend variants, reference merging across modules sharing a msgid, --from-attributes --merge, custom :priv, --check-up-to-date staleness via the incremental compile, umbrella extraction, and prod-clean BEAMs.

[josevalim]

Do we even need the module attribute? Maybe instead we could simply start the Agent on demand during compilation with the existing mode? Then later we can swap it to use ETS or whatever.

[oliver-kriska]

but how will we persist cross compilation? So it would mean we will have to still do force compile which can be pretty expensive. So basically your suggested solution is how it works now if I'm not wrong, with that add_message function.

[josevalim]

Oh, sorry, I have completely misread it. I see.

The reason I don't like the module attributes is that, IIRC, they are loaded once the code is loaded, and that's also a runtime cost. Another option would be to track additional files per module? And then we prune those files when assembing the .po if they don't have an equivalent .beam.

[maennchen]

@josevalim What do you think about just disabling those attributes for prod? For dev, the runtime cost should be negligible.

I like the approach with module attributes because you have the same incremental compilation as normal and you can extract at any point by enumerating the modules / their attributes. You could for example extract while there's a running phoenix dev server.

[oliver-kriska]

it should be already there, see those lines

defp extraction_environment? do
    cond do
      not Code.ensure_loaded?(Mix) ->
        false

      is_nil(Mix.Project.get()) ->
        false

      true ->
        gettext_config = Mix.Project.config()[:gettext] || []

        extraction_environments =
          gettext_config[:extraction_environments] ||
            Application.get_env(:gettext, :extraction_environments) ||
            [:dev]

        Mix.env() in extraction_environments
    end
  end

https://github.com/elixir-gettext/gettext/pull/437/changes#diff-b815910ebc5507c67ea8edc182bd8917455114497fbbf5427cc26255b3a30d6bR135-R140

[josevalim]

I don't understand why we need all of this. We can compile and assume that already puts everything in place? Or it may be better if we use separate tasks maybe? mix gettext.generate or something?

[oliver-kriska]

make sense, trying something with mix gettext.extract_from_attributes it's too long but .generate is "too wide meaning". Open for other ideas

[maennchen]

Do we want to maintain two extraction features in parallel? I trust that this approach will be stable.

[josevalim]

I’d maintain both for now. This package is likely a decade old, this is an essential part, id not necessarily assume we won’t have blockers and regressions. We can merge them together later.

[josevalim]

Ok, agreed then. I would probably make it like this though:

# config/dev.exs
config :my_app, MyGettext.Backend, automatic_extraction: true

So we are not accessing Mix from deep within Gettext.

It may also be worth treating extraction with and without this feature as completely different things? They can share some steps (such as the merging and fusing) but it may be clearer to treat them as separate code paths execution wise.

[oliver-kriska]

fyi I updated code but I want to properly test it with our big translations, but feedback welcome. Also with this oliver-kriska#2