Automated docs translations with LLMs

[IgnacioHeredia]

Hi all,

this issue stems from two facts:

• as noted somewhere else ([Proposal] Keeping translations up-to-date #70, [COLLAB // FRENCH] Keeping the Translation updated #69), docs versions in other languages tend to fall quite behind the English version.
• LLMs are getting very good at translating

So it is only natural to try to leverage them for automating docs translation.

Possible workflow:

• Keep the English version of the docs a ground truth.
• Use GitHub Actions to automate everything.
• When commit is detected in the English version, GA calls an LLM (e.g. Gemini Free Tier) to translate the commited file into other languages.
• GA commits directly or opens a PR with the changes for manual approval.

Translation does not need to be done from scratch. One could prompt the LLM to reuse the existing non-English translation as a baseline for the new translation. This could allow for automatic docs translation, but still curated by humans. For example, if a docs contributor spots an error in the LLM translation, they could fix it in the markdown of that language and the LLM would take that into account in the next translation.

Does that make sense?

Caveats:

• I'm not sure how cross-file linking would work. Cross linking to another file is not a problem because as far as I can tell all languages share the same file names. But linking to another file header might break from time to time because header names might change with translations. But in any case, that is an error that is easily detected during docs compilation and can easily be fixed manually .
• If GA fails and some file remains untranslated, I'm unsure what would be the best way to keep track of it so that it is also translated next time the GA runs.

[nathanlesage]

While I agree with the sentiment and believe that a better maintained community section in the documentation, I am wondering if adding LLM based translations would really provide tangible benefits for our users.

My points of concern would be:

• All browsers support some form of translation built in, so I don't know if we would not simply add a redundant new step to the process which users can already use.
• How would we support LLM based translations in the current approach where users can also deliver some improvements? I don't yet see a meaningful way of auto-translating only changed parts, given that the main benefit of having actual, community-provided separate documentation also gives them the ability to restructure the documentation where needed to maintain understandability for users in the target language.
• We cannot really deploy local LLMs for that, and I am hesitant to throw money at corpos for a feature that, as I said, essentially comes built in for most users. And we WILL exceed free tier token limits very quickly, since the documentation has grown considerably since the humble beginnings.

What are your suggestions?

EDIT: Also I just realized: how would quality assurance work? While LLMs can get 95% right, there will always be a few instances which are wrong. I would like to keep potential bug requests due to wrong documentation translations to a minimum.

[krisfremen]

@nathanlesage If we get the docs under weblate, it would make human user approvals and review of translations(whether user submitted or LLM) much easier to manage

[IgnacioHeredia]

Hi @nathanlesage ,

• All browsers support some form of translation built in, so I don't know if we would not simply add a redundant new step to the process which users can already use.

Completely agree with this. My point is that since Zettlr went down the road of having dedicated docs for each language, I rather have those docs as updated as possible to avoid misleading people with old documentation. But Zettlr could definitely made the decision to remove the non-English docs and rely on built-in browser translation. This would be the easiest option IMO, ensuring consistent documentation in all languages at zero maintenance cost.

Possible downsides: Maybe you might loose some SEO positioning for not having dedicated docs in that language? Anyway, this is mitigated somehow because nowadays many people rely on LLMs to ask questions on the docs, and I believe many times English docs are used to provide non-English answers. So SEO is not that important anyway.

• How would we support LLM based translations in the current approach where users can also deliver some improvements? I don't yet see a meaningful way of auto-translating only changed parts, given that the main benefit of having actual, community-provided separate documentation also gives them the ability to restructure the documentation where needed to maintain understandability for users in the target language.

Inside a specific markdown file, I don't see either a simple way to only translate changed parts. So each time you would need to regenerate the whole translated markdown. But that does not mean user curation will be lost each time because you can force the LLM to respect human curation.

You could have a basic system prompt like:

You will be provided an English documentation that you need to translate. You will be also be provided a reference translation in the target language. When the reference translation already contains a piece of information from the English  translation, please leave it as it is, even if you don't like the wording. Only translate the English information that was not found in the reference translation. 

(more details can be added to the prompt for better context ).
and user prompt like:

English docs
------------
<English latest markdown>


Translated reference docs
-------------------------
<previous non-English translated markdown, possibly curated by humans>

Hopefully this would respect human curation (not sure what would happen if the translator heavily modified the markdown structure though).

Variations of this could be providing the LLM only the English git diff, instead of the whole markdown, to force it to focus only on the translated part. Although the whole non-English file would still need be regenerated, as in the previous case.

• We cannot really deploy local LLMs for that, and I am hesitant to throw money at corpos for a feature that, as I said, essentially comes built in for most users. And we WILL exceed free tier token limits very quickly, since the documentation has grown considerably since the humble beginnings.

Not fan either of throwing money to them. But you would be surprised how far you can go with the free tier. For example, using Gemma 3 models (small compared to frontier models but more than enough for translation) you can have up to 15K daily requests (not tokens!) for free. So unless you plan to translate more than 15K docs per day, it's more than enough.

EDIT: Also I just realized: how would quality assurance work? While LLMs can get 95% right, there will always be a few instances which are wrong. I would like to keep potential bug requests due to wrong documentation translations to a minimum.

No quality assurance that comes right now to my mind, I'm afraid. But I assume that people that are technical enough to open a GitHub issue (in English), are already consuming the English docs, so I don't foresee a flood of new bug requests. And in any case, the number of bug reports would still be fewer with LLM-generated not-perfect docs compared with human-generated outdated docs (like currently).

If I were to make the call myself, leveraging pros and cons, I would remove non-English docs and rely on built-in translation. Low maintenance, little downsides. But maybe it makes sense to do a small pooling around people that use non-English docs.