Describe the problem
I would like to integrate self-hosted LLMs which are proxied by litellm which provides a API in OpenAI format.
The API endpoint URL is not configurable in the OpenAI machinery. Currently this can be worked around by setting the environment variable OPENAI_BASE_URL to the custom API endpoint which applies globally.
But the models names which are offered by litellm via the API do not match the hardcoded common OpenAI model names available in the Weblate machinery settings dropdown. This results in the error "Could not fetch translation: Unsupported model: <model>".
Describe the solution you would like
It would be great if the OpenAI machinery could:
- provide the option to optionally override the default endpoint
- populate the model dropdown from the API. This would also remove the need to hardcode the model choices in
OpenAIMachineryForm.
Describe alternatives you have considered
-
Add another machinery for litellm: there are probably a lot of other OpenAI compatible services, so to it should be as generic as possible to keep the amount of machineries in check.
-
Use the litellm python package instead of the openapi package to broaden LLM support: Probably more prompt customization options would be needed plus it would make Weblate dependant on a package from a VC funded startup in the seed phase.
Screenshots
No response
Additional context
The relevant litellm proxy config looks like this:
model_list:
- model_name: mistral-7b-instruct
# [...]
router_settings:
model_group_alias:
gpt-4: mistral-7b-instruct
gpt-3.5-turbo: mistral-7b-instruct
Describe the problem
I would like to integrate self-hosted LLMs which are proxied by litellm which provides a API in OpenAI format.
The API endpoint URL is not configurable in the OpenAI machinery. Currently this can be worked around by setting the environment variable
OPENAI_BASE_URLto the custom API endpoint which applies globally.But the models names which are offered by litellm via the API do not match the hardcoded common OpenAI model names available in the Weblate machinery settings dropdown. This results in the error "Could not fetch translation: Unsupported model: <model>".
Describe the solution you would like
It would be great if the OpenAI machinery could:
OpenAIMachineryForm.Describe alternatives you have considered
Add another machinery for litellm: there are probably a lot of other OpenAI compatible services, so to it should be as generic as possible to keep the amount of machineries in check.
Use the litellm python package instead of the openapi package to broaden LLM support: Probably more prompt customization options would be needed plus it would make Weblate dependant on a package from a VC funded startup in the seed phase.
Screenshots
No response
Additional context
The relevant litellm proxy config looks like this: