What does the "<|translate|>" mean ?

[zhhl9101]

Hi Whisper:

When used Italian model, I got text like "Un anno dopo restituisce<|translate|> alla banca a tutte i capitali con gli interessi composti. Dopo aver<|it|><|transcribe|> rapp나 sull' kobbiere iljcie modularisso del pi taskedio ottenete la praличna."

So, what do "<|translate|>" and "<|it|><|transcribe|>" mean ?
How can I avoid getting results like this ?

Thanks

[jongwook]

It appears to be a bug in the sampling process. Are you using Whisper on the command line or in Python? It'd be helpful if you could share the audio and the command(s) you're using to call Whisper.

UPDATE: fixed in #1033

[zhhl9101]

@jongwook, Thanks for your clarification.

Using Whisper in Python, code like:

import sys
import whisper

wave_path = sys.argv[1]
locale = 'it'
model = whisper.load_model('small')
reco_json = model.transcribe(wave_path, language=locale)
print(reco_json)

The audio: it.zip

Do other language models also have this issue ?

Thanks

[jongwook]

Hi! Thanks for providing the example. I've identified the problem where the model was not disallowed to sample <|transcribe|> and <|translate|> tokens (which are only meant to be used as prefixes) and fixed it in #1033. You can apply this change by reinstalling Whisper at the latest commit:

pip install --upgrade --no-deps --force-reinstall git+https://github.com/openai/whisper.git

The update will be available in the next version on PyPI (as openai-whisper) in the near future.