Several issues introduced in version 20230306 on audio with silences (repeated text, segment "id" not unique/increasing)

[Jeronymous]

This issue looks more general than #1046 and it might be related to #730 (however this was after v20230306).

---

Take this audio: bonjour_vous_allez_bien.mp3
where "Bonjour, est-ce que vous allez bien?" (French) is said twice, with a long delay in between.

For this command:

whisper --model tiny --language fr \
   --beam_size None --temperature_increment_on_fallback None --best_of None \
    bonjour_vous_allez_bien.mp3

below are the differences between outputs of previous version 20230124 (left), new version 20230306 (middle) and 20230306 with --word_timestamps True (right). We can see two issues:

• 20230306 is producing a lot of hallucinated repetitions and this is much worse with --word_timestamps True
• the "id" of each segment is not necessarily unique in 20230306. It can be several times the same in a row (it gets stuck at "0" in the example below)
  

---

This became obvious when I used greedy decoding, but a similar thing can be observed with less options (with beam search).

whisper --model tiny --language fr bonjour_vous_allez_bien.mp3

On this command, the differences between outputs of previous version 20230124 (left), new version 20230306 (middle) and 20230306 with --word_timestamps True (right) are:

---

I also noticed that the following command is particularly long to run:

whisper --model tiny --language fr --word_timestamps True /home/jlouradour/src/whisper-timestamped/tests/data/bonjour_vous_allez_bien.mp3

It takes 1 minute with 4 CPU, whereas it takes 6 sec without --word_timestamps True, or 16 sec if adding options --beam_size None --temperature_increment_on_fallback None --best_of None

[Jeronymous]

I've just tested 20230307 and the two problems are still here:

• not unique "id" (not necessarily increasing)
• the "text" field of the transcription include hallucinated repetition, bu actually that those are not present in the segments (so if we join all the "text" of the segments, we get a result that is different from the main "text")

[glangford]

This looks like the repetition is unique to the .json output, the others (.txt, .srt, etc) look correct. Do you agree?

[glangford]

I can reproduce this problem only with the tiny model, and only for the .json output. Small and medium and other outputs seem to be ok. word_timestamps does not have to be True.
whisper --model tiny --language fr bonjour_vous_allez_bien.mp3

If I don't supply --language, the tiny model detects the audio as Russian.

[SalimovAlbert]

Had the same issue with JSON, used "".join(map(lambda x: x['text'], result['segments'])) to get text output

[Jeronymous]

This looks like the repetition is unique to the .json output, the others (.txt, .srt, etc) look correct. Do you agree?

Yes, correct.
It only affects json output (or the output of the transcribe function when using python and not the CLI)

[jongwook]

Hi! Thanks for reporting this. The non-unique "id" is my total oversight and should be an easy fix; wasn't paying attention to it much.

About the discrepancy between ["text"] and segment["text"], I think it has something to do with these lines:

whisper/whisper/transcribe.py

Lines 345 to 356 in aac47c9

	# if a segment is instantaneous or does not contain text, clear it
	for i, segment in enumerate(current_segments):
	if segment["start"] == segment["end"] or segment["text"].strip() == "":
	segment["text"] = ""
	segment["tokens"] = []
	segment["words"] = []
	current_tokens[i] = []
	all_segments.extend(current_segments)
	all_tokens.extend(
	[token for segment in current_tokens for token in segment]
	)

EDIT: the culprit was actually this line:

whisper/whisper/transcribe.py

Line 290 in aac47c9

all_tokens.extend(tokens[: last_slice + 1].tolist())

will push a fix soon.

[jongwook]

This was an issue where the new transcribe() was mishandling the all_tokens variable which affected the prompts to be more prone to repetitions and also caused the discrepancy between the top-level "text" field and the segment-level "text" fields in the JSON response.

I have a fix merged in #1060, so hopefully this resolves your issue! Please let me know if it continues.

[Jeronymous]

Great. Thank you @jongwook

[timothyaveni]

I think the condition_on_previous_text flag was recently broken, and my hunch is that the cause is some changes in #1060; in particular, I think all_tokens is off by one window at the point where the prompt is reset, causing the prior window's tokens to be added to the prompt anyway. @jongwook would you mind having a look? Thanks!

[guillaumekln]

A PR was opened to fix this issue: #1224.

[timothyaveni]

oop, didn't think to look there. thanks for the pointer!