Authoring Media Overlay with text fragments instead of an ID

[HadrienGardeur]

Section

https://w3c.github.io/epub-specs/epub34/authoring/#sec-media-overlays

Describe the problem

This was originally raised by @richwaters in #2917 (comment)

Authoring Media Overlays using SMIL can put a heavy toll on the HTML that you need to produce.

With an increased number of tools to do automated alignment or to generate audio (with TTS), this is quickly becoming the heaviest task for any of these tools. It can be very invasive and heavily impact the markup for HTML.
Unlike other features of our spec, this is often done as a separate task after the EPUB has been authored.

Our spec is fairly open when it comes to fragments. In the definition of the text element, we define src as:

Refers to the associated EPUB content document and, optionally, identifies a specific part of it.

The value MUST be a path-relative-scheme-less-URL string, optionally followed by U+0023 (#) and a URL-fragment string.

We also say in a note that:

This specification places no restriction on the src attribute of a text element […]

In practice though, things are a little different:

• in all of our examples we use fragment IDs
• and the entire ecosystem expects them as well

Describe the fix or new feature you propose

First of all, I think that we could be a little more explicit and openly say that we recommend using a fragment ID. This is documented further down in the spec (https://w3c.github.io/epub-specs/epub34/authoring/#sec-media-overlays-fragids) but can be easily missed if you focus on the definitions.

But with the addition of text fragments for annotations, we could also consider them as well.

Unlike IDs, text fragments would allow production tools to create SMIL files without changing the HTML at all.

[richwaters]

It's great to see interest in addressing this issue. The problem description states the issue well, and "invasive" is exactly right.

Even if done perfectly, without visible changes to the book's appearance, modifying the html content still feels wrong. And it really is difficult to get it perfect (like when CSS depends on direct children). When formatting issues creep in, readers can assume the book itself is faulty, when the defect was introduced by the alignment process. To make matters worse, some workflows discard the original unaligned epub. That makes it impossible to clearly attribute when a problem was introduced.

Anyway, I think keeping alignment data completely separate from the book content provides a much cleaner model all around and I hope we can get it done.

[richwaters]

The text-fragments approach makes sense to me. If I understand correctly, the spec already allows for this and it's a matter of implementation by the reading systems and producers.

One thing that might be worth considering is that with word-level granularity, these text-fragments might explode. Disambiguating words like "a", "of", and "the" on a per-chapter basis might require a lot of repeated context.

In addition to flat text-fragments, does it make sense to use the nested seq from Issue 2917 and use that outer container (sentence) to provide scope for the smaller chunk (word), such that the fragment for the word would only need to be unique for the sentence it appears in? That would seemingly cut these smils down quite a bit. I don't know if nested text fragments are outside the scope of this proposal and/or if that would make the burden lighter or heavier on reading systems, but I thought it worth mentioning.

On a similar note, different strategies for constructing text-fragments can result in a very different fragment sizes. Should there be a recommendation of what to optimize for when creating them? Absent that, it seems like you'd want to optimize for smallest smil, but would that have an impact on reading system performance or it doesn't matter?

[HadrienGardeur]

The text-fragments approach makes sense to me. If I understand correctly, the spec already allows for this and it's a matter of implementation by the reading systems and producers.

That's right, the spec says that you can use other media fragments but that they might not be properly supported by RS.

One thing that might be worth considering is that with word-level granularity, these text-fragments might explode. Disambiguating words like "a", "of", and "the" on a per-chapter basis might require a lot of repeated context.

In addition to flat text-fragments, does it make sense to use the nested seq from Issue 2917 and use that outer container (sentence) to provide scope for the smaller chunk (word), such that the fragment for the word would only need to be unique for the sentence it appears in?

I don't think that this is too different from TTS though:

• with TTS engines, you know the entire utterance and where it's located in the document
• but the TTS engine only returns a single word at a time as it's being read (boundary event on the Web with equivalent native APIs in Swift/Kotlin for mobile)

If we already know the context (full sentence or equivalent using <seq>), then the <par> for each word doesn't need anything more than the word(s) being read and we can follow them sequentially.

But if the context is missing, this gets much trickier and you'd need a few words before and after to properly identify where each word is located. This is a case where things would quickly get very bloated SMIL-wise.

I wouldn't recommend doing word-by-word alignment using text fragments without using another container element.

On a similar note, different strategies for constructing text-fragments can result in a very different fragment sizes. Should there be a recommendation of what to optimize for when creating them? Absent that, it seems like you'd want to optimize for smallest smil, but would that have an impact on reading system performance or it doesn't matter?

Right, this could affect performance but also the ability to highlight text at all.

There's a non-zero risk with text fragments that the RS could fail to find and highlight some text fragments.

[smoores-dev]

This is a case where things would quickly get very bloated SMIL-wise

It's worth noting that there's a significant cost to the alternative (wrapping every word in a span) in terms of EPUB size, too.

<p>This is a sentence.</p>

Becomes something like:

<p>
  <span id="ch1-sent1">
    <span id="ch1-word1">This </span><span id="ch1-word2">
    is </span><span id="ch1-word3">
    a </span><span id="ch1-word4">
    sentence.</span>
  </span>
</p>

Which is an increase in size from 26 to 161 characters.

[HadrienGardeur]

Well let's take an example to illustrate all of this with text fragments:

It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way—in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only.

This definitely qualifies for text-range-large so the entire thing would be in a <seq> that contains a <par> per word.

For the containing <seq>, we just need a few words from the beginning and the end in the text fragment, while for each <par> we'd only need the text fragment to contain the word.

<seq epub:type="text-range-large" epub:textref="chapter1.html#:~:text=It%20was%20the,of%20comparison%20only.">
  <par>
    <text src="chapter1.html#:~:text=It" /> 
    <audio … />
  </par>
  <par>
    <text src="chapter1.html#:~:text=was" /> 
    <audio … />
  </par>
  <par>
    <text src="chapter1.html#:~:text=the" /> 
    <audio … />
  </par>
</seq>

Compared to the addition of markup in the HTML, I think that this would be by far the most optimized representation.

[smoores-dev]

Hm... is this true, though? Within that text-range-large, this wouldn't be at all unique, right?

  <par>
    <text src="chapter1.html#:~:text=was" /> 
    <audio … />
  </par>

[HadrienGardeur]

It doesn't need to be unique to work. Once again, things are ordered in SMIL so you basically have a cursor that you can move around as you progress through a paragraph/sentence.

This is what I was describing for TTS, we also get a word returned by the Web Speech API.

[smoores-dev]

From an implementation standpoint, it seems like it would be challenging to implement this literal case correctly, though I admit that I may be missing something. I guess when seeking for the specific text fragment identified by #:~:text=was, you would use the preceding pars' text fragments as a prefix of a sort? That sort of presumes that you haven't skipped any text in your media overlay, which I suppose is more safe at the word-size granularity, but definitely not safe at the sentence-size granularity. Even if audiobooks didn't sometimes skip some sentences, any algorithmic alignment may sometimes fail to find the audio for a given sentence.

[HadrienGardeur]

Here's how you could handle it in a RS:

• with the <seq> element containing an epub:type="text-range-large" the RS would highlight the entire sentence here, using the text fragment from epub:textref
• from that point, we would also place a cursor (character index) inside this sentence, at the very beginning of it and we know that we're strictly looking for text within that element
• for the first <par> element, we look for the first match of "It" by looking forward after our cursor, when we find it it gets underlined and the cursor gets moved right after "It"
• moving on to the next <par> element, we look for "was" the same way and underline it
• if at some point we can't identify an element, then the audio plays without moving the cursor forward in the sentence or underlining anything
• skipping words would also be OK since we would always look for the first match after our cursor

This is slightly more bothersome than TTS where we get a character index into an utterance instead of having to look for text, but it's roughly the same idea.
To try it out with TTS (just make sure that you dont use a Google voice since they don't return boundary events): https://readium.org/speech/demo/

Of course, the author of this SMIL is always free to provide additional context as well and repeat words before/after what we're trying to synchronized, if deemed necessary.

[richwaters]

Well, you both know more about this than I do, but relying on non-unique, position-oriented word fragments feels not quite right to me. It seems more like an optimization and implementation choice than something that should be encouraged.

In my earlier comment, I was suggesting an additional layer of scope for uniqueness if nested seqs are used. So, the text frag for the text-range-large would be unique within the document and the text-range-small would be unique within the text-range-large. To me, this feels truer to the spirit of current text-fragments than allowing repeated non-unique words and depending on internal cursor state to disambiguate. Also it seems like requiring uniqueness within the limited scope leaves open more implementation options for the reading system. And the difference in smil size between those two options shouldn't be too significant.

On the other hand, I think the position-oriented solution would work for storyalign, which provides a smil for every single word. I haven't been able to think of a case where a mismatch would result in anything worse than skipping highlights at the end of a chunk instead of the middle.

In case it's of interest, here are some numbers from the Dragonsteel Prime book using the current span-wrapping technique:

original m4b: ~784MB
original epub: ~2MB
original xhtml: ~1.5MB
smils (sentence granularity): ~4MB
xhtml (sentence granularity): ~3MB
smils (word granularity): ~49MB
xhtml (word granularity): ~13MB

• Note that epub and m4b are compressed, while the smils & xhtml are not.

[HadrienGardeur]

In my earlier comment, I was suggesting an additional layer of scope for uniqueness if nested seqs are used. So, the text frag for the text-range-large would be unique within the document and the text-range-small would be unique within the text-range-large.

How is that any different from the example that I shared above? Do you have something else in mind for scoping?

In #2934 (comment):

• I didn't use text-range-small at all, which IMO is fine, you're free to use neither, one or both
• the text fragment referenced in the <seq> element is unique
• but the word-level text fragments are not unique

I'm not encouraging any sort of best practice for producing files here, just pointing out that this is the minimum level of info that we could receive with a text fragment.

In the case of RS, we have zero control over the way that content is authored, which means that we need to prepare for the worst case scenario and design solutions that can work anyway.

This is not truly the worst case scenario BTW, because at least we're scoped down to the sentence-level (albeit a very long one).

The truly worst scenario would drop the <seq> element completely with a very large HTML document that contains many repeated words.

original m4b: ~784MB
original epub: ~2MB
original xhtml: ~1.5MB
smils (sentence granularity): ~4MB
xhtml (sentence granularity): ~3MB
smils (word granularity): ~49MB
xhtml (word granularity): ~13MB

This is very useful info.

So the total EPUB with fragment ID would be:

• ~790MB for sentence level
• ~846MB for word level

With text fragments it would probably be slightly larger for SMIL but it would have zero impact on the HTML size, so something like:

• ~789MB for sentence level
• ~840MB for word level

Impact of sentence level vs word level or ID vs text fragment would be minimal for the file size.

But working with large SMIL and XHTML files could have a fairly noticeable impact on parsing and rendering and affect overall RS performance.
That said, resolving text fragments would be slower than ID, but this has a lesser impact than rendering (everyone is affected by rendering, even if they don't use Media Overlays).