Index/Track Fixes

[JPKribs]

Summary

Resolves: #676
Resolves: #900
Resolves: #926
Resolves: #1531
Resolves: #1554
Resolves: #1728
Resolves: #1854
Resolves: #1889
Resolves: #1904

---

This PR is very dense but I think this solves a lot of our playback quirks! The crux of the issue is that we have two sources of truth for track selection. Jellyfin and the actual files read via VLCKit.

• Jellyfin handles this as External Audio > External Subtitles > Video > Audio > Subtitles.
• VLC handles this as File Track Order > External Audio > External Subtitles.

Previously, we were resolving this by just moving tracks around at the beginning. This ended up causing issues for us with the video player rewrite as we have a kind of 'sidecar' system for external files now. So, when remapping indexes, we didn't know the order of these files but just a guess. On top of this, we were not accounting for files types that Jellyfin was hiding in the MediaSource but exist in the file. Particularly, when Text or Image subtitles types are disabled. This then causes issues as our index we are using in Swiftfin doesn't match anything in Jellyfin anymore. To work around this, I build an indexMap with local and server indexes that are built first based on the Jellyfin provided MediaSources that are then revised as the 'sidecar' subtitles and other information load in. Round 1 is used to get the default / selected track for playback and Round 2 sures these up in case there are issues like Image or Text subtitles being hidden or things like external audio existing in the MediaSources but being unusable to us. Transcoding also causes issues as all internal subtitles because external and move down the row.

The other issue with this is that we can request stream based on default tracks. So, if the default track is AAC and the user switches to MLP / Atmos, a transcode will not be requested because we were given the raw file and we are changing tracks locally. This ave the impression that Swiftfin was not transcoding at all. I've updates this so, as tracks change, audio always requests a new stream to trigger a transcode if needed. Subtitles only triggers a new stream if we are requesting a PGS / Image Subtitle in an external format as those cannot be handled locally and need to be encoded into the video. Documentation and Device Profiles have been updated to reflect this!

Finally, we cannot play external PGS or image subtitles saves as .sup file. These need to be encoded into the file or embedded into the container. For this reason, if I have compatibility mode set to DirectPlay only, I hide these options from the drop down to prevent issues.

I've fully tested every variation and have ZERO issues or mis-selections. The only issue is external audio tracks do not play but the have never played in Swiftfin. I left a TODO to figure this out but basically, these external audio tracks can DirectPlay so Swiftfin doesn't request a transcode but Jellyfin doesn't provide a delivery URL so we need to encode these into the stream to play this.

Testing files with subtitles can be found below. 25mb was the GitHub limit so I had to break this into 2 batches:

Tests 1-3.zip
Tests 4-7.zip

Happy to discuss any of what I've done here! It's all functional and clean (in my eyes) but I've been looking at this too long so any cleanup I can do to make this more universally understandable is welcome feedback!

---

Videos:

Hopefully this shows what I mean but this is transcodes & directplays with various subtitles counts and audio counts all working without issue. All from a test file with the exception of the MLP / TrueHD which is LOTR since that's like my only file I left that in.

MLP / TrueHD Transcoding (AC3 -> TrueHD)

1.4.1:

If my starting track does not require transcoding, and then I switched to a track that does require transcoding, VLC just switches the track inside of the file, resulting in no playback.

If my starting track does require transcoding, or transcoding occurs at any point, I am only shown the audio track that is included with the new stream. Additionally, the subtitle indexes are incorrect, so you cannot select subtitles while transcoding.

trim.DFEB39B9-DFD2-4797-BF00-42989207AEA1.MOV

All tracks are always shown to you and are always selectable. If you select a track that requires transcoding a transcode occurs, but if you select a track that does not require transcoding, it will just switch within the file itself. The position of playback is preserved.

trim.2E0F026E-1499-4A87-BEA5-21DB12060B22.MOV

External PGS File Transcoding (Text Embedded Subtitles Only)

Simulator.Screen.Recording.-.iPhone.-.2026-02-20.at.00.34.33.mov

Transcoding File Test (Text Embedded Subtitles Only)

Simulator.Screen.Recording.-.iPhone.-.2026-02-20.at.00.35.17.mov

Transcoding File Test (Text Embedded Subtitles Only & DirectPlay Only)

Simulator.Screen.Recording.-.iPhone.-.2026-02-20.at.00.35.57.mov

Transcoding File Test (Image Embedded Subtitles Only & DirectPlay Only)

Simulator.Screen.Recording.-.iPhone.-.2026-02-20.at.00.36.55.mov

[JPKribs]

@LePips if you have a chance I'd appreciate a second set of eyes on this before the next release. This resolves a lot of outstanding issues that come form VLC being frustrating but, to be completely frank, I was having trouble understanding the best way to do this in relation to the media player manager. All of the components are there and I have spent a ton of time and scenarios testing (per the main post) so I know the solution is good but I feel as though my solution is just kind of tacked onto the player as opposed to being integrated with it. The changes can be grouped into 2 categories:

1. Recreate the HLS stream on Track change. Some users have reported Swiftfin not transcoding. This is because if a file can be played natively, we get the full file with all tracks. If I switch to a track that cannot be natively played, we do not request a transcode because we are doing a local track change. This PR moves that to request the new stream which forces transcoding if the track doesn't work. Every audio track forces a new HLS stream but only external image subtitles force one as internal anything or external text subtitles will work as switches locally. External images need to be burned in and VLC does not work with them in my testing.

I moved this to the manager since HLS changes seemed more like a manager item than a proxy item? I was thinking maybe this is both? Like, manager creates the new stream then the proxy selects it? What I have now on the manager works but honestly, I'm feeling a bit lost. Sorry to toss this over the fence to you like this. I have it working in this PR but there is a layer of abstraction here I understand is required for VLC v AVPlayer v etc that I'm not positive if I'm breaking with this PR. Also, calling a new HLS will break downloaded playback so I'm not sure how to approach that.

I believe this HLS route will also work for bitrate changing so the structure is there for that but no real work.

---

2. This is a continuation/rework of [iOS] Fix External Subtitle Selection #1445. The issue at the end of the day is that VLC indexes based on the location of the track (internal v external) and also based on type/container while Jellyfin is just a straight global list. The reason this kept getting us into trouble was we were just overriding the old indexes for mapping. If the indexes every varied from what Jellyfin has/provides we could get off track. Some examples of where this can occur:

• Library is set up to exclude certain subtitles (Image, Text, All Internal) which meant we get the file with all subtitles but we're supposed to ignore some.
• The file is being transcoded so some of the subtitles that were IN the file are now external which changes the order. Also, some internal subtitles are not suitable as external subtitles and have to be dropped altogether.

The way I got this to work is by creating an index map based on the original file in Jellyfin. This is based on the information that Jellyfin shows (Tracks we are supposed to have access to and where they are located). Once we receive the information about the file provided to us (Actual tracks), we have to re-map to help prevent issues with indexes on tracks we're not supposed to use. I think the index map is the correct move as it gives us an internal index for local track changes but also an external track for recreating the HLS stream (part of why these are bundled together). The reason this is two processes is this lets us request the stream first then this lets us remap based on what is actually provided to us. So like, we are requesting index 4 but it's 6 in the file because there are two internal image subtitles we are not allowed to see per library settings. As a result, without first mapping using information provided via API, we will start playback on the wrong track or with no subtitles/audio at all.

I'd be happy to help field questions or rework any of this! I think this is a big improvement/fix on the original #1445 solution

---

As a final note, this method for changing audio tracks would enable us to change tracks in AVPlayer as well as we are no longer reliant on the single audio track provided in HLS and instead would re-request HLS for track changes.

IMO, I think this track switching is the appropriate solution for what is being done here: #1562

[JPKribs]

I have a branch here I've been testing this change with AVPlayerProxy using the existing UI:

https://github.com/JPKribs/Swiftfin/tree/avplayer

I can confirm that these changes work for AVPayer to enable subtitles and track selection even in PiP!

trim.5947BDCC-16DF-4522-8949-A8EB9FC7BC16.MOV

Ignore the contents of the commentary lol I purely pulled this as a test of everything in a hurry while I wrapped up today. I'll pull a better video of this next time I'm home.