Implement DRM auto-selection

[CastagnaIT]

Description

This is the last big step of my drm rework

This PR completely rewrites the code of how DRM info provided by manifests are handled, and how DRM/CDM are initialized and managed.

The main goal is to introduce a common place where manage DRM info and CDM so in the DRMEngine,
this avoids the need for each manifest parser to have the task of choosing and managing DRM info (which parsers should not do, as is self-explained by the name) and along this introduce the automatic selection of DRM.

To reach this point, i started working on this 2 years ago, initially with the RFC test PR #1415, then with PR #1672 and in between other minor PRs not mentioned here, thousands of changes were needed to achieve something of acceptable quality with this PR, although there are some poor code parts that require more attention and much more time.

The amount of work to achieve this is very high, and despite my efforts to do dozens of tests, i will never have the opportunity to verify every single use case, so it is possible that this imlementation introduces some bugs that do not allow correct playback. It would be oppurtune to have some feedback from some users who use ISA with some video add-ons, if you read this please give your feedback.

Main changes:

DRM auto-selection (new)

This requires a manifest that supports multiple DRMs,
in this case you will have to provide each DRM configuration by using the new property:
inputstream.adaptive.drm
as explained on Wiki https://github.com/xbmc/inputstream.adaptive/wiki/Integration-DRM (that i will update later time)
therefore ISA will automatically determine which DRM to use, but you will also be able to give some sort of priority to the type of DRM to be used by making use of the priority parameter.

HDCP (breaking changes)

In my research it seems to me (but i could be wrong since i haven't found comprehensive docs) that a media player should not have a way to handle HDCP, take task is inherent possibly in the DRM (CDM), which should block non-usable Keys and nothing more.
The possibility of managing the rejection of keys seem not so good handled on ISA, and cannot be done on this PR, also because there is a problem in stream ID selection in Kodi core, that prevent a fallback an appropriate stream ID when one fails, which should be fixed before that.

Breaking changes 1:
The ISA setting "Ignore HDCP status" is now removed,
the current behavior is that the HDCP "status" is never verified, but can be re-enabled by using the new check_hdcp, read below.

Breaking changes 2:
In the unwrapper_params, child parameter of the license parameter,
the path_hdcp and path_hdcp_traverse has been renamed into path_hdcp_res and path_hdcp_res_traverse (that handle resolution limit),
and there is a new path_hdcp_ver and path_hdcp_ver_traverse, that handle an HDCP Version comparison with the manifest value.
Details on wiki page https://github.com/xbmc/inputstream.adaptive/wiki/HDCP-resolution-limit

New property parameters

The inputstream.adaptive.config have two new parameters:

• resolution_limit - Allow to limit the playable resolutions provided by manifests. This is a kind of workaround since currently we cant handle appropriately HDCP use cases. Just set the max resolution that can be used e.g. "1024x768"
• check_hdcp - Allows to enable the check of the HDCP values provided in the license response. Details on wiki page https://github.com/xbmc/inputstream.adaptive/wiki/HDCP-resolution-limit

The inputstream.adaptive.drm have a new parameter:

• force_single_session - Allow to enforce a single DRM session, when possible. (while with new "drm" prop this is disabled by default, in the old properties and "drm_legacy" still use the enforced single session. more details below)

DRM configuration changes

1. When a manifest include DRM protections that dont require additional configurations, it will be played directly, so there is no need to set inputstream.adaptive.drm or inputstream.adaptive.drm_legacy property.
   This is the case for example for PlayReady, or ClearKey case that embed its license url in the manifest.

2. In the ClearKey configuration with inputstream.adaptive.drm or inputstream.adaptive.drm_legacy, its not only supported to set the license URL, but now its also supported to set a URI with "data" scheme. The data follow the W3C standard, so: data:application/json;base64,REPLACE_WITH_CK_LICENSE_RESPONSE_AS_BASE64

DRM Sessions (breaking change)

With older ISA versons (older kodi versions) sessions created most of time was a single one (when license provide all KIDs) and on android was always enforced into a single session, that was leading to possible decryption problems by displaying corrupted/pixellated videos.

Now by using the new properties to configure the DRM so inputstream.adaptive.drm and inputstream.adaptive.drm_legacy
the DRM sessions sessions are distinguished by media type and will be reused when:

• They have same init data (default KID)
  or when:
• The obtained license support multiple KIDs, but this only applies to DRMs that allow it to be known, for example on Widevine for android this is not possible and so this option will be excluded (means no session reused if there are different initdata/KIDs).

So if there are multiple DRM sessions opened means also multiple license requests,
if this is a problem for your provider and you are sure that the license response provide all KIDs in a single request,
you can enforce the use of a single session, by using force_single_session parameter of inputstream.adaptive.drm,
this way you can also save the time it takes to request multiple licenses and start playback faster.

Brief summary session breaking-changes:

• Nothing is changed if you are using old DRM properties (e.g. inputstream.adaptive.license_key)
• inputstream.adaptive.drm_legacy unlike Kodi 21, it no longer forces single session
• inputstream.adaptive.drm never force a single session, if not specified by using force_single_session parameter

Stream selector type: Ask quality (improvement)

In multi-codec manifests, all codec variants will now be shown. I had to include this change here out of test necessity.

Deprecated old DRM properties

All following old DRM properties:

inputstream.adaptive.license_type
inputstream.adaptive.license_key
inputstream.adaptive.license_data
inputstream.adaptive.server_certificate
inputstream.adaptive.license_flags
inputstream.adaptive.pre_init_data

Are now --officially-- deprecated, in favor of the more advanced methods:
inputstream.adaptive.drm_legacy and inputstream.adaptive.drm
Its recommended to use the new properties from Kodi v22, see Wiki for more details.
Old properties still work, for now, but will be removed on future Kodi versions.

Other changes under the hood

• Now the initialization of DRM/CDM Sessions are done on streams to be played only. where previously sessions were created for all streams even if unused, this improve also the playback start speed (with the exception of HDCP license based is enabled). Changes restored to old behavior due to lack of Widevine CDM Secure decoder for audio media preventing code optimization
• Add support to SmoothStreaming to allow override PlayReady DRM with ClearKey DRM
• Removed HLS parser hack that used DRM objects to store AES data, now the AES data is stored on each segment, with an appropriate cache for downloads
• Fix JSON traverse feature used on JSON license response to find data by path, now correctly traverse also json array type cases

To do / tests

[x] Rework drm info on Dash parser
[x] Rework drm info on HLS parser
[x] Rework drm info on Smoothstreaming parser
[x] Reimplement CheckHDCP
[x] Adapt test project, if needed
[x] Enable multi-drm support to "drm" property
[x] Deprecate old properties
[x] Test stream with single drm widevine
[x] Test stream with single drm playready
[x] Test stream with clearkey
[x] Test stream with widevine "to" clearkey
[x] Test multi drm's stream widevine+playready, between windows and android
[x] Test HLS AES-128
[x] Test HLS AES-128 - key on each segment
[x] Test HLS multi-period, encryption transitions: unencrypted->AES-128->unencrypted
[x] Some tests on android (tested SS+PR, some widevine)
[x] Some tests on linux (under Ubuntu widevine)

fix #1585 about to separate audio/video sessions

Motivation and context

This PR will finish the multi-drm configuration support for inputstream.adaptive.drm property

What is auto selection of DRM?

in this case we are talking about manifests (usually HLS/DASH) that support multiple DRMs at same time.
Until now, every python video add-on was forced to manually set the appropriate DRM for the type of operating system in use
by setting appropriate inputstream.adaptive.license_type and inputstream.adaptive.license_key (and all other properties).
for example a manifest that support playready and widevine,
on Windows OS is needed that the python addon set the drm configuration for widevine, and on android could be for playready.

This implementation allows a video add-ons to provide all the DRM configurations supported by the provider in a distinct way, so that ISA can automatically select a DRM by finding a match between DRM supported in the operating system in use and the manifest, so a video add-on will no longer have to check if a DRM is compatible with the operating system in use, to configure ISA.

This will be even more useful in the future when a support to new DRMs can be implemented.

Types of change

• Bug fix (non-breaking change which fixes an issue)
• Clean up (non-breaking change which removes non-working, unmaintained functionality)
• Improvement (non-breaking change which improves existing functionality)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that will cause existing functionality to change)
• Cosmetic change (non-breaking change that doesn't touch code)
• None of the above (please explain below)

Checklist:

• I have read the Contributing document
• My code follows the Code Guidelines of this project
• My change requires a change to the Wiki documentation
• I have updated the documentation accordingly

[CastagnaIT]

@glennguy are you still here? please let me know
i need help about CBCS, is broken from Kodi 21 and i have no clear idea

for now it would be enough to have it working by using this PR (Kodi 22)
have you time to check the problem?

by using "InputStream adaptive samples" addon, im pretty sure that following stream was working:
List CBCS streams menu --> Google Tears [CBCS]

EDIT: fixed by #1790

[Uukrull]

I did a quick test and the only regression I noticed is that UrnToKeySystem returns "" if
<ContentProtection schemeIdUri="urn:uuid:edef8ba9-79d6-4ace-a3c8-27dcd51d21ed">
doesn't exist, but <ContentProtection schemeIdUri="urn:mpeg:dash:mp4protection:2011" value="cenc" cenc:default_KID="xxxxxxxxx" />
exists so it should support those cases, otherwise it won't work because keySystem is empty.

[CastagnaIT]

in this case the target DRM is unknown because "urn:mpeg:dash:mp4protection:2011" means only CENC protection that should be compatible with all drm's
i think the better thing could be add an additional check that when there is a single DRMInfo with no KS match to the first DRM (in priority order), otherwise another idea is check the initial segment, but this require much more changes, maybe later time

[CastagnaIT]

@Uukrull fixup added let me know

[Uukrull]

Working fine here. Many thanks!

[CastagnaIT]

@Sandmann79 i have updated code, but i tested only on windows, can you give a feedback also on android?

i added new DRM Sessions (breaking change) section on PR description
that explain more or less about the drm sessions changes, anyway am@zon should be compatible with both new/old session methods
anyway i suggest you to try add force_single_session parameter to True in your addon when you use inputstream.adaptive.drm

PS -- OT:
there is somewhere am@zon bug, but dont happens always,
it happens (on windows in my tests) to reproduce usually i do -> play a video -> close kodi with "X" on window frame
happens that kodi try to close itself but it hang when it try to terminate amazon service, it dont free resources and kodi executable become a "zombie" (you need to kill process)
when this problem happens you can see "[Amazon VOD] Service stopped" on log, but its missing "[Amazon VOD] Service: Proxy server stopped"
the python code is freezed on one of these calls
https://github.com/Sandmann79/xbmc/blob/12c1771994c2294d661e8a227fe25eee0bbc46c2/plugin.video.amazon-test/resources/lib/service.py#L56-L59

EDIT: perfect on android is full broken

[Sandmann79]

Sorry for the late reply, can you please check the Windows OS again, I have the problem again that several attempts are necessary to start a stream. Even though force_single_
session is set. In the attached log it only works on the 3rd attempt.

Thanks for the tip about the service not terminating. I'll have to take a look at that.

kodi.log

[CastagnaIT]

@Sandmann79 hi i was trying investigate in these hours
and i found 2 bugs, one on inputstream.adaptive.drm that i have just pushed fixup,
and one on your addon, you are sending wrong drm configuration
at least was wrong in the addon version that i have locally, please check your sources
this is the fixed playback.py:

        req_data = "widevine2Challenge={CHA-B64U}&includeHdcpTestKeyInLicense=true"
        drm_cfg = {'com.widevine.alpha':
                       {'license':
                            {'server_url': licURL,
                             'req_headers': urlencode(lic_headers),
                             'req_data': base64.b64encode(req_data.encode('utf-8')).decode(),
                             'wrapper': "none",
                             'unwrapper': 'json,base64',
                             'unwrapper_params': {'path_data': 'license', 'path_data_traverse': True,
                                                  'path_hdcp': 'hdcpEnforcementResolutionPixels', 'path_hdcp_traverse': True}
                             }
                        }
                   }
        listitem.setProperty('inputstream.adaptive.drm', json.dumps(drm_cfg))

with these 2 fixes, playback on windows works with old and new drm properties
the playback problem on Android i will have to investigate better another day...

[Sandmann79]

I am currently using the following DRM config, with which playback was possible:
https://github.com/Sandmann79/xbmc/blob/master/plugin.video.amazon-test/resources/lib/playback.py#L418-L428

However, the above-mentioned effect also occurs when using the old properties.

[CastagnaIT]

ok discovered now that some videos works and others not
this thing is becoming a nightmare i will investigate on next days

[CastagnaIT]

Okay, i think i found the problem affect windows,
this was difficult to understand since the mechanism was not so directly understandable

in short am@zon have some of his audio streams that require CDM "secure path"

since on ISA its missing "secure path" implementation for audio media
will be trigged SSD_INVALID, silently

inputstream.adaptive/src/decrypters/widevine/WVCencSingleSampleDecrypter.cpp

Lines 202 to 207 in 2364435

	LOG::LogF(LOGDEBUG, "Single decrypt failed, secure path only");
	if (media == DecrypterCapabilites::SSD_MEDIA_VIDEO)
	caps.flags |= (DecrypterCapabilites::SSD_SECURE_PATH |
	DecrypterCapabilites::SSD_ANNEXB_REQUIRED);
	else
	caps.flags = DecrypterCapabilites::SSD_INVALID;

and silently also remove all audio streams!!

inputstream.adaptive/src/Session.cpp

Lines 556 to 558 in 2364435

	if (session.m_decrypterCaps.flags & DRM::DecrypterCapabilites::SSD_INVALID)
	{
	m_adaptiveTree->m_currentPeriod->RemovePSSHSet(static_cast<std::uint16_t>(ses));

inputstream.adaptive/src/common/Period.cpp

Lines 69 to 83 in 2364435

	void PLAYLIST::CPeriod::RemovePSSHSet(uint16_t pssh_set)
	{
	for (std::unique_ptr<PLAYLIST::CAdaptationSet>& adpSet : m_adaptationSets)
	{
	auto& reps = adpSet->GetRepresentations();
	for (auto itRepr = reps.begin(); itRepr != reps.end();)
	{
	if ((*itRepr)->m_psshSetPos == pssh_set)
	itRepr = reps.erase(itRepr);
	else
	itRepr++;
	}
	}
	}

over the years i have forgotten about this oddity in the removal of repr, also because i didn't know where to get a suitable test stream
and this print nothing on log, so its not possible check from log, this is another bad hack shit from the old code

this lead to BIG PROBLEMS:

1. i have no idea how much efforts are required to implement CDM "secure path" for audio side, because if you try use it ffmpeg complains and throws you out...
2. this lead to difficult in to skip these audio streams that require CDM "secure path"
3. nullifies all good improvements made here about speedup playback startup and avoid opening unnecessary CDM sessions for a not real use

audio "secure path" is old feature lack in ISA forgotten over all the years,
a kind of solution can be check if there are DRM encrypted audio streams, if so, initialize all streams to CDM in similar way of the old ISA behavior

[CastagnaIT]

@Sandmann79 ok i pushed changes, need some cleaning but should be ok

unfortunately this lack in a single decryptor (widevine CDM for non-android) affects the whole DRM management, i had to reintroduce the old hack in a different way (of course that messes up a bit the code again) but at least now its more clear and print on log the disabled (unsupported) audio streams

I tested a couple of videos on windows and android and now it seems to be working fine with old and new props on windows/android

please do some tests so you can give me a feedback

i confirm also that add force_single_session parameter to True on Android will allow you to keep a single drm session (as on windows) anyway by using old props its enabled by default

[CastagnaIT]

@Sandmann79 i pushed last cleanups, tested again seem ok
when you will have time let me know

[Sandmann79]

I've tested several videos on different systems and couldn't find any issues.

Thanks for the really good work and your patience.

[CastagnaIT]

Thank you i finally got to the end of this colossal PR

[CastagnaIT]

hmf too soon i found now that at least on android there are still problems with some videos
atm its unclear whats wrong

[CastagnaIT]

ok, i found that enable 4K on am@zon addon cause decryption problems, whats weird thing
i disable it and now works perfectly as before

[mkreisl]

Hi, thanks for the tireless work here.

Could it be that with this PR the previous Python addons no longer work? In my specific case it is the simpliTV addon
On my Trixie test installation, a Kodi 22 alpha built 2 days ago with all relevant binary addons no longer works
Kodi no longer even shows an error message when starting a stream but simply does nothing except write this Depreciated warning in the log

With a slightly older Kodi 22 alpha version with correspondingly older binary addons, however, streaming works perfectly, so it must be due to some new commits

By the way, the version of libwidevine.so is 4.10.2662.3

[mirh]

Can you try #1824?

[mkreisl]

@mirh
I will try it tomorrow and report back. I'm too tired to do it today.
I built the 22.1.13 version again to be sure and tested it with the Kodi 22 Alpha from today, that works anyway

[mkreisl]

@mirh
Now I've done it after all.
Unfortunately it doesn't help, the stream doesn't start again, but I had the feeling that it was doing a bit more
I looked in the log and found this:

Apr 12 22:47:37 kmpifive 2025-04-12 22:47:37.401 T:30470   debug <general>: AddOnLog: inputstream.adaptive: Download finished: https://simplitv-live.mdn.ors.at/live/eds/servustv_fhd-1/dash/servustv_fhd-1.mpd (downloaded 16368 byte, speed 765611.00 byte/s)
Apr 12 22:47:37 kmpifive 2025-04-12 22:47:37.407 T:30470    info <general>: AddOnLog: inputstream.adaptive: Manifest successfully parsed (Periods: 1, Streams in first period: 4, Type: live)
Apr 12 22:47:37 kmpifive 2025-04-12 22:47:37.407 T:30470   debug <general>: AddOnLog: inputstream.adaptive: [Repr. chooser] Stream selection conditions
                                                   Screen resolution: 1920x1080 (may be limited by settings)
                                                   Initial bandwidth: 196188005 bit/s
Apr 12 22:47:37 kmpifive 2025-04-12 22:47:37.407 T:30470   debug <general>: AddOnLog: inputstream.adaptive: Initialize crypto session
Apr 12 22:47:37 kmpifive 2025-04-12 22:47:37.407 T:30470   error <general>: AddOnLog: inputstream.adaptive: Cannot find the cdm_aarch64_loader.so file
Apr 12 22:47:37 kmpifive 2025-04-12 22:47:37.407 T:30470   error <general>: AddOnLog: inputstream.adaptive: The DRM decrypter cannot be initialized
Apr 12 22:47:37 kmpifive 2025-04-12 22:47:37.407 T:30470   debug <general>: AddOnLog: inputstream.adaptive: GetCapabilities()
Apr 12 22:47:37 kmpifive 2025-04-12 22:47:37.407 T:30470    info <general>: Creating Demuxer

but the libcdm_aarch64_loader.so file is present

/usr/local/lib/kodi/addons/inputstream.adaptive $ ll
insgesamt 2208
lrwxrwxrwx 1 root root      28 12. Apr 22:44 inputstream.adaptive.so -> inputstream.adaptive.so.22.0
lrwxrwxrwx 1 root root      30 12. Apr 22:44 inputstream.adaptive.so.22.0 -> inputstream.adaptive.so.22.2.0
-rw-r--r-- 1 root root 2180176 12. Apr 22:44 inputstream.adaptive.so.22.2.0
-rw-r--r-- 1 root root   67192 12. Apr 22:44 libcdm_aarch64_loader.so

hth

[CastagnaIT]

there was wrong filename "cdm_aarch64_loader.so"
fix on #1825

[mkreisl]

Yeah, great 😄 Works again
Thanks