F-Droid and IzzyOnDroid can't build

[licaon-kter]

Checklist

• I am able to reproduce the bug with the latest version given here: CLICK THIS LINK.
• I made sure that there are no existing issues - open or closed - which I could contribute my information to.
• I have read the FAQ and my problem isn't listed.
• I have taken the time to fill in all the required details. I understand that the bug report will be dismissed otherwise.
• This issue contains only one bug.
• I have read and understood the contribution guidelines.
• I have read and understood the AI policy. The content of this bug report is not generated by AI.

Affected version

0.28.6

Steps to reproduce the bug

• Build APK
• Verify reproducibility

Expected behavior

It's reproducible

Actual behavior

Builds fine but the APK does not verify: https://gitlab.com/fdroid/fdroiddata/-/merge_requests/37678 and @IzzySoft sees the same issue

if I take your APK and delete MANIFEST.MF SCHABI_K.RSA SCHABI_K.SF then the resulting file matches our CI (or local) built unsigned APK

if I copy the signature from your APK to our unsigned:

apksigcopier copy NewPipe_v0.28.6.apk org.schabi.newpipe_1011.apk org.schabi.newpipe_1011_signedtest.apk

then I do:

diff-zip-meta.py NewPipe_v0.28.6.apk org.schabi.newpipe_1011_signedtest.apk > difftest.log (from here)

difftest.log: diff.log

when doing the copy or comparison via apksigcopier we see that 2915 offset between positions

looking at your CI .yml history I don't see anything changed since .5, and I've looked at the changelog of ubuntu-latest image but no Android tooling was changed iirc

Any ideas? Is the APK built by the GHA CI?

Screenshots/Screen recordings

No response

Logs

No response

Affected Android/Custom ROM version

No response

Affected device model

No response

Additional information

No response

[Stypox]

The APK does come from the CI, in particular this run: https://github.com/TeamNewPipe/NewPipe/actions/runs/25217410277.
I guess the SCHABI_K.RSA and SCHABI_K.SF have to do with the signature itself, so maybe something changed in the way we signed the APK?
Thanks for letting us know

[Stypox]

Yeah those three files are indeed not present in the unsigned APK but only in the signed one. The content of MANIFEST.MF and SCHABI_K.SF is SHA-256 digests of all other files in the APK (apparently), and the content of SCHABI_K.RSA is DER Encoded PKCS#7 Signed Data which can be read with openssl pkcs7 -in SCHABI_K.RSA -inform DER -print and I guess is a signature. I wonder how they ended up there...

[licaon-kter]

We are lost in translation? 😄

your upstream APK is signed, as expected: https://github.com/TeamNewPipe/NewPipe/releases/tag/v0.28.6

both F-Droid and IzzyOnDroid, use that APK to verify reproducibility against it.

we build the unsigned APK, extract the signature part from your APK then we try to put it in our unsigned APK.... to reproduce ... yours back (eg. via https://github.com/obfusk/apksigcopier#copyextractpatch-android-apk-signatures--compare-apks)

we've done this since forever for NewPipe and we do this for other apps too, works fine

but not today

[IzzySoft]

To put it into other words: Your APK and ours are matching, except for the signature part. Comparing the signed ones, it seems your certificates are padded with a bunch of zeros, see the difftest.log in this post. I assume this might be new shenanigans introduced by apksigner (like it introduced other padding in the past). So first question here is: Do you use a different version of apksigner than you did with previous builds? Or changed the signing step in other ways?

[Stypox]

Sorry for my previous comment, I completely misunderstood 🤦

Anyway, the person who signed the APK reports apksigner --version -> 0.9, and supposedly it hasn't changed since previous versions.

[licaon-kter]

Where does that come from? Eg. Debian has versions matching Android platform tools: https://packages.debian.org/trixie/apksigner so 35 is expected, but 0.9 sounds odd

[IzzySoft]

Might sound odd, but…

$ apksigner --version
0.9

That is jammy, same on trixie.

Asking dpkg:

ii  apksigner      31.0.2-1     all          command line tool to sign and verify Android APKs
ii  apksigner      35.0.2-1     all          command line tool to sign and verify Android APKs

So both, 31+35, identify as 0.9. I bet so do 36+37. Odd as it sounds…

Can "the person who signed" check with e.g. dpkg -l apksigner which one is installed, @Stypox ?

[IzzySoft]

As it doesn't seem we can expect any answer soon, we'll now mark this release "failed" at IzzyOnDroid. Looking forward to the next one, which hopefully has this fixed.

Btw, according to the commit history, last changes to apksig were more than a year ago, with the last changes to apksigner itself in 10/2024. So it's rather unlikely the issue is due to changes at that part (especially as we didn't see any comparable thing with any of the other almost 900 apps we build here). Which means, it could rather be something in the signing process itself that changed, maybe?

[licaon-kter]

Which OS is the signing being done? Maybe this is like the zlib-ng issue on latest Fedora

[TheAssassin]

The signing system is currently operating Ubuntu 24.04.4 with apksigner 31.0.2-1ubuntu1.

[IzzySoft]

Strange. That's basically what we have here for quite a while. Did anything change in the signing process itself? Any other steps performed after the GHA spit out the unsigned APK? And while it might sound stupid, just to make sure: if you take the same unsigned APK from the artifacts here, sign it again, and run sha256sum *.apk – is the resulting hash identical to that of the APK attached to the release here? Just wondering if "something strange slipped in", and how. So if you could list all steps performed from the unsigned APK to the one attached at the release, that would be much appreciated, and hopefully shed some light. Together with the sha256sum "experiment". Thanks in advance!

[licaon-kter]

In my test, fyi, just deleting META-INF from the upstream signed APK created a sha256sum identical with my local unsigned built APK.

Will retest asap to confirm