Readit News logoReadit News
Posted by u/nativeforks 11 days ago
F-Droid build servers can't build modern Android apps due to outdated CPUs
On August 7, 2025, a new build problem started hitting many Android apps on F-Droid. Many Android apps on F-Droid have been unable to publish updates if they use Android Gradle Plugin (AGP) 8.12.0 or Gradle 9.0.

The root cause: Google’s new aapt2 binary in AGP 8.12.0 started requiring CPU instructions (SSE4.1, SSSE3) that F-Droid’s build farm hardware doesn’t support. This is similar to a 2021 AGP 4.1.0 issue, but it has returned, and now affects hundreds of apps.

As an example, my open-source app MBCompass hit this issue. I downgraded to AGP 8.11.1 with Gradle 8.13 to make it build, but even then, F-Droid failed due to a baseline profile reproducibility bug in AGP. The only workaround was disabling baseline profiles and pushing yet another release.

This has led to multiple “maintenance” versions in a short time, confusing users and wasting developer time, just to work around infrastructure issues outside the developer’s control.

References:

- F-Droid admin issue: https://gitlab.com/fdroid/admin/-/issues/593 - Catima example: https://github.com/CatimaLoyalty/Android/issues/2608 - MBCompass case: https://github.com/CompassMB/MBCompass/issues/88

csdreamer7 · 11 days ago
This means their servers are very old ones that do not support x86-64-v2. Intel Core 2 Duo days?

https://developers.redhat.com/blog/2021/01/05/building-red-h...

Think of how much faster their servers would be with one of those Epyc consumer cpus.

I was about to ask people to donate, but they have $80k in their coffers. I realize their budget is only $17,000 a year, but I am curious why they haven't spent $2-3k on one of those Zen4 or Zen5 matx consumer Epyc servers as they are around under $2k under budget. If they have a fleet of these old servers I imagine a Zen5 one can replace at least a few of them and consume far less power and space.

https://opencollective.com/f-droid#category-BUDGET

Not sure if this includes their Librapay donations either:

https://liberapay.com/F-Droid-Data/donate

bayindirh · 11 days ago
> This means their servers are very old ones that do not support x86-64-v2. Intel Core 2 Duo days?

This is not always a given. In our virtualization platform, we have upgraded a vendor supplied VM recently, and while it booted, some of the services on it failed to start despite exposing a x86_64v2 + AES CPU to the said VM. Minimum requirements cited "Pentium and Celeron", so it was more than enough.

It turned out that one of the services used a single instruction added in a v3 or v4 CPU, and failed to start. We changed the exposed CPU and things have returned to normal.

So, their servers might be capable and misconfigured, or the binary might require more that what it states, or something else.

lucb1e · 11 days ago
A developer on the ticket writes: "Our machines run older server grade CPUs, that indeed do not support the newer SSE4_1 and SSSE3"
ignoramous · 11 days ago
> about to ask people to donate, but they have $80k in their coffers

I'd still ask folks to donate. £80k isn't much at all given the time and effort I've seen their volunteers spend on keeping the lights on.

From what I recall, they do want to modernize their build infrastructure, but it is as big as an investment they can make. If they had enough in their "coffers", I'm sure they'd feel more confident about it.

It isn't like they don't have any other things to fix or address.

csdreamer7 · 11 days ago
I would too but do you have a link to them talking about it?
Timshel · 11 days ago
$2-3k ? That’s barely the price of a lower end Threadripper bare cpu not a full Epyc server ???
wongarsu · 11 days ago
At our supplier $2k would pay for a 1U server with a 16 core 3GHz Epyc 7313P with 32GB RAM, a tiny SSD and non-redundant power.

$3k pays for a 1U server with a 32 core 2.6GHz Epyc 7513 with 128GB RAM and 960GB of non-redundant SSD storage (probably fine for build servers).

All using server CPUs, since that was easier to find. If you want more cores or more than 3GHz things get considerably more expensive.

c0balt · 11 days ago
Low end EPYC (16-24 cores) especially for older generations are not that expensive 800-1.2K ime. Less when in a second hand server.
doublepg23 · 11 days ago
Perhaps the servers run Coreboot / Libreboot?
pclmulqdq · 11 days ago
I'm not even sure mainline Linux supports machines this old at this point. The cmpxchg16b instruction isn't that old, and I believe it's required now.
cwillu · 11 days ago
CMPXCHG8B is required as of a month or two ago, not 16B (i.e., the version from the 90's is now required)

See https://lkml.org/lkml/2025/4/25/409

csdreamer7 · 11 days ago
32 bit Linux is still supported by the kernel and Debian, Arch, and Fedora still supports baseline x86_64.

RHEL 8 is still supported and Ubuntu is still baseline x86_64 I believe for commercial distros. Not sure about SuSE.

FirmwareBurner · 11 days ago
>they have $80k in their coffers but I am curious why they haven't spent $2-3k on one of those Zen4 or Zen5 matx consumer Epyc servers

I would also like to know this.

Perz1val · 11 days ago
Yeah and everybody was complaining how slow the builds are for years. I really want to know too
pastage · 11 days ago
I would much rather they spent that on having the devs network and travel, the servers work.
lupusreal · 11 days ago
Probably a case of "don't fix it if it ain't broke" keeping old machines in service too long, so now they broke.
benrutter · 11 days ago
This is pretty concerning, especially as FDroid is by far the largest non-google android store at the moment, something that I feel is really needed, regardless of your feelings about google.

Does anyone know of plans to resolve this? Will FDroid update their servers? Are google looking into rolling back the requirement? (this last one sounds unlikely)

dannyw · 11 days ago
I agree it’s a bit concerning but please keep in mind F-Droid is a volunteer-run community project. Especially with some EU countries moving to open source software, it would be nice to see some public funding for projects like F-Droid.
berkes · 11 days ago
> please keep in mind F-Droid is a volunteer-run community project.

To, me, that's the worrying part.

Not that it's ran by volunteers. But that all there's left between a full-on "tech monopoly" or hegemony, and a free internet, is small bands of underfunded volunteers.

Opposition to market dominance and monopolies by multibillion multinationals shouldn't just come from a few volunteers. If that's the case, just roll over and give up; the cause is lost. (As I've done, hence my defaitism)

Aside from that: it being "a volunteer ran community" shouldn't be put as an excuse for why it's in trouble/has poor UX/is hard to use/is behind/etc. It should be a killer feature. Something that makes it more resilient/better attuned/easier/earlier adopting/etc.

benrutter · 11 days ago
Hope I didn't come across as criticising FDroid here- It seems sucky to have build requirements change under your feet.

It's just I think that FDroid is an important project, and hope this doesn't block their progress.

nativeforks · 11 days ago
> Nice to see some public funding for projects like F-Droid

Definitely, SSE4.1 instruction set based CPU, for building apps in 2025, No way!!

happosai · 11 days ago
Maybe if f-droid is important to you, donate, so they can buy newer build server?
benrutter · 11 days ago
I'm not quite sure if I'm over reading into this, but this comes across as a snarky response as if I've said "boo, fdroid sucks and owes me a free app store!".

Appologies if I came across like that, here's what I'm trying to convey:

- Fdroid is important

- This sounds like a problem, not necessarily one that's any fault of fdroid

- Does anyone know of a plan to fix the issue?

For what it's worth, I do donate on a monthly basis to fdroid through liberapay, but I don't think that's really relevant here?

nativeforks · 11 days ago
This has now become a major issue for F-Droid, as well as for FOSS app developers. People are starting to complain about devs because they haven't been able to release the new version for their apps (at least it doesn't show up on F-Droid) as promised
chasil · 11 days ago
Is Westmere the minimum architecture needed for the required SSE?

Server hardware at the minimum v2 functionality can be found for a few hundred dollars.

A competent administrator with physical access could solve this quickly.

Take a ReaR image, then restore it on the new platform.

Where are the physical servers?

ratdragon · 11 days ago
Did and doing regularly.
Jyaif · 11 days ago
> FDroid is by far the largest non-google android store at the moment

Not even sure it's in the top 10

msgodel · 11 days ago
I think we only know about F-Droid because it's the only high quality one.

Low quality software tends to be popular among the general public because they're very bad at evaluating software quality.

benrutter · 11 days ago
Wait really? What other ones are there!? Somebody's already pointed out Samsumg Galaxy store, but I don't think I know of others?

Edit: searching online found this if anyone else is interested https://www.androidauthority.com/best-app-stores-936652/

Suppafly · 7 days ago
>This is pretty concerning, especially as FDroid is by far the largest non-google android store at the moment

That's almost certainly not true.

charcircuit · 11 days ago
>FDroid is by far the largest non-google android store at the moment

Samsung Galaxy Store is much much bigger.

ykonstant · 11 days ago
Funny true story: I got my first smartphone in 2018, a Samsung Galaxy A5. I have it to this day, and it is the only smartphone I ever used. This is the first time I hear about Samsung Galaxy store! (≧▽≦)
ozim · 11 days ago
Largest not run by the corporations then ;)
benrutter · 11 days ago
Yup! I missed that one because I didn't realise it still existed. Woops!
lucb1e · 11 days ago
> Are google looking into rolling back the requirement? (this last one sounds unlikely)

That's apparently what they did last time. From the ticket:

"Back in 2021 developers complained that AAPT2 from Gradle Plugin 4.1.0 was throwing errors while the older 4.0.2 worked fine. \n The issue was that 4.1.0 wanted a CPU which supports SSSE3 and on the older CPUs it would fail. \n This was fixed for Gradle Plugin 4.2.0-rc01 / Gradle 7.0.0 alpha 9"

1oooqooq · 11 days ago
why you read "google build tools cannot be built from source and it was compiled with an optional optimizations as required" and assume the right thing to do is to buy newer servers?
benrutter · 10 days ago
I'm not assuming anything, this is from a ticket for fdroid on google:

> Our machines run older server grade CPUs, that indeed do not support the newer SSE4_1 and SSSE3.[0]

I.e. the problem is because fdroid have older CPUs, newer ones would be able to build. I only mentioned it in terms of what the plans to fix might be. I have zero idea if upgrading servers is the best way to go.

[0] https://issuetracker.google.com/issues/438515318?pli=1

ivanjermakov · 11 days ago
Why not recompile aapt2 to correct target? It seems to be source available.

https://android.googlesource.com/platform/frameworks/base/+/...

munchlax · 11 days ago
Have you tried building AOSP from available sources?

Binaries everywhere. Tried to rebuild some of them with the available sources and noped the f out because that breaks the build so bad it's ridiculous.

zoobab · 11 days ago
"Binaries everywhere"

So much for "Open Source"

rbanffy · 11 days ago
Yes. Sources available means nothing without a reproducible build process.
ivanjermakov · 11 days ago
So open source is only in the name, noted
pwdisswordfishz · 11 days ago
Debian also seems to have given up.
ethan_smith · 11 days ago
Using Docker with QEMU CPU emulation would be a more maintainable solution than recompiling aapt2, as it would handle future binary updates automatically without requiring custom patches for each release.
mjevans · 11 days ago
https://en.wikipedia.org/wiki/Streaming_SIMD_Extensions#Late...

Even my last, crazy long in the tooth, desktop supported this and it lived to almost 10 years old before being replaced.

However at the same time, not even offering a fallback path in non-assembly?

wtallis · 11 days ago
> However at the same time, not even offering a fallback path in non-assembly?

There's probably not any hand-written assembly at issue here, just a compiler told to target x86_64-v2. Among others, RHEL 9 and derivatives were built with such options. (RHEL 10 bumped up the minimum spec again to x86_64-v3, allowing use of AVX.)

shadowgovt · 10 days ago
Or even, a compiler told to target nothing in particular, and a default finally toggled over from "Oh, we're 'targeting x86'? So CPUs from the early 2000s then" to "Oh, we're 'targeting x86'? So CPUs from the mid-2010s then."
vocx2tx · 11 days ago
Looking at the issue their builders seem to be Opterons G3 (K10?)[0]

[0] https://en.wikipedia.org/wiki/AMD_10h

SSLy · 11 days ago
at this point they're guzzling so much power the electricity is more expensive than replacement platform
ozim · 11 days ago
I can imagine this has to be like that as they usually get $1500 per month in donations.

You could buy a newer one but I guess they have other stuff they have to pay for.

yonatan8070 · 11 days ago
I have a home server with a 9th gen i7 that's doing jack sh!t most of the time, is there a way to donate some compute time to build F-Droid packages?
CJefferson · 11 days ago
The problem with offering fallbacks is testing -- there isn't any reasonable hardware which you could use, because as you say it's all very old and slow.

Deleted Comment

pestatije · 11 days ago
I'm sure theyll appreciate your old desktop donation
karteum · 11 days ago
I don't fully understand: aren't gradle and aapt2 open-source ?

If you want to build buildroot or openwrt, the first thing it will do is compiling your own toolchain (rather than reusing the one from your distro) so that it can lead to predictable results. I would have the same rationale for f-droid : why not compile the whole toolchain from source rather than using a binary gradle/aapt2 that uses unsupported instructions?

a2128 · 11 days ago
SDK binaries provided by Google are still used, see https://forum.f-droid.org/t/call-for-help-making-free-softwa...
mid-kid · 11 days ago
I agree, this should be the case, but Gradle specifically relies on downloading prebuilt java libraries and such to build itself and anything you build with it, and sometimes these have prebuilt native code inside. Unlike buildroot and any linux distribution, there's no metadata to figure out how to build each library, and the process for them is different between each library (no standards like make, autotools and cmake), so building the gradle ecosystem from source is very tedious and difficult.
1oooqooq · 11 days ago
having worked with both mvn and gradle, i always have a good chuckle when i hear about npm "supply chain" hacks.
jchw · 11 days ago
Apparently it was fixed upstream by Google?

https://gitlab.com/fdroid/admin/-/issues/593#note_2681207153

Not sure how long it will take to get resolved but that thread seems reassuring even if there isn't a direct source that it was fixed.

AnssiH · 11 days ago
It is not fixed.

In the thread you linked to people are confusing a typo correction ("mas fixed" => "was fixed") as a claim about this new issue being fixed.

The one that was fixed is this similar old issue from years ago: https://issuetracker.google.com/issues/172048751

jchw · 11 days ago
Oh, that's unfortunate, very confusing thread.
nativeforks · 11 days ago
Still haven't. Currently, most of the devs aren't aware of this underlying issue!
micw · 11 days ago
As far as I can see, sse4.1 has been introduced in CPUs in 2011. That's more than 10 years ago. I wonder why such old servers are still in use. I'd assume that a modern CPU would do the same amount of work with a fraction of energy so that it does not even make economical sense to run such outdated hardware.

Does anyone know the numbers of build servers and the specs?

eadmund · 11 days ago
> I'd assume that a modern CPU would do the same amount of work with a fraction of energy so that it does not even make economical sense to run such outdated hardware.

There are 8,760 hours in a non-leap year. Electricity in the U.S. averages 12.53 cents per kilowatt hour[1]. A really power-hungry CPU running full-bore at 500 W for a year would thus use about $550 of electricity. Even if power consumption dropped by half, that’s only about 10% of the cost of a new computer, so the payoff date of an upgrade is ten years in the future (ignoring the cost of performing the upgrade, which is non-negligible — as is the risk).

And of course buying a new computer is a capital expense, while paying for electricity is an operating expense.

1: https://www.eia.gov/electricity/monthly/epm_table_grapher.ph...

wang_li · 11 days ago
You can buy a mini pc for less than $550. For $200 on Amazon you can get an N97 based box with 12 GB RAM and 4 cores running at 3 GHz and a 500 GB SATA SSD. That’s got to be as fast as their current build systems and supports the required instructions.
adrian_b · 11 days ago
It has been introduced in Intel Penryn, in November 2007.

However the AMD CPUs did not implement it until Bulldozer, in mid 2011.

While they lacked the many additional instructions provided by Bulldozer, also including AVX and FMA, for many applications the older Opteron CPUs were significantly faster than the Bulldozer-based CPUs, so there were few incentives for upgrading them, before the launch of AMD Epyc in mid 2017.

SSE 4.1 is a cut point in supporting old CPUs for many software packages, because older CPUs have a very high overhead for divergent computations (e.g. with if ... else ...) inside loops that are parallelized with SIMD instructions.

cjaackie · 11 days ago
I haven’t seen the real answer that I suspect here - the build servers are that one dual socket AMD board which runs open firmware and has no ME/PSP .
ffaser5gxlsll · 11 days ago
On the server side, probably not, but I'd like to point out that old hardware is not uncommon, and it's going to be more and more likely as time passes especially in the desktop space.

I was hit by this scenario in the 2000s with an old desktop pc I had, also in the 10ys range, I was using just for boring stuff and random browsing, which was old, but perfectly adequate for the purpose. With time programs got rebuilt with some version of SSE it didn't support. When even firefox switched to the new instruction set, I had to essentially trash a perfectly working desktop pc as it became useless for the purpose.

LukeShu · 11 days ago
I was going to say that I assume that the reason for such old CPUs is the ability to use Canoeboot/GNU Boot. But you absolutely can put an SSE4.2 CPU in a KGPE-D16 motherboard. So IDK.
whizzter · 11 days ago
Because setting up servers is an annoying piece of grunt-work that people avoid doing more than absolutely necessary, there's an reason the expensive options of AWS,Azure and Google cloud make money because much "just works" when focusing on applications rather than the infra (until you actually need to do something advanced and the obscure commands or clicking bites you in the ass).
heavyset_go · 11 days ago
Hardware after the first couple of generations of x86_64 muliticore processors are perfectly capable machines to use as servers, even for tasks you want to put off to a build farm.
Pyrodogg · 11 days ago
A few months ago Adobe finally updated Lightroom Classic to require these processor extensions. To squeeze all of the matrix mults it can for AI features also in CPU mode.

It's amazing how long of a run top end hardware from ~2011 has had (just missed the cutoff by a few months). It's taken this long for stuff to really require these features.

nativeforks · 11 days ago
benrutter · 11 days ago
The Catima thread makes FDroid sound like a really difficult commmunity to work with. Although I'm basing this on one person's comment and other people agreeing, not on any knowledge or experience.

> But this is like everything with F-Droid: everything always falls on a deaf man's ears. So I would rather not waste more time talking to a brick wall. If I had the feeling it was possible to improve F-Droid by raising issues and trying to discuss how to solve them I wouldn't have left the project out of frustration after years of putting so much time and energy into it.

eptcyka · 11 days ago
F-droid are thoroughly understaffed and yet incredibly ambitious and shrewd around their goals - they want to build all the apps in a reproducible manner. There’s lots of friction around deviating from builds that fit within their model. The system is also slow, takes a long while before a build shows up. I think f-droid could benefit immensely from more funding, saying that as someone who has never seen f-droid’s side, but have worked on an app that was published there.
ohdeargodno · 10 days ago
There's a bunch of stupid behaviors all around (running AGP in alpha being one), but F-Droid asking maintainers to disable baseline profiles because it breaks reproductibility for them is thoroughly stupid and demanding.
typpilol · 11 days ago
I saw that too and was wondering what kind of drama happened in the past