Though individual phone manufacturers have been accused of cheating on benchmarks in the past, it's usually a relatively localized problem: Only certain phones from certain companies were disingenuously using a higher-performing mode when running certain apps. But according to a report by Anandtech, MediaTek has integrated this sort of "cheating" into the software it gives phone manufacturers to use its chipsets. Essentially, allmost all MediaTek phones running certain chipsets are "cheating" on benchmarks.
It's a long and great story that I encourage anyone interested in technical details to read, but the short version is that Anandtech noticed that MediaTek's Dimensity 1000L was outperformed in benchmarks by the older, lower-end Helio P95 in an OPPO Reno3 Pro. Shenanigans were confirmed when a version of the benchmarking app tweaked to prevent it from being identified by the system returned lower results for the P95. That means something in the Helio P95-powered phone was dramatically cranking performance in response to the presence of the benchmarking app. And this isn't a small difference, Anandtech observed a 30% drop in general tests, with up to a 70% change in specific workloads.
Table of devices Anandtech tested. Image via Anandtech.
The logic that controls this behavior is baked into the firmware and targets plenty of benchmarks, including highly niche ones only used by a few sites. Taking a wider look at other MediaTek-powered devices in recent years, Anandtech found this behavior was present on basically all of them. It wasn't exclusive to the Reno3 Pro they first tested, nine different devices with nine different MediaTek chipsets had it to varying degrees. That consistency across devices meant that it wasn't just OPPO acting on its own, this behavior originated from MediaTek itself, and more recent changes indicate the company may even be trying to hide it.
MediaTek responded to Aanadtech's story with its own statement explaining its actions, included in a blog post that further details its position. Essentially, the company's arguments boil down to:
- Chipsets already scale in performance and power for other tasks, so this isn't different.
- Phone manufacturers can disable it if they don't want it.
- Other companies also do it.
The first point isn't an apples-to-apples comparison, it's not as if this high-performance mode is user-facing, and it's not really equivalent to the dynamic scaling you might observe in other situations. But the other two claims, unfortunately, are true. Huawei, OPPO, and Samsung were all previously accused of similar behavior, and Samsung even had to settle a lawsuit over it last fall — so maybe "everyone else is doing it" isn't the best defense. And no matter who left the feature enabled or who does it, it's still misleading.
With many of us hoping and waiting on MediaTek to bring a high-performing chipset to the US, "cheating" on benchmarks in this way does more harm than benefit. We should all be talking about how the Dimensity 1000 stacks up against Qualcomm, but instead, we're focusing on how MediaTek is manipulating software to lie about performance in benchmarks. Hopefully, the company reconsiders the practice in the future so we can all focus more on what actually matters: Bringing competition back to the US smartphone chipset market. Especially given there's a real chance MediaTek could even get sued over this like Samsung did.