We always say that benchmarks don't matter as much as the overall experience of using a phone, but they do still matter enough that device makers sometimes put a thumb on the scale to improve scores. A recent report from Anandtech accused Huawei of configuring phones to produce artificially high benchmarks. Now, the maker of 3DMark has banned several Huawei devices in response.
The ban covers the Huawei P20 Pro, P20, Nova 3, and Honor Play. These devices will no longer appear in the list of top devices in UL Benchmarks apps (remember, UL recently acquired Futuremark). Essentially, Huawei was caught changing device performance when the phone detected a benchmark app. The phones would engage a much higher power limit for the SoC while running benchmarks in order to eke out a higher final score.
UL Benchmarks says it confirmed Anandtech's testing on three of the four devices and took its word on the fourth (the P20). By using a private version of the 3DMark benchmark, UL was able to confirm that Huawei's phones were operating differently when they detected the benchmark. In some cases, scores were 47% higher on the public app. This is considered cheating by UL because the resulting score is not indicative of the performance people can expect when using the phone.
UL says it reached out to Huawei, which has pledged to add a performance mode that lets users turn on the higher power limit. It's unclear when that will happen or how it will affect benchmarks. We've pinged Huawei for more information and will update if we hear back.
Huawei has issued a response to UL. The full text follows.
Huawei and UL (creators of 3DMark) have held comprehensive discussions on benchmarking practices this week, and have reached a positive agreement on the next steps in working together.
In the discussion, Huawei explained that its smartphones use an artificial intelligent resource scheduling mechanism. Because different scenarios have different resource needs, the latest Huawei handsets leverage innovative technologies such as artificial intelligence to optimize resource allocation in a way so that the hardware can demonstrate its capabilities to the fullest extent, while fulfilling user demands across all scenarios.
UL understands the intent of Huawei’s approach, but is opposed to forcing the use of a “Performance Mode” by default when a benchmarking application is detected by the device. UL rules require a device to run the benchmark as if it were any other application.
Huawei respects consumers’ right to choose what to do with their devices. Therefore, Huawei will provide users with open access to “Performance Mode” in EMUI 9.0, so that the user can choose when to use the maximum power of their device.
Huawei and UL have also discussed current common benchmark testing methodologies in general. UL and Huawei would like to participate in an industry movement to develop benchmarking standards that best serve the needs of manufacturers, press, and consumers.
To prevent confusion around current benchmarking results, after discussion, UL and Huawei have temporarily delisted the benchmark scores of a range of Huawei devices, and will reinstate them after Huawei grants all users of Huawei handsets access to the Performance Mode.
- UL Benchmarks