Samsung was the first to selectively boost system performance when a benchmark app was run, but it was forced to backpedal pretty quickly on that one. The latest OEM to try and sneak one past the benchmarks is Huawei with its new-ish Ascend P7. Futuremark is wise to this game, though, and has pulled the P7 from the 3DMark top phone charts.

2014-08-29 10_09_13-Best Smartphones and Tablets August - 2014

The Ascend P7 isn't running a top-of-the-line ARM chip like the Snapdragon 801 and Tegra K1 devices that dominate the charts. Instead, it packs a budget-friendly Kirin 910 with four Cortex-A9 cores clocked to a maximum of 1.8GHz and a Mali-450 GPU. Even with the modest hardware, the Ascend P7 was doing respectably in benchmarks, but that was all a farce. As detailed by Anandtech, the device detects when a known benchmarking app is running and manually keeps all four CPU cores cranked until the test has completed. This is not representative of the workload performance in other apps.

The shenanigans were verified by changing the package name of 3DMark and comparing the resulting score to the one obtained in the regular Play Store version. Futuremark got a baseline of 5816 in the renamed version and 7462 in the Play Store version. The company has a set of rules for OEMs which require they avoid benchmark-specific optimizations, so Huawei's device is out. Bummer, but maybe they'll knock off the cheating now. The device is at the bottom of the phone list with no score, right along side other devices that failed to follow the rules, including the HTC One M8 and Galaxy Note 10.1.

[3DMark Top Devices]

Ryan Whitwam
Ryan is a tech/science writer, skeptic, lover of all things electronic, and Android fan. In his spare time he reads golden-age sci-fi and sleeps, but rarely at the same time. His wife tolerates him as few would.

He's the author of a sci-fi novel called The Crooked City, which is available on Amazon and Google Play.

  • Ajb.esquire

    I've had a huawei before, OK device for what you pay, but you get what you pay for. That cheap internals combined with the companies questionable practice's in the past will definitely keep me from buying one of their products again.

  • Melad360

    how do people call this cheating?? its not cheating she you're benchmarking your phone to see what the processor is capable of doing. and unless the processor and GPU are both at max frequency you're not gunna see what your devices full potential is.

    • Luke

      I think the problem is the device will never run that way any other time. It will only do so in selected benchmarks. Thus the user never gets a true view of how the device performs in every other application they may encounter. Example; It could be faster than __ device in benchmark but slower than __ device in every single non benchmark app.

      • Melad360

        true, but what other daily use application is as processor intensive as a benchmarking app? for example my nexus 5 maybe doesn't get the best benchmark scores for CPU or GPU, especially with its heavy thermal throttling, but in daily use with stock android it runs smoother and with less lag than like pretty much any other phone out there

        • Jason B

          And that's because it's not being throttled under light loads. Thermal throttling is just that - to control heat. If you disable the power saving features, and the SoC reaches 70C, what then? It gets extremely unstable as heat increases electrical resistance.

          • DarkStarr

            The stock throttle values for many devices is actually higher than 70C so...

    • URABUS0924

      I don't see what the issue is either. No one has a problem with this when doing PC benchmarking, unless ramping up your overclocked, liquid cooled CPU/GPUs is what everyone does when they log into Facebook. Maybe a better solution would be for the app to show what clock speed the CPU and GPU are running at during the benchmarking and if that is stock or not.

      • Melad360

        yeah exactly lol, or maybe having an option in the benchmark to set the cores to the max frequency or something, butt hat probably won't happen. either way though nowadays people don't tend to make buying decisions based on benchmarks nearly as much as they did years ago when dual core phones were becoming the new thing , but rather on software and everyday use. that's me anyway

    • Đức Thành

      Obviously because OTHER DEVICES don't do the same (try to push the device to its full potential capability) when taken under the same test. It is like saying doping in sports isn't cheating because it only helps the athletes display their full potential. The problem is, while they are capable of doing that when at their full potential, they aren't normally, and definitely can't keep that up in everyday use.

      The purpose of benchmark tests is to see how well a device performs and compare that to other devices, so they must be exposed to the same condition. If you don't consider this cheating, I don't quite agree but you and I can still come to a compromise: Perhaps should dope every athlete to make the competition fair, and max out every device that's put to the same test to "see their full potentials". After all, when everybody cheats, nobody really does.

      • Melad360

        I completely agree with you, however this is a benchmark app, and benchmark apps measure the full potential of your system. a benchmark isn't an every day app, so it shouldn't really be using the processor at its every day state, but rather in its max potential state. if you want to compare day to day activities and see how it performs in real world situations, just go down to your local phone store and play around with them for a few minutes. every day tasks should be dealt with every day processor clock speeds and such, but benchmarking apps that are meant to see the maximum power your setup can deliver should be met with the maximum clock speeds etc. so yes if every manufacturer "cheats" and puts their CPU in the max frequency then it'll be fair across the board vs only one or 2 manufacturers "cheating" and the rest doing it without. now if the manufacturer over clocks the CPU past what it's advertised when it sees a benchmark app, then yes, that should be considered cheating

        • Đức Thành

          That's why I proposed that "compromise". I only intended that as a joke, mind you, because no matter what I still see this as cheating. The problem here is that they purposefully clocks this device to make it seem better than it would be if it were to perform normally in normal conditions, like how every other devices do, thus gain itself a superficial advantage over other products. Thought that was obvious already. Oh wait, basically I am just paraphrasing myself... Here, lemme make the explanation of what I was getting at a bit simpler:

          It's not cheating if the test is just to see the potential performance of a certain device and only itself, yes. (And I think this is the point why you insist on saying it's not cheating.)

          However, when its score is compared to other devices' that don't run in the same advantageous condition, it is cheating.

          Here, it IS being compared using its score. And that is my point.

          • Melad360

            yeah I agreed with you on that, that's why I said if everyone puts the CPU to the max clock on benchmarks then it'll be fair

    • Serge Cebrian

      Exactly what i say when i see the benchmaks i want to know full potential ... Not regular prosessing poder..

    • Jason B

      Because they're forcefully bypassing the SoC's power saving features and TDP. An SoC with a rated TDP meets that heat rating through various means, usually clock/voltage gating, frequency scaling, and core control.

      If the SoC has a rated TDP of 3W, and you forcefully bypass all of the power saving features built-in, you also force the SoC to have a higher TDP out of its specified range.

      So, say that doubles its TDP to 6W, how do you control the heat in a phone without active cooling? You can't. How do you meet advertised battery life? You can't. It's cheating because you can't run the SoC like that all the time (unless you like power cords, but then it's not truly mobile, is it?).

    • DarkStarr

      Exactly. I have said it before and ill say it again. You don't see professional overclockers on PCs letting the CPU downclock during benching, they disable all of the powersaving and have it running at the full speed (plus quite a bit usually). Benchmarks (especially synthetics) are fucking horrible to test real world performance, they are designed to test the maximum performance of the hardware.

  • Matthew Fry

    When will these OEMs learn... to not just key off of the application name alone.

  • mustbepbs
  • Reg Joo

    The problem with cheating with benchmarks, is that the score is used as a comparison to other phones. There has to be a level playing field between mfg's phones, or you couldn't compare real world performance. Stringent standards protect the public from being duped, and this is going in the right direction for us. Đức Thành , is fairly right, and depends on real-world performance, which is all that matters.

    • Đức Thành

      Thank you, couldn't have phrased it better myself.

  • Reg Joo

    Why would you use your phone running at full tilt, all the time? Not only would it overheat, but the battery would have to be charged all the time. Not a real-world scenario, and who could trust a benchmark that doesn't understand real-world usage? The people that make these bench apps, understand this, and would lose credibility, if they did otherwise.

  • AntiCheat
  • fonix232

    Again then. Benchmark should represent the TOP of what a specific phone can do, not average. I think the manufacturers cranking up the phones to do their best on benchmarks is a kinda good thing - otherwise the default governor results in a kind of hit-and-miss result. Just for example, using ondemand will result in 5 tests 5 heavily different results - depending on how much the device is used by other apps, background services and so.

    While if you do a little trickery and kill all apps and bg processes, and put the performance governor on, it shows what the hardware is really capable of - and that is the goal of the hardware.

    Based on this, any modified laptop/PC results must be removed from the database, Futuremark! Even if it's a slight under/overclock!