04
Mar
vin

Update: Developer Wanam has confirmed on Twitter that the code responsible for the "boosting" behavior has been removed in the Android 4.4 ROMs for both the Note 3 and Galaxy S4.

Months after the Galaxy S4 was released last year, allegations began surfacing from Anandtech that Samsung was essentially "gaming" its devices' CPU and GPU benchmark scores by leaving cores at "full throttle" when such benchmarking applications were launched. This allowed Samsung to achieve marginally higher scores in such tests than its rivals, and gave the Galaxy S4 and Note 3 the appearance of being a little quicker than a typical benchmarking environment would otherwise show them to be.

Benchmark-gate initially made waves in the tech press, but was rather quickly forgotten because, well, they're benchmarks - who really cares? Was the practice a bit deceptive - scummy, even? Sure. But there are no established rules for benchmarking, and really, if you're considering buying a phone based on a benchmark score, you probably need to evaluate your priorities.

Still, it was a dumb, dastardly thing to do, particularly considering the only people really paying close attention to benchmarks are rabid fanboys and in-depth technical reviewers like the team over at Anandtech. In short, it was a big PR gaff for an extremely small - practically nonexistent - payoff.

Samsung never really fully accepted responsibility for the practice, or admitted to any outright wrongdoing, but as this story goes to show: actions speak louder than words. The Android 4.4 updates to the Galaxy S4 and Note 3 don't just bring a healthy dose of KitKat, no no - Samsung's benchmark shenanigans have been eradicated according to tests conducted by arstechnica and Geekbench.

gs4

Geekbench's data for Galaxy S4 scores by Android version

The results of the testing were pretty conclusive, too; Geekbench scores for both the Note 3 and Galaxy S4 running Android 4.4.2 are significantly lower than those obtained when the devices ran Android 4.3. Geekbench assisted Ars' investigation by examining not just the scores of individual devices, but rather the full dataset of GS4s and Note 3s Geekbench has accumulated, making the results fairly difficult to dispute.

The conclusion, then, is pretty obvious: Samsung removed the offending code which caused the S4 and Note 3 to behave differently in benchmarking apps. Somehow I doubt they'll be trying that again anytime soon.

arstechnica

David Ruddock
David's phone is whatever is currently sitting on his desk. He is an avid writer, and enjoys playing devil's advocate in editorials, and reviewing the latest phones and gadgets. He also doesn't usually write such boring sentences.

  • Sean Lumly

    I would hardly call the reduction in geekbench score significant. It's around 1% for the single core, and 10% for the multi-core test. These are insignificant values, which was the conclusion that Anandtech originally made for the case against benchmark cheating in the first place.

    In any case, it's good to know that the benchmark results are more trustworthy. All major devices are close enough in performance that these small differences would hardly matter in actual practice.

    • http://www.androidpolice.com/ David Ruddock

      Statistically, they're significant. Practically, of course not.

      • Sean Lumly

        Fair point. I hope this becomes a trend as 'benchmark optimizing' seems like an awful waste of resources, and really unnecessary code bloat.

        • curious goerge

          I agree it's pointless, but it's a few, very simple lines of code. If by "resources" you mean 5 minutes worth of time, then yes, they wasted resources.

  • Fatal1ty_93_RUS

    Serves Samsung right, cheats are for losers

    • Sean Lumly

      At this point, your irrational hatred towards Samsung can only be described as impressive.

    • http://dexterslab.wikia.com/wiki/Mandark Mandark

      How is pushing the CPU and GPU to its full potential in benchmarking cheating???

      Yours Sincerely,
      Mandark, Evil Boy Genius

    • invinciblegod

      I don't know how this counts as "serves them right" considering the fact that they were not punished, and they removed them on their own accord.

  • curious george

    I still don't understand all the outrage over this. I think all of the people out there yelling with pitchforks in hand just don't know exactly what the "cheat" does.

    • NorCalAngler

      I think most of us know what it does. It raises the thermal threshold for gpu/cpu throttling for those specific programs. The performance allowed to those programs is not allowed under any other circumstance without a custom kernel so the results are inflated beyond how the device performs for any other app. Graphics cards tried to pull this same crap years ago and consumers were rightfully ticked off and they changed their practices. This isn't game-specific optimization, this is allowing specific programs to surpass thermal thresholds, which are in place for power saving and device longevity.

      • Grayson

        Exactly. If they raised the thermal threshold for all 3D games and CPU intensive applications, then I wouldn't have a problem with them doing it for benchmarks (although battery life would probably suffer), but doing it ONLY for benchmarks is just pure deception.

  • http://dexterslab.wikia.com/wiki/Mandark Mandark

    How is pushing the CPU and GPU to its full potential in benchmarking cheating??? I still don't understand.

    Yours Sincerely,
    Mandark, Evil Boy Genius

    • Grayson

      Because it was only pushing the CPU and GPU to their full potential in benchmarking apps, and it wouldn't do that for any other app. Either it pushes the CPU and GPU to their full potential for every app, or it does it for no apps, but doing it only for benchmarking apps doesn't help the consumer at all... it just makes consumers think their regular apps will perform better than they will on other phones that don't do this, even though they won't. Deception.

      • ProductFRED

        ...but that's how it works on a computer too... It's called scaling...

        • Grayson

          No, Samsung was actually raising the thermal throttling limits for benchmarking apps, in addition to disabling scaling and forcing the max frequency for those apps.

        • andy_o

          No, on a modern i7/i5 PC it's more akin to turbo mode, which is throttled. It would be the same if a PC manufacturer had code that detected benchmarking programs and when they're running forced turbo mode in order to say "my i7-4770 is faster than the competitor's i7-4770".

          • Steve W

            Actually, you're 100% wrong. I've got an i7-2600 that I've got OC'ed on the processor to 15% greater than stock frequency, RAM is OC'ed to 10% greater frequency than stock and my system is configured to dedicate 385MB of the RAM to the integrated graphics (aka IntelHD 2000).
            With that configuration I get the same performance as a non-OC'ed system utilizing a Nvidia GTX 640 graphics card with 1GB RAM.
            I've confirmed this on the games I test.

          • andy_o

            What are you even talking about? Did you read the thread you were responding to?

        • Stacey Liu

          Yes. Scaling that works the same way for any app.

          Intel SpeedStep doesn't detect a benchmark and lock into its max turbo frequency. It exposes it to the same scaling algorithm any program would get. If the benchmark is intensive enough to make it reach max turbo for extended periods, so be it.

          Samsung is using a different algorithm entirely specifically for the benchmark. That's wrong.

    • redsnowfox

      They were hard coding to achieve clock speeds which can never be achieved by any application no matter how high the load. If this higher clock was achievable by all the applications and not just the benchmarking ones, then it would have been fair.

    • gord

      Hi Mandark, glad you asked

      1) It is cheating because Samsung both removed thermal restrictions & hard coded a higher power-state exclusively for select benchmarking programs
      2) If the author had intended the benchmark to test the maximum power-state he would put load on the processor before starting the timer
      3) If every manufacturer used a white list you could never make a meaningful comparison between different makes or even generations of the same phone

  • ProductFRED

    A benchmarking app is supposed to test peak performance. Not average performance. Peak performance. Processors scaling lowers and highers the frequency depending on the load. Your shiny new 4 GHz PC is not always running at 4 GHz. It works the same way on a phone. All Samsung did was program the phone to run at full speed when it detects benchmark apps. Since there is no way for a phone to recognize what "type" of app is running, they punched in the package names of the most common apps (e.g. AnTuTu).

    The people that are complaining have no idea what benchmarks are or how they work. Samsung removed the code because of the negative press. Not because they did anything wrong.

    • http://dexterslab.wikia.com/wiki/Mandark Mandark

      Finally! Someone who agrees with me! Pushing yourself to the limit is not cheating!

      Yours Sincerely,
      Mandark, Evil Boy Genius

    • Den

      Run full speed just for sake of benchmark app? Why not treat the benchmark app like normal app that stress the CPU/GPU?

      • ProductFRED

        Because most apps don't push the CPU/GPU to its limits.

        I'll put it to you this way:

        By your logic, and the logic of everyone saying this is wrong, the Galaxy S4 should not be advertised as having a "1.9 GHz, Quad-Core Processor". It should be a "1.24 GHz, Dual-Core Processor" (random example), because the processor runs at 1.24 GHz on average, and usually, only two of the cores are being used while the other two are idle/at low usage.

        • Stacey Liu

          Samsung was over clocking the GPU beyond the speed available to normal apps. That's cheating.

          Also, you shouldn't be changing the governor. The point is to benchmark the phone, not the SoC. If you want to benchmark to SoC, grab one of Qualcomm's reference designs.

          Part of benchmarking the phone is allowing the default governor to handle a benchmark test. If it provides the SoC's peak performance to the benchmark, that's great. If it doesn't, that's a limitation of the phone. The benchmark shouldn't be exposed to power other apps aren't. If the benchmark has to push clocks up, it should do so by making the test more intensive, not by artificially locking frequencies.

          When you run 3DMark 11 on a graphics card, the Nvidia/AMD driver doesn't do anything special for the benchmark. It exposes the benchmark to the same governor and scaling any game would be exposed to.

          Same with CPUs. An i7 doesn't detect Cinebench running and lock into max turbo speed. It exposes Cinebench to the same conditions any app would be given, and if that requires Turbo, it will use it.

    • blahmoomoo

      That's not really the point though. The phone can run at full speed, but when all cores are running at full blast, that can generate too much heat, so it throttles back a bit. So the phone is running at its allowable peak. Even if it could theoretically go faster, it won't, unless it's rigged to allow that. Shouldn't the benchmark app reflect the peak usage available to any app, rather than treated special? I'm sure that's preferable.

      Take BOINC for example. If you tell it to, it will use up all processor cores at full execution speed. It will push the phone to its peak performance. But that peak performance is slower than the theoretical speed because the CPU needs to throttle back, otherwise it can reach dangerous temperatures. If the phone let any app continuously run at absolute max, it could be destroyed. Benchmarks should be treated the same way, because that is the actual peak any app will reach.

      • ProductFRED

        See my reply above.

    • andy_o

      You're rationalizing. It's very simple. They deviated from normal operation (which includes throttling thresholds) in order to game the benchmark reviews. Forcing full speed is not normal operation. You would be right if the phone ran at full speed all the time, what would be normal, but they would be murdered in the reviews by battery life and heat.

      • ProductFRED

        Alright, so what if the phone got warm (from the CPU) after the reviewer had been testing out all of the features? The CPU would be throttled because of the heat generated, which would result in lower scores. I can't speak for Samsung, but the logic is pretty simple; in addition to the reply I posted a few comments down, it makes sense since the average benchmarking app runs for, what, a minute to two minutes? That's not going to cause harm to the phone's components. In contrast, for the average user juggling tasks all day on their device, the CPU needs to be thermal throttled to avoid permanent damage.

        • andy_o

          I think you're missing the point. Your premise is not completely correct. "A benchmarking app is supposed to test peak performance", but on normal operation. If code specifically tells the CPU to deviate from normal operation when benchmarking is occurring, then it's cheating. I think NorCalAngler's response explains it well.

        • gord

          Its really not a hard concept to grasp

          1) If the author had intended the benchmark to test the maximum power-state he would put load on the processor before starting the timer
          2) If every manufacturer used a white list you could never make a meaningful comparison between different makes or even generations of the same phone
          3) See one

          Samsung fooled you twice "ProductFRED"

      • Jabbathewocket

        That depends on if the code was in place *only* for benchmarks or if it was for all things that they knew most users would want "boosted"

        For example if they had simply made it a toggle in settings to enable "gaming mode" or whatever and explained that the current state of governers and power saving tech made it easier to "white list" apps that should be allowed to use all the resources of the device, rather than simply allow the device to adjust on its own.. this would not have been a thing..

        But near as I can tell they got caught with a benchmark score improving hack instead..

        I for one would welcome a toggle for times when I am running on mains power and want to run full bright screen/full speed.. etc.

    • Su-Au Hwang

      Yes a benchmarking app is supposed to let my shiny new 3.5GHz i7-4770K PC run at full load so it runs at 3.5GHz. It's not supposed disable all throttling, OC to 8GHz and then benchmark. The point is, only hard coded benchmarking apps can get that OC (or users with a custom kernel and rom) no one else, no matter how much load your app puts on that cpu. It's totally normal and desirable for a mobile cpu to throttle for the sake of battery life and temperature, but hardcoding ? Really ?

    • TY

      http://dictionary.reference.com/browse/benchmark

      Sorry, but your idea that "benchmarking app is supposed to test peak performance" is blatantly false. It really depends on the intention of the creator of the benchmark app.

      For instance, 3D Mark is AGAINST the action of detecting the app and scaling the processors to maximum performance. They even removed those devices they considered cheating, including Samsung and HTC devices.

      On the other hand, Antutu seems to be against such behaviour too, releasing an app that is supposed to prevent "cheating". Why? Because their app is intended to reflect REAL LIFE performance, not POTENTIAL MAXIMUM performance of the chip.

    • jiduz

      Benchmarking apps are designed for not only to test peak performance but also to COMPARE performance with other phones..

    • Android Developer

      you should read more about what benchmarking is:
      http://en.wikipedia.org/wiki/Benchmark_(computing)

    • SammyScum

      I thought Samsung had stopped paying people to comment on blogposts and forums? No? Where can I sign up?

      - ProductFried is absolutely right.

      where's my check?

    • ToTeX71

      But when nobody else uses that method, lets make sure you read that sentence,
      WHEN NO OTHER COMPANY USES THAT METHOD, it will not be fair and the results will be useless, unless everybody did it, then it would be fair benchmarking.

    • gord

      1. If the author had intended the benchmark to test the maximum power-state he would put load on the processor before starting the timer
      2. If every manufacturer used a white list you could never make a meaningful comparison between different makes or even generations of the same phone
      3. See one

      Samsung was caught in an act of deception and a bunch of contrarians who couldn't put 1+2 together have all been fooled twice

  • puyu

    Well thats strange cause under kitkat I get at least 1000 points more over the score on jelly bean on my S4.

    • Matthew Merrick

      That's because kitkat has lower system requirements.

  • jpd514

    Cheating, cheating.

    Are you going to call a cheater, the Exynos 5422 that comes with Heterogeneous Multi Processing, which means that a combination of any cores can work in conjunction, or all eight cores can run simultaneously if a workload requires it.

    Proof, it wasn't a cheat!

  • Tim

    Have the other OEM done the same thing with their KitKat update?

  • jamcar

    Who cares? The idea is that people could unlock the CPU/GPU if they're rooted, which is unfair to the consumer, but the average consumer would never look at a benchmark or even know that Samsung did this. Only advance users would, the same kind of people that would use root / a custom ROM. Then that same user would be able to achieve this on their own. I would like it if I didn't use a custom ROM/kernel, but sense I do I already tweak it to run at or above its full potential.

  • Blab Blab

    Benchmarking should be about testing peak performance like other comments said. But I think Samsung only making it available for benchmarking apps to use the tools, make the results less relevant. The performance boost is not available to all apps. This is what the press is talking about because benchmark results are one way of conveying how fast the phone is when doing reviews. Without these numbers, the reviews cannot tell how fast is comparing to other devices..

    • Android Developer

      but if no other app can reach this peak performance, the benchmark is worthless since it doesn't give you any result that you can reach, on any app.

      it's best to let the OS handle the need for tweaking the CPU in a global way, so that all apps would be able to enjoy a fair performance, and not just benchmarking apps.

      if you wish to get peak performance, overclock your device. for real world, be fair as you are with any other app.

  • Anthony Johnson

    Don’t know even don’t care about what
    benchmark they are talking about! I have Galaxy S4 which I bought last year and
    fully satisfied with its features that includes not only software but hardware
    too and the main feature which I like the most in it by considering hardware
    aspect is that it support wireless charging feature. And if we talk about software part Samsung introduce
    latest update time to time for Galaxy S4 device in order to make it one of the
    best phones available in the world.

Quantcast