04
Oct
gnote3

Update: We've heard back from Sprint on the "Restriction to benchmark sites removed" line. Here's what a representative told us:

During the development phase of this device, we had blocked benchmarking sites/apps. Now that it is released to our customers this fix will allow users to download benchmarking apps on their note 3. Hope that answers your question.

So presumably any favorable treatment that the Galaxy Note 3 demonstrated in review units, as shown by the Ars Technica report below, is still in effect.

--

Samsung has been in hot water for the last few days thanks to a minor controversy over benchmarks on the Galaxy Note 3. Android Police alumnus Ron Amadeo found that Samsung has implemented a whitelist that sends the CPU of the US Galaxy Note 3 into a "boost mode" for most popular benchmark apps, disabling some of its power-saving features and locking all four cores of the Snapdragon 800 at 2.3GHz. Shady, to be sure. Yesterday an interesting notice popped up for the first Galaxy Note 3 software update on Sprint:

Enhancements/Fixes

- Voicemail audio restored when connected to Galaxy Gear

- Restriction to benchmark sites removed

Important Notes

- Software version is MI5

- Refer to the Samsung Galaxy Note 3 Software Update article in Support for install instructions

The thing we're interested in here is the "Restriction to benchmark sites removed" line. We've reached out to Sprint and Samsung for clarification and have yet to hear back.

What that could mean is that the whitelist for benchmark apps on Sprint's version of the Galaxy Note 3 will be removed almost immediately, so that the phone will return benchmark scores that more accurately reflect day-to-day usage instead of the highest theoretical performance. That's how benchmarks should be run, but Samsung isn't the first company to be found with its finger on the scales, and this isn't the first benchmark controversy to come out of the Korean company.

The current benchmark results aren't technically incorrect, but they are misleading, due to the fact that normal apps simply don't have access to the kind of processing power that the benchmark results would suggest. In "benchmark mode" the Galaxy Note 3 is unable to idle or scale down on any of its cores - see below. By the way, this is why we tend to report on real-world performance in our reviews instead of relying solely on benchmark results. You can see a review of the Sprint Note 3, presumably with the skewed benchmark feature, here.

2013-09-29-05.251

Ars Technica screenshots show the "Benchmark Mode" on a Snapdragon-powered Galaxy Note 3.

I'm sure there will be plenty of users who'll be willing to poke around the updated software to see what's actually going on, especially since it's being disseminated to Sprint customers on day one. It will be interesting to see if a similar update goes out to the Snapdragon-equipped Note 3 models on the rest of the American carriers.

Source: Sprint Community, Ars Technica

Jeremiah Rice
Jeremiah is a US-based blogger who bought a Nexus One the day it came out and never looked back. In his spare time he watches Star Trek, cooks eggs, and completely fails to write novels.
  • TY

    Manufacturers, stop this benchmark cheating already.
    1. Most people expect benchmark scores (as covered in reviews) to (somehow) reflect real world performance, not the maximum performance. This will be considered cheating by many people.
    2. The score only improves by, like, 10-20%. Is the reputation loss worth it? Especially looking at you, LG, lol. Your thermal throttling mechanism completely overrides the benchmark mode, so what's the point?
    3. The boosting should be visible by the user, not sneakingly
    implemented. Might as well put a "benchmarking mode for 10 minutes"
    option in settings, lol

    • Roger Siegenthaler

      Well an app that has a need for the full resources should actually be getting the full resources.

      • TY

        ...and only benchmark apps need full resources?
        And benchmarking does not necessarily mean "testing the device at its full potential". It can mean "testing the device at its normal condition to reflect its REAL WORLD performance".

        • Roger Siegenthaler

          Well sadly benchmarks only pile of shit-code which won't trigger the device into running a benchmark application at full-speed. If however a game happens to require the full power of a cpu/gpu it "should" be given that.

          Emphasis on the should. I hope that, that'd be the case. And because of this OEM's have to force the benchmarks to return results that are actually more real-world then what the benchmarks would elsewise return.

          tl;dr an app that actually needs the full power should be getting that full power and that's the real-world performance that a benchmark is made to define.

          • TY

            You're argument is that:

            1. Ideal performance = real world performance that the benchmark wants to measure.
            2. When using high-demand apps, such as benchmarks, the CPU governor *should* ideally ramp up the CPU to its full power (i.e. all cores turned on, max frequency).

            ^This is the "ideal" case. But if it isn't the normal behaviour, that means the governor is not "ideal" and the benchmark should reflect this fact. Moreover, during normal usage, the governor may very well tune down the frequency to conserve battery.

            3. Benchmark apps are not written well enough to force the device to run at full speed.

            ^Where did you get this assumption from? If even an app which deliberately wants to test the device at its full potential fails to do that, what else can success in doing so?

            =====================

            Your conclusion:

            Turning the CPU to max when launching benchmarks fixes the benchmarks' problem and let it run as it should.

            Sorry but I don't think your argument is sound.

  • Ishimaru

    Like PC benchmarks, I really couldn't care less about phone benchmarks. Does it perform well? Yeah? That's all I need to hear.

    • Jeremiah Rice

      Eh, I think accurate PC benchmarks are useful - if you're spending $400 on a graphics card, you want to make sure it's at least $100 better than the $300 option. Likewise a dramatically more powerful processor can reduce rendering times on unwieldy programs like video and CAD editors.

      But for a smartphone, an intensely personal and ergonomic item both in hardware and in software, it's almost pointless to boil performance metrics down to individual numbers. Never mind the fact that there are so few apps that actually need the kind of ludicrous number-crunching power that top-end ARM chips are providing. RAM and battery life are so much more important for a good user experience.

    • MayaAyala

      Benchmarks are only good with ALL THINGS BEING EQUAL. For instance, how does your phone compare to a similar device -- all things equal (OS, standard OTA interface, all bloatware installed and running, etc...)? We are getting to the point where benchmarks need asterisks.

  • Cherokee4Life

    I love Ron...

  • GM

    counter point. perhaps turning everything to "10" is the only real way to measure a phone's capability. "real-world" measures are useful to an extent but every user experience is different based on how they use their phone. Knowing a device's upper limit is perhaps just as useful as knowing what a reviewer was able to do with their phone when the reviewer and the reader are using different apps, etc.

    • Jeremiah Rice

      I think the biggest problem is that the "Benchmark mode" is only applicable to those whitelisted benchmark apps. You'll never see that kind of completely unrestricted performance in the vast majority of apps, even if they might benefit from them (like 3D games or high-res video playback). It's all marketing and showboating anyway, but it still rubs a lot of people the wrong way, including yours truly.