07
Jul
2012-07-07_12h05_06

A week ago, I posted a head-to-head comparison/buyer's guide of the Asus Transformer Pad (TF300), Transformer Prime (TF201), and Transformer Pad Infinity (TF700). The most upvoted comment: how is the internal storage performance? So I sat down to benchmark 6 devices.... and with the help of the team, ended up benchmarking 11:

  • HTC One S (S4)
  • HTC One X (T3)
  • Motorola Droid RAZR Maxx
  • Samsung Galaxy Nexus (Android 4.1 - Jelly Bean)
  • Samsung Galaxy Nexus (Android 4.0.4 - Ice Cream Sandwich)
  • Samsung Galaxy S III (S4)
  • Asus Transformer Pad (TF300)
  • Asus Transformer Prime (TF201)
  • Asus Transformer Pad Infinity (TF700)
  • Nexus 7 (Android 4.1 - Jelly Bean)
  • Toshiba Excite 7.7

The Tests

At Mekerz86's suggestion, we used both RL Benchmark and AndroBench, both available for free on the Play Store. Before running any benchmarks, all devices were turned off, turned on, and let sit unused for a few minutes. Although this may have impacted some scores negatively (due to a lack of caching and the like), it was a necessary trade-off to ensure that background apps/processes were limited and fairly consistent; letting the device sit unused after boot up helped ensure all boot-up tasks, syncing, and updates had been completed. Three consecutive trials were run of each benchmark, and those 3 scores were then averaged.

RL Benchmark

2012-07-07_08-20-57

RL Benchmark is an SQLite benchmark that runs thousands of insert, select, update, and delete functions - the same functions Android uses to store data. The resulting number is the total amount of time (in seconds) that it took for your device to run through all the tests. The faster it completed it, the faster your device stores and retrieves data; thus, lower is better.

AndroBench

2012-07-07_08-20-27

AndroBench is a little more complicated, and tests Micro, SQLite, and Macro performance. For our tests, we only used the Micro and SQLite scores. Micro measures sequential read/write and random read/write speeds of your device's storage, while SQLite (again) tests insert, update, and delete performance. Sequential speeds are reported in MB/s, and Random speeds are reported in MB/s and input/output operations per second (IOPS). SQLite tests are measured in both transactions per second (TPS) and seconds, though seconds were not used in the test since they directly correlate to TPS and thus overlap.

2012-07-07_11h04_25

Not easily graphed.

The resulting output from AndroBench was 12 different numbers per benchmark per device - clearly too many to be individually charted and graphed. The solution: combine the sequential speeds (1) , random speeds (into MB/s, 2, and IOPS, 3), and SQLite scores (4) to come up with 4 different totals. It was the average of these totals I then graphed - much cleaner, don't you think?

2012-07-07_11h10_16

Easily graphed.

Finally, for those who don't know, this is the simplest depiction of the difference between sequential and random access:

1000px-Random_vs_sequential_access.svg

Image source

The Results

The results can be downloaded in .XLSX (Excel 2010) format here, or the full data set can be viewed on Google Docs.

RL Benchmark

Again, this is the total amount of time it took in seconds to execute thousands of SQLite actions; lower is better.

2012-07-07_11h23_56

Obviously the winners here are the Galaxy SIII, Galaxy Nexus (ICS), and One S, all about one second apart. With the exception of the Galaxy Nexus (JB), it seems that phones generally lead the pack in SQLite performance. And while this isn't the most scientific test, it appears that Jelly Bean slows things down a bit - the Nexus 7 and Galaxy Nexus (which both run JB) are substantially slower than all the other ICS phones.

AndroBench

Again, these scores are the sum of the various averages.

Sequential Read/Write

2012-07-07_11h39_34

The Galaxy Nexus absolutely crushed the sequential benchmarks on both ICS and JB, though ICS was more than twice as fast - definitely an outlier. Next up were the One S and One X, followed by the SGSIII.

Random Read/Write: MB/s

2012-07-07_14h49_28

Rather surprisingly, the MB/s scores for random read/write speeds were all over the board, ranging from 203.08 (GNex ICS) to 3.86 (Razr). It's a little hard to understand why the range is so massive, since we have 2 "outliers" out of 11 results - and even the best non-outlier (SGSIII, 11.22) is 3 times higher than the lowest score. Let's eliminate the outliers just to get a clearer picture:

2012-07-07_14h50_21

All told, the Galaxy Nexus crushes again on both ICS (1st place) and JB (2nd place); the SGSIII checks in at third place. The other devices are all substantially lower.

Random Read/Write: Input/Output Per Second (IOPS)

2012-07-07_11h48_11

Once again, it's pretty hard to get a clear picture with the Galaxy Nexus as an outlier. Something is definitely going on with those scores - let's remove them and regraph:

2012-07-07_11h49_52

Without the Galaxy Nexus dominating the picture, it's a little easier to see the differences. Excluding the Galaxy Nexus, the SGSIII, One X, and One S once again lead the pack by a wide margin.

SQLite: Transactions Per Second (TPS)

2012-07-07_11h53_31

Once again, the Galaxy Nexus dominates so much that it throws off the entire graph. Obviously, it takes first and second place, but so that we can get a clearer comparison of the rest of the scores, let's take it out (... again).

2012-07-07_11h54_42

Excluding the Galaxy Nexus, the SGSIII, One X, and One S lead the pack.

Conclusion

There are a few broad conclusions that can be drawn from these tests, general as they may be. First, ICS (Android 4.0) seems to be better about storage speeds than JB (Android 4.1). Second, phones tend to be faster than tablets.

With that said, it's important to keep in mind that these are synthetic benchmarks and are not a direct reflection of real-world performance. The TF700 and Excite 7.7 feel every bit as snappy in real-world use as do the One X and One S. The benchmarks do help show performance differences and can help explain the quality of the experience, such as the likelihood of experiencing "Application Not Responding" (ANR) errors.

Aaron Gingrich
Aaron is a geek who has always had a passion for technology. When not working or writing, he can be found spending time with his family, playing a game, or watching a movie.

  • atinsley

    Makes me a proud GNex owner (best phone of all time?), but it's curious to see the difference between ICS and JB. The most important piece of the Android puzzle for me is responsiveness, though, and JB definitely beats out every previous version of Android.

    • niv

      i feel just like you. proud to be a Gnex owner.

    • http://twitter.com/ToysSamurai Toys Samurai

      I also feel that JB on GNex is much more responsive than ICS on GNex. However, that proves that hardware speed means nothing in real world. iOS users often say their devices are fast, but in fact it's often due to the optimization of their UX. I can clearly see that Google is doing similar thing on JB. For example, there is an annoying bug in ICS with the Face Unlock: http://code.google.com/p/android/issues/detail?id=23197

      On JB, the lag seems to be gone, but looking closer reveals that the camera does not come up any faster. It's just that JB shows an animation before the camera is activated -- it changes the user perception even though the underlying problem still exists.

      • atinsley

        I totally agree that the addition of the animations makes it feel faster, but I also believe that they have improved beyond just that.

        As far as the animations go, I love the animation when you click on a notification and the app flies on to the screen.

    • Ron

      The results seem odd to me. I'm shopping for a new phone and compared the S3 with GNex yesterday. EVERYTHING was so much more fluid and bright on the S3 compared to GN. It seemed like the GN screens were stuck when swiping and opening apps was slower. The GN was soooo dark and difficult to see even with brightness turned all the way up. I really really want to get stock Android and a Nexus device off contract, but the S3 and HTC One just seemed more visibly friendly and responsive. Am I missing something?

      • atinsley

        I've yet to use either of those phones, but I haven't had an issue with the brightness of my screen. It's been just as bright as my 4S and my old Droid 1.

        I personally mess around with my phone way too much to go with a skinned version of Android, but if that's not true for you, just go with your favorite phone of the 3. Like I said, I haven't used either of the new flagships from Samsung and HTC, but the Galaxy Nexus is by far the best phone that I have used.

        • Tyler Chappell

          On my dad's GNex, he has complained a few times about how dim the screen is when you have it set to Auto, which I noticed myself several times when using it. My Thunderbolt's auto brightness mode seems to do a better job than the GNex.

      • http://wyldtek.com/ wyldtek

        My I/O GN came with ICS and I used it for a few hours before updating it to JB. The UI is drastically faster and smoother with JB. The only complaint I have is the external speaker (not the phone speaker) is much quieter than on my GS2. No issues with brightness either. This is my second Nexus (had a N1) and I plan to always have an off-contract Nexus from here on out.

  • http://twitter.com/rohanXm Rohan Mathur

    I'm amazed at how well the GNex did!

  • http://www.facebook.com/people/Timm-Eddy/880640297 Timm Eddy

    are these apps compatible with jb? with how much smoother/faster it is over ics i find it hard to believe it performs worse then ics

  • http://twitter.com/Tempie007 Tempie007

    the effects of slow storage are unfortunally the cause of lag and ANR messages on all of the asus tablets. reviews of those devices really arent on par with the end user experience. many people have performance problems ... fast IO really makes a difference , samsung does a good job, their devices fly compared to the asus products.....

  • Andreas

    Aaron, are those JB devices using a final (ie. non beta version of JB)? there might be data logging taking place, which would explain the performance regression on those benchmarks.

    • http://www.androidpolice.com/ Artem Russakovskii

      They're using whatever software is available to us now from Google.

  • Mekerz86

    Hi Aaron,

    Thanks for taking the time to benchmark all these devices in response to my comment on your article. You could have taken offence to my feedback/constructive criticism and defended your article, but I'm glad you didn't.

    As you can see, your results back up my experience of the poor I/O performance of my Asus TF201 tablet and it also confirms my fear that the TF700 also performs as poorly. You're right that general smoothness of the TF201 is good but the poor I/O performance makes the browser and games crash on a regular basis. In comparison, I've not had a single "Application not responding" message whilst web browsing on my Galaxy Nexus.

    Whilst benchmarks are not the be all and end all of a devices' performance, these benchmarks back up and reveal one of the causes of the poor web browsing and gaming performance on Asus tablets. It also effects the device when application updates from the Google Play store are being written to the /data partition.

    Asus have made no attempt at fixing these issues in the past 6 months, despite two firmware updates and the 28.5 firmware currently being beta tested by a bunch of XDA users does not fix these issues as the testers have still experience the "Application not responding" message. What's frustrating is the I/O performance really is the Elephant in the room, by that I mean the issue is really obvious and can be replicated in benchmarks like the one you've ran.

    Now to hold a software developer and/or engineer role at Asus, I'm guessing you must have a brain in your head somewhere. Why are they not fixing this obvious issue? Gary Key's (Asus America employee/representative on XDA) has not mentioned the I/O issues despite how well documented these issues are on XDA. It's almost like they're ignoring it as they know it can't be fixed.

    There's rumour the MMC code in Asus' kernel is two years older than that in the nVidia Tegra kernel repository and that the new code does improve I/O performance. What I don't get is why Asus' do not update their kernel as nVidia's repository is open to the public as per GPL license.

    We need potential Asus consumers to be aware of these issues, so that they vote with their wallet. These big corporations are happy to release devices that are not fit for retail and fail to quality test them to public consumption standards. They'll continue to ignore their responsibilities to current consumers and wait for time to sweep us under the carpet unless people start voting with their wallets and take their custom elsewhere.

    • http://www.facebook.com/profile.php?id=523907787 Maxx Tan

      So much truth in this. I really don't know what ASUS is really updating... Pretending to be competent or actually doing their job

    • Striderevil

      Perhaps you guys should check this out.
      http://youtu.be/mrQRYmYip6Q

      Its real nice the way he explains about where Googles focus is at the moment. I was laughing.

    • Unni

      Just to update in case anybody stumbles on this comment: Bought a TF700T just 1 week back. This so called IO issue is a killer (right word doesn't come to my mind right now). Do not buy this tablet. I have already printed out the return form from Adorama (from whom I purchased after seeing the 100$ discount deal here in AP). If not for the IO issue, it is a beautiful tablet. To understand what I am talking about, just download a torrent. The tablet becomes unusable.

  • Tyler Chappell

    Thank you for doing these tests, after first hearing about the poor I/O performance of the Asus tablets just within the past few weeks, I was worried that my Nexus 7 might suffer the same issue. I'm really glad to see it performs substantially better than Asus's more expensive tablets, making me all that much more excited that I cancelled my TF201 order back in January and decided to wait for the "perfect" tablet. =] It looks like I made a wise decision and saved myself a few hundred dollars, as well as lots of headaches.

    • Ramiro Fernandez

      I'm still concerned, don't know whether to cancel my preorder, that performance is still not very good. Have there been any reports of ANR on the nexus 7?

  • hot_spare

    I have my doubts about Gnex performance.. I would like to know what is causing the numbers to be skewed so much. Some Gnex numbers are similar to those of SSD's. Strange that the numbers are again 2x in ICS compared to JB. How can numbers change so much between ICS and JB? I would assume that the tests bypass the file system.

  • duplissi

    hurray gnex!

  • bung1

    As a GNEX owner I love to see it so far ahead of the other devices, but as said, it doesn't necessarily represent the real-world feeling.

    After using stock JellyBean "Preview" on this phone I have to say I can't wait for some tweaked custom kernels and the logging shut down properly. I see myself using this phone for a long time.

  • joeybags59

    I'll be happy to let Asus know how much I love my TF101 and how I don't have any problems that he and Mekerz86 want me to think I have, especially since Aaron didn't happen to include it in his tests. Thanks, Asus, keep making great products.

    • AaronGingrich

      I love all my Transformers as well whether they run the benchmarks or not. They're good products that I use daily and have no complaints about :)

    • Mekerz86

      Not sure how you've come to conclusion that I'm forcing an opinion of the TF201 seeing as 1) I'm not forcing any opinion at all, all my findings are backed up by these benchmarks and the 100's of unhappy XDA TF201 users and 2) I never mentioned the TF101 in my comment and have no idea if it suffers from the same I/O issues.

      All I know is the TF300 and the TF700 exhibit the exact same I/O performance and that the Asus kernel is a mess.

      I'm glad you like your product, I want to like my TF201 too especially as I have no interest in other Android tablets currently on the market.

    • dupo24

      So let me ask you this then Joey, have you ever visited XDA and seen the problems with the TF201? Broken GPS, bad wifi, light bleed, lcd spots....the list goes on. For a flagship device, it surely has its fair share of problems. Mekersz86 is dead on with his posts concerning the I/O issues and lack of support from Asus. He as well as many many others own and love our TF201's ... we just are annoyed at the problems the device has.

  • http://codytoombs.wordpress.com/ Cody Toombs

    I admit, I'm far from an expert...I don't even have much experience in the area beyond learning a bit about performance improvements for a desktop app I wrote years ago...but huge variances like this are almost always related to under-the-hood write buffering and I/O caching that may not be properly/completely accounted for by the benchmark apps.

    The most basic description of write buffering, for those who might not be familiar, is that there's a small amount of RAM (also called 'cache') that is used for performing operations more quickly. When an app wants to write something to storage, it is first written to this RAM (which is REALLY fast), and then the I/O controller takes responsibility to write that data to long-term storage (a much slower operation). Occasionally, read times can also be affected if they try to get data and find that it's still available in the write cache. For this to be useful, it has to be enabled (in the kernel) and code written to handle the read/write operations.

    It's possible the Galaxy Nexus (on ICS) is making full use of the I/O cache while nobody else is. Since this explanation is a software issue, it would further suggest that Jellybean doesn't have all of the optimizations turned on yet.

    Another possible explanation is that the Galaxy Nexus may have an uncommonly large cache and data isn't being purged as quickly as it is on other devices. This possibility is hinted at by the differences in RL Benchmark and AndroBench. If AndroBench is writing relatively small amounts of data, they might not be writing enough to cause old data to be purged from the cache prior to reading it.

    Disclaimer: please keep in mind, I'm just speculating. I'm not saying these are the causes, nor am I saying write caching is being done or not done on any given device, only that it could explain the abnormal scores and that it would be my best guess without proper investigation :)

    • Jeff Brown

      Bingo!

    • chlo1ber

      Yep - 200MB/s random read/write is simply impossible. You don't even reach those numbers on modern SSDs. It also depends on what exactly the benchmark measures. Is it 4k or 8k, does Android support NCQ (i.e. is queue depth of importance)?

      But I can certainly say that you were measuring cache on the GNex.

  • Jeff Brown

    I believe there are several problems with the benchmark methods used in this article and the conclusions that were drawn.

    You can't compare the relative I/O performance of Jellybean and Ice Cream Sandwich just by looking at one device.

    To begin with, the performance of embedded flash controllers varies tremendously. To make these benchmarks meaningful, we need to know what underlying flash controller, firmware, and memory parts are being used by the various devices. The choice of components on the board has a much bigger impact on I/O performance than the Android system software version.

    This is where things get complicated. Manufacturers often use multiple sources for their components. The same product might be shipped in a variety of configurations with different eMMC parts, touch panels, radios or other components. They are not easily distinguished.

    For example, the GSM and CDMA variants of the Galaxy Nexus have different eMMC parts with different performance characteristics. The article does not specify which ones were tested.

    These differences are measurable but often insignificant. There is a point at which the I/O performance is "good enough." User-perceived performance and satisfaction is mainly determined by other factors: UI latency, smooth scrolling, transitions between applications, how the system behaves under heavy memory pressure, overall reliability, availability of regular updates, etc.

    Moreover, flash memory performance is about much more than just read/write speed and average IOPs. Suppose you have one flash part that achieves 20MB/s random writes but every now and then one write operation blocks for 800ms while the flash controller performs garbage collection and wear leveling. Suppose you have a different flash part that manages 10MB/s random writes but it only ever pauses for at most 100ms for garbage collection and wear leveling. Which is better?

    (I'll give you a hint: responsiveness is often more important than throughput.)

    I'd like to point out that some of these benchmark numbers in the article are just flat out wrong.

    Let's take the claimed 200+MB/s random read/write performance shown above. That number is impossible! Typical eMMC parts have a maximum theoretical throughput of around 50MB/s. Consequently, AndroBench must be measuring the performance of the system cache (RAM) or something else rather than the true cost of I/O operations.

    It's also a bad idea to sum read and write throughput numbers together. Read throughput might marginally affect how fast an application will launch but is largely compensated for by the system cache. Write throughput, especially random write and synchronous write IOPS (refer to fsync() / fdatasync()), significantly affects transactional operations such as synchronizing your inbox, creating lots of little files in the browser cache, installing applications, etc.

    Anyhow, I want to leave you with some positive news. SQLite transactional operations such as inserts are somewhere around 30% faster in Jellybean than in Ice Cream Sandwich. That said, the actual impact of these optimizations may vary from one device to another for the reasons I described above.

    • Mekerz86

      The Nexus is GSM, simply because the only Jelly Bean build in the wild is the developer build that came on the GSM Nexus' handed out at Google IO. I don't believe the point of the article was to compare ICS and JB, I guess JB was included as it's new and there's a Nexus build available so Aaron though "hey, why not throw it into the mix".

      The article was written in response to my comment on another article regarding Asus' tablets, although I think there could have been more commentary regarding the fact Asus' devices consistently performed badly in every/most tests.

      • AaronGingrich

        > "I don't believe the point of the article was to compare ICS and JB, I guess JB was included as it's new and there's a Nexus build available so Aaron though "hey, why not throw it into the mix""

        Exactly. I'm sorry these benchmarks aren't good enough for you Jeff, but these are the ones I was asked to run and these are the ones we have. Benchmarks are sort of inherently flawed, so I'm not really sure what your diatribe was about there. I made it a point to basically say "these are benchmarks, take it for what it is."

  • squiddy20

    I would just like to point out that on the second AndroBench Random benchmark, the one where you have the Gnexii "removed", they're still both listed, while while the Nexus 7 and Razr are missing. I assume the data is correct, but the wrong names were listed at the bottom. Just pointing this out because it confused the h*ll out of me for a few minutes and I'm sure it might for others as well.

  • GazaIan

    DAMN, the Galaxy Nexus is serious, but what's with the half speeds in Jellybean?

  • thepeddle

    I don't know how accurate this RL Benchmark is, I just ran it four times and all four came in under 24 secs on my Galaxy Note.

Quantcast