12
Nov
art thumb

By now you've probably heard about ART and how it will improve the speed and performance of Android, but how does it actually perform today? The new Android Runtime promises to cut out a substantial amount of overhead by losing the baggage imposed by Dalvik, which sounds great, but it's still far from mature and hasn't been seriously optimized yet. I took to running a battery of benchmarks against it to find out if the new runtime could really deliver on these high expectations. ART is definitely showing some promise, but I have to warn you that you probably won't be impressed with the results you'll see here today.

art

Reality Check

Let's be honest, benchmarking apps tend to be inaccurate and unreliable, often giving wildly varying results even when run in precisely identical situations. However, they are the only option available for recording meaningful and measurable values on performance. Further, since most popular benchmarks are built on the NDK (Native Development Kit), they won't gain any benefit from running under ART. Despite these limitations, there are some interesting and unexpected results that help us learn a little more about the current state of performance.

Screenshot_2013-11-08-11-42-32

How The Benchmarks Were Run

Each benchmark was run at least 4 times on a completely stock Nexus 5 (it isn't even rooted) with both Dalvik and ART. To ensure there was no interference from apps at startup, a minimum of 5 minutes was given after a reboot before any tests were run. In addition to the 6 benchmarking apps listed below, I also tried 2 browser benchmarks (SunSpider & BrowserMark) in Chrome, but neither displayed significantly different scores. So, let's get to the results.

Linpack for Android

One of the key factors in getting good test results is knowing that the tools are measuring the right thing. While many of the benchmark apps target the NDK, a few stick to the SDK. The first and most consistent among them is Linpack for Android, a port of the already popular benchmarking app used throughout numerous computing platforms. It produces a score by performing a series of calculations on floating point numbers. I think this is an obvious choice after reading the description, "This test is more a reflection of the state of the Android Dalvik Virtual Machine than of the floating point performance of the underlying processor." Thanks to ART, scores are 10%-14% higher than they would be with Dalvik. Not too shabby… (link)

Real Pi Benchmark

Calculating digits of Pi is another popular way of stressing a processor, and particularly suitable because most methods stick to integer calculations and avoid floating-point math entirely. Along with Linpack, this gives us coverage of both basic mathematical operations. On top of it, Real Pi happens to use native code to perform the AGM+FFT formula, but uses Java for Machin's formula. On the native side, ART came out about 3.5% faster, probably due to interface optimizations rather than mathematical performance. More importantly, testing with the java code turned out to be 12% faster. (link) Note: in this test, lower numbers are better.

Quadrant Standard

The previous tests are highly specific to mathematical performance, so it's time to branch out to test more of the system. Both Linpack and Real Pi show some positive improvement with ART, but Quadrant gave a result that borders on the amazing, perhaps even too good. The CPU score is off the charts for ART, almost doubling that of Dalvik, which is substantially better than even the most optimistic estimates we've heard so far... While tests for I/O, 2D, and 3D rendering show fairly negligible differences, Dalvik does take an oddly high 9% advantage in the memory test. (link)

Screenshot_2013-11-08-11-58-31Screenshot_2013-11-08-11-57-42

3D Mark

I was leery of using a benchmarking app that clearly focuses on the NDK, as it theoretically shouldn't be affected very much by ART. However, as the tests were run, an interesting pattern emerged where the Dalvik runtime repeatedly held a slight advantage. It's difficult to attribute a reason for Dalvik to do better here, but I'm open to theories. (link)

AnTuTu Benchmark

Breaking performance down even further, AnTuTu helps to expose a pattern. It's increasingly clear that ART is making significant strides with floating-point operations, but doesn't usually turn out huge gains for integers. A strong showing in "RAM Operation" also hints at better use of caching as opposed to just raw memory I/O. These high scores indicate areas where the Dalvik virtual machine was probably very expensive, causing more extensive overhead. The other results weren't particularly remarkable except for the Storage I/O, which might suggest a couple of specific optimizations. One significantly low score appears for UX Dalvik, but it's not clear what AnTuTu is measuring, so this may not be particularly relevant. (link)

CF-Bench

For the ultimate in number production, Chainfire's own benchmark tool takes out a lot of the guesswork by performing tests built on both the SDK and NDK. Again, native code displays a small but curious advantage on Dalvik. Here we can see the integer calculations are swinging back towards Dalvik, as well. Mostly confirming the pattern, floating-point operations demonstrate a significant speed gain, this time in the 23%-33% range. (link)

Other Interesting Measurements

Measuring the first boot after switching runtimes isn't your typical test, no doubt, but the time it takes is quite striking. I wanted to record just how long it took to complete both the App Optimization step and then the total time to actually reach the unlock screen. When I ran this test, I had 149 apps installed.

The Other Stuff

While numbers can be helpful, they don't tell the full story. Benchmarks usually push the hardware to work as hard as possible for a few seconds, then switch to a new test that does the same thing. Sadly, this ignores details that aren't easily measured. I don't have a good way to measure the smarter timing of memory management (especially garbage collection) or better handling of multiple threads. While I can't show numbers for these things, I can demonstrate them. The classic test for a browser simply requires flinging the page as fast as possible and watching it try to keep up. After stress testing Chrome for Android with the mobile version of David's gigantic HTC One review, it turns out that even the supercharged SoC of the Nexus 5 can't quite keep up while running on Dalvik… ART, on the other hand, never lost a pixel. Take a look for yourselves.


Left: running on Dalvik, right: running on ART

To be fair, switching to the desktop version and giving a single fling will easily send you into blank screen territory, but it's still obvious that the renderer catches up faster on ART than on Dalvik. When more optimizations are in place, maybe we won't be far off from flawless scrolling even in the desktop version. For another demonstration, a user by the name of spogbiper has posted his own side-by-side comparison with two Nexus 7s. The one running ART seems to be more responsive.

Summary And Conclusions

The numbers and the videos together paint a picture of where ART stands today. It will definitely make a difference, but its current incarnation just hasn't matured enough to deliver significant gains. Floating-point calculations and basic responsiveness are obviously reaping the benefits of the new runtime, but that's about it. There's little or no overall improvement for integer calculations, most regular code execution, or much of anything else. In fact, it looks like gamers would be better served by sticking to Dalvik, for now.

Why aren't the benchmarks blowing us away? If I were to make a guess, it's probably because the first goal in developing ART was to make sure it was functional and stable before the heavy optimizations came into effect. If that's the case, there is probably quite a bit of code for error-checking and logging just to ensure everything is operating as it should, which might even be responsible for more overhead than we had with Dalvik. Even in the places where ART doesn't outperform Dalvik, the numbers tend to remain reasonably close. As subsequent versions of the runtime emerge from Mountain View, we should expect to see the performance gap growing wider as ART pulls ahead.

Now for the real question: is it worth switching to ART right now? Google obviously isn't recommending it for regular users, and I tend to agree. While ART seems very solid and I feel like responsiveness is better - possibly just the placebo effect - there are still circumstances where it is unstable and causes apps to crash. If there is even a single instance where you have to switch back to Dalvik to get an app to run correctly, that inconvenience far outweighs the minimal performance gain you might have had. Once I've finished this series, I will probably stick to Dalvik for the remainder of KitKat; and I imagine most people will be better served by doing the same.

Cody Toombs
Cody is a Software Engineer and Writer with a mildly overwhelming obsession with smartphones and the mobile world. If he’s been pulled away from the computer for any length of time, you might find him talking about cocktails and movies, sometimes resulting in the consumption of both.

  • Trent Callahan

    Can't wait until it's stable!

    • abqnm

      Waiting until we see ART/Dalvik toggle as an xposed module (once that gets handled for 4.4, that is). Then maybe we could choose which apps use which runtime. Or we can just wait it out...

      As long as it doesn't rebuild for all apps every time one is installed or uninstalled, I am OK with this. I just don't want to go back tohe days the days of the BlackBerry that took 8 minutes to reboot and you had to reboot after most app uninstalls.

      • ProductFRED

        You can't choose while the system is on. I mean you can but since the system itself relies on the run time, you'd need to reboot to see any changes.

        • abqnm

          Certainly. I wasn't advocating for real-time switching as that would be pointless. It already does things in real time with Dalvik. I am saying to set the runtime for specific apps that may benefit from it.

          Chances are though, Google will throw an API out there giving app developers the option of supporting ART at first. If they add whatever API or manifest entry the app will be compiled with ART else it would default to Dalvik. This is how I see the next step going. This is assuming that the runtimes will even run concurrently. Oh well. At least we have more stuff to spin rumors about until the next OS release.

          • Dmitri Smirnov

            This would mean running 2 instanses of JVM, one - dalvik, one ART, which would give a performance hit much greater than any advantage.
            Most apps will work with ART just fine, the rest will need to be upgraded.

          • Steven Z

            I dunno if you understand the way this works. You can't just switch runtimes on an app-by-app basis.

          • http://www.androidpolice.com/author/cody-toombs/ Cody Toombs

            The two runtimes will not run side-by-side and there will be no special API. Google is specifically using the word 'replace' to describe ART, which means Dalvik will be going away at some point.

            There is no reason to have any special APIs because...well...the runtime implements the API. It is the thing that makes the method calls have meaning. Whatever runtime is used should be transparent to the apps that run on it.

            Now, that doesn't mean there can't/won't be compiler flags that might force an app to be built somehow differently, but I haven't heard of such a flag yet. I doubt this will happen, but it might become necessary for architecture-specific situations.

      • JayQ330

        I'm pretty sure Google will have the apps already convertedto art in Google play, once they art is out of beta & is perfected. IMO I believe that all anxious apps will be art straight from Google play.

        • Koblavi

          Probably not, given that AOT compilation which isn't done on the destination device can't take advantage its hardware optimizations at compile time. It's still a good idea to keep the binary in some sort of intermediate form and compiled on device at install time

  • aatifsumar

    Wow new AP Writer? Are you by any chance the Ron replacement Artem had started a sub-reddit about a while back?

    • Phil Oakley

      Cody's been here for a while AFAIK.

    • http://www.androidpolice.com/ Artem Russakovskii

      Cody's first post was in February: http://www.androidpolice.com/2013/02/28/amazon-mobile-for-tablets-breaks-into-europe/

      He's so far written over 100 posts, mostly of the more technical nature, since he's a developer: http://www.androidpolice.com/author/cody-toombs/.

    • Simon Belmont

      He's not that new. He's been around a while, and in fact, I remember when he was just another commenter like you and me on AP.

      He generally handles the more technical topics of Android and he gives us a developer's perspective. I, for one, love his articles because I'm a developer and a behind the scenes, lets get down to the nuts and bolts, type of guy.

      • http://www.androidpolice.com/author/cody-toombs/ Cody Toombs

        Thank you, Simon!

        That's what I've been going for. Writing content for an audience that ranges from casual-but-interested Android users through to enthusiasts and on to developers can be...daunting :) But I've had a lot of contract work where I find myself explaining topics to both developers and business people in the same meetings, so I try to strike a good balance in getting the point across and introducing the right level of technical details. I'm glad to see the developers are enjoying it, too!

        • Simon Belmont

          You're quite welcome. Thanks for the reply.

          It's sometimes difficult to strike that balance, depending on the audience, but it's a great skill to have when it's needed. So far, you've been spot in, so keep up the great work.

    • ssj4Gogeta

      Before that he used to comment here.

      IMHO, he is one of the best writers at AP. His articles are almost always technically correct and don't make me cringe.

    • http://www.androidpolice.com/author/cody-toombs/ Cody Toombs

      I think we can all agree, there's no replacing Ron ;) However, I have been working with Artem on the teardowns and I started the Bug Watch series after Android 4.3 launched. I'm working on part 3 and a lot of other stuff coming pretty soon, so you'll be seeing some more of me.

      • aatifsumar

        Oops. Looked back at your article list and realised I read (and enjoyed) most of them but didn't pay attention to the author name!

  • Michael Pahl

    Can someone confirm that these benchmarks are in fact running properly in ART?

    • http://www.androidpolice.com/ Artem Russakovskii

      What do you mean running properly? They're making calculations and arriving at results. As per Google, ART or Dalvik should be completely transparent to app developers - they shouldn't do anything special to get their apps to run on ART compared to Dalvik.

      • Markus Ressel

        No? But why are some apps (like whatsapp) incompatible? Are there simply errors in ART code that result in wrong calculations? But then again, why are only some apps affected by that?

        • Thomas’

          Questionable bytecode optimization of some apps might be the reason.

          • ssj4Gogeta

            You mean optimizations which don't conform to the JVM spec, but target Dalvik's quirks specifically? Otherwise if they conform to the JVM spec, they should run fine on ART, unless ART itself doesn't conform to them.

        • http://www.androidpolice.com/ Artem Russakovskii

          According to Dan Morrill, that's the case, unless those apps are doing something naughty and unsupported (using hidden APIs or patching Dalvik on the fly).
          http://www.reddit.com/r/androiddev/comments/1qd1f0/advice_on_art_how_do_i_make_sure_my_apps_are/cdbm89f

          • arathkone

            It's a shame Dan doesn't seem to post on G+ (publicly) any more. I enjoyed his postings.

        • StateOfTheART

          I think it's due to the fact that Dalvik compiles the code on the fly (Just-In-Time) and may never reach the unfinished functions and classes if they are not being used by the app (the Work In Progress code), where ART compiles the whole code to native at boot, meaning if there are any errors or unfinished functions it will fail... not an expert though ;)

      • Michael Pahl

        Just wondering if ART could possibly skew results? Is it possible until the app is "optimized" for ART that the results using Dalvik can't be compared to ART?
        just wondering - but you're saying no.
        The 95% increase in CPU score using Quadrant seems outrageous.

        • http://www.androidpolice.com/author/cody-toombs/ Cody Toombs

          I was equally skeptical about the Quadrant results. Frankly, if I weren't already planning to write the 'Reality Check' disclaimer, I would have written it just because of Quadrant.

          To be fair, the whole point of running benchmarks, in this case, is to identify that something is running faster or slower. In a manner of speaking, the results are supposed to be skewed, and that's what I want to measure. Some benchmarks, particularly Quadrant, appear to be written with code that gained massive benefits from the optimizations already in ART, while others saw almost no gains.

  • Smithers_Jones

    Running it here, no problems whatsoever, but then I don't use Whatsapp.

    • http://www.androidpolice.com/ Artem Russakovskii

      Titanium Backup crashes too, if you use that.

      • Daniel Pogue

        As well as running Google goggles and then camera app or other way around will cause them to constant force close. Solution is to force close goggles manually. I'm not sure if this is only running ART haven't check on dalvik. (Nexus 5 stock rooted)

        • Sean

          Theoretically you could use Tasker as a workaround, just force close the respective one when you close it or open the other.

          • Abdullah

            But does tasker itself run in ART?

          • Sean

            It would have to; an app doesn't decide whether to run in ART or not, it's an all or nothing affair (save for something written in native code).

  • didibus

    Can't wait for Android 4.5 or 5.0, hopefully ART will be matured, optimized and stable and become the default. I'm hoping a consistent 30% improvement, that would be awesome!

  • david leinhogon

    Another good test would have been app opening time. By using AOT a new jit instance doesnt need to be created. And since the majority of app opening is boiler path ART can do some heavy optimizations on it.

  • NeedName

    Great writeups about ART. . . THX

  • Phillip Maiden

    Scrolling in apps just feels much better for me using ART runtime, maybe just a placebo.

    • h4m

      Not a placebo. In ListViews with large images there is a big improvement with ART. Did not measure, but it is so obvious.

      • http://www.androidpolice.com/author/cody-toombs/ Cody Toombs

        I completely agree, the differences are obvious (if you're looking), even if they can't be accurately measured. I didn't bring it up in the article, but I had a friend help me do a blind test by picking the runtime for me and then I'd use it for about 10-15 minutes. I guessed correctly almost every time because of scrolling. I can't wait for it to mature, because this will really bring Android up to the responsiveness what we would see in the iPhone or maybe even Windows Phone.

      • rober34

        do you have by chance a measure for TableViews? They are SO TERRIBLE in Dalvik!

  • benjdm

    Could you do a part 3 evaluating ART's effect on battery life?

    • Matt

      Purely anecdotal, but my standby time on the Nexus 4 is substantially improved. I run a lot of background syncing (Google+ location sharing, location history, sync of all Google services, sync of Any.do and the prolific wakelock creator Facebook Messenger). I assume this might have something to do with how quickly these processes run and finish on ART, but that's really conjecture.

      Either way, losing Titanium Backup/Xposed/Wikipedia app features is worthwhile compared to the battery life increases in my book. Then again, I have so many apps that switching to ART took ~35 minutes on first boot instead of 10...hopefully I don't have to switch back and forth frequently.

      • benjdm

        My anecdotal experience is similar to yours.

      • http://www.androidpolice.com/author/cody-toombs/ Cody Toombs

        Wow, 35 minutes?!?! This is on the Nexus 5? That is brutal

        • Matt

          On the Nexus 4 actually. Either way, it's definitely not quick (this is with ~250 apps). I decided to time it because all the approximations online have been ~10 minutes; I was beginning to wonder if I was delusional in thinking mine took much longer.

          • h_f_m

            It's probably much faster on the N5

          • SorinDobrin

            Of course it depends on how many apps you have...tried it today, nexus 5, 125 apps ,12 minutes

          • Matt

            I know; I never said it wasn't app number-dependent. I just hadn't seen anybody else mention 30 minutes+.

      • Chris

        Oh most definitely. The processor isn't under constant stress trying to compile each application you open.

      • https://plus.google.com/108596272537415356460/posts Jason Farrell

        Using ART means losing TiBu and Xposed? Can't live without those two, so I wouldn't use ART even if it was 50% faster.

        • Matt

          I use TiBu *a lot*, but for now I'm just reverting to Dalvik when I want to backup (e.g., to flash a new ROM). As for Xposed, I wasn't really using it much...mainly for the Greenify GCM allowance feature, but I'm getting better battery life with ART anyway. All the other features of Xposed seem to be similar to those found in custom ROMs and Xposed isn't working that well on 4.4 right now anyway...

          • https://plus.google.com/108596272537415356460/posts Jason Farrell

            Yeah, I understand that Xposed doesn't explicitly support 4.4 yet, but it will soon, and I can't live without its Music2SD module in order to pin my dozens of gigs of GoogleMusic to my external SD. As for TiBu, I suppose I could bear to use it like you mentioned, but it'd be a pain to have revert to dalvik all the time for backup/restore.

            Hope this all gets worked out in future updates.

          • Matt

            Yeah, definitely. Considering how the demographics are likely to overlap between people using Xposed, Titanium Backup, and ART (otherwise known as power users), I imagine the devs for TiBu and Xposed are a little more apt to fix this than the typical Android developers would be.

    • Whyzor

      I agree, need to run one of those battery drain tests, to test how efficient the CPU can race-to-idle to save battery. Basically have a script that loads a web page, clears cache, watches a video for some seconds, opens another app, do some normal app related stuff, repeat until battery goes from 100% to 5% (or device auto-shutdown). Compare Dalvik & ART runtimes.

    • Adrian Meredith

      I've seen a huge jump on my nexus 5 but that might be only because the batterys now worn in

  • fonix232

    The difference won't be noticeable on current high-end devices (yes, the '12 N7 is high-end compared to my tests).

    I've been running CM11.0 on my Alcatel OT-995 (MSM8255T Scorpion, Adreno 205, etcetera, basically a Galaxy S Plus with an Alcatel logo and slightly different baseband), and it is much more visible. With Dalvik, I get the usual lag (even with the lowmem device optimizations). Switching to ART, and suddenly everything became faster. Apps open at least twice as fast, switching between activities is a snap (compared to the hill of lag I got before), and overall it indeed helps the performance.

    Otherwise, nice comparison. I only can't understand one part, the Chrome comparison. As far as I know, a good chunk of the Chrome for Android app is native JNI code, not Java-Dalvik based. Including the renderer. Then how the hell can ART make a difference?

    • Simon Belmont

      Thanks for the information. Another feather in the cap of KitKat (and later versions) working better on lower spec hardware it seems.

      I'm hoping to see ART really speed things up over the course of the next few versions of Android. The scrolling in Chrome is a fantastic demonstration of its power, despite probably having tons of error checking assertions in there weighing it down, as it is now.

    • http://www.androidpolice.com/author/cody-toombs/ Cody Toombs

      If I understand correctly, the javascript engine and the renderer are written in native code, but the UI is still written against the SDK. That explains why things like SunSpider and BrowserMark show negligible differences, since they only measure javascript execution and rendering performance, but something like scrolling and rendering from the cache are going to gain from ART. I could be wrong, and the scrolling video is certainly anecdotal, but there is still some kind of difference and I think this explanation seems pretty logical.

      Oh, and it's awesome to hear an example of ART making a big difference on older hardware. I don't expect to hear about a lot of older devices getting an extra year or two because of ART, but I think it'll result in more current devices getting a longer life.

      • fonix232

        I doubt there will be any official updates from Alcatel though... We got 4.0.3 last December (some got it like, this March, thanks to carriers in the EU), and there are talks about an upcoming 4.1, but nobody knows anything for sure.

        My custom build does not exactly function very well. The old kernel (3.0.101 at the moment, getting small, squished updates by minor version only) for some reason refuses many old functions to work with 4.4 (ioctls are bugging out like crazy, we can't even use any android.media related app), so it is a long way from now. But yes, KitKat definitely brought some life into my old phone.

        • rober34

          argh Alcatel my previous employer. Never expect anything on time from them!

  • OSagnostic

    I will be honest, i did not read much of this article. Can anyone tell me if Adobe reader is any quicker than current software ?
    I would not expect Chrome or Gmail to run quicker.

    • Gabernasher

      shoo

    • Steven Z

      Why do comments like this exist?

  • Deinde

    Excellent pun on the title!

  • yahyoh

    i think ART going to be great when its fully optimized....maybe in next Android update :D
    lower memory footprint + faster execution = win

  • ginobili1

    Great series Cody. Loving reading all about the future of Android, and it does look sweet (no pun intended). :P

  • Daeshaun Griffiths

    improved ART + latest linux kernel + next gen x8 system ... cmon

  • Testraindrop

    Nice writeup thanks, but benchmark are crap :p

    As some said app opening times for example would be great and maybe trying on lower end devices (I know for now only on custom roms, but maybe something for the future)

    • Michael Pahl

      "Benchmarks are crap" - Apple fanboys

      • Gabernasher

        iDevices do quite well in benchmarks.

        • Michael Pahl

          iPad Air ranked #15, iPad Mini ranked near the very bottom.
          iPhone 4 one of the worst ranking devices. Many Android devices from the same time frame scoring 2-3 times better.

          http://community.futuremark.com/hardware/mobile

          • folkrav

            Yet iPhone 5S on its dual-core processor blows almost every Android phone out there. iPhone 4 is an old, old, old phone. 3 years back. If you're going to compare an iPhone 4 to something, compare it to 3 years old Android phones.

            iPad Air as 15th isn't bad at all, only two 10 inch tablets in front of it (TF701 and Note 10.1 2014).

            iPad Mini is ranked similarly to it's 10 inch counterpart, the iPad 2. No surprises here.

            Be fair. I love Android, but it's not as optimized as iOS.

      • Testraindrop

        Sorry but how is my sentence in any way related to Apple? o.O

        I meant that most of the time benchmarks do not represent anything useful in regards to normal usage...

        Like in the good old GS2 days I remember some devs "tweaking" Kernels for Quadrant, so that the GS2 was the fastest device in this benchmark, but it had absolutely not correlation to real world usage.

        And as already reported some weeks ago, some Manufacturers are also "optimizing" their devices to score especially good in benchmarks, and only in benchmarks.

        tl;dr real world experience counts.

  • Stacey Liu

    Games and 3D tests should are native. Dalvik/ART should have no appreciable effect on performance.

    Maybe other processes are consuming more CPU cycles because of ART. Otherwise I can only think of statistical variation to account for the difference.

    • rober34

      Yes but even a 100% native application needs a small java stub to run, that receives some messages from the native layer. I suspect the context change to call the java layer from native still needs some polish!

  • dcdttu

    All that talk abuot 4.4 making Android good for lower-powered devices, you'd wonder if this was one thing they really wanted to have ready by release, but they just couldn't get there.

    • Justin W

      I'm pretty sure this will be completed in the next x.0 update (5.0, probably), but I'm not sure how long we'll have to wait for that. IMO, that would be a major change in and of itself, but I feel like the masses wouldn't agree, and there would probably be some heavier UI changes or something. Can't wait to see what else Google does with ART, though.

  • Matthew Fry

    What's with ART's low native code scores?

    • http://www.androidpolice.com/author/cody-toombs/ Cody Toombs

      That's been bugging me, too. I'm going to do some more research and maybe a good explanation will surface. My best guess at the moment is that the native runtime had to be modified to allow ART to hook into it, and maybe part of the modifications added some overhead. It's either that or there is some service or code that doesn't run quite right under ART and it's chewing up cycles somewhere, but I don't think that seems as realistic.

  • dextersgenius

    When I first heard of ART, I thought it stood for ARTEM. Seriously!

  • Tim

    Could it be that these benchmark test apps are designed for Dalvik? This could potentially cause ART benchmarks to be lower than expected.

  • Dipish

    But shouldn't one of the big improvements be decreased app startup time?

    • http://twitter.com/ryocoon Kurtis Whittington

      App Optimization - or making of runtime caches, takes REALLY long on ART right now. So installing new stuff, or first time after you reboot when you switch to ART from Dalvik takes a long time. Launch times after that are supposedly snappier (I'm on Galaxy Nexus, and our stuff is buggy enough as it is on 4.4, no way am I trying to use ART on top of it.)

  • Eszol

    More than 30 seconds for boot time? No thanks I rather kill a kitten

    • http://www.androidpolice.com/author/cody-toombs/ Cody Toombs

      Keep in mind, those times are strictly for the first boot. Subsequent reboots will happen at virtually identical speeds. In fact, ART should boot a bit faster, though I really couldn't tell and there's not enough time difference to bother measuring. It's just that the initial conversion of apps from ODEX to OAT takes a very long time.

      • Michael Pahl

        Not only the first boot but also when system cache has been wiped right?

        • http://www.androidpolice.com/author/cody-toombs/ Cody Toombs

          Yeah, I guess I could have mentioned that, too. Anytime the OAT files would be deleted, then they will have to be regenerated on the first reboot.

    • Gabernasher

      QQ

  • spogbiper

    The speed reduction seen when interfacing with native code is quite likely due to immature code or code with some time consuming debug routines built in and yet to be disabled for production. In fact, all of ART may yet to be streamlined and optimized. I think its great that we see some improvement now and I quite like how responsive my ART test device is. However, to draw any conclusions at this point about what the final system will be like is difficult. All we can really prove is that there is definitely room for improvement over Dalvik in some areas, we cannot yet say how much room is there or how much improvement ART will deliver.

    PS - sorry about those crap videos. they seem to be getting posted in all the Android sites. Can someone with more (any) video editing skill please do better ones? :)

    • http://www.androidpolice.com/author/cody-toombs/ Cody Toombs

      Lol, I would have made some better videos myself, but the best camera I have handy is the HTC One, and that's not great for catching things on a small screen. I'm just glad you had two N7's handy for doing this and took the time to do it right. Thanks!

      Fortunately, there aren't very many places where ART significantly slower than Dalvik, and Google's engineers have been forthcoming in saying that this version isn't close to being finished, so we know a lot of stuff will get faster and more stable. I wanted to give people an objective view of where ART stands as of its first public appearance, and then give some perspective that it's not just about the numbers we can measure, but the other little speed-ups that you can easily observe with your own eyes.

      • spogbiper

        Its great that you put together these benchmarks, I didn't mean to be negative at all. It is difficult to capture the small improvements in latency on video, so some clear numbers are very helpful. FWIW all I have for a camera is an S4 literally duct taped to box to get the right angle/steady. That and Windows Movie Maker, and you've got a spogbiper approved video production studio :)

  • Steve knipe

    Here's a list I started of apps/games and whether they work or not with ART. Please feel free to use the form to add any performance issues you notice...
    https://plus.google.com/102960627217331612470/posts/7poPLnSnjna

  • mgamerz

    Why didn't you boot into safe mode instead of waiting 5 minutes?

  • GeeKLoRD

    This is ART!

  • Android Developer

    i hope google will make it possible for users and/or developers to set this per app, and not just a global feature for all apps.
    the reason is to avoid having ART making bad performance on some apps that don't work well with it.
    a similar approach was done on Honeycomb, where developers could choose if they want the app to be hardware-accelerated or not.

    • rober34

      per-app will need to have both VM's running at the same time and possibly fighting for low-level resources, I don't see it happening.

      • Android Developer

        as far as I know, each app that you open, creates its own dalvik VM already, so I don't think what you've written is true.
        Every app runs on its own process, with its own instance of the dalvik VM and using a different user ID.
        this is how much apps are sandboxed on Android.

        Here's a link about it:
        http://elinux.org/Android_Dalvik_VM

        • JayQ330

          I read an article that started iOS was a better mobile is because it's completely sandboxed, but you just said it is sanboxed, is iOS sanboxed in a different way or better single? Please entire back thanks.

          • Android Developer

            both are sandboxed well enough.
            Apps on Android are very well sandboxed. any app cannot read or write data to another app's private data unless the other app allows it in a special way, or if the device is rooted.

            However, Android apps have the ability to write data into the external storage (usually sd card), which is public for all apps to read from.
            So, if the developer is stupid enough to put sensitive data there, you have a bad protection on your privacy. but again, that's the developer's fault...
            Also, if the app writes the data into a path on the external storage that isn't intended for it (which is also allowed), this data will never be deleted automatically when you uninstall the app (example:like camera apps - you wouldn't like to delete the images you've taken just because you've uninstalled the camera app...), so this means you can also get junk files if the developer didn't think about it...

          • JayQ330

            This could be the reason that android is going the none SD route? One more question I just rooted my n5, but you have to give any root app access & the app that gives the root access is password protected, would it still be safe? Like if an asp tries to get root access on its own if that's possible or still just as dangerous.

          • Android Developer

            about the no - sd card route, that's one reason - the junk that is left because of apps. google has at least added the ability to write&read to the external storage without any permission in case your app will only do it on its own paths (but it's this way only on relatively new Android versions - kitkat and above).
            there are better reasons that google doesn't want it (i don't agree with any of them, though). do note that external storage doesn't always mean it's really the SD card. the galaxy S3, for example, has an external storage even if you don't put any SD card into it. In fact, I think most devices work this way today, much more than in the past.

            About the root question, my guess is that it's an extra protection, maybe more against people who install apps on your device,rather than protecting you from malicious apps. There are even games that check for root access (Gamevill does this a lot, maybe to protect against cheating).
            Only when you grant other apps the root permission, you allow it to do so much more that it can access ANY file on your system, including where the password is stored. They can do other things too, of course, like using permissions that are not allowed for normal apps (like reading the system logs). As long as you block an app, it shouldn't be able to use the root permission in any way (unless there is a security hole, of course).

          • JayQ330

            Thanks for the wick replies, I've always known about the internal SD cards since I first bought my galaxy vibrant from T-Mobile. Making it ext4 format to speed up the phone & all that. Then sg3 & I believe sg4, it's cheaper but slower. But is getting faster, so I'll just be careful of what root access I give to an so & review them before I decide to install one. These root app's should have a page with app's certified root certified, lol maybe I would of started one of I knew something about writing app's & noticing things that shouldn't be there. Thanks for everything :)

          • Android Developer

            certified root apps? and who will put each to the list, and how will it be decided? not all android apps are even open source...
            it's like in life - sometimes you just need to trust people.
            if an app shouldn't have root permission (like gamevil apps), block it.

          • http://www.androidpolice.com/author/cody-toombs/ Cody Toombs

            I think the first sign of something wrong is that the article stated iOS was better. I don't mean that in a snarky, fanboy way, but rather that it's nearly impossible to judge the two objectively.

            A core focus of iOS was to effectively keep apps restricted in every conceivable way, while Android was designed from the beginning to allow apps to interact with each other and the operating system in complex ways. Each approach has it's advantages and drawbacks.

        • rober34

          you have a point here!

  • Michael Moussa

    Compare ART and Dalvik when attached to the debugger. ART is painfully slow when debugging, I reverted to Dalvik for that reason alone.

  • AndroidThief©

    You can also benchmark on ti backup as it supports ART. Try backing up some apps or startup loading performance.

  • s0m3f00l

    Art Is extremely buggy I ran into many many issues with it.

  • Carl “C4RN4GE”

    I have a galaxy s4 original us release i337 and recently switched from iB4STiD's KANGAKAT 4.3 ROM (which is an awesome ROM) to danvdh's GOOGLE EDITION 4.4 for one reason. I had to test drive ART. I have been in ART runtime now for 2 days and I have almost decided to give up the insane tweaks provided by the sadly incompatible Xposed app because ART seems to be faster now than when I first loaded it. Last night I opened 18 tabs on chrome and started scrolling.... Zero hangs or blank screens! I Had ran antutu and 3DMark and was unimpressed by the results (No overall significant difference between dalvik and ART but after reading this article I installed and ran Quadrant for the first time. I then ran the full benchmark WITHOUT RESTARTING, CLOSING APPS, OR MAKING A SINGLE ADJUSTMENT OR CHANGE! I had 8 applications in active use including a ram intensive Spartan Warfare game. My phone scored a 9935 which DOUBLED every other device scored including HTC One X! I agree with everything you say in the article, I will continue to run ART for now just to see how it does over time. I have a screenshot of my score but don't know how to make it show here haha!

    • Tralala

      My Note 3 scores 22000 :)

  • Sharmil SK

    A great series of articles..loved it..gonna stick with Dalvik on my N5 for now..after switching back to Dalvik from ART..

  • Danny

    Can I enable ART in developer options, then uncheck the box for using developer options? I've already rebooted and had all my apps optimize.

Quantcast