21
Mar
Google-Thumb

For over a few hours on Monday, several Google services came crashing to a halt. Users all over the world were unable to send messages via Hangouts, engage in video chats, or check Google Voice. Some people trying to create spreadsheets with Sheets were met with 502 errors, and people taking advantage of the multi-player aspect of Google Play Games were also affected. All of this apparently resulted from an oops during a routine hardware maintenance event where the company miscalculated available capacity.

Oops

During these maintenance events, Google redirects traffic away from certain backend servers to a new set while they perform their work. Due to this slip-up, the new servers lacked enough capacity to handle the redirected traffic. Google Engineers started running the maintenance procedure at 8:25 AM and realized something was up roughly twenty minutes later.

Oops2

The team then brought in additional capacity, halted the maintenance process, and started bringing users back online in waves to avoid overwhelming the system.

These things happen, but if you need the reassurance that Google's learned its lesson, here is a dry list of bullet points the company's provided to show what it's taking away from the experience.

  • Review memory requirements and increase the memory capacity for the affected backend
    servers to meet peak load needs.
  • Implement better monitoring for memory utilization and usage tracking to ensure that servers
    have sufficient capacity available.
  • Lower the alert threshold for errors with the Hangouts service to improve Engineering
    response time.
  • Review internal procedures for bringing up emergency capacity to speed mitigation efforts.
  • Continue work in progress to improve the resilience of Hangouts service during high load
    conditions.

You can read the entire incident report for yourself at the link below.

Google Apps Incident Report - March 17, 2014

Bertel King, Jr.
Born and raised in the rural South, Bertel knows what it's like to live without 4G LTE - or 3G, for that matter. The only things he likes sweeter than his tea are his gadgets, and while few objects burn more than a metal phone on a summer day, he prefers them that way anyway.

  • Thomas Dussan

    "These things happen, but if you need the reassurance that Google's learned its less,"

    Lesson*

    • http://bertelking.com/ Bertel King, Jr.

      Thanks. I've learned my lesson too.

  • dan

    Two questions:

    1- where is android 4.4.3, Google?

    2- Why is Google Maps on iOS better than on Android? I keep getting GPS location lost and Maps' freezes on my Nexus 5 while it works very smoothly on the iPhone 5s !

    • Scott

      Yes by all means bring android updates into the article. We don't get enough of it every other day of the year.

    • PiLoT

      Why are you asking for an update that we dont know what it contains? It'd be like me whinig at microsoft for not giving me windows 10

    • Mayoo

      Obvious unrelated post is obvious

      • Bob_Poutine

        Because, Canada

      • duck hairs

        Cinema sins?

        • http://petercast.net Peterson Silva

          Love Cinema sins, but "adjective noun is adjective" is an old meme, mate.

          • duck hairs

            Righto

    • http://bertelking.com/ Bertel King, Jr.

      1 - It was lost during routine maintenance.

      2 - Have you tried restarting it?

      • dan

        Sorry for the off-topic question. I am waiting for the play services update or os update or anything to fix Maps's problems.
        I have already tried restarting, cleaning cache and data, reinstalling. None of that works, GPS location occasionally lost in the middle of LA and LV ! Worse than that are the frequent freezes in the middle of the road or after every turn

        • PiLoT

          Firstly, why complain here and not to google directly?

          Secondly have you tried installing anything else that may use gps data? this would tell you if its an error with gps or maps.

          • Roger Siegenthaler

            Because google has the worst customer support in the world, seriously... I've never gotten more of a reply from them than "We have forwarded your request to the applicable team, please contact them directly from now on." -.-

          • PiLoT

            then ranting in public is so much more likely to get them to listen to?

      • A A Ron

        ..

    • KingofPing

      Probably not the app, but the device.

      The app itself does not manage your GPS connectivity at all.

      Try a different maps app on the same device.

      I know several devices were plagued with bad GPS...or you may be unlucky enough to have caught a bad one in a good batch. It happens.

    • trjlive

      Entirely out of scope for this article. But I'll just say there is nothing wrong with Maps app on the Nexus 5, it works perfectly for me. In fact the newest version of the app is much faster than before. So if you're having problems with it, it's something to do with your device. Either faulty hardware, or you've made some tweaks or custom ROM or something that's causing problems. But that's not Google's fault.

    • Austin Akens

      Hi, I am Google. I have your answers:

      1- Yes.

      2- Yes.

      Anything else we aren't doing as well enough on Android as we are with iOS?

      Yes.

    • Roh_Mish

      The transparent bars make iOS one look good. But android one works good. Try using other apps and check if gps is lost. If yes then problem lies in your phone

  • myculito

    As someone in the IT Audit industry, I appreciate Google publishing their findings to the world.

    It's always good to do something like this for transparency purposes, especially if you're counting on them for your business.

    • Richard Yarrell

      Google has been so full of shit in so many aspects lately they have become a pretty SAD CONCLUSION.

      First the purchase of Motorola against your business partners over a year ago. Then the pitiful joke of losing over 400 million every quarter for over a year then just simply dumping Motorola later just scandalous.

      Now the inability to inform it's users of the android platform about the DRASTIC CHANGE in the operating system of the new Kitkat 4,4,2 update.

      Creating new permissions where people's favorite third party applications aren't working properly and restrictions with sdcard usage which has been the hallmark of android since it's inception.

      I wouldn't trust GOOGLE as far as I can spit they are definitely a sad farce from a business standpoint period.

      The funny part is the overall joke will be on it's pitiful Kitkat 4,4,2 update numbers which won't add up.

      A lot of users were smart enough NOT TO UPDATE TO KIT KAT 4.4.2 and the ones who weren't have started to take updated Kitkat handsets BACK TO CARRIERS for 4.3 jellybean handsets.

      Thanks to the WARNING given here by android police back in January/February I made up my mind to deny any Kitkat 4,4,2 update on my Jellybean 4.3 Note 3.

      At the end of the day it's that individual user who decides what kind of day to day user experience they want on the device they paid for.

      GOOGLE can FUCK it's cloud storage and go FUCK THEMSELVES TOO.

      • Danny

        Why so angry? You can always choose to not use Google Products. ;)

      • jesuguru

        If you take "Yarrell" and remove "Yell" you're still left with "arr".......

      • squiddy20

        Please explain how Google has been "so full of shit lately".

        As it was stated on virtually all tech blogs at the time of the purchase, Google bought Motorola almost purely for the patent portfolio the latter held. This is evidenced by the fact that, when Google sold Motorola to Lenovo, Google kept practically all of the patents and a few specialized groups (like Project Ara). Further, you didn't see Google playing the favorite child with Motorola. There was no Motorola Nexus device, Google execs had basically no say in what Motorola did/didn't do, and it was stated many many times that Motorola was run under it's own power, and that Google just owned Motorola in name. Please check your so called "facts".

        Since when has Google ever informed users, especially users of non-Nexus/GPE devices, about "drastic changes" to the OS? With the exception of during the release/preview of a new Android version, they don't. Why were you expecting them to start now? What a little cry baby. Do you need mommy and daddy to hold your hand because big bad Google did something you think is "wrong"? Newsflash: Google had wanted it this way since 2011, as the brilliant AP article on the KitKat SD card issue stated. It was only just recently that Samsung started playing ball with Google. The Note 3 and other affected devices are Samsung's products, not Google's. Why should Google have to notify users of Samsung devices, about changes that Samsung is finally instituting after years of Google asking them to make it this way? What a joke.

        "I wouldn't trust GOOGLE as far as I can spit they are definitely a sad farce from a business standpoint period." 1) You clearly don't know what a "farce" is. Come back when you've either looked up the definition, or are making billions of dollars yourself. 2) And yet, you probably typed all of this up on that Galaxy Note 3 that is your only gateway to the Internet, which is powered by Google's Android OS, and uses many of Google's apps and services. What an ignorant Samsung fanboy.

      • http://www.modminecraft.com/ Nick Coad

        "A lot of users were smart enough NOT TO UPDATE TO KIT KAT 4.4.2 and the ones who weren't have started to take updated Kitkat handsets BACK TO CARRIERS for 4.3 jellybean handsets.

        Thanks to the WARNING given here by android police back in January/February I made up my mind to deny any Kitkat 4,4,2 update on my Jellybean 4.3 Note 3."

        All of that is hilarious, and nothing more than utter nonsense. That's not happening at the levels you describe at all.

      • DirkBelig

        Enjoy your iPhone, Bub. Apple is totally transparent and consumer-friendly.

  • Defenestratus

    These pain points are shared by service providers and IT departments the world over. I can only imagine the nightmare of dealing with it on the scale that Google has to.

  • Michał Droździewicz

    You have extra line break in your dry list ;)

    • Kostas

      true

  • Theratchetnclank

    As a systems engineer i can tell you these things happen sometimes, bet the engineers feel guilty lol.

  • Ritalmumbai

    What you predicte after seeing this http://m.youtube.com/watch?v=xVVeoAIJLDU#

  • FrillArtist

    This kind of makes you realize something. What if Google just went down and out without a single trace left? Just disappeared into thin air. It makes you realize how many people would be stranded.

    • SSDROiD

      Life isn't worth living if it doesn't have Google in it.

  • mesmorino

    Wait, this is the same Google that's trying to force everyone to the cloud, right?

    Just so we're clear.

    • PiLoT

      you want an online service thats 100% fool proof. are you even aware of googles uptime?

    • Bonzo99

      "Force" you to use the cloud? Do you know how to use that phone in your hand. There is nothing that Google designs yet that "forces" you to use the cloud.

      • mesmorino

        Yes, hence my use of the word "trying"

        • Bonzo99

          That doesn't make sense. How are they trying to force you to use the cloud when everyone of their services that can use cloud based storage also offers a local storage option? Google Music, Google Keep, etc. You're reaching for something that isn't there. Regardless, even if they are pushing for cloud it's only to offer you more options.

          One more point. Just like was mentioned before. Look at googles uptime compared to other cloud services.

          Last year Google had 99.984 ℅ uptime for gmail and Google apps and cloud. Microsoft was close with 99.94 %.

          Not bad huh? How much do you pay for those services? I'll bet you're one of those entitled people that refuse to pay for apps and Google services right?
          But you expect everything to work 100% of the time.

          Get a clue and do some research or go troll somewhere else

          • GraveUypo

            do you know what i miss? downvoting.
            freaking disqus

          • mesmorino

            You must be new here, because if you weren't you'd have seen the countless other comments I have made on other articles on this very website.

            To counter your irrelevant point, when I say that Google is forcing cloud use, it is not through pushing anything but rather their insistence on limiting your other options, to wit: sd cards and internal storage. Nobody expects 100% uptime, that is a physical impossibility because even if your gear is totally bulletproof shit happens - from earthquakes to lightning strikes. This is why I ignored the other guy, because that's not what I'm saying at all and that's an unreasonable request.

            But if they would quit interfering with these other options, then your dependence on the cloud is reduced and during those random downtimes you're not stranded. Pushing for the cloud is all well and good, but not at the expense of other options.

            And what's that crack about not paying for apps and services? I'm trying to be civil here but seriously. Go fuck yourself with the horse you rode in on, I do what the fuck I want with my money. It is not up to you to be making baseless, arrogant and insulting comments about me. And for your information i do pay for the apps and services I like, starting with SwiftKey over three years ago, and ending with Ultimate Rotation Control last night.

            But of course your dumbass wouldn't know anything about that seeing as you don't know me. How about YOU get a clue and take your trolling somewhere else?

          • Bonzo99

            LMAO! Thank you you have just confirmed it for me! Oh, and it's "go fuck yourself AND the horse you ride in on" not WITH the horse you rode in on.

            I really don't understand why everyone's crying about SD card support. they are not trying to limit SD card support that has to do with the way that Kit Kat and the newer operating systems along with the change from dalvik runtime to ART that is interfering get a clue.

            Not new to reading here. I just choose not to troll like you.

            When you start the curse on a public forum you show your intelligence level. thank you and have a nice day!

          • mesmorino

            So now you're drifting- Alright, what have I confirmed for you? That I (along with many others) do not appreciate their beef with sd cards? That's hardly a secret, take a look any number of sd card or cloud related articles on AP and you'll find me talking shit about Google's could predilections. You MUST be new here otherwise you would have seen this previously.

            And no, when you make random, baseless and insulting assertions about people you don't know just because you disagree with them, THAT is when you show your lack of ability to carry on a civil discussion. My language choice is not an indication of anything, it is merely a direct response to your stupidity.

            And since you're not the language police, go fuck yourself WITH the horse you rode in on. You have brilliant day my friend.

          • Bonzo99

            Again, I have seen everything about the SD card complaint and while I disagree I understand the complaint.

            What is "could predilections"?

            You're a douche bag!

            Take your SD Card and shove it up your ass!

            My assertions are far from random or baseless. ( I did and continue to do research before speaking unlike yourself)

            You sound like someone that should buy an Iphone.

            I am in fact the language police when someone isn't smart enough to type it properly.

            I hope you have a terrible day!

          • Roger Siegenthaler

            ok, 1. Deal with it, some people want SD cards... they make sense, no matter how you want to twist it and google's banning of the usage of them in any meaningful way (deleting all content when the app is uninstalled without warning, camera app --> absolute disaster for most people). Google fucked up on that one, they should have communicated it better and in the opinion of almost everyone not changed anything, but enough of that.

            I want to get to your statement about how the SD Card change was brought upon android by the change from a JIT compiler to an AOT compiler... please enlighten me about how that is connected.

          • kilteddroid

            Blimey. Mr angry got out of bed the wrong side today. I would hate to see the reaction to something that was actually important....

  • Erica Crawford

    they need another bullet point: schedule regular maintenance for 2 a.m. like every other website

    • trjlive

      Except that 2am Pacific time would be during one of the highest usage times of Google services. You have to remember that Google is an international service provider, and as such, there is no such thing as the "middle of the night" maintenance window. I am a special contractor for Google in Los Angeles, and I review usage pattern reports before planning and scheduling certain processes to occur during the lowest usage window. Believe it or not, 8am Pacific time is actually one of the statistically lowest usage times for those services. There are more Google users in Europe and Asia than in the United States, so they actually did plan the maintenance for the best time. While the outage affected a lot of American users, it was still a small percentage of Google's global user base.

  • A2theC

    Go easy on them, it was St Patty's day!

  • her
  • MistiXF

    As a user who tries to walk away from all Google services, it makes me happy to realize that I did not notice any of those failures at all. Looks like there is life besides Google after all, holy crap.