02
Apr
alienstiny

Way back in the day, if you wanted to know if a piece of software was safe, reliable, or usable before you opened the package, you were out of luck. Once you rip that shrink wrap, you're stuck with it. The app markets of today are, by comparison, a breath of fresh air. A litany of user reviews let you know up front if a piece of software is crap or not before you download. Unfortunately, sometimes developers, like Noosoft Games, abuse this system by, as an example, using the Mechanical Turk system from Amazon to pay for 5-star reviews.

What Is Mechanical Turk

For the uninitiated, here's a little background: Amazon's Mechanical Turk system (named after an 18th century chess-playing automaton) is a platform for businesses to farm out minor, tedious tasks (called HITs) without hiring full-time employees, and lets users perform those small tasks for a pittance. It's not going to earn you $500 a day working from home like those email ads are promising, but it's a handy way to make some spare cash while proofreading some crap that a company doesn't want to hire editors for.

How It's Being Used

turk2 turk1

Left: Mechanical Turk HIT for app reviews. Right: Presto! Perfect 5.0 rating!

The advantages are obvious: cheap labor and easy money. However, as one of our eagle-eyed turking tipsters (turksters?) pointed out, developer Noosoft Games submitted the task of rating its app 5-stars and leaving favorable reviews. This isn't necessarily evil in itself. It's not like users are prohibited from actually using the app and providing an honest review. However, at $1.15 for 30 seconds worth of work, it's hard to imagine too many people will take the extra time, or turn down the HIT if they believe the app is crap.

The above HIT has since been removed (hence the hasty screenshot), but even offers a $0.20 bonus for a "successful" result. That means that, for the low price of $150, a company could buy over a hundred 5-star reviews. Needless to say, this creates some problems for users looking for honest experience. Since this isn't an automated system, it's harder to spot the usual tells of fake reviews: bad grammar, identical sentences from multiple users, vague reviews with no useful information. These reviews are written by real people with a bit of time on their hands.

2012-04-02 11h50_35

This game, The rise of the Aliens, has 121 5-star reviews, one 4-star review and two 1-star reviews as of the time of this writing. It's rare that a game maintains such a high ratio of positive reviews. It's not impossible, but even Angry Birds, the world's most popular mobile game, has only managed an average rating of 4.7 stars on its Rio incarnation on the Play Store. Angry Birds and Angry Birds Seasons are slightly lower. These numbers are pretty obviously inflated.

What To Do About It

For starters, keep in mind that this is just a shady promotion move. Full disclosure: I never downloaded this app and as a general rule, I wouldn't advise anyone else to, either. It should still be remembered, though, that using shady tactics to get reviews doesn't necessarily make an app unsafe. It does, however, call into question the motivations of the developer.

If you see something like this report it to Google/Amazon immediately. Google's official developer policies clearly state that developers "should not attempt to change the placement of any Product in the Store by rating an application multiple times, or by offering incentives to users to rate an application with higher or lower ratings." Which, by the way, also means that app that requires you to rate it 5-stars to unlock new levels/items is also in violation of Google's policies.

The Mechanical Turk policy also specifically prohibits HITs that require users to download software, which the Google Play Store requires before a review can be posted. This means that these tasks not only directly violate Google's policies on Play Store behavior, they also violate Mechanical Turk policy.

A quick search on the Mechanical Turk system doesn't return any similar HITs right now, but that doesn't mean everything is in the clear. Amazon's system isn't the only one out there that provides services like this, nor is it hard to imagine that this kind of shady dealing could be done without the need for a central platform. It's also worth noting that this same type of thing could happen on the iOS App Store, Windows Phone Marketplace or with any other system. Since this tactic involves directly paying a real human being to write a review, it's virtually impossible to distinguish from a legitimate review.

The best way to protect yourself, then, is to keep your wits about you. Remember that nothing online is universally liked. If you spot an app with crummy-looking screenshots, but a thousand reviews stating it's the best thing ever, something might be up. And if everyone says it's amazing, you might be getting played. It doesn't matter how good an app is, someone on the internet will always hate it.

As always, if you find something suspicious on the Play Store or anywhere Android-related, hit us up at tips@androidpolice.com. We are here to serve and protect, after all.

Thanks, Denise!

Eric Ravenscraft
Eric is a snarky technophile with a taste for the unusual. When he's not obsessing about Android, you can usually find him obsessing about movies, psychology, or the perfect energy drink. Eric weaves his own special blend of snark, satire, and comedy into all his articles.

  • Sam

    Thanks for the PSA. This is valuable information. But I always depend on AP/AC or XDA for app reviews/news.

  • tarman

    this was good article...fun and unlaggy....highly recommend

    • Ashkan

      Who payed you to write this? :D

  • Brian

    A fine job you are doing, Android Police(the only cops I respect!) Keep up the excellent work.

  • Marcus

    I though there was some suspicious activity going on with the File Expert app after its devs removed a root-explorer feature, which initially earned the app a wave of disgruntled 1-star reviews in the Play Store.

    After a while there was suddenly a glut of generic 5-star reviews that praised the overall functionality of the app (much like the Turk reviews in the pics above) but made no mention of the offending move by the devs at all.

    My first thought was that the devs had mobilised friends and family to help stem the tide of bad reviews. But who knows what really happened...

    • Mark

      Well, to a degree, that kinda makes sense. Initially there's going to be a bunch of pissed off users who like the old feature. But once those die down a bit, new users will continue to download the app, not knowing that such a feature ever existed, and rate it highly again.

      • Marcus

        I know what you mean. But if you saw the content of those 5-star reviews (which were insipid, totally un-descriptive and sometimes inane) you'd understand why I'm alluding to the possibility of some mechanical-Turk shenanigans going on...

    • Randy

      I noticed that too. I definitely left a 1-star rating after they did that. They promoted it as a free root explorer for so long, then removed that functionality after winning "best app" award or some crap like that.

      I still have the app, but haven't updated it to the version without the root explorer. (Although I bought ROM Toolbox this weekend when it was on sale, so I guess I don't need FE anymore.)

      • Marcus

        +1

        Same here. I'd uninstall it, but I'd miss its FTP server function.

        Does ROM Toolbox's root explorer feature work as well as FE's?

        • junkdog

          Give Total Commander a try if you're still looking: it supports ftp and smb via offical plugins.

          http://www.ghisler.com/android.htm

        • junkdog

          ahh... nvm, i didn't see the server part.

  • Tom

    This review was fun to use and the graphics were amazing. Worth the money - one of the best reviews on the market!

  • andy

    Interesting. I always found that a good, reasonable negative review was worth about 100 favorable reviews.

  • Zepplin76

    This also explains comments like "awesome pinball game" on non pinball games...

  • http://www.androidapphighlights.com Leif

    In the last Google friday review show they mentioned that they usually take a look at apps which have a rating that is too high. The highest common rating seems to be about 4.3-4.5 but everything above is mostly not realistic. It's the same with apps who ask user for a 5 star rating which isn't also really allowed. Asking for a rating is okay, but not promising anything for a 5 star rating like some apps like to do. No app can check which rating you've given to it so you always should give a 1star rating if an App promise you anything for a 5-star.

    • Dandmcd

      Besides paying for reviews from some company likely contracted overseas, this is the 2nd most obnoxious thing about the current review system. I tend to stay away fro many game that shows a hundred 5 star reviews every day, and the responses are typically simply "Great game", "Awesome, so cool!", or "Amazing, add me herpderp1111!!!". Game Insight is one of the worst offenders, but it seems more and more games are pushing 5 star reviews for new content or free stuff now and it definitely is not policed.

  • Andy in Indy

    I distrust the honesty of developers that take artwork from unrelated sources (in this case, the main character from the anime show Cowboy BeBop) and put them in their games. Seriously, you have the artwork from the game, why use someone else work?

  • spydie

    You guys are forgetting, you can't review an app you haven't installed.

    • Dandmcd

      You might wish to reread, real people were paid to buy and review the app, it only cost the developer $150 to get all those 5 star reviews, which is cheap advertising to bump it up on the search charts in the Market.

    • dappwl

      Although this is a real post, now you can rate an app without installing it. I don't know why Google enabled this feature and why the hell aren't they trying to set up a quality limit as apple does

  • Souvik

    What goes around comes around... The original "Mechanical Turk" was not an automaton, but an elaborate hoax http://en.wikipedia.org/wiki/The_Turk . It only seems natural that someone is using the new Mechanical Turk with similar motivations.

  • http://kennydude.me Joe Simpson

    Google should add "Trusted Reviews" from proper vetted Android blogs :D

    • http://mattcasto.info Matt

      Or, like Amazon reviews, even a "was this review useful" system or "verified purchase" system would be much more useful.

      • Bobby Tables

        Isn't there a button for useful/not useful/spam already? But I couldn't find where to see such ratings...

  • Sonotocentertainment

    Since when did jesus Diaz start writing under the name Eric Ravenscraft? ;)

    • http://androidpolice.com/author/eric-ravenscraft Eric Ravenscraft

      Like Diaz would be caught dead in a place like this.

  • ChumbleSpuzz

    @SouvikL: Motivations aside, the name has to do with how the system works. Instead of scripts and programs performing a task, the work is actually done by humans, like the original Mechanical Turk. Though,
    I see your point in that the original Turk was also used to deceive.

  • Botman

    Great store, great product, would recommend - will shop again! A++

  • Icefreez

    Very fun and simple game. Would play again. 5 Stars!!!

  • Dan

    1-star review for Rise Of The Aliens coming up.

    • Dan

      ...and a "thumbs up" vote for every 1-star review so they'll get pushed higher on the list.

  • 5*Review

    I did this HIT and got paid for it. I wish there were more like this. Very little work and I uninstalled the game without even playing it!

    Payola is and always will be part of capitalism.

  • android ratings
  • Jose

    Check out this website, they offer ratings. I've ordered there and they deliver fast and secure: http://www.ratingsboost.com/Android-Ratings.php?pid=6

  • AndroidFan

    http://www.RatingsBoost.com for Android Ratings and Android Accounts in bulk quantities.