Support our Sponsors:

Go Back   Touch Arcade > Developer Discussion > Public Game Developers Forum

Reply
 
Thread Tools Display Modes
  #1  
Old 09-01-2014, 04:16 PM
WalterM WalterM is offline
Junior Member
iPad (4th Gen), iOS 7.x
 
Join Date: Aug 2014
Posts: 3
Default Help needed in deciphering strange App Store rating activity

Dear Friends,

This is an appeal for crowd-sourcing help in trying to understand the strange behavior in user ratings for an app of ours happening since May 21, 2014. We know that there are some brilliant minds out here that can detect patterns and discern algorithms much better than we can.

Background: We suspected that there was some persistent "manipulative" activity going on against our app for more than a year (not the typical occasional hit reviews every app gets) and started some extensive logging and testing of ratings and reviews. We will have a full report later this year on our findings, which to us are disturbing concerning the integrity of the Apple app Store rating/review system.

But, for now, rather than predispose you to any conclusions, just want to present one type of raw data we have been collecting to leverage your minds/skills to arrive at an explanation for this data and what it might suggest to you. It will be obvious when you look at it that it is very odd at the minimum. Any thoughts on what might be happening, ideas, suggestions for further testing/logging are welcome and much appreciated.

If developers here can do similar logging of data for their own apps and confirm whether it is happening to them as well (which might indicate a bug rather than an intentional activity) or not (which would imply it is targeted), it will be helpful. Getting to the bottom of this is for the good of the entire community especially if malicious activity is the cause. We have benchmarked against a number of apps and have not found this in any of those apps.

We will be happy to publicly credit any help/findings/insight here from individuals in our report to be publicized later in the year.

We have tried to bring this to the attention of App Store folks at Apple but have only received form responses ("Users may delete or change their ratings at any time"). You can convince yourself that this is not the case here.

The app is a niche market one in crosswords: Across Lite Crosswords (not something that would be an obvious target for ratings manipulation)

https://itunes.apple.com/us/app/across-lite-crosswords/id480513184?mt=8

The logging we have been doing in real time and continuing can be found at:
(although once we have made this public here, the behavior will likely change if what we suspect is true and so the patterns prior to this date and time will be the most valid to study)

http://icrossword.com/monitor/

The logging is done as follows:
1. At 30 minutes past every hour we fetch the number of ratings for each * for the current version of the app via http call to iTunes (in the same way third party app store listing services do, nothing unusual here).
2. If there was any change in the ratings (the ratings sweep at the App Store typically happen at the top of the hour) from the previous fetch, we record the deltas for each * ratings along with the time stamp and the current average rating in each line listing deltas from 5*s to 1* in that order.
3. The logs for many weeks prior to May 21, 17:30 PDT (exactly when this odd behavior started with a large number of ratings disappearing in the same sweep) can be found at

http://icrossword.com/monitor/archive.log

The prior archive does not have time stamps but is the same procedure with before and after average ratings for many weeks. This log provides a baseline case for the kind of ratings we were getting prior to this date, so you can see the stark change in behavior since May 21.

The odd behavior obvious in the above is the persistent behavior of a rating hopping from one rating to another in almost every sweep that appears algorithmic and intentional than random or related to normal user behavior of changing their ratings which occurs very rarely in practice let alone several times every day for more than 3 months.

We have noticed several patterns here that rule out randomness and show particular intent but will leave it to the best minds here to come up with their own independent conclusions. Be happy to share our observation in an ensuing discussion.

Here are some specific questions for now:
1. Do you think this is random or a bug or an intentional automated/algorithmic activity? What is your rationale for that opinion?

2. Do you think there is an intentional attempt here to affect the average ratings either to hold it down (sort of a "rev limiter") or even to slowly walk it down in a way that is not easily detectable (note that average ratings affect search rankings on the app store and so can affect the downloads of a product that depends on App Store visibility)? We have noticed some patterns in this "bot behavior" correlated with an increase in average rating and in appearance of ratings that uncharacteristically appear to reduce the average and totally out of character with the weeks before May 21.

3. If this appears to be an intentional activity, is it possible for someone to do this from OUTSIDE the App Store rather than it happening from inside the App Store itself? Not concerned with why anyone might want to do it for now.

Many thanks for your help and indulgence in this curious matter.

Look forward to hearing your best ideas and brilliant deductions....

Our apologies if you saw this in another forum. We selected the top 3 forums which we thought would reach the maximum numbers in the community that can help in this effort, not to spam any and all forums.
Reply With Quote
  #2  
Old 09-01-2014, 06:01 PM
OnlyJoe OnlyJoe is offline
Member
iPhone 5s, iOS 7.x
 
Join Date: Sep 2013
Location: Auckland
Posts: 92
Default

I am not sure how to really read the data you have given. But it looks like your average rating is pretty much the same. So how are you matching up individual ratings to see if they have actually changed? I would assume you are using the username of who did the rating, not the order you get them back (because the order they get returned is going to change every time, its a very active database). And it makes little sense for someone to run a bot that keeps your rating the same.

I know that apple is always removing reviews for various reasons, most common is that the user has given a high review, and then deleted the app. Pretty sure apple expects the app to remain on the persons phone for awhile after giving a 5 star review. This is to stop fake reviews, or companies that offer mass review services.
Or a review is given, but the app is never actually opened, or the review is done in iTunes on a computer, and the app never installed on a device. Apple seems to have made a lot of improvements to how the reviews work recently.
Reply With Quote
  #3  
Old 09-01-2014, 07:02 PM
WalterM WalterM is offline
Junior Member
iPad (4th Gen), iOS 7.x
 
Join Date: Aug 2014
Posts: 3
Default

Quote:
Originally Posted by OnlyJoe View Post
I am not sure how to really read the data you have given. But it looks like your average rating is pretty much the same.
Thanks for the reply and the opportunity to clarify. Sorry if my post was not clear on this.

Each line represents the new number of ratings removed or added since the last sweep when there was a rating added/deleted. So, for example,

Sep 1 09:30:01 |1 0 1 -1 0 |4.61651

denotes 1 new 5* added, 1 2* removed and 1 3* added since last change in ratings count and this happened at the ratings sweep at the top of the hour at 9AM on Sept 1. The total number of ratings do not matter. It is the deltas when new ratings get added or deleted.

The latter may happen if for example, someone changed their 2* rating to a 3* rating OR someone's 2* rating was deleted for whatever reason and someone else added a 3*.

While one might think this latter case is possible (and it does happen perhaps occasionally), such a matching addition/deletion happening several times a day, everyday for more than 3 months and with more frequency than actual user ratings is extremely improbable to the point of being impossible. As I mentioned, in my earlier post, this does not happen for any app we have similarly tracked. Why would our app be different?

Moreover, it started to happen exactly from the sweep on May 21 at 5:30PM for our app but not for months before it.

If this was "normal", then it should be happening to more apps than not, no?

On the other hand, if you take a hypothesis that it is a single rating that is hopping around algorithmically, it makes a lot more sense, especially compared to the period just before that May 21 "event".

So the more likely explanation is that it is a single bot rating that for whatever reason is hopping around. It is that reason we are trying to determine and asking for help.

We have noticed certain patterns around this hopping that appears to be algorithmic. Just this frequent hopping observation alone should raise eyebrows.

It is not random and how it hops around is influenced by the ratings from users as they come in. So, for example, if the ratings start to go up from good ratings, the hopping behaves differently than if the ratings were not going up. When it hops to a 5* rating itself and jumps back, there is a correlation with what happens to ratings (there are some uncharacteristic lower ratings that immediately follow to lower the average) and it happens regularly enough that it does not appear to be random. How does this "bot" know that there will be lower ratings in the next few sweeps when it moves to a 5* unless.... :-)

This is what we are trying to determine, because it appears to have some carefully designed logic/reasoning behind that hopping and not as crude as spamming an app with 1* ratings. And there is enough data in there to get statistical validity that it is not random. For example, it has never hopped to a 4* and back with one exception over 3 months that may be an actual user changing the rating. We have our own hypothesis why but do not want to pre-dispose others to it yet. A random behavior over that length of a period would make it highly improbable.

The ratings have been monotonically decreasing which can happen to any app but with this strange activity, we would like to determine if this activity is contributing to it or at the least limiting the ratings from going up.

It is statistical forensics.

Quote:
So how are you matching up individual ratings to see if they have actually changed? I would assume you are using the username of who did the rating, not the order you get them back (because the order they get returned is going to change every time, its a very active database).
We do not need to know who left what ratings for the above observation. There is no way to know.

Quote:
And it makes little sense for someone to run a bot that keeps your rating the same.
You are right, except that it makes sense if the activity contributes to the ratings not going up (and hence affecting the app's search ranking which does not round it like what is published on the app page but the latter is a different topic) or it contributes to the rating going down slowly over time so as to be unnoticed and mistaken for normal activity while it should be going up to a higher level.

We do not depend on search ranking inside the App Store for downloads (although it is affected by it) as we have brand awareness outside the App Store that leads to downloads within our niche market. But for an app that entirely depends on being found via search in the App Store, such a behavior that reduced its ranking slowly would death spiral very quickly as lower rankings reduced its visibility which led to less downloads which resulted in less ratings reducing ranking further, etc. This is the reality for most indie apps because they do not have marketing presence. So, if this activity is manipulative, it can quickly make apps die in the App Store. This is why we believe it is important to investigate, not just for our sake but for all developers.

It would be unnoticed except for this odd activity that is targeting our app.

Quote:
I know that apple is always removing reviews for various reasons, most common is that the user has given a high review, and then deleted the app. Pretty sure apple expects the app to remain on the persons phone for awhile after giving a 5 star review. This is to stop fake reviews, or companies that offer mass review services.
Or a review is given, but the app is never actually opened, or the review is done in iTunes on a computer, and the app never installed on a device. Apple seems to have made a lot of improvements to how the reviews work recently.
You are absolutely right on this. Except that it does not explain why such "user behavior" should be happening to our app with such frequency but not to any app within our benchmark sample that allows for similar audience in demographics. .

Anyone can verify this for their own app, which is why we have asked people to check if it is happening to their own apps. If this was part of the normal Apple cleaning up, then it should be happening to more than just one app regularly, no? We have not found one yet within our sample space. That is very odd.

How many people have seen 1* - 3* ratings appear and disappear for short periods for their apps every day and more often than ratings that stick around? We would like to see if anyone notices the same.

Thank you for taking the time to look at the logs and reply. Appreciated.
Reply With Quote
  #4  
Old 09-03-2014, 04:40 PM
HLW HLW is offline
Senior Member
iPad Air, iOS 7.x
 
Join Date: Sep 2010
Location: UK
Posts: 1,007
Default

Quote:
Originally Posted by OnlyJoe View Post
I know that apple is always removing reviews for various reasons, most common is that the user has given a high review, and then deleted the app. Pretty sure apple expects the app to remain on the persons phone for awhile after giving a 5 star review.
Is this true? If so, that's a shame - I pretty much only rate games in the App Store once I've completed them so I often delete them soon after (or even before) I rate them.

Sorry WalterM I can't help you with your problem.
Reply With Quote
  #5  
Old 09-03-2014, 05:28 PM
OnlyJoe OnlyJoe is offline
Member
iPhone 5s, iOS 7.x
 
Join Date: Sep 2013
Location: Auckland
Posts: 92
Default

There is another possibility. And that is fake review bots. Not saying that you are using them, but maybe another app on the same search terms, or category is.
What fake review bots do is give a number of random games a review, say 5, to try and mask the actually game they are trying to review. Would look kind of obvious which app paid for the review if all the bots just reviewed that one app. So they tend to give random reviews to a number of other apps. I have had some random reviews show up on some of my games before, which then later get removed (by apple, I assume).
So what you might be seeing is fake reviews on your app, and then apple slowly catching up to remove them.
And because your app has a reasonable number of reviews, and is in a category where the get rich quick app builders are going to go, there is a high chance that your app might be used to try and mask fake review bots, or to make them seem more real.
Reply With Quote
  #6  
Old 09-05-2014, 01:47 AM
Destined Destined is offline
Senior Member
iPhone 5, iOS 6.x
 
Join Date: Aug 2013
Posts: 689
Default

Quote:
Originally Posted by OnlyJoe View Post
There is another possibility. And that is fake review bots. Not saying that you are using them, but maybe another app on the same search terms, or category is.
What fake review bots do is give a number of random games a review, say 5, to try and mask the actually game they are trying to review. Would look kind of obvious which app paid for the review if all the bots just reviewed that one app. So they tend to give random reviews to a number of other apps. I have had some random reviews show up on some of my games before, which then later get removed (by apple, I assume).
So what you might be seeing is fake reviews on your app, and then apple slowly catching up to remove them.
And because your app has a reasonable number of reviews, and is in a category where the get rich quick app builders are going to go, there is a high chance that your app might be used to try and mask fake review bots, or to make them seem more real.
That was my immediate thought.

Apple isn't doing anything to intentionally hurt your app. Not in their interest.
Reply With Quote
  #7  
Old 09-05-2014, 04:12 AM
Jez Hammond's Avatar
Jez Hammond Jez Hammond is offline
Developer
iPad, iOS 5.x
 
Join Date: Oct 2012
Location: UK
Posts: 36
Default

Quote:
Originally Posted by HLW View Post
Is this true? If so, that's a shame - I pretty much only rate games in the App Store once I've completed them so I often delete them soon after (or even before) I rate them.

Sorry WalterM I can't help you with your problem.
This sounds close to what I think is happening. A player makes a spontanious decision to rate low quite early on - say when some bug effects their experience - then uninstalls the app. The system still processes the actions over a longer period of time, rather than in real time. That's my feeling.
Reply With Quote
  #8  
Old 09-05-2014, 08:23 PM
WalterM WalterM is offline
Junior Member
iPad (4th Gen), iOS 7.x
 
Join Date: Aug 2014
Posts: 3
Default

As an update, we have progressed quite a bit in analyzing the data thanks to some helpful reader tips and an extensive analysis of the data from a member of another forum.

We are able to establish fairly conclusively that an automation/calculated activity of some sort is at work here and have a couple of hypotheses that needs to be tested to see what the intent might be and where the source of this might be (e.g., from outside the App Store or from inside). We have been able to rule out normal user behavior being behind this. Just a matter of time before we get to the bottom of this.

We have also contacted a number of media outlets that find the data very odd and interesting and some willing to publish the evidence and conclusions if and when we have a compelling report. That should get Apple's attention for sure, since we have not been able to do so by direct e-mail because this is such a niche app from their perspective.

If anyone wants to get involved in this group effort and help test out a couple of hypotheses that will make this process go faster, please PM me. We need people with US App Store accounts for this. People who are willing to actively work on this rather than those that are just curious to know. We will have updates here as we learn more for the latter and will not be able to answer a lot of e-mail for that purpose. Hope you understand and very much appreciate any offer of help and active co-ordinated participation. Nothing that violates any Apple policy to do here.

Quote:
Originally Posted by OnlyJoe View Post
There is another possibility. And that is fake review bots. Not saying that you are using them, but maybe another app on the same search terms, or category is.
What fake review bots do is give a number of random games a review, say 5, to try and mask the actually game they are trying to review. Would look kind of obvious which app paid for the review if all the bots just reviewed that one app. So they tend to give random reviews to a number of other apps. I have had some random reviews show up on some of my games before, which then later get removed (by apple, I assume).
So what you might be seeing is fake reviews on your app, and then apple slowly catching up to remove them.
And because your app has a reasonable number of reviews, and is in a category where the get rich quick app builders are going to go, there is a high chance that your app might be used to try and mask fake review bots, or to make them seem more real.
If we were using any fake ratings/reviews ourselves, we would hardly be the ones putting out the entire logs and asking for analysis and trying to get Apple's attention to look at our rating activity. :-)

I think you mean user * ratings than (text) reviews. What you suggest is possible. However, the evidence from the pattern seems to point to a single source rather than multiple sources. Otherwise, statistical probabilities with multiple sources acting independently would result in numbers more than just 1 and -1 in this pattern with this consistency over more than 3 months. Moreover, the removal and re-rating in the SAME sweep period (several times a day for over 3 months with a correlation close to 1) would preclude Apple's action in removing ratings subsequently. How would these sources know in which hour's sweep the rating is going to be removed, so they can re-register again exactly in the same sweep, not before and not after?

The pattern indicates that it is the same source/account removing and adding a rating in each sweep. Otherwise, statistical probability would stagger these between sweeps if they were from two independent sources rather than the addition and deletion in the same sweep with a correlation close to 1.

Quote:
Originally Posted by Destined View Post
That was my immediate thought.

Apple isn't doing anything to intentionally hurt your app. Not in their interest.
We agree with you. But we are not assuming or ruling out who or why at this point. It is for Apple to investigate and find out what is going on when this activity is made evident that questions the integrity of the App Store and the lack of protection around access to their rating web API not to detect and block what appears to be a single source of continuous activity, several times a day.

Perhaps, our app happens to be just a randomly picked test app for some external entity perfecting an algorithm in stealth using a niche app for wider scale ratings manipulation later, in which case investigating this now might prevent a large scale attempt later on. The reason isn't really important for us as much as establishing that there is user unrelated activity that violates App Store ratings integrity in a particular fashion.

If it is coming from inside the App Store, it could be any number of other reasons - a buggy or a runaway ratings or approval software that has got stuck for some reason with our app, a rogue App Store employee with a vested interest that has found a way to do this undetected for whatever reason even if getting found might result in being fired, etc. Not saying any of these are the case. Apple can investigate and determine this in a minute or two at any time. We just need to catch their attention with strong evidence and perhaps a media report that it is anything but normal user rating activity.

Quote:
Originally Posted by Jez Hammond View Post
This sounds close to what I think is happening. A player makes a spontanious decision to rate low quite early on - say when some bug effects their experience - then uninstalls the app. The system still processes the actions over a longer period of time, rather than in real time. That's my feeling.
This is what we thought at first, but this can be statistically eliminated. Some of the probabilistic reasoning was provided above that precludes multiple sources of ratings acting independently. The statistical correlations (e.g., one single rating added or removed per sweep) are too high to indicate multiple sources.

1. Users for our app are not different for other similar apps and there are a lot of apps similar to ours with their own bugs. So, one would expect to see this or something similar for many apps. The fact that we have not found a single instance of this yet (anyone can do this scraping for any app since it is all public) seems to indicate that it is something targeted at specific app(s). And why would they suddenly start behaving differently starting exactly May 21 when there was no update to the app or any device/iOS release that may have changed the experience?

2. We have also found a reasonable but not perfect statistical correlation between the average rating and the direction of this activity (addition and deletion to a higher or lower rating). The difficulty is the "noise" introduced by normal user activity. This is why the logs in the prior period (before May 21) activity is important. They allow statistical subtraction of this "noise" from the later period, to expose the pattern more clearly for analysis.

But it does needs some hypotheses testing to deduce the intent or algorithm, hence our request for participation in this investigation.

Anyone who finds scientific/mathematical forensics interesting and have an aptitude for it should like this challenge... :-)

Thanks for your interest in this matter.
Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Copyright 2012, TouchArcade.com, LLC.

Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.
Copyright 2008 - 2011, TouchArcade.com. Privacy Policy / DMCA Copyright Agent