Skip to main content

Indie game storeFree gamesFun gamesHorror games
Game developmentAssetsComics
SalesBundles
Jobs
TagsGame Engines

Rating systems vs bad-faith actors

A topic by UPPAJUPP created Mar 30, 2026 Views: 790 Replies: 20
Viewing posts 1 to 5
(+2)

Rating systems are inherently vulnerable to bad-faith actors. There is a lot of people who leave a 1/5 review because a game doesn't match their political views, (why is one person in your game gay? 1/5), or simply because they can. This kind of griefing can tank a game's rating and exists mostly to make a developer sad, regardless of a games actual quality.

So here's a small mechanical change to itch's rating system that I think could help:

If a 1/5 rating represents less than X% of a game's total ratings, instead of counting as a 1/5, it cancels out the highest existing rating.

Example: A game has 9 players who rated it 5/5. One bad actor rates it 1/5. Rather than dragging the average down, the 1/5 simply removes one of the 5/5 votes — leaving the game with 8 five-star ratings and an unchanged average.

And bad games will still suffer. Like have a two votes of 1/5, three of 2/5 and two votes of 3/5. Now if the developer gives himself a 5/5, it will not really matter that much, since one of the 1/5 votes will just cancel it out. The game will still average below 3/5.

The obvious counterargument: won't griefers just switch to voting 2/5?

Maybe some will. But most won't — partly because this change wouldn't need to be loudly advertised, and partly because the psychological reward of griefing comes from feeling like you dealt damage. And a 2/5 probably doesn't scratch that itch the same way a 1/5 does.

It's not perfect, but it's a low-cost change to make targeted harassment significantly less effective.

(+2)

The real issue is, that most games do not have many ratings. Solving an issue for games with less than 10 ratings is not gonna happen, imho. And games with 100 ratings do not have this issue.

Downvoting a game for personal taste or opinions is actually a good faith rating. Players are not quality testers giving a paid review in alpha test. They tell why they would not like the game, or in most cases, just rate negative without saying why. Same with many positive ratings: they do not reflect the quality of the game.

Your way of pruning ratings is not improving the rating system. Worse, it would destroy the little trust it has. Taking the average already evens out good and bad ratings. You need a lot of ratings to start considering removing statistical outliers. But if you do have enough ratings to have a statistic, taking the average is sufficient and what's expected for a 5 star rating system.

Simply canceling out a good rating, because a bad rating is coming in, is unfair (even with a threshold). You math does not work out. The average is going down, if you remove a good rating. Your example only works, because you chose a bad example of a game with a perfect 5.0. Of course 5.0 is staying at 5.0, if you simply change the number of 5s there are.

Try it with 2 3 4 5 5 5 5 5 . That's 4.25. Add a "bad faith" 1 and it would be 3.89. Remove the "bad faith" 1 and a "good faith" 5 and you have 4.14. The rating went down from 4.25 to 4.14.

And that's if you would be able to detect an incoming bad faith rating in the first place. But if you could do that ... the solution would be to remove the bad faith rating - be it a 1 or a 5. And apart from fake ratings by fake accounts, a bad faith negative rating cannot be distinguished from a good faith negative rating. Just as you cannot distinguish a genuine 5 star rating from your friends rating your game up for enouragement. 

(2 edits)

But there are not really any "good faith 1s", and when there are, then the average rating would we very close to 2 anyway.

Your example: 2 3 4 5 5 5 5 5, adding a one here is not deserved and can be seen as in bad faith.
Your bad faith 1 example only reduced the average from 4.25 to 4.14, and saved one bad actor from taking it to 3.89.

An example of a "good faith 1s" would be the reverse of your 2 3 4 5 5 5 5 5, making it 5 4 3 2 2 2 2 2. In this case a 1 could be justified.

This example changes the score from 2.75 to 2.55 using the original rules, but reduces it to 2.42 with these new rule, Making that 1 have an even bigger impact when a game TRULY is bad.

And this works for exactly as good if there are many ratings too. Like if there is a forum of "pure christianity+" that gets together to downvote games with bisexual contents, or having people give "1/5, didn't play, as you stated it contains AI use", or "1/5 why are there so many horror games on my main page, go away!", or "Why did you state you vote X on your Facebook, I vote Y, so you get a 1/5".

The comparison between rejecting the use of the Plagiarized Slop Machine and hating bisexual people is, uh... iffy.

(5 edits) (+1)

If you don't like something, why play it? Or should we rate things 1/5 just based on our beliefs?

I made the post because of a campaign against one person that received so many "1/5 reviews", even on his straight-games, that he published the reviews... and the reason for the "1/5" votes was that some of his OTHER games contained gay themes. So while he had a lot of 4/5 and 5/5, a bunch of accounts added a lot of 1/5 just to try to force him to focus on straight games only. 

But even so, I still don't think games should be rated 1/5 without playing them because of AI either. Because I know a lot of games get that too. For a new developer, that cannot draw and has no budget, having them use some AI art can still be a way for them to be creative. Like you could probably use to learn some free 3d-models too and make some visual novel by setting these models in different poses, BUT such developers do not give artist money either, they just don't get review bombed. In the end, the main problem is that the artists receives less money? (and sure.. that AI slop looks ugly, etc, etc). But in the end, these artists would not have gotten it from like 95% of these people anyway - the only difference is that these creators would just not have been able to be as creative and make their games.

But enough about AI, it was not the reason for this thread.

(1 edit)

"But even so, I still don't think games should be rated 1/5 without playing them because of AI either. "

Well, I do believe that it is entirely fair to give plagiarized stuff a low rating without playing it, because plagiarism is bad and artistically bankrupt. I also believe that it is fair to give a game a low rating without playing it if it is already clear from the screenshots and product page that it is lazy, effortless slop. Both of these rules apply whether AI has been involved or not, but these days, most slop is automated.

You're the one who brought this up as an example of invalid opinions that need to be algorithmically suppressed and as a similar level of injustice as brigading someone for being queer and making queer content. I believe it is fair that I get to argue why I do not see those things as remotely comparable.

(+1)

Sure, but other people can ALSO see that it is AI-generated. And they might not care about your 1/5 vote, they just want to know if the game is good or not? And these votes just makes that harder?

I agree that big companies using AI is bad and sad, but I personally do not care if a small kids first garage made game contains AI images.. as he just would not have hired an artist anyway.

I just don't agree with you that you should hand out a bunch of 1/5 without playing a game. Just don't vote? As those votes you are adding just doesn't help anyone? They can make people feel sad I guess, so if that is you goal? Like increase the sadness in the world? No artist will receive money from you adding that review. And if some people really like AI games, they will probably still play them, and then notice that AI games rated 3/5 is somehow better than regular games rated 4/5 and just start searching for more AI games - and just enforce their view that AI-games are clearly superior?

I still don't want to defend AI games though... but there are good uses. Like I have played a game from one developer that uses 3-d to generate his scenes, but added some AI use for generating books in bookshelves, to not have all of them look the same. And that was a fast and smart way to solve a problem, using modern tools. But I guess I should change my 5/5 rating to a 1/5 because of AI use, as that will surely help people in the world somehow. I mean he did use AI once, and no artist received any money - and his game is at least somewhat well known - yeah, he really should have a 1/5, even though the game was really great.

> they just want to know if the game is good or not

And I rate those games 1/5 because I think the game is bad and want to let other people know about that. That's what ratings are for.

 a bunch of accounts

I believe you meant: fake accounts, not bunch of accounts ;-)

Itch does remove those, but it can take very long. And for some ratings, they might disagree. Or just forgot in all the swamp of other issues they get requests about.

I agree that people should only rate games they are interested in. They need not necessarily have played them, but it's just lame to rate things for which you are not the target audience.

For those situations with few votes you should also not underestimate counter votes. People that might not have rated at all normally, but see the game having a bad average and deciding to give it a bump, because they think the game is (literally) underrated. It's also a reason why this downvoting business will might achive the opposite what people think it would. A situation with a bunch of most likely fake accounts is completly different though.

(+1)

Sorry for all the text. I got carried away and I can type fast. If you want to see how Itch handles ratings adjustments, there are some discussions about how ratings in jams are handled. There are systems in place to deal with the number of ratings. A 4.5 with 10 ratings is not the same as 4.5 with 20 ratings. There they do have solutions for small amount of ratings. But only to compare them to other games in a similar situation, like being in the same jam.

But there are not really any "good faith 1s"

What does 5 or 1 (or 2, 3, 4) even mean?

On Steam you have a clear question: would you recommend the game? Yes/No? There is no middle ground. Does it mean that a "No" translates to a "1-Star out of 5 - would not play again" rating? Unfortunately, in some way, yes, it does.

Do you rate 1 on games you do not like (for whatever reasons, which includes disliking the company for including a root kit drm system or other meta things, like their dlc policy), do you rate them 2, or 3, or not at all? Everyone will have a different answer here.

Or do you rate 4 on the games you like and only 5 on your favorite games? Wait, there are favorites and there are all time favorites. So better rate 3 on good games, 4 on your favorites and 5 on the things you play again and again. That leaves 2 for the average games and 1 for games you did not like.

I hope you get my point here. Because I very strongly disagree with your notion that there are no good faith 1 star ratings. Players are not game critics that are paid to give consistent ratings of games according to some defined criteria. Everyone has their own valid views on what the 5 different numbers mean and when to use them. And some people will only give 1 or 5 stars. While others will only give 5. People giving the full spectrum are rare in my estimate. Giving a rating at all is already a big deal on Itch.

and when there are, then the average rating would we very close to 2 anyway.

So your assumption seems to be, that a game has an objective rating and players rating that game will only deviate from that rating by very little. Like the objective rating of a game is 2.5, so with the standard deviation of 1.5, people can rate it from 1-4. Whereas this other game has an objective rating of 4.5 and obviously players rating it less than 3 deviated too much and therefore gave a rating in bad faith.

Your solution to just weed out the bad faith ratings and try to preserve the apparant average when doing so, is very similar.

But it is the other way round. The "objective" rating emerges after many (>100 at least) people give their subjective ratings. And this includes people that will downvote a game for lame reasons. Or upvote it for reasons that are not any better.

2 3 4 5 5 5 5 5, adding a one here is not deserved and can be seen as in bad faith

You do not know why the ratings are that way. So you cannot claim any knowledge at all about how deserved a new rating would be. Even if you disagree with the newest rating of 1, it would just be your subjective opinion. People have different opinions. One person's trash is another person's favorite game and vice versa. For all you know the ratings of 5 might have been undeserved.

Giving a rating of 1 is not the defining attribute of a bad actor.

2 3 4 5 5 5 5 5, making it 5 4 3 2 2 2 2 2. In this case a 1 could be justified.

Ohhh, you assumed the ordering of the ratings were of significance. I just counted up to have every different rating, except 1 and added some 5s. Then I added a 1 and then removed a 1 and a 5 to show that the average still goes down if you remove a 1 and a 5 and thus showing that this system does not preserve the average. 

But no, you cannot claim any knowledge about justification here either and it also goes again into this assumption of a correct objective rating. Where any nonconforming rating is somehow unjustified.

The average is calculated by looking at the data. You do not look at the average to say some data is in bad faith.

that gets together to downvote games

That's an abuse of the system. A review bombing. It should be reported to the platform, so they can take action. Itch does take action, but they remove fake ratings in bulk and that can take a good while.

Also, if the accounts are not fake, but genuine persons, they might rule, that this is no different from a book club reading a book and collectively voicing their opinion about that book. Though if they do that systematically to just downvote certain things, Itch might still take action. They also do take action against fake upvoting.

But you were talking singular "bad faith" ratings and how to remove them from cacluation of an average. I shall repeat my opinion about this. A solution for games with only 10 ratings ain't gonna happen. And games with 100 ratings don't need it. And actual abuse of the rating system should be reported to Itch. Giving a 1 star to a game is not abuse of the system. Giving it to hundreds of games, just because they share a tag you do not like, would be abuse of the system though, imho.

To use your example of the 1 star with the gay comment. Maybe the rater would have wanted 2 persons to be gay. Can't be gay by yourself, so the 1 person was seen as a token character and received very negative. Or the person did not read description and was surprised to encounter the topic and was disgruntled by that and left a 1 star. I assume you paraphrased examples. For an actual bad faith example, it is not very bad faith. Bad actors usually would hurl insults and worse in harassment ratings. Or use fake accounts. For both the solution is to report them, and not change the rating system itself.

(3 edits)

"A 4.5 with 10 ratings is not the same as 4.5 with 20 ratings" - absolutely, and itch will not know what you would think, that could be a 1 or a 5 or something in-between. So in reality if you see a game with only one vote of 4, you cannot make much assumptions, as after you played it and added your vote, it could be anything between 2.5 to 4.5. And that itch has this is good, I'm just saying my change would not make it worse.

Well, maybe a better solution would be to use the median, and add some extra calculation layer on top of that so that not all games have too similar scores - maybe X% of the score comes from median, Y% from top Z%-percentile, Y% from bottom Z%-percentile, etc, etc.

Another change could be that you would need to have downloaded a game, and waited at least 5 minutes after that, to be able to vote it. Probably still wouldn't really turn down trolls though.

And sure, there are good faith 1s. But also note that a 1 will always affect the score (except in my kind of bad and unrealistic all 5's example). But if you see the following examples, the system only makes it so that if the scores are centered high (2 3 4 5 5 5 5 5), your 1 vote affects the score less and if the scores are really low for a game (5 4 3 2 2 2 2 2) your 1 vote actually affects the score more.

"Ohhh, you assumed the ordering of the ratings were of significance. I just counted up to have every different rating, except 1 and added some 5s. Then I added a 1 and then removed a 1 and a 5 to show that the average still goes down if you remove a 1 and a 5 and thus showing that this system does not preserve the average. " Yes, it is intended that the score goes down in both cases. See above, the 1s just have less or more of an impact depending on the vote distribution. The vote still counts. Lets say a game has an average of 4. As it is now you could vote 5 and slightly increase the average, or vote 1 and tank it. I.e. inherently your vote affects the score more if you vote 1. This change just adjusts that.

"But no, you cannot claim any knowledge about justification here either and it also goes again into this assumption of a correct objective rating. Where any nonconforming rating is somehow unjustified." - again, it just weights the value of 1s lower if the distribution is high and higher if the distribution is low. So nonconforming rating are still counted. It is not like your vote doesn't count.

From my view, you propose a solution to a non-issue. And to make it worse, I do not think your solution would work.

The solution to rating abuse is to remove the abusive ratings.

So, would your solution make it any better? The harrasment ratings would still be given. The developer would still be harrassed. The score would still go down, no matter how you calculate the average. And if the issue is already "fixed", why bother with removing fake ratings. We have implemented a solution after all.

So, would your solution make the rating system better for everyone? Like developers with very few ratings? Or users expecting the rating score to say what exactly? An important point. What do users expect the rating score to say? Do they expect a simple average or do they expect some convoluted math that flattens the rating and makes it nice?

On Itch, giving a rating is already a rating in itself. 1.0 million of the 1.3 million game here do not have a rating score. My advise to people trying to downrate the thing they do not like, is meant serious, even if it might read tongue in cheek. The smirk between the lines stems from the irony that people downrating things are actually promoting the thing they do not like. But again, if they used fake accounts to manipulate ratings, that should be reported. Each rating has a report button, in case you did not know. But that should only be used for fake and abuse and harrasment and not to try cull 1 star ratings.

To me, the core problem with star ratings on itch.io is that very few people actually rate things, so you do not get decent sample sizes and thus a single bad faith review can heavily distort the end result.

(I got two ratings on my game over 10k views and 400 downloads. One of them was genuine, the other one was a retaliatory bad rating by some spammer who didn't like my review of their slop.)

If review counts were higher, individual retaliatory reviews would have far less impact than they do. With review counts being as they are, review scores are functionally useless, with or without your mechanism for filtering out bad faith reviews. Your mechanism also couldn't possibly counteract brigading - where bad faith reviews may outnumber genuine ones - which is unfortunately a real problem that marginalized creators face.

So while I appreciate your effort, I think a better approach would be to just hide the review score on store pages until there's at least X individual reviews, with a relatively high X. Even then, this is a technological solution to a social problem and those tend to be... not great.

Game creators who want to opt out of this nonsense can just disallow ratings for their product.

(+2)

To anyone reading here, I strongly advise against up- or downrating games to promote or discourage content. You should only rate games you want to play. Ratings on Itch are mostly for the person rating them, as the reviews are not attached to the game for everyone to see.

If you start rating down lgbt ai pixel art games, the algorithm would think you are interested in lgbt ai pixel art games, but have just not found the right game and will recommend such games to you. After all, you constanly browse the ai-generated tags and the lgbt tags and the pixel art tag. It's obvious those are topics you are interested in.

And if you really dislike certain content so much, giving them ratings is the wrong way to discourage it. It increases popularity of the games in the eyes of the system. You browsed the game and were engaged enough to bother with a rating. Also, everyone knows that pixel art games get downvotes, so you will not let anyone know that the game is bad for using sub par resolution. Instead you improve the exposure of the game. And ruin your recommendations.

Ah,  we're still pretending that disliking AI and hating queer people are somehow comparable

(1 edit) (+2)

We are talking about rating games here, and not about how justified someone is in their dislike. Ai is legal. In some countries lgbt is not legal. People from those countries can justify that they are more justfied to dislike lgbt, than you can justifiy disliking ai. But why you dislike something or if you are in any way justified to dislike something, is not relevant in this discussion.

Try to justify it here anyway, and you are offtopic. Go to other threads where you can discuss your views on pixel art and other things.

If you systematically downrate games, because you dislike an aspect of those games, you are abusing the rating system and it borders harassment. And of course, it would be "bad faith" to just list up a tag you dislike and downrate all those games.

If you merely downrate games you played, based on your preferences - including dislike towards lgbt topics, ai usage, usage of low resolution graphics, or because the developer uses the "wrong" social platforms - that would be ok.

But regular people would just remove points from the 5 score for their dislike triggers and not go out of their way to evaluate games that feature their personal no-deal triggers.

(1 edit)

"AI is legal" - sure. But reviews are for expressing opinions, not for enforcing the law. "AI is unethical and artistically bankrupt" is an acceptable opinion. "Gay people need to shut up" is hate speech.

Dislike needs no justification. It is preference in games. We can discuss your views in another topic. But be warned, that flame war topics are against the rules. It might be a rather short discussion though, since I dislike ai graphics pixel art about gay characters. I would still play such games, but it would not be bonus points to peak my interest, and I would not vote games down because of such elements.

You came here, triggered by an example list of ratings that have nothing to do with the games itself and misrepresented this listing of examples as a comparison between the dislike reasons. Right there you started acting like one reason for dislike would be better than any other. This was not the point of the examples, this is not the point of this thread. But you try to make this about ai. Please go talk about ai somewhere else.

(1 edit)

No, I am not "trying to make this about AI". That's an example someone else brought into the discussion as an example for something that needs to be protected from "bad faith" reviews. They brought that here, not me.

We cannot come to an agreement about how to prevent "bad faith" ratings if we cannot fundamentally agree what makes a review "bad faith", or if such a thing even exists. If you do not want to discuss what constitutes "bad faith" reviews, you simply should not discuss how to sanction those reviews. Period. Those two topics cannot be separated from one another.

Also, the idea that a game being made with AI has "nothing to do with the games itself" is just patently absurd.

(2 edits) (+1)

My thoughts on this idea—

  • It adds confusion to a system that needs more clarity.

  • I agree with the comment above that a bigger problem is with how rare voting is.

  • The suggested calculation ignores that there are valid reasons for a dissenting vote, like when a game that is no longer maintained breaks, making the first one-star vote a warning, or when the first five star is given after a group targets the project with three stars and nasty comments.

  • What number is X? What happens when the next one– or five–star rating is given outside that percentage?

My thoughts on AI companies for parts instead of using stock assets or commissions for trade from actual creators— who gets money for what is a concern of big publishers. No one else focuses on that every day. It’s so far down the long list of concerns about general AI use for most indies that it’s laughable.

To put this simply (tho if you’re very young, you won’t likely believe me): kids in the garage who choose to put AI slop in their first games are going to be embarrassed and disgusted by that decision years from now. They could use anything like the free assets a bunch of artists here on itch.io offer. We share without charge not because we’ll somehow make money off the downloads but because we’re hoping someone, somewhere, will make a cool thing with it. But it’s not cool to be willingly associated with the humongous AI tech industry scheme we were all pulled into.

Edited for formatting.