1/4 points

*Any* adjustments to the weight of individual characteristics (i.e. changing the weight of Drinkability and Appearance) affect the consistency of reviews, especially when they apply retroactively. On a large scale (e.g. over 1000 reviews), these purportedly "minor adjustments" can alter reviews in a significant manner.

You're suggesting that introducing more depth wouldn't affect existing data/scores, but it absolutely would - especially if future reviews have more minute scoring options. Changing any review system on-the-fly reduces its consistency - if not its accuracy.

Example. If we go to restaurants reviewing French fries on a scale of 1-10, and then change our scale to 1-12 after 100 reviews, consistency and accuracy is demolished. If we don't change our scale, but instead change the weight of characteristics within that scale (e.g. "saltiness" and "crispness" of French fries switch percentage weights), consistency is reduced. If we allow half points within that scale, previous reviews are not as accurate and future reviews are inconsistent. In fact, merely changing the name of one of the characteristics (e.g. "Colour" of French fries is changed to something seemingly synonymous like "Vibrance" or even "Yellow-ness") also affects accuracy because the score given applied to the prior descriptor, not the new one.

Every alteration to the review system damages it.

 

re: changing weighting of appearance and such, this only damages anything if people were doing it wrong in the first place. you shouldn't be giving, say, mouthfeel, a number based on how you want to influence the whole.

re: finer grain, it makes future reviews more accurate than older ones. how is that bad?

 

It's not bad provided you make everyone go back and re-review all of their beers. I know I have (and I know other reviewers have too ) said something like "The aroma is not quite a 4.5 but not a 4 either. Since the rest of the beer is good/bad, I'll give/not give it the benefit of the doubt and score that a 4.5/4". Given the option of a 4.25, I would have gone with that, but instead, I chose an option that was not truly an independent measure of the characteristic being evaluated. As a result, my old scores are potentially incorrect.

And not to go off on a tangent, but going from Drinkability to Overall was a pretty big shift, and I think that is what kojevergas is talking about.

 

drinkability always meant your overall subjective perception of the beer to begin with (e.g. why AALs aren't auto-5s and english barleywines auto-1s)... and some people still don't understand this--they think "overall" means "take a guess at what the weighted average of your other scores will be."

 

Pahn, please consider that I've taken issue with two different factors: one is indeed accuracy, the other consistency. Somewhat analogous to accuracy versus precision in the sciences.

Regarding the drinkability versus overall debate, I have little to say that would do any good. I have my gripes with past changes, but I've made peace with them and moved on. I want to see the site become better; I do not want to gripe about its past iterations. To that end, I oppose future changes to the review system. I've tried here to illustrate the principles behind my viewpoint on changes in general rather than to attack specific changes.

I think everyone wants to see the website as accurate and consistent as possible. I just hope my .02 means something towards that end.

 

I kinda agree, but that's not the core issue here. The first part of my original response to you still remains unaddressed.

 

not going to respond too in-depth because this ruination 10th anniversary packs a kick but...

i see your concerns, but i think you may be guilty of the same mild fallacy as dyan. it's great to care and think enough about it to raise the objections that you do, but sometimes improvements-making-the-past-look-worse looks like "making things worse"... but that's not what's happening. e.g. dyan's post about how in the past one may have wanted to give 4.25, but instead chose 4.0 or 4.5... this is evidence for the future being better served by .25 increments, not evidence that the whole thing will be destroyed by change (even if no one goes back and re-reviews!).

if you care about BA and your past reviews, it makes a lot of sense to be watchful for changes like "1 to 5 being changed to 1 to 7" or something like that. but increasing future precision doesn't hurt the past, nor does adjusting weight when you shouldn't have had weight influencing your scores in the first place.

people would accuse you and me both of overthinking, which i think is silly (if it's "overthinking" to you, the best course of action is to underthink right out of the thread, b/c who cares?). i'm all for trying to keep reviews meaningful... i just think that .25 change is nothing but a help.

 

I dont want to go back and adjust my almost 500 reviews. We had a debate in college in how to score students on their performance, that choosing from a set will always be less accurate than fill in the blank with your own score. now Fill in your own score leads to a hassle for programmers and formulas. Just let it be man.

 

Not a big problem so long as you recognize that this change will decrease the reliability of the ratings even though people will be happier with the increased number of points on the scale. See example of research on this topic here:

https://docs.google.com/viewer?a=v&...FZ03Gr&sig=AHIEtbSKSnTKNx_oiTlffa4zt1IweIB4rQ

 

Don't discount the findings of that study simply because I used Google to give you easy access to the information. If you like I'll find you several dozen articles and/or books on rating scales and measurement theory which support same conclusion that you can find in any reasonable University Library.

 

Apologies for misreading the tone of your post. Usually when many folks say you can find almost anything with Google they intend to communicate their doubts about what has been found. I've been spending too much time on the internet....