It's Time to Embrace Ratings

Discussion in 'BeerAdvocate Talk' started by Todd, Jan 10, 2022.

  1. Todd

    Todd Founder (6,558) Aug 23, 1996 California
    Staff Moderator Fest Crew Society Trader

    There's been lot of talk over the years about beer ratings (1-5 and less than 150 characters) without reviews (150 characters or more). Some have suggested that those who just rate don't give it much thought (they're just ticking), that ratings are probably significantly higher versus those attached to reviews as a result, that we should give reviews more weight, or not count ratings at all.

    Let's take a look by pulling some numbers...

    Historical
    These numbers reflect when we had an app, which brought in a ton of ratings and reviews. What's a ton? From May 2014 to July 2016 (app lifespan), the app and site pulled in over 3,200,000 ratings and reviews with 86% being ratings.

    (Yeah, I know. We need an app. Along with being able to rate a beer multiple times, attaching a photo to your rating with ease, adding serving type back, and much more.)

    Anyway, when we look at everything, there's a 0.07 differences between the average rating w/o reviews and the average rating w/ reviews, and 55.7% more ratings over reviews.

    Total Ratings: 10,221,853
    Total Average: 3.9
    Average Rating w/o Reviews: 3.92
    Number of Ratings w/o Reviews: 7,080,810
    Average Rating w/ Reviews: 3.85
    Number of Reviews: 3,141,043

    2019 (The Before Times)
    Back when people were going out all the time, there was a 0.05 difference between averages with 58.8% more ratings over reviews.

    Total Ratings: 423,544
    Total Average: 3.98
    Average Rating w/o Reviews: 4.0
    Number of Ratings w/o Reviews: 299,932
    Average Rating w/ Reviews: 3.95
    Number of Reviews: 123,612

    2020 (Pandemic)
    We saw a significant drop in overall site activity due to the pandemic, the gap between averages closed a bit with a 0.03 difference, and users posted 44.6% more ratings over reviews.

    Total Ratings: 297,068 (down 29.9%)
    Total Average: 3.99
    Average Rating w/o Reviews: 4.0
    Number of Ratings w/o Reviews: 191,123 (down 36.3%)
    Average Rating w/ Reviews: 3.97
    Number of Reviews: 105,945 (down 14.43%)

    2021 (Pandemic Limbo)
    The number of ratings and reviews were both down with 40.1% more ratings over reviews, averages went down for both, and the difference between averages widened back to 0.05.

    Total Ratings: 270,169 (down 9%)
    Total Average: 3.98
    Average Rating w/o Reviews: 4.0
    Number of Ratings w/o Reviews: 169,003 (down 11.6%)
    Average Rating w/ Reviews: 3.95
    Number of Reviews: 101,166 (down 4.5%)

    Ratings Need a Hug
    The ability to rate a beer without reviewing has been around for over a decade, it's clearly more popular amongst users, aggregated ratings are very useful, and ratings act as a gateway to reviewing and contributing elsewhere on the site. Without ratings, we'd have a lot less user activity, less beer data, and many beers wouldn't get noticed on the site.

    Like posting in the forums, posting a beer review requires the user to feel comfortable articulating their thoughts. We can't force people to review. We've tried. And we've failed. It needs to happen organically. Same with rating, but to a much lesser extent. So, I plan on making some adjustments to encourage more users to log in and rate beers, while continuing to explore ways that we can guide users toward reviewing overtime without it feeling intimidating.

    I'm already projecting over 300,000 total ratings in 2022 without changing a thing. That said, I'd like to aim for 500,000. Open to any ideas that you might have, but confident that we can collectively make it happen by embracing ratings instead of hating on them.

    Thanks for reading.
     
  2. Rug

    Rug Meyvn (1,450) Aug 20, 2018 Massachusetts
    Society Trader

    I don't often post in NBW and WBAYDN threads, but maybe we could encourage posters in there to at the very least give the beer they post a quick rating
     
  3. DIM

    DIM Poo-Bah (3,246) Sep 28, 2006 Pennsylvania
    Society

    I think the possibility of getting some kind of feedback would help generate reviews. Just a simple "like" would be sufficient.

    Tagging other users in rates or reviews?
     
  4. SFACRKnight

    SFACRKnight Poo-Bah (1,948) Jan 20, 2012 Colorado
    Society Trader

    I use this site to talk about beer. It is my Facebook. Or meta. Or wtf ever. Here are some reasons why I have little interest in rating Or reviewing a beer...

    I don't feel like over thinking. I want to drink the beer and not focus on the why's of the experience. Why do I think this beer is better than Or worse than another beer within the same style.

    Ease of access, @Todd you are right in that an app has and would continue to facilitate this part of BA. I am lazy. I don't even try to post pictures any more due to laziness. If rating and reviewing were easier to access I would do it more often. Same with pictures. An app could facilitate both issues.

    Sometimes I have issues articulating what it is that I am tasting. A subforum of how to taste, issues we run into with descriptors etc could be helpful. I promise not to snark it up. If I felt more confident with my descriptions I would likely start up again.

    Just talking out loud as to the reasons I don't do it as often as I should. I really don't care for personal data points and was using them as a tool to help hone my homebrewing. Can't fix something you don't perceive right? Now that my kids are of the age that I need to spend that time with them I don't brew much any more. I may step back into it soon, but for now it is what it is I suppose.
     
  5. BigIronH

    BigIronH Meyvn (1,111) Oct 31, 2019 Michigan
    Society Trader

    We’ve got your word not to “snark it up”, now we just need the word of the dozen other users who are bound to do so…:wink:

    Cheers.
     
  6. Shanex

    Shanex Poo-Bah (1,916) Dec 10, 2015 France
    Moderator Society Trader

    This about sums it up for me.

    I mostly post on the forums, often in WBAYDN. Sometimes in NBW. Have rated 457 beers and reviewed about half of them.

    And I do it less and less.

    I’m mostly here to enjoy the site, the discussions, the beers and the camaraderie.

    To not overthink it is a key point, BA is designed to make us learn about beers, breweries and also make fellow new beer geeks friends (I have met three venerable advocates and traded beers with each of them).

    I just enjoy the site for what it is. I can’t think of a better forum to discuss anything beer related and lord knows it’s a wide range of topics.

    I’m happy here. If I run out of new beers to rate or review, I’ll just buy some old reliable, go post it in the daily thread and that’s good enough for me.
     
  7. zid

    zid Poo-Bah (1,669) Feb 15, 2010 New York
    Society Trader

    @Todd , Who would be your primary targeted group for any initiatives to increase ratings?
    - existing users who already rate
    - existing users who do not rate
    - people w/o an account currently

    One would think incentives can drive results. "What's in it for me?" needs a good answer... whatever that may be. I am ignorant about how badges on Untappd work, but I am assuming that they set more obtainable goals far more clearly and concretely (and on-going) for the user than the karma system here. "Tracking your beer" might not be incentive enough with Untappd in the space. Any reward also has to work well with whatever burden is being placed on the user. If the burden is high, the incentive/reward needs to match it... and rewards like badges/karma are immaterial. A low burden can drive more interaction. If someone rated a beer on BeerAdvocate, would they currently have to enter 5 different numbers (look, smell, etc.) or can they just enter 1 number?
     
  8. GratefulBeerGuy

    GratefulBeerGuy Poo-Bah (3,241) May 20, 2006 New Hampshire
    Society Trader

    I'm in it for reviewing. BA for me, since 5/20/06 has been the place I rate and share my thoughts on the beers I have. It's more like my own catalog of beer, with exactly the aspects I see worthy of putting into words. I love this site for that, more than anything else. I think rating without a review really doesn't hold much weight. That's just my feeling. A rating without at least, a few words of what's good, what's bad doesn't mean anything to me. Thank you for what you do here at BeerAdvocate.
     
  9. bret27

    bret27 Poo-Bah (1,976) Mar 10, 2009 California
    Society Trader

    [​IMG]

    I used to review most beers on here. Then I rated everything to essentially log and keep track. I then switched over to untappd. I double-logged everything for a while but that got tedious
     
    MTKATL, Ranbot, eppCOS and 8 others like this.
  10. donspublic

    donspublic Poo-Bah (2,123) Aug 4, 2014 Texas
    Society Trader

    Honestly, most people are not going to be comfortable spending the time to rate a beer. When I first started, I lingered a while, then started giving numbers, then said fuck it, no one knows me, I don't really give a shit what they think and I started writing reviews. The main reason for that was I wanted to go back and see what I liked about the beer or didn't like. I think my % of reviews to ratings has increased now that I drink more from home, but I am not drinking as many different beers as I did when I first got into this. One thing that might get people to give you more of a review would be the usage of tags. For example for look, there would be a list of tags they could select. For smell, taste another set of tags. So basically a review would look like below:
    Look: #clear #2fingerhead #goodretention #excellentlacing #golden
    Smell: #coconut #ripepineapple
    Taste: #pineapple #citrus #coconut
    Mouthfeel: #lowbitterness #dry
    Review: like this beer because it was easy to drink and was tropical in nature, and I am a sucker for anything with sabro

    I know this isn't something that most would like, but it gives you a feel for what the beer is like with little effort on the reviewers part. Honestly I am a shit reviewer, I review for myself and if you looked at my reviews, they could be decompiled into tags like above. One advantage of this, with the usage of tags, you could then offer a search that could go against these tags. For example if I know I like the pineapple/citrus/coconut thing I can search out for it.
     
  11. bubseymour

    bubseymour Poo-Bah (3,338) Oct 30, 2010 Maryland
    Society Trader

    @Todd agree with others, I think if you marketed/sold it to new comers and existing members alike with “here on Beer Advocate we provide you the tools to record tasting ratings and notes (then explain all of the various bullet points as to how useful the data stored will be to the individual). I think you know most of the pros from the forum comments over the years but we can always elaborate again and share enhancement ideas.
     
    Whyteboar, FBarber, Rug and 2 others like this.
  12. chipawayboy

    chipawayboy Devotee (465) Oct 26, 2007 Massachusetts
    Trader

    Reviewing a beer on BA is not intuitive - in order to entice others it would certainly help to have a search/rate interface right on the homepage. Perhaps this is too radical and expensive to code -- but it's the most obvious solution. Personally - I'm in the untappd camp since 2019 - given the ease and ability to post photos quickly but I have often crossed ref to BA if I'm on the hunt for a beer -- doing it less and less these days. BTW - thx for sharing the data- particularly surprised at the minimal avg. score delta between ratings for those w/ and w/out a review.
     
  13. elNopalero

    elNopalero Poo-Bah (4,189) Oct 14, 2009 California
    Society Trader

    I think we all need a hug!

    One of my BA goals for this year is to contribute by reviewing more--and posting my reviews in a timely manner. My "system" is not at all efficient. I take/save notes to write up/post down the road, ideally. In practice, it leaves me with a backlog of things to post and edit. I could streamline this by composing and posting in BA, true, but I am not always online or in a situation where that makes sense.

    While these reviews are a contribution to the site, they are also for me to keep track of what I've had, which in turn helps me track what works and doesn't work for me.

    So, thinking about how I approach reviewing, how to make the site user-friendly/efficient, and how to set BA apart from Untappd, along with what @donspublic wrote, what about a menu with a limited set of options? "Like" or "dislike" plus an overall score might be the bare minimum, along with some overly general and deliberately vague descriptors for flavor/aroma/appearance: "hoppy," "sweet," "roasty," and whatnot.
     
  14. MNAle

    MNAle Poo-Bah (2,262) Sep 6, 2011 Minnesota
    Society

    If you really want to embrace rating, then you need to make it a rating, not a half-review.

    Look -Smell-Taste-Feel-Overall without any words of explanation is no better (or at best, very little better) than a simple 1-5 rating.

    IOW, add a quick 1-5 rating without L-S-T-F-O. Probably leave a place for a short comment or two.

    Leave the current system for those who want to at least say something more, full fledged review or not.

    BTW, I ignore ratings. And it has nothing to do with any of the aspects you gathered stats for, @Todd.

    It is because I am not interested in merely someone's view on whether or not THEY liked the beer. I want to read the words for some explanation of WHY they liked (or didn't like) the beer. For example, if someone doesn't like an English-style barley wine because it is too malty, well, I know what to do with their rating.

    If I am looking up a beer on BA, I first hide ratings and then sort by "top reviewers." Then, I scan for high and low numbers and read.

    BTW, I consider the ratings on UT to be less than useless. Most don't add any comment at all, and it is too tedious to search for those than do.
     
    #14 MNAle, Jan 10, 2022
    Last edited: Jan 10, 2022
  15. mudbug

    mudbug Defender (611) Mar 27, 2009 Oregon

    I really don't care as I ignore ratings, if it helps the site great.
     
    2beerdogs, mactrail, beergoot and 3 others like this.
  16. MNAle

    MNAle Poo-Bah (2,262) Sep 6, 2011 Minnesota
    Society

    Once I got over the mental hurdle of not having the finely-tuned olfaction and palate some reviewers have, and just started describing what I smelled and tasted within my simple smell and taste vocabulary, it wasn't that hard.

    Just describe what YOU smell and taste. Don't worry if you can't pick out the specific hop being used or yeast strain. If you've never tasted a dragon fruit, who cares? Maybe it just tastes like a pear to you.
     
  17. maximum12

    maximum12 Poo-Bah (5,136) Jan 21, 2008 Minnesota
    Society Trader

    I've enjoyed writing reviews for well over a decade now. They are incredibly useful to me as a record of what I've had & how I've felt about it in the moment (& sometimes what kind of socks I was wearing). I also often dial up other user's opinions at times before I buy a beer; reviews have saved me from buying infected and/or trash beer, which I appreciate.

    That said, there are a lot of barriers to significantly increasing reviews. People have already mentioned some. One big one is venue. I very rarely review beers on draft as I'd rather enjoy the atmosphere & people I'm with than tap away on my phone like the geek I am. I know a lot of people feel the same. And almost doubling the ratings year-over-year seems unrealistic on the face of it.

    As for increasing reviews...I don't see it happening in a major way. By highlighting the benefits one could encourage a small percentage to jump on the bandwagon & I like the 'rewards' angle noted above & the ability to react to reviews or tag other users. I miss the 'serving type' because that was also part of my research in looking into new beers.

    To summarize my unhelpful post: I believe BA would need to create major incentives or make reviews hugely more useful to the browsing public to make any noticeable increase in reviews. A free box of beer for rater of the month? Or for the most likes on a review (though that could be manipulated)? Increased 'status' for those who rate X beers a month?
     
  18. jonphisher

    jonphisher Crusader (739) Aug 9, 2015 New Jersey
    Trader

    Most of us, if not all of us who participate in NBW do this. The times I post in WBAYDN I try to as well.
     
  19. BillAfromSoCal

    BillAfromSoCal Devotee (418) Aug 24, 2020 California

    Ditto...A mere numeric rating means little to me. I use the review to discern something about how thoughtful and qualified the rater is in addition to see if their written description is a match for what I am looking for in a beer. However,I am actually surprised at how closely the average review score is to a rating score. It may suggest to me that I need to rethink how often I minimize the value of ratings while focusing on reviews.

    I would value some way of commenting on reviews, either with a thumbs up or better yet, a written comment either supporting or not supporting the reviewer's observations.
     
  20. BigIronH

    BigIronH Meyvn (1,111) Oct 31, 2019 Michigan
    Society Trader

    For me personally, a rating is a review without words. I score the beer just as I would in a review, and many others do as well which is made obvious by the numbers.
     
    RMW66, LXIXME, Mista_Carta and 9 others like this.
  21. Harrison8

    Harrison8 Poo-Bah (4,860) Dec 6, 2015 Missouri
    Society Trader

    I'm firmly in the 'review first' camp, and have expressed that during my time on these forums, but anymore, I'd just like to see more user participation. Especially in going to active vs. inactive beers. If locals to me are not rating beers as they get released and re-released, then many active beers may fall off the page.

    It would also be nice to formulate a BA score for more local beers. Even with the reduction to five ratings, it still feels like a massive task due to the lack of local participation.

    This is one feature I would not want added. Part of trying to increase ratings/reviews should be positive feedback. If reviewers are putting themselves in a spot to be critiqued, it may discourage them from reviewing. Lots of timid users when it comes to reviewing, and I wouldn't want to make it even more intimidating.
     
  22. bret27

    bret27 Poo-Bah (1,976) Mar 10, 2009 California
    Society Trader

    Another thing...
    I don’t think going on to rate a beer is that time consuming or tedious. It’s pretty quick.
    However when I was drinking a lot of local brews I found myself having to add beers to the site surprisingly frequently. This was a little more time consuming.

    I’m also the opposite of many rater haters. I’m going to look at the numbers almost every time and not read reviews. Exception is when there’s only a few ratings and not enough data to get a reliable average consensus. Then reviews can possibly be more helpful to me.
     
  23. MNAle

    MNAle Poo-Bah (2,262) Sep 6, 2011 Minnesota
    Society

    Plus, it would turn reviews into forum threads.
     
  24. ATL6245

    ATL6245 Crusader (772) Aug 16, 2018 Georgia
    Society

    I like the "How to Review a Beer" section you have. I think that's a good start and way to help users quickly get comfortable reviewing a beer. If you can somehow make that real easy to locate & reference while rating/reviewing, that might help. One of the things I hate about Untappd is people who rate and leave zero comment about the rating. If you gave the beer a 2.0, I'd like to know why. I've seen a lot of people rate a beer a 3.0 and then comment that the beer was outstanding. That comment at least gives me an idea of where that person's rating system is even if it doesn't match up to my own.
     
  25. PapaGoose03

    PapaGoose03 Poo-Bah (3,286) May 30, 2005 Michigan
    Society

    This is what I think will get me doing more. I typically add a rating for my local breweries, but not for most other beers that I drink. When I rate I give an across-the-board number, so a method to provide a single across-the-board number is needed. However, the individual characteristics number choices should be kept. If a characteristic about a beer stands out as being very good or very bad, I'll use that category and increase or decrease that number from what I gave the other characteristics. I'll then add a note to why I changed that number, and some of my ratings turn into reviews because of my explanation on that item.
     
  26. SFACRKnight

    SFACRKnight Poo-Bah (1,948) Jan 20, 2012 Colorado
    Society Trader

    I get that. My reviews reflect that, I found myself leaning on the same descriptors all the time. Almost every ipa I reviewed mentions grapefruit or other Citrus pith. It became like an echo chamber.
     
  27. BillAfromSoCal

    BillAfromSoCal Devotee (418) Aug 24, 2020 California

    Re: my suggestion of allowing comments that either support or don't support the reviewers comments....
    I guess I can see that happening, although in my mind I was thinking it would be along the lines of discussion rather than something that would read as disagreement or argument. As much as I would like to provide comments and feedback on some reviews I guess trying to manage the content and tone of the comments in text form could be a nightmare and inevitably lead to something that is perceived as overly negative. If nothing else, though, toggling a selectable star rating (1= "this review sucks", 5 = "this is a shining example of a great review") could help authors improve their reviewing style. Personally, if authors are writing cr*p reviews worthy of multiple 1-star designations from readers I don't care if they get offended and don't come back. The star system gives feedback without being wordy or confrontational. As a side benefit it would make identification of truly great reviewers much easier every year...just look at the average star ratings and number of 5 star ratings.
     
  28. cavedave

    cavedave Poo-Bah (3,050) Mar 12, 2009 New York
    Society Trader

    Have breweries offer some kind of inexpensive recognition directly to us for reviewing all the current beers in their portfolios. Make it something that has no cash value, and is simple enough to attract people to want to earn it without encouraging people to make fake ratings/reviews. Model it like states do with visiting all the state's breweries, and in such a way as it might attract lots of new ratings/reviews to the site. I have no real clue how this might work, if it's a good idea, or if it could be tailored to fit with BA, but here it is, my best suggestion
     
  29. cavedave

    cavedave Poo-Bah (3,050) Mar 12, 2009 New York
    Society Trader

    I often think people are overcritical of their ability to write reviews. I was surprised once to learn that a certain contributor to NBW/S thought his reviews were lacking, and I had found them honest and useful. I think what is missed in the pursuit of trying to be @BEERchitect is that a review works well when it is simple, honest, and hits the main features of the beer in such a way as the reader can relate to it.. This is what's lacking in ratings, the honest description of what's going on in the beer. We get instead an honest description (hopefully, no way to tell without a review) of how much the rater likes the beer. I think if certain words express things well, those are the words to use, repetitive as certain of them are likely to be when giving honest descriptions.
     
    MacMalt, jadpsu, Gemini6 and 20 others like this.
  30. spersichilli

    spersichilli Aspirant (269) Apr 26, 2018 California
    Trader

    A lot of new beers also don't have entries here. Untappd is just a better resource for ratings due to the sample size
     
  31. jonphisher

    jonphisher Crusader (739) Aug 9, 2015 New Jersey
    Trader

    I couldn't agree with this more. A rating to me doesn't really mean anything for exactly the reason you said. However; if a rating is bad but accompanied by a review you get the "why" as well. If it is an honest description and describes things that I find appealing in a beer, even if it is rated low I'm still inclined to want to try it or take a gamble on it.

    The reason I tried to start making it a point to review as much as possible is because of how many times I've looked up a beer on here wishing it had more than just ratings. I wanted, as you said an honest description of what the beer was actually doing. Yes I know reviews can disagree too, but enough of them saying the same thing is usually a good indicator of whatever trait is consistent.
     
  32. wadamslll

    wadamslll Initiate (57) Nov 25, 2021

    I really enjoy rating the beer I buy.
    I also like the what beer you drinking threads. The pics are awesome!
     
    RMW66, Chaz, beer_beer and 4 others like this.
  33. SFACRKnight

    SFACRKnight Poo-Bah (1,948) Jan 20, 2012 Colorado
    Society Trader

    After reading some of his reviews I find myself more reticent to review. Thanks Dave. :grin:
     
  34. Todd

    Todd Founder (6,558) Aug 23, 1996 California
    Staff Moderator Fest Crew Society Trader

    We'll eventually fix this once BeerAdvocate and Untappd (and all Next Glass brands) share a common beer database.
     
    Rug, Beersnake1, meefmoff and 13 others like this.
  35. shelby415

    shelby415 Meyvn (1,040) Oct 10, 2011 Oregon
    Trader

    What if you were to replace the drop-down numerical rating system with a clickable rubric that has verbal descriptors for each trait. It could pretty much be the same as what you have on the "how to rate a beer" page but fleshed out so there's less guesswork about what a 3 in aroma, for instance, actually means. This could have the effect of making ratings more accurate and less subjective and make new raters more comfortable while exposing them to language they could use if they crossed over into reviewing. I think the biggest problem with reviews is that the way it's set up now, the numbers are pretty disconnected from what they represent.

    I don't know how easy it is to add that, but as a teacher I come across them all the time.
     
    cavedave and Singlefinpin like this.
  36. papposilenus

    papposilenus Poo-Bah (1,896) Jun 21, 2014 New Hampshire
    Society

    I use the raw ratings in a very specific way: if I'm going anywhere, I'll go into 'Places' and sort by 'Beer Avg' descending, just sort of discarding the low '# Beers' by eyeball. So, for my purposes, the more ratings the better and, given a large enough universe of ratings, I'm not super concerned with how much care and consideration went into generating each of those individual ratings.

    But I also value the ability to write my own reviews and refer back to them. That's how I can remember anything for more than a day or so. And I enjoy reading other peoples' reviews, not all the time but, at least sometimes.
     
    RMW66, Rug, Whyteboar and 6 others like this.
  37. papposilenus

    papposilenus Poo-Bah (1,896) Jun 21, 2014 New Hampshire
    Society

    You're not wrong and I've thought exactly the same myself many times.

    But that's exactly the joy of four-packs - I usually just go ahead and drink the first couple cans and enjoy them (or not, as the case may be), review the third one one and then just enjoy the fourth can (or, depending, pour it down the sink).
     
    #37 papposilenus, Jan 11, 2022
    Last edited: Jan 11, 2022
    Rug, Whyteboar, hopsputin and 7 others like this.
  38. Jwale73

    Jwale73 Poo-Bah (5,005) Aug 15, 2007 Rhode Island
    Society Trader

    Reviews help me remember, whereas just rating does not. That said, I’m not as militant about rating everything new I try, as was the case when I first joined.
     
  39. papposilenus

    papposilenus Poo-Bah (1,896) Jun 21, 2014 New Hampshire
    Society

    The NYT Cooking app has something like an 'X' number of viewers found this review helpful. Make that a sortable field and it would seem both useful and reasonably non-confrontational.
     
    Whyteboar, LXIXME, meefmoff and 5 others like this.
  40. Mindcrime1000

    Mindcrime1000 Meyvn (1,472) Apr 30, 2016 South Dakota
    Society Trader

    I like to review for three reasons: (1) it provides context if someone does happen to read it; (2) it forces me to actually think about why I like a beer; and (3) if I drink it again later I can compare it "now" versus "then" without relying solely on my memory.

    However, no hate for those who rate only.