Does the Beer Trading Feedback System work? Improvements?

@zoober911 @mythaeus and @jbaker67

The issue with scales is always how people use them, not the detail of the scale itself. Also, I imagine the tardy or slightly crappy traders would also be late leaving feedback, so when they get a 4 or a 3 from the other person, they simply respond in kind and leave a lower score because they feel slighted.

I suspect what would end up happening, is that people would just get all 5's across the board unless they were pretty bad. There would be the occasional 4 on something when they probably should be getting a 2 (say shipping a week or 2 late - to me this is a 2 out of 5, I bet people wouldn't use the scale that way).

Don't get me wrong, a 5 point scale on 4 elements as proposed above would be great, but if people aren't using a simple 3 point scale correctly, I suspect they will be even worse at using a 5 point complex one...

 

As far as having "open" trades where the user doesn't leave feedback even though the trade is done I've probably been guilty of this myself. I don't trade that often and forget to look at my trading stats until either I do another trade and look at the trade request or the other user asks me about leaving feedback. It would be helpful (at least to me) if once one person leaves feedback the other would get the "notification" flag that they had an open trade that needs to be looked at.

 

LMK if you need any support as to your "quality" as a trader, Matt.

 

If the issue is with the rating scale or proposed rating scales, why not eliminate that portion of the trading system.

Keep the trade request portion locking in the trade and when the trade is completed on both ends, it will log as a completed trade. It will be on the traders to do the necessary research to determine if they are potentially trading with a bad trader and or a trader with a not so great reputation here on BA. It doesn't take long to shoot off a BM requesting a reference from the most recent trade partner. Or look at the trade partner history and see who they have traded with in the past.

 

@pagriley I pretty much agree that this is mainly a problem of people not using the existing system rather than a flaw in the system. I notice you post often in the Beer Trading info section, as do I, but I think there are a ton of people who trade here that don't ever see these discussions as to how to use the system. I have frequently seen comments to the effect that unless someone just flat out steals beer, they aren't a bad trader, and I couldn't disagree more. I certainly don't advocate for being a petty dick and marking someone down because of a brief delay or something, but people act like the "okay" category doesn't exist at all.

Perhaps one of Todd's trading update messages that will show up as an alert for everyone could clarify the intent of the feedback system, with a general example of how it should be used.

 

It would be a radical departure, but the only way I see of making it objective and 'forcing' honesty would be to disconnect the rating scale from the respondent and make it based on some simple survey questions that then translated into the score. Lets take the criteria others have already proposed as the example, say we want a 5 point scale for each of the following:

1) communications
2) shipping time
3) packaging
4) overall

Obviously you would need a different set of criteria for an In Person trade, but lets just focus on mail trades for now.

In addition, you would have to set it up so that the trade ratings don't show up for anyone until both people have left feedback (to prevent retaliation) and then lock the responses.

To get the 1-5 rating for each you would ask a few simple questions with defined categories; I took a rough pass at something below to give the idea:

Communications:
Question 1:
Did your trade partner respond to all messages quickly (within 24 hours) and keep you fully informed throughout the process with no gaps or delays?
Yes/No
(If yes, rating = 5, if no, ask question 2)

Question 2 (not asked if Q1=yes):
Which of the following best describes the communication you had with your trade partner?

• Just the one break in communication but overall kept me informed and I never had to send a reminder or follow up (rating = 4)
• A couple of breaks in communication and I had to follow up to get a response (rating = 3)
• Regular breaks in communication and I had to send multiple follow up messages to get a response (rating = 2)
• Frequent breaks in communication and I had to send multiple follow up messages and didn't get responses (rating =1)

Shipping time:
Question 1:
Did your trade partner ship their package on the agreed on day and provide you a tracking number?
Yes/No
(If yes, rating = 5, if no, ask question 2)

Question 2 (not asked if Q1=yes):
How long after the agreed to ship date did it take for your partner to ship their package and provide a tracking number?

• 1-2 days (rating = 4)
• 3-7 days (rating = 3)
• 7-14 days (rating = 2)
• 14+ days (rating =1)

Packaging:
Question 1:
Were any of the bottles you received from your partner damaged or leaking?
Yes/No
(If yes, rating = 1, if no, ask question 2)

Question 2 (not asked if Q1=yes):
Which of the following best describes the packaging of the beer you received from your trade partner?

• Bombproof - they used a foam shipper or a very sturdy box with each bottle individually wrapped in bubble wrap and tightly packed (rating = 5)
• Solid - They used a regular box and padded the bottles, but bottles could move around and / or the padding was thin (rating = 4)
• Okay - Wrapped the key bottles, but materials were inadequate and bottles could touch and were loosely packed (rating = 3)
• Poor - Just a box with some loose padding (peanuts / newspaper) and nothing to separate the bottles. It is a miracle something didn't break (rating =2)

Overall:
Question 1:
Would you trade with this person again without hesitation and describe this as a perfect trade - were all the bottles correct, and the interactions with your partner quick, clear and easy?
Yes/No
(If yes, rating = 5, if no, ask question 2)

Question 2 (not asked if Q1=yes):
Which of the following best describes your overall trade experience?

• Good - no major issues. We did have a few minor things but they were resolved quickly and easily. I would still absolutely trade with this person again (rating = 4)
• Okay - A few things came up that weren't ideal and I would probably trade with this person again (rating = 3)
• Poor - This trade was a hassle and I probably would not trade with this person again (rating = 2)
• Disaster - There were multiple problems with this trade and I would not ever trade with this person and recommend that others don't trade with them (rating = 1)

The above seems complex, but it is actually very simple. 90% of trades go just fine, so people would get 4 simple yes/no questions to answer and 1 rating on the quality of the packaging - it would take them a minute to fill in.

If there were issues, there are only 8 total questions, and they would all be objective, so there is little wriggle room.

It seems over the top, but if people really wanted to get good data out of the trade review system, something like this is just about the only way to get at an objective and useful set of data.

 

Generally I agree with you on the bolded, and you have expertise in the field. And what others have said about using the comment box better.
Perhaps a single four point scale:
All Expectations met (Great)
All but one (pretty good)
An issue, resolved (ok, eventually)
An issue, unresolved or had to start bad trader thread that confirmed bad trader (bad)

Check boxes in different categories also sounded pretty simple.

Please not this.

Lastly, this open trade request or feedback thing is something I've dealt with, even a year after the trade is done. However, I've also been part of trades where one party has asked the other to hold off on shipping (be careful this winter, folks!), so one side has left feedback and the other won't for a few weeks/months.
Maybe a report ignored feedback button can be made and it will let Todd or the Mods restrict a users account til feedback is left?
Using the feedback system is part of trading now, and leaving feedback is as well. If you can't leave feedback, you're not a great trader.

 

Great first pass. At this point, we're still spit balling, but nice work.

That said, some modifications would likely be due to give some flexibility on trading style. I tend to trade with people who have say over 20 feedback, so oftentimes we'll exchange messages like "I'll check in and tell you what I could get this weekend," or "I'll drop the box off early next week so you have it for the weekend." So while doing details a 1-24 hour response is mostly expected, other times traders can be laid back (as long as both sides are cool with it!).

Also, a damage or leak shouldn't be an automatic 1. I had a bottle leak once (8 oz from a bad cap iirc) in a 9 shipper and immediately offered to replace the bottle and ship in the next 48 hours, with some locals. The trader and I resolved it a different way in a few nice messages, to the satisfaction of my recipient. I'd think that's a 4 or 5. Also, the shipping companies can beat up or lose/discard boxes, so packaging might be great, but a bottle breaks.

With refinement, I think your scale should be good and simple enough for anyone able to trade beer.

 

I think overall is really all that matters. And to me there is no difference between poor/disaster (either way it's a no) and yes/good (either way it's a yes).

 

I'm down to have a better feedback system that provides more information. But I think 4-5 scales (ebay style) would be too much. It could be narrowed down to two 1. Beer received, 2.Experience with the second point covering all the communication, shipping delays, etc issues. I really don't see a reason to break those downs. If you screw in one aspect you're killing the whole experience so most people wouldn't want to trade with that person regardless of which aspect was wrong.

Now, the current system in place allows for that! Neutral feedback is basically that... "received beer but".... yet most people still give POSITIVE feedback in these circumstances...WHY? . . . Pity? Superb Extras? Overnight shipping after the delay? If people actually start using this tool a lot of conflicts can be prevented. Right now there is an active thread in Bad Traders where a lot of previous traders are coming out to say that they also had a bad experience with the same user in the past yet the dude has 24 positives reviews for a whooping 100% score. Even if you write down what happened why rate that as a positive? C' mon people, get though! Remember that the mods will have your back if you get retaliation feedback, if you had a crap experience rate accordingly.

 

Agreed - the overall is all that matters, but it is a sum of the parts, and unless you force objective criteria, people will default to what they are doing today and just give a high rating every time.

In the above example, I would actually weight the scores:
1) communications - 20%
2) shipping time - 30%
3) packaging - 30%
4) overall - 20%

by parsing out the elements and setting hard criteria (with no numbers on the page) you force people to actually respond honestly about their experience and calculate the score on the back end - just asking the overall is going to get everyone a bunch of 4's and 5's and not actually dig deep enough to get the right answer.

 

Fair - but by splitting things up and giving objective criteria you provide context and prompt people to actually think about their experience. I agree completely, people should give more 'okay' ratings, but they don't because there isn't objective criteria to rate on.

 

I've thought about the bolded and I think the only way to avoid that would be to make the feedback hidden until you send feedback yourself. Instead of receiving feedback and being able to go into your trade history and see what the person said, you would just receive a notification saying that someone left you feedback and in order to see it, you will need to leave feedback as well. I know users could still edit the feedback they left to lower the scores but maybe that should have to be mod approved. I have a hard time believing that there are that many feedback scores that honestly need to be changed that it would tie up the mods very much.

 

I am the one that said the feedback system can no longer be relied on in a post in one of the BT threads. And I didn't mean that in a negative way and it's no fault of the system nor is there anything that can really be done about it IMO. All good suggestions above but some people with perfect feedback can just flake out in a moments notice and not live up to their obligations. It's human nature and the world we live in.
Don't expect to trade with a complete stranger and expect him to be a stand up guy or gal just because he (or she) has good taste in beer and came through on their obligation with someone else.

 

Agreed - it should only be shown once both people put it in - you could also lock it so people can't change / edit once they see they got crappy feedback.

 

I would definitely leave "okay" feedback since it falls into the "Expectations were met...but" category. It seems to me people are having trouble throwing out that "but." Out of my 19 logged trades, I can say I properly evaluated them all as positive trades. There was one where I could've said "the communication was a bit patchy and the packing job could've been better but still doesn't warrant neutral feedback because the box arrived when promised with all bottles intact." But that quoted sentence is 186 characters long (current limit is 140 characters), so instead I was very brief and neutral in my written feedback while still leaving a positive rating.

My proposal is that we leave the ratings as they are but raise the limit of characters to around 200 or so. Simply give the community more space to police and voice itself properly. Also, I think the mods are on the right track with penalizing or at the least notifying people that are failing to leave feedback or not doing so in a timely manner. Right now in "Beer Trading Feedback System, Explained" it reads

• Failing to leave feedback is bad form

It should read something like

• Failing to leave feedback in a timely manner will result in a penalization of x

We just need to figure out what the "x" will be.

That's how it was done in the old days before the feedback system existed and the community was forced to police itself, but I like the feature of looking at written feedback for a user's past trades without having to bug any of their former trade partners with a BM. I feel like simply giving more space to thoroughly explain the trade and include small dissatisfactions will be a sufficient solution. Let's face it, sometimes a trade can get complicated with so many factors to take into account; communication, packing job, promptness of delivery and quality of, or lack of, extras being the main ones. I feel like, if things get wacky with a trade, it can easily require more than 140 characters to properly articulate that, especially in the case of "Expectations were met...But." It's easy to explain all the positives in a positive trade briefly as well as all the negatives in a negative trade. But accurately explaining both the positives and negatives in a neutral trade can sometimes take close to a paragraph.

 

I don't necessarily agree that if something minor is wrong that people will not trade with them again. I recently traded with a guy that said he was going to ship on Monday and didn't end up shipping until Thursday with no excuse and while that wouldn't necessarily make me avoid future trades with him, it would be nice to have been able to see what other people said about his shipping to put my mind at ease. Maybe if a few people before me gave him a 3 out of 5 for shipping, I would have relaxed and just chalked it up to a slow shipper. Instead I messaged him daily asking for a tracking number because I was afraid to get screwed out of a big bottle. Everything else was great, communication was great, packing was bomb proof, and the extras were awesome. I would have given him a 5 in everything but shipping and I wish people would know that about him, and maybe it's something he should know so he can work on it. With the current format, there's no way I could have left him anything but positive feedback and not just looked super petty about a 3-4 day delay.

This is why I think a 1-5 scale on a few different things would work best. Very few people will leave negative feedback or even neutral feedback simply because shipping was delayed by a week or because the communication was slow(we've all had that one guy that doesn't respond until you send a follow up about everything). People are afraid of tarnishing someones rep over something that minor. On the other hand, I would have no problem leaving someone a 3 out of 5 for communication if the guy took days to respond or send my tracking number.

I traded on the beer exchange before coming here and as much as that site sucks and the trading just isn't convenient, their ratings work. I've traded with multiple traders that had less than a 5.0 rating without hesitation but I would never trade with anyone on here with less than a 100% rating. All I had to do on bex was go in and see that a couple people gave him a 3 or 4 for communication and I realized that the guy would get me my beer, but he just might not be super responsive and the trade may take 2 weeks to complete instead of a few days. It's also a good way to do some self-reflecting when you receive less than a 5. Here, it's very difficult for a new trader to learn how to become a great trader if there is no realistic feedback.

 

There are a lot of excellent points raised in this thread, and I haven't had time to read every last one, but I do share a couple of my own thoughts. I figure if I'm duplicating what some have already said, that perhaps serves as an indicator to the guys running the site that it is a popular request...

For starters, I do think the system itself is part of the problem. We have a system that is, for all intents and purposes, binary. Yes, there are three options, but two of them provide the same result on the percentage score. If you select Good/Okay, then that trade gets tallied in the overall percentage and the user's rating remains at 100% and if you select Bad, the user's percentage drops below 100%.

This is a problem because as has been noted several times on this thread, traders and trading interactions don't exist on a two-point scale. Even a three-point scale (good/okay/bad) would be better, but right now the final score isn't reflective of three options. The percentage score would be more useful if it only related to the ratio of "good" trades to your overall trades, but even then you're just trading one problem for another, instead of lumping mediocre traders in with good traders, you're lumping them in with bad traders. We could certainly have a debate about which side they deserve to be lumped in with, but I think a better solution would be to alter the score portion of feedback to be on a scale instead of a percentage.

If you keep the same three data points, then a Good is worth 5.0, an Okay is worth 3.0 and a Bad is worth 1.0. Then each user gets a total score based on averaging all their prior scores.

Of course, that only fixes the issue with the system itself. It does nothing to address what is a larger source of concern: the way the users actually use the system. As has been established, the fundamental problem is the reluctancy to use unfavorable reviews for anything outside of a trader completely flaking out and stealing beer, at which point it is moot because they are usually banned anyway. I believe the best way to fix this is to give users enough flexibility so that they don't believe they are dooming someone to "bad trader" status just for some lackluster communication or delays. Users need a way to distinguish between severity of bad trading habits.

The best way to do this would be to do a multi-point scale on each key attribute of trading. You could even keep the number of points the same so that you aren't giving users selection paralysis.

• Communication: Good Okay Bad
• Timeliness: Good Okay Bad
• Packaging: Good Okay Bad
• Follow-through/lives up to deal: Good Okay Bad

You wouldn't even have to weight them, imho. Because this gives enough variability in the final score that people who slack on one area of a trade won't see a large impact on their scores and traders who consistently do everything right will look elite by comparison--getting 5s across the board. And traders who consistently mess up will see their overall scores dip.

Think about it: guy who packages his beer well, ships in a timely manner, and makes good on the agreement but goes long stretches without necessary communication gets a really good score on a trade (4.5), and even has the benefit of that single trade averaging out with a bunch of other 5.0 trades if his lack of communication was a one-time thing amongst a whole catalog of pristine trades.

Of course, the problem with this approach is twofold: 1) I imagine it's complicated from a coding standpoint and may not be worth the effort, and 2) how do you handle grandfathered scores? Just treat all "Good" trades as a 5.0 and move forward with the multi-point system?