Ratings systems have returned to haunt the gig economy

Andrew Hill:

This shift is mere common sense. Any review system is prone to what experts call the “idiosyncratic rater effect”, which is a polite way of saying that bias and discrimination can pollute the outcomes. That applies in particular to “rank-and-yank” assessments, but also to poorly presented feedback. As Marcus Buckingham, a consultant, and Cisco’s Ashley Goodall wrote in Harvard Business Review earlier this year: “Because your feedback to others is always more you than them, it leads to systematic error, which is magnified when ratings are considered in aggregate.”

Having tested such methods to soul-destroying destruction in some of the biggest organisations in the world, it is not merely perverse but positively dangerous to disinter their flaws so they can haunt the gig economy.

Discrimination has been one of the first ghosts to re-emerge. Researchers who looked at Uber, concluded that while its rating system was outwardly neutral, it could be a vehicle for, say, racial bias. Academics feel the ratings effect personally. The authors of another paper about Uber pointed out that their own students’ evaluations were “relevant for the renewal of teaching contracts, promotions or future applications”, and are also suspected of bias. Their study suggested solutions could include giving Uber drivers the opportunity to challenge a bad rating, or appointing a third party who could audit reviews for potential bias.

Uber does let drivers rate users, who can themselves be kicked off the app if their bad behaviour pushes their rating below par. This leads to the mutually assured insincerity of high ratings on both sides (the flaw in Airbnb’s review system identified in the Boston study) and leaves neither customer nor provider much the wiser.

The grim alternative is not much better, though, and I will bear it in mind before I next submit a low mark. It is that everyone slips back into a swamp of personal performance ratings, where customers are cast in the role of rankers-and-yankers, remotely and unwittingly ruling on the fate of individuals just like them.