“The internet breeds inequality.” In January 2023, weeks after the launch of ChatGPT, I speculated on how AI could change the distribution of what gets seen, liked, and shared online. Historically, the web tends to produce power-law distributions: A handful of creators get most of the views. A handful of songs get most of the streams. As Spotify’s former chief economist Will Page pointed out, listeners spend 90% of their time on less than 2% of songs.
This inequality, I argued, is not a fact of nature. Instead, it’s a byproduct of how we discover content online. Google’s founding mission was to “organize the world’s information”, and that’s exactly what it did. The key insight behind Google’s search engine was to treat the web as a popularity contest: pages with more incoming links rank higher; links from higher-ranked pages count more than others. Later, algorithmic feeds on apps like Facebook, Instagram, and Twitter embraced a similar logic: showing people what others liked.
Popularity became a proxy for relevance. The result was a content landscape that looked very different from the one of old media gatekeepers and quality standards. As I put it in 2023, “in the past, content became popular because it was good; today, content is considered good because it is popular.” The attention-rich get richer by design. And as attention converted to revenue, the same inequality affected actual incomes.
I thought AI could offer a different way to organize information by evaluating content based on its actual relevance rather than a popularity proxy:
Artificial Intelligence could make expertise scalable. Instead of evaluating content the way Google would (popular = good), the next iteration of ChatGPT might be able to evaluate it the way an expert would (good = good). […] Scalable expertise could usher in a world of better content, lower inequality, lower polarization, and more opportunity for more people.
And I made a testable claim: a shift from “social” to “intelligent” search, I wrote, would “limit the size and occurrence of massive winners.”
This was the theory in 2023. Last week, some interesting evidence arrived.
What should you watch next?
A team of Netflix economists just published the results of a remarkable study. The streaming app is famous for its influence on what people watch. Whatever you stream is automatically followed by what the algorithm thinks you should watch next, and the home screen is full of suggestions for your next binge.
How does the quality of these suggestions affect user behavior? How does it affect the kind of content people watch and the distribution of attention between the most popular shows and everything else?
To find out, the researchers devised a simple experiment. They separated 8.5 million Netflix subscribers into two groups. For 60 days, the first group received content recommendations from a “frozen” version of the algorithm. Meanwhile, a second group of subscribers received a series of updates that constantly improved the algorithm’s intelligence and its ability to understand each user’s taste. Then, the researchers compared what each of the two groups watched.
Did smarter recommendations make any difference? Yes. The recommendations for the second group shifted away from the biggest hits (the top 5% of titles) toward moderately popular titles (which the researchers describe as the “middle tail”). This, in turn, led to an actual shift in what people watch. Concentration fell across the board: the Herfindahl–Hirschman Index, the standard measure of market concentration, dropped 5.7% for recommendations and 1.2% for plays. This was a modest but meaningful result for just two months of accumulated algorithmic improvements.
Attention did not just shift; it also increased. Better (and less hit-focused) recommendations increased user engagement. Subscribers in the second group watched more hours, played 1.2% more distinct titles, completed more titles, and relied more heavily on the app’s recommendations.
The Crutch of the Crowd
The experiment’s result was similar to what I predicted, although the mechanism was slightly different. Netflix’s improved algorithm did not become an expert judge of quality; it became better at understanding individual taste. A weaker algorithm uses popularity as a crutch. A stronger one needs less crowd data to recognize a good match.
In other words, Intelligence reduces the informational advantage of popularity. As the researchers put it:
“As recommendation technology matures, then, the plausible set of targetable titles continues to shift toward the middle-tail.”
It’s important to note that, on Netflix, the better algorithms only pushed attention away from the very top to the middle, but not toward the bottom. The least popular titles did not benefit; only the bulk of those in the middle of the popularity curve. This was different from my original prediction about intelligence benefitting the long tail, the vast number of niche content that is only relevant for a few people.
Still, even better algorithms might push attention further down the attention frontier. In addition, Netflix’s “long tail” is limited, as the platform is heavily curated. Seeing a similar experiment on a platform like YouTube might produce different results.
Either way, we now have strong evidence that better machine matching can counteract the popularity bias of online platforms. This has consequences well beyond Netflix and online content. Online recommendation systems already determine people’s next job interview, travel destination, and date. In all of these fields, the rise of online platforms has made outcomes more extreme and polarized, favoring the most popular contenders over everyone else.
Can AI reduce the inequality? In at least one important setting, yes.
Will that make the economy more equal? Probably not. AI could produce fewer superstars and more medium-sized winners within individual markets, while making wealth and power even more concentrated across the economy. As I wrote about in The Nonlinear Economy, there are other forces that make our economy inherently less equal. Countering this tendency while maintaining economic dynamism would require bold new policy ideas.
Have a great week.
Dror
How will AI reshape your company, industry, and career?
My speaking schedule for the fall and winter is filling up. Visit my speaker profile and get in touch to learn more.
Click here to book a keynote or learn more.
This concentration of what is recommended which comes out of this self reinforcing loop is bad. Your article is encouraging. Getting discovery into the middle and long tails is better for artists and regular people and society and the distribution of income. I love jazz. Jazz has never and will never be super popular which is fine with me but not so good for the artists 🧑🎨