Activision made a surprising revelation over the weekend about Call of Duty’s skill based match-making.
The company shared a document explaining the reasons they implement skill based match-making in their games, and in doing so they shared their surprise.
In their words:
“…in early 2024, we ran the Deprioritize Skill Test in Call of Duty®: Modern Warfare® III, where we used our A/B test framework to loosen the constraints on skill in matchmaking.
It’s important to note that skill, as a factor in matchmaking, was decreased for this test, but not removed entirely from the matching algorithm.”
And the results? They found that more players at low and medium skill levels quit playing. But even worse than that, they believe that if they allowed this to keep happening, eventually the top players would stop playing too.
Again, to quote Activision’s document here:
“With deprioritized skill, returning player rate was down significantly for 90% of players. The 10% of highest skilled players came back in increased numbers, but in aggregate, we see meaningfully fewer players coming back to the game.
This effect may appear small, but this change was observable within the duration of the test. This will compound over time, just like interest, and will have a meaningful impact on our player population.
This is a concern for all players, including the top 10%, as if this pattern is allowed to continue, players will exit the game in increased numbers. Eventually a top 10% player will become a top 20% player, and eventually a top 30% player, until only the very best players remain playing the game.
Those original top players will become increasingly likely to not return to the game. Ultimately, this will result in a worse experience for all players, as there will be fewer and fewer players available to play with.”
For your convenience, we’ll share a table showing their results here. From left to right, players were ranked at lowest to highest levels, with the columns indicating how many players left or came back. Those columns below the line indicate how many players left.
Activision’s test results certainly stand in stark contrast to the conventional wisdom surrounding skill based matchmaking. It’s also quite interesting, as XDefiant famously took a stand to not have any form of skill based matchmaking for the game’s casual modes.
The XDefiant dev team also clarified that they would have skill based matchmaking for the game’s ranked modes, which arrived at the start of this month with it’s 1st Season. This is definitely a different skill based matchmaking system than what the Call of Duty games have, but as you can imagine, players are complaining about XDefiant ‘s skill based matchmaking too.
But perhaps attention now has to go to the possibility that a lot of the complaining about skill based matchmaking are coming from top players, who happen to be in a position to influence and convince lower skilled players, perhaps fans who don’t even play the game. If the call to end skill based matchmaking was coming from YouTubers and Twitch streamers, it makes sense for Ubisoft to prioritize their players at large over them. It’s not Activision’s job to make games that are good for streaming, maybe streamers should stick to FromSoftware’s output for that instead.
In any case, if you’re interested in learning more details on Activision’s study, you can download and read the document here.
