These are the notes and details from our Steamworks Blog post Data Deep Dive: How are new releases on Steam performing?
We thought through a few different ways of studying the health of new releases before deciding on our methodology. Here's a look into our reasoning.
New releases instead of all games
We decided to focus on new releases in this post to address the questions of developers who either recently released a game or are about to release a game. Including data on all our back-catalog games in these statistics wouldn't be as relevant for that audience.
Median earnings instead of mean earnings
Games earning a ton of money can have an outsized effect on mean earnings, and those games don't reflect the budgets and realities for most developers. Using the median reduced the effect of those outliers compared to the mean, and did a better job of showing how regular games are doing. We also looked at other percentiles besides the median – aka the 50th percentile – to get a broader view of the health of games on Steam.
Initial earnings instead of total earnings during the year
We chose to study a fixed period of initial earnings to give even weight to games released over the course of each year. Using total earnings during the year would have meant that games released at the beginning of each year had 12 months to accrue revenue, while games released at the end of the year had hardly any time.
Since immediate earnings are a crucial metric for many developers, we chose the first two weeks of earnings for our main analyses. To check our work, we also tested other periods – 30 days, 60 days, etc – to make sure our results weren't sensitive to that choice. (They weren't.)
Excluding non-game content
We filtered out all apps besides games - so we're not counting non-game software, videos, comic books, soundtracks, demos, or other non-game content. Earnings for DLC were added to the base game, but only if they were earned within two weeks of the base game's release date.
Comparing 2019 to 2018 in our distributional analysis
We were initially tempted to compare 2019 to all previous years to study changes in the median game and other percentiles, but we quickly realized that a comparison of the same percentiles across policies was meaningless: the distribution of games on Steam had changed so much between the various release policies that the median game (or the 25th percentile game, or any other percentile) under one policy was nothing like the median game under a different policy.
To get around this, we tried other ways to make the comparison across eras, like comparing games that had implemented similar features and supported similar languages. However, when we eyeballed the games we'd be comparing, it was again apples to oranges. The reason is that many features have changed in popularity or become easier to implement than in the past, and the number of games supporting multiple languages has exploded over time as Steam's market has become increasingly international.
In the end, we decided that the most accurate thing to do would be to compare how things had changed since Steam Direct had been implemented. Since 2018 was the first full year of Steam Direct, this meant comparing 2019 to 2018.
Other ways to measure the success of new releases over time
We mentioned in the main post that we looked at various ways of slicing the new releases data. If you are curious about the trends for other revenue benchmarks, we've included them here:
We also tested other success thresholds and initial earnings periods besides 14 days. In each case, we saw similar trends and found that more games achieved success in 2019 than in any previous year. As an example, here is the graph of games earning at least $10,000 during their first 30 days of initial earnings.