> With their latest data measurements specific to the game, the developers have confirmed the small number of players (11% last week) using mechanical hard drives will witness mission load times increase by only a few seconds in worst cases. Additionally, the post reads, “the majority of the loading time in Helldivers 2 is due to level-generation rather than asset loading. This level generation happens in parallel with loading assets from the disk and so is the main determining factor of the loading time.”
It seems bizarre to me that they'd have accepted such a high cost (150GB+ installation size!) without entirely verifying that it was necessary!
I expect it's a story that'll never get told in enough detail to satisfy curiosity, but it certainly seems strange from the outside for this optimisation to be both possible and acceptable.
> It seems bizarre to me that they'd have accepted such a high cost
They’re not the ones bearing the cost. Customers are. And I’d wager very few check the hard disk requirements for a game before buying it. So the effect on their bottom line is negligible while the dev effort to fix it has a cost… so it remains unfixed until someone with pride in their work finally carves out the time to do it.
If they were on the hook for 150GB of cloud storage per player this would have been solved immediately.
The problem they fixed is that they removed a common optimization to get 5x faster loading speeds on HDDs.
That's why they did the performance analysis and referred to their telemetry before pushing the fix. The impact is minimal because their game is already spending an equivalent time doing other loading work, and the 5x I/O slowdown only affects 11% of players (perhaps less now that the game fits on a cheap consumer SSD).
If someone "takes pride in their work" and makes my game load five times longer, I'd rather they go find something else to take pride in.
It is a trade-off. The game was developed on a discontinued engine, the game has had numerous problems with balance, performance and generally there were IMO far more important bugs. Super Helldive difficulty wasn't available because of performance issues.
I've racked up 700 hours in the game and the storage requirements I didn't care about.
I'm not sure that's necessarily true... Customers have limited space for games; it's a lot easier to justify keeping a 23GB game around for occasional play than it is for a 154GB game, so they likely lost some small fraction of their playerbase they could have retained.
> I’d wager very few check the hard disk requirements
I have to check. You're assumption is correct. I am one of very few.
I don't know the numbers and I'm gonna check in a sec but I'm wondering whether the suppliers (publishers or whoever is pinning the price) haven't screwed up big time by driving prices and requirements without thinking about the potential customers that they are going to scare away terminally. Theoretically, I have to assume that their sales teams account for these potentials but I've seen so much dumb shit in practice over the past 10 years that I have serious doubts that most of these suits are worth anything at all, given that grown up working class kids--with up to 400+ hours overtime per year, 1.3 kids on average and approx. -0.5 books and news read per any unit of time--can come up with the same big tech, big media, economic and political agendas as have been in practice in both parts of the world for the better part of our lives--if you play "game master" for half a weekend where you become best friends with all the kiosks in your proximity.
> the effect on their bottom line is negligible
Is it, though? My bold, exaggerated assumption is that they would have had 10% more sales AND players.
And the thing is, that at any point in time when I, and a few I know, had time and desire to play, we would have had to either clean up our drives or invest game price + sdd price for about 100 hours of fun over the course of months. We would have gladly licked blood but no industry promises can compensate for even more of our efforts than enough of us see and come up with at work. As a result, at least 5 buyers and players lost, and at work and elsewhere you hear, "yeah, I would, if I had some guys to play with" ...
Gamers are quite vocal about such things, people end up hearing about it even if they don’t check directly.
And this being primarily a live-service game drawing revenues from micro-transactions, especially a while after launch, and the fact that base console drives are still quite small to encourage an upgrade (does this change apply to consoles too?), there’s probably quite an incentive to make it easy for users to keep the game installed.
Studios store a lot of builds for a lot of different reasons. And generally speaking, in AAA I see PlayStation being the biggest pig so I would wager their PS builds are at least the same size if not larger. People knew and probably raised alarm bells that fell to the wayside because it's easier/cheaper to throw money at storage solutions than it is engineering.
I started my career as a software performance engineer. We measured everything across different code implementations, multiple OS, hardware systems, and in various network configurations.
It was amazing how often people wanted to optimize stuff that wasn't a bottleneck in overall performance. Real bottlenecks were often easy to see when you measured and usually simple to fix.
But it was also tough work in the org. It was tedious, time-consuming, and involved a lot of experimental comp sci work. Plus, it was a cost center (teams had to give up some of their budget for perf engineering support) and even though we had racks and racks of gear for building and testing end-to-end systems, what most dev teams wanted from us was to give them all our scripts and measurement tools to "do it themselves" so they didn't have to give up the budget.
That sounds like fascinating work, but also kind of a case study in what a manager's role is to "clear the road" and handle the lion's share of that internal advocacy and politicking so that ICs don't have to deal with it.
It's because patting yourself on the back for getting 5x performance increase in microbenchmark feels good and looks good on yearly review.
> But it was also tough work in the org. It was tedious, time-consuming, and involved a lot of experimental comp sci work. Plus, it was a cost center (teams had to give up some of their budget for perf engineering support) and even though we had racks and racks of gear for building and testing end-to-end systems, what most dev teams wanted from us was to give them all our scripts and measurement tools to "do it themselves" so they didn't have to give up the budget.
Misaligned budgeting and goals is bane of good engineering. I've seen some absolutely stupid stuff like outsourcing hosting a simple site to us, because client would rather hire 3rd party to buy domain and put a simple site there (some advertising), than to deal with their own security guys and host it on their own infrastructure.
"It's a cost center"
"So is fucking HR, why you don't fire them ?"
"Uh, I'll ignore that, pls just invoice anything you do to other teams"
...
"Hey, they bought cloud solution that doesn't work/they can't figure it out, can you help them"
"But we HAVE stuff doing that cheaper and easier, why they didn't come to us"
"Oh they thought cloud will be cheaper and just work after 5 min setup"
In an online services company, a perf team can be net profitable rather than a "cost center." The one at my work routinely finds quantifiable savings that more than justify their cost.
There will be huge mistakes occasionally, but mostly it is death by a thousand cuts -- it's easy to commit a 0.1% regression here or there, and there are hundreds of other engineers per performance engineer. Clawing back those 0.1% losses a couple times per week over a large deployed fleet is worthwhile.
11% still play HD2 with a spinning drive? I would've never guessed that. There's probably some vicious circle thing going on: because the install size is so big, people need to install it on their secondary, spinning drive...
Even though I have two SSDs in my main machine I still use a hard drive as an overflow for games that I judge are not SSD worthy.
Because it's a recent 20TB HDD the read speeds approach 250MB/s and I've also specifically partitioned it at the beginning of the disk just for games so that it can sustain full transfer speeds without files falling into the slower tracks, the rest of the disk is then partitioned for media files that won't care much for the speed loss. It's honestly fine for the vast majority of games.
It is no surprise to me that people still have to use HDD for storage. SSD stopped getting bigger a decade plus ago.
SSD sizes are still only equal to the HDD sizes available and common in 2010 (a couple TB~). SSD size increases (availability+price decreases) for consumers form factors have entirely stopped. There is no more progress for SSD because quad level cells are as far as the charge trap tech can be pushed and most people no longer own computers. They have tablets or phones or if they have a laptop it has 256GB of storage and everything is done in the cloud or with an octopus of (small) externals.
I don't find it surprising at all. A ton of developers do optimizations based on vibes and very rarely check if they're actually getting a real benefit from it.
I expect it's a story that'll never get told in
enough detail to satisfy curiosity, but it certainly
seems strange from the outside for this optimisation
to be both possible and acceptable.
From a technical perspective, the key thing to know is that the console install size for HD2 was always that small -- their build process assumed SSD on console so it didn't duplicate stuff.
154GB was the product of massive asset duplication, as opposed 23GB being the product of an optimization miracle. :)
How did it get so bad on PC?
Well, it wasn't always so crazy. I remember it being reasonable closer to launch (almost 2 years ago) and more like ~40-60GB. Since then, the devs have been busy. There has been a LOT of reworking and a lot of new content, and the PC install size grew gradually rather than suddenly.
This was probably impacted to some extent by the discontinued game engine they're using. Bitsquid/Stingray was discontinued partway through HD2 development and they continued on with it rather than restarting production entirely.
>It seems bizarre to me that they'd have accepted such a high cost (150GB+ installation size!) without entirely verifying that it was necessary!
You should look at COD install sizes and almost weekly ridiculously huge "updates". 150gb for a first install is almost generous considering most AAA games.
Game companies these days barely optimize engine graphical performance before release never mind the package size or patching speed. They just stamp higher minimum system requirements on the package.
From a business perspective the disk footprint is only a high cost if it results in fewer sales, which I doubt it does to any significant degree. It is wasteful, but can see why optimization efforts would get focused elsewhere.
I think certain games dont even bother to optimize the install size so that you cant fit other games on the hard drive, I think COD games are regularly hundreds of gigs
I'd bet any amount of money a demo ran slow on one stakeholder's computer, who happened to have a mechanical hard drive, they attributed the slowness to the hard drive without a real investigation and optimizing for mechanical hard drive performance became standard practice. The demo may not have even been for this game, just a case of once bitten twice shy.
IIRC this has been the “done thing” forever. I’m not in game development, but I think I recall hearing about it in the Xbox 360 era. Conventional options are picked by default, benchmarks are needed to overturn that. Looking at my hard drive, massive game installations are still very much the industry standard…
I have heard that in many scenarios it is faster to load uncompressed assets directly rather than load+decompress. Load time is prioritized over hard drive space so you end up with the current situation.
You need very fast decompression for that to work these days when io speeds are so high, and decompression takes compute that is being used for game logic.
Very fast decompression often means low compression or multicore. I have seen libjpgturbo vastly outperform raw disk reads though
There have been plenty of times where the opposite is true: Storing highly compressed data and decompressing it in RAM is much faster than loading uncompressed assets.
Which is the primary problem: Computers are so varied and so ever changing that if you are optimizing without hard data from your target hardware, you aren't optimizing, you are just doing random shit.
Add to that, game devs sometimes are just dumb. Titanfall 1 came with tens of gigabytes of uncompressed audio, for "performance", which is horse shit. Also turns out they might have been lying entirely. Titanfall 1 was made on the source engine, which does not support the OGG audio format their audio files were in. So they decompressed them at install time. They could have just used a different audio file format.
High cost to who though. We see the same thing when it comes to RAM and CPU usage, the developer is not the one paying for the hardware and many gamers have shown that they will spend money on hardware to play a game they want.
Sure they may loose some sales but I have never seen many numbers on how much it really impacted sales.
Also on the disk side, I can't say I have ever looked at how much space is required for a game before buying it. If I need to clear out some stuff I will. Especially with it not being uncommon for a game to be in the 100gb realm already.
That all being said, I am actually surprised by the 11% using mechanical hard drives. I figured that NVME would be a lower percentage and many are using SSD's... but I figured the percent with machines capable of running modern games in the first place with mechanical drives would be far lower.
I do wonder how long it will be until we see games just saying they are not compatible with mechanical drives.
That already happened :) Starfield claimed to not support HDDs and really ran bad with them. And I think I saw SSDs as requirement for a few other games now, in the requirement listings on steam.
Latest Ratchet and Clank game relies heavily on ps5’s nvme drive. Its PC port states that SSD is required. And IIRC, the experience on mechanical drives is quite terrible to the unplayable level.
All of that takes time.abd you never have enough time.
At any given point if it wasn't vital to shipping and not immediately breaking, then it can be backburnered.
Messing with asset loading is probably a sure fire way to risk bugs and crashes - so I suspect this mostly was waiting on proving the change didn't break everything (and Helldiver's has had a lot of seemingly small changes break other random things).
The game is released on both PC and PS5, the latter of which was designed (and marketed) to take advantage of SSD speeds for streaming game content near real time.
The latest Ratchet and Clank, the poster child used in part to advertise the SSD speed advantage, suffers on traditional hard drives as well in the PC port. Returnal is in the same boat. Both were originally PS5 exclusives.
The HDD performance suffers very much during the portal loading sequences in Ratchet and Clank, but even the entry level SSD performs fine, with little visible difference compared to the PS5 one. It’s more about random access speed than pure throughput
I played Rift Apart from HDD and apart from extra loading time during looped animations it was fine. On the other hand Indiana Jones Great Circle was barely playable with popping-in textures and models everywhere.
Optimizing for disk space is very low on the priority list for pretty much every game, and this makes sense since its very low on the list of customer concerns relative to things like in-game performance, net code, tweaking game mechanics and balancing etc.
Apparently, in-game performance is not more important than pretty visuals. But that's based on hearsay / what I remember reading ages ago, I have no recent sources. The tl;dr was that apparently enough people are OK with a 30 fps game if the visuals are good.
I believe this led to a huge wave of 'laziness' in game development, where framerate wasn't too high up in the list of requirements. And it ended up in some games where neither graphics fidelity or frame rate was a priority (one of the recent Pokemon games... which is really disappointing for one of the biggest multimedia franchises of all time).
a one time cost of a big download is something customers have shown time and again that they're willing to bear. remember that software is optimized for ROI first and all other things second. Sometimes optimizing for ROI means "ship it today and let the first week of sales pay salaries while we fix it", sometimes ROI means picking between getting the file size down, getting that new feature out and fixing that game breaking edge case bug. Everything you do represents several things you choose not to do.
It’s the same sort of apathy/arrogance that made new Windows versions run like dogshit on old machines. Gates really should have had stock in PC makers. He sold enough of them.
I don’t think it’s always still the case but for more than ten years every release of OSX ran better on old hardware, not worse.
Some people think the problem was MS investing too eagerly into upgrading developer machines routinely, giving them a false sense of what “fast enough” looked like. But the public rhetoric was so dismissive that I find that pretty unlikely. They just didn’t care. Institutionally.
I’m not really into the idea of Helldivers in the first place but I’m not going to install a 150GB game this side of 2040. That’s just fucking stupid.
It would be ironic if incidents like this made Valve start charging companies for large file sizes of their games. It would go to show that good things get abused to no end if limits aren't set.
> These loading time projections were based on industry data - comparing the loading times between SSD and HDD users where data duplication was and was not used. In the worst cases, a 5x difference was reported between instances that used duplication and those that did not. We were being very conservative and doubled that projection again to account for unknown unknowns
Unfortunately it's not only game development, all modern society seems operate like this.
Back of the envelope, in the two years since the game was released, this single bug has wasted at least US$10,000,000 of hardware resources. That's a conservative estimate (20% of people who own the game keep it installed, the marginal cost of wasted SSD storage in a gaming PC is US$2.50 per TB per month, the install base grew linearly over time), so the true number is probably several times higher.
In other words, the game studio externalised an eight-figure hardware cost onto their users, to avoid a five-to-six-figure engineering cost on their side.
Data duplication can't just be banned by Steam, because it's a legitimate optimisation in some cases. The only safeguard against this sort of waste is a company culture which values software quality. I'm glad the developers fixed this bug, but it should never have been released to users in the first place.
Steam compresses games as much as possible, so in the case of Helldivers 2, you had to download between ~30 and ~40 GB, which was then unpacked to 150 GB (according to SteamDB[0])
You are missing that each update takes AGES while it tortures your disk for patching the files (on my machine it takes 15min or so, and that's on an SSD). So I agree that this is careless and reminds me of the GTA5 startup time that was fixed by a dedicated player who finally had enough and reverse engineered the problem (see https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times...). I still find these things hard to accept.
In this case, the bug was 131 GB of wasted disk space after installation. Because the waste came from duplicate files, it should have had little impact on download size (unless there's a separate bug in the installer...)
This is why the cost of the bug was so easy for the studio to ignore. An extra 131 GB of bandwidth per download would have cost Steam several million dollars over the last two years, so they might have asked the game studio to look into it.
Makes sense, initial claim was that HD2 size was mainly because of duplicated assets, and any compression worth it's salt would de-duplicate things effectively.
> Originally, the game’s large install size was attributed to optimization for mechanical hard drives since duplicating data is used to reduce loading times on older storage media. However, it turns out that Arrowhead’s estimates for load times on HDDs, based on industry data, were incorrect.
It wasn't a bug. They made a decision on what to optimise which was based on incomplete / incorrect data and performed the wrong optimisation as a result.
As a player of the game, I didn't really care that it took up so much space on my PC. I have 2TB dedicated for gaming.
Why not offer 2 versions for download and let the user choose, whether they want to block their whole disk with a single game, or accept a bit longer loading times? Or let the user at installation time make an informed decision by explaining the supposed optimization? Or let the user decide before downloading, what resolution (ergo textures) they want as the highest resolution they will play the game at and only download the textures they need up to that resolution?
> the marginal cost of wasted SSD storage in a gaming PC is US$2.50 per TB per month
Out of curiousity, how do you come up with a number for this? I would have zero idea of how to even start estimating such a thing, or even being able to tell you whether "marginal cost of wasted hard drive storage" is even a thing for consumers.
I'd be very interested in hearing alternative estimates, but here's my working:
The lowest cost I could find to rent a server SSD was US$5 per TB-month, and it's often much higher. If we assume that markets are efficient (or inefficient in a way that disadvantages gaming PCs), we could stop thinking there and just use US$2.50 as a conservative lower bound.
I checked the cost of buying a machine with a 2 TB rather than 1 TB SSD; it varied a lot by manufacturer, but it seemed to line up with $2.50 to $5 per TB-month on a two-to-five-year upgrade cycle.
One reason I halved the number is because some users (say, a teenager who only plays one game) might have lots of unused space in their SSD, so wasting that space doesn't directly cost them anything. However, unused storage costs money, and the "default" or "safe" size of the SSD in a gaming PC is mostly determined by the size of games - so install size bloat may explain why that "free" space was purchased in the first place.
> whether "marginal cost of wasted hard drive storage" is even a thing for consumers
As long as storage has a price, use of storage will have a price :-)
Maybe average cost of next-size-up SSD price divided by a SWAG of a gaming PC lifetime? So if I had to buy a 2 TB NVMe stick instead of a 1 TB stick it's an extra $70 and I upgrade after 5 years that's only about $1 per TB-Month. I don't game I have no idea if those are good numbers.
The cheapest storage tier on s3 with instant retrieval is $.004 per GB-Month which implies AWS can still make money at $4 per TB-Month so $2.50 for consumer hardware sounds reasonable to me.
“On a higher PC it wouldn’t be an issue. On a medium or moderate PC, it wouldn’t be an issue, it’s that on a two-core [machine] with where our min spec is, we couldn’t dedicate those resources to audio.”
It's also bullshit. Half Life 2 didn't need uncompressed audio. MP3 decompression is damn near free.
Half life also did it with half the required minimum cores, and 1 ghz less clock speed on that CPU. It released a decade before Titanfall 1. Sure sure, it's got so much more going on, but uh, that much?
For a reference of how trivial it is for CPUs to decode MP3 files, software decoders take tens of MIPS. Remember that unit? Less than a percent of one of those minimum spec CPUs required.
You know what's funny? The source engine only supports MP3 compressed audio. Do you know what titanfall 1 downloads and decompresses to to create 30gb of audio data? Lossy compressed, 160kb/s OGG format audio.
It was BS considering countless other games having no problem with sound. Decoding something like Opus takes ~30MHz of a single CPU core[1], meaning even an unreasonable situation of decoding 16 simultaneous uninterrupted 128Kbit Stereo streams would only eat half of one core.
Possibly a similar process to when you go into an AWS account, and find dozens of orphaned VMs, a few thousand orphaned disk volumes, etc., saving like $10k/month just deleting unused resources.
It's not a case of forgotten data, it's duplicated for access time reasons, like with optical media.
It follows in the footsteps of trading in storage for less compute and/or better performance.
An opposite approach in the form of a mod for Monster Hunter: Wilds recently made it possible [0] for end-users to decompress all the game textures ahead of time. This was beneficial there, because GPU decompression was causing stalls, and the trading in of compute for less storage resulted in significantly worse performance.
I've been really curious precisely what changed, and what sort of optimization might have been involved here.
Because offhand, I know you could do things like cute optimizations of redundant data to minimize seek time on optical media, but with HDDs, you get no promises about layout to optimize around...
The only thing I can think of is if it was literally something as inane as checking the "store deduplicated by hash" option in the build, on a tree with copies of assets scattered everywhere, and it was just nobody had ever checked if the fear around the option was based on outcomes.
(I know they said in the original blog post that it was based around fears of client performance impact, but the whole reason I'm staring at that is that if it's just a deduplication table at storage time, the client shouldn't...care? It's not writing to the game data archives, it's just looking stuff up either way...)
I'm not entirely clear what you're trying to say, but, my understanding is that they simply put lots of copies of files in lots of places like games have done for a long time, in the hopes it would lower seek times on HDDs for those players who use them.
They realised, after a lot of players asking, that it wasn't necessary, and probably had less of an impact than they thought.
They removed the duplicates, and drastically cut the install size. I updated last night, and the update alone was larger than the entire game after this deduplication run, so I'll be opting in to the Beta ASAP.
It's been almost a decade since I ran spinning rust in a desktop, and while I admire their efforts to support shitty hardware, who's playing this on a machine good enough to play but can't afford £60 for a basic SSD for their game storage?
HDDs also have a spinning medium and a read head , so the optimization is similar to optical media like CDs.
Let’s say you have UI textures that you always need, common player models and textures, the battle music, but world geometry and monsters change per stage.
Create an archive file (pak, wad, …) for each stage, duplicating UI, player and battle music assets into each archive. This makes it so that you fully utilize HDD pages (some small config file won’t fill 4kb filesystem pages or even the smaller disk sectors). All the data of one stage will be read into disk cache in one fell swoop as well.
On optical media like CDs one would even put some data closer to the middle or on the outer edge of the disc because the reading speed is different due to the linear velocity.
This is an optimization for bandwidth at the cost of size (which often wasn’t a problem because the medium wasn’t filled anyway)
> HDDs also have a spinning medium and a read head , so the optimization is similar to optical media like CDs.
HDDs also have to real with fragmentation, I wonder what the odds that you get to write 150 GBs (and then regular updates in the 30GB range) without breaking it into fragments...
The game installer can't control the layout on an HDD without doing some very questionable things like defragging and moving existing user files around the disk. It probably _could_ but the risk of irrecoverable user data loss or accidentally corrupting a boot partition via a bug would make it completely not worth it.
Even if you pack those, there's no guarantee they don't get fragmented by the filesystem.
CDs are different not because of media, but because of who owns the storage media layout.
I recently downloaded Hunt showdown. I think it was around 70 gigs. About a month later, I had to update it. The download was the same size. I think they literally just overrode the entire game because they were too lazy to update it properly.
The opposite happens in some games where the update will be a gigabyte or two, but the patching takes so long, that some players with fast enough download speeds will swear by wholly uninstalling and reinstalling the game.
I observed this too with some other games. Really annoying, when you need to redownload tens of gigabytes , because they cannot be arsed to put a proper updater. Things that were solved by most games way before Steam became even big.
To be fair, they started working on the game before it was discontinued. I'm sure someone made a decision that it made more sense to continue on without support rather than start from scratch.
You can, but on the other hand they've been battling bugs from it with every release. The game is notorious for breaking things constantly. I played for quite a while and the sound engine was always awful with things not in your line of sight frequently not making any noise at all, and every month or two a new major bug relating to host client desyncing is found out by the community who then has to have big campaigns badgering the devs to notice and fix it. Very fun game still but if they had started with a supported engine a lot of stuff would probably work way better
Yeah but what evidence is there that what you describe is an engine problem rather than an Arrowhead can't software engineer problem?
As others point out, several other good games were made on this engine, and they do not have such abysmal engineering quality. Vermintide 2 is on the same engine and very much doesn't have problems with constant bugs and regressions.
Arrowhead struggled to fix the lock on for the missile launcher for months, and then claimed that "raycasts are hard* and I just don't think they should be for a game dev. Several patches seem to show that they just do not test at all the release build, and that they don't seem to have functioning version control to keep from breaking things they've already fixed.
>and every month or two a new major bug relating to host client desyncing is found out by the community who then has to have big campaigns badgering the devs to notice and fix it.
This is not a game engine problem. Does fire damage properly work across multiplayer yet?
If im remembering correctly, I think its a variant of the engine that vermintide and darktide use, some old autodesk-acquired engine. Given the successes of vermintide and darktide, the engine still has plenty of legs left.
It was legit faster to delete and redownload this game than update it since steam considered my SSD too full (WITH 200 GIGS FREE) to download the files to said SSD, instead opting to use my SLOWEST HDD as the cache drive for the download.
It would then proceed to download the update in 5 minutes and spend 8 HOURS UPDATING.
A full download of the game? 10 minutes.
Glad to see this update. I hope more games follow suit
I remember how pre-loading some games turned out to be slower for me because downloading it at launch meant decrypting it directly from the network, but decrypting files on drive that are already encrypted doubled the IO and that was too much random access for my hard drive
Readit News
It seems bizarre to me that they'd have accepted such a high cost (150GB+ installation size!) without entirely verifying that it was necessary!
I expect it's a story that'll never get told in enough detail to satisfy curiosity, but it certainly seems strange from the outside for this optimisation to be both possible and acceptable.
They’re not the ones bearing the cost. Customers are. And I’d wager very few check the hard disk requirements for a game before buying it. So the effect on their bottom line is negligible while the dev effort to fix it has a cost… so it remains unfixed until someone with pride in their work finally carves out the time to do it.
If they were on the hook for 150GB of cloud storage per player this would have been solved immediately.
That's why they did the performance analysis and referred to their telemetry before pushing the fix. The impact is minimal because their game is already spending an equivalent time doing other loading work, and the 5x I/O slowdown only affects 11% of players (perhaps less now that the game fits on a cheap consumer SSD).
If someone "takes pride in their work" and makes my game load five times longer, I'd rather they go find something else to take pride in.
I've racked up 700 hours in the game and the storage requirements I didn't care about.
I'm not sure that's necessarily true... Customers have limited space for games; it's a lot easier to justify keeping a 23GB game around for occasional play than it is for a 154GB game, so they likely lost some small fraction of their playerbase they could have retained.
I have to check. You're assumption is correct. I am one of very few.
I don't know the numbers and I'm gonna check in a sec but I'm wondering whether the suppliers (publishers or whoever is pinning the price) haven't screwed up big time by driving prices and requirements without thinking about the potential customers that they are going to scare away terminally. Theoretically, I have to assume that their sales teams account for these potentials but I've seen so much dumb shit in practice over the past 10 years that I have serious doubts that most of these suits are worth anything at all, given that grown up working class kids--with up to 400+ hours overtime per year, 1.3 kids on average and approx. -0.5 books and news read per any unit of time--can come up with the same big tech, big media, economic and political agendas as have been in practice in both parts of the world for the better part of our lives--if you play "game master" for half a weekend where you become best friends with all the kiosks in your proximity.
> the effect on their bottom line is negligible
Is it, though? My bold, exaggerated assumption is that they would have had 10% more sales AND players.
And the thing is, that at any point in time when I, and a few I know, had time and desire to play, we would have had to either clean up our drives or invest game price + sdd price for about 100 hours of fun over the course of months. We would have gladly licked blood but no industry promises can compensate for even more of our efforts than enough of us see and come up with at work. As a result, at least 5 buyers and players lost, and at work and elsewhere you hear, "yeah, I would, if I had some guys to play with" ...
And this being primarily a live-service game drawing revenues from micro-transactions, especially a while after launch, and the fact that base console drives are still quite small to encourage an upgrade (does this change apply to consoles too?), there’s probably quite an incentive to make it easy for users to keep the game installed.
It was amazing how often people wanted to optimize stuff that wasn't a bottleneck in overall performance. Real bottlenecks were often easy to see when you measured and usually simple to fix.
But it was also tough work in the org. It was tedious, time-consuming, and involved a lot of experimental comp sci work. Plus, it was a cost center (teams had to give up some of their budget for perf engineering support) and even though we had racks and racks of gear for building and testing end-to-end systems, what most dev teams wanted from us was to give them all our scripts and measurement tools to "do it themselves" so they didn't have to give up the budget.
> But it was also tough work in the org. It was tedious, time-consuming, and involved a lot of experimental comp sci work. Plus, it was a cost center (teams had to give up some of their budget for perf engineering support) and even though we had racks and racks of gear for building and testing end-to-end systems, what most dev teams wanted from us was to give them all our scripts and measurement tools to "do it themselves" so they didn't have to give up the budget.
Misaligned budgeting and goals is bane of good engineering. I've seen some absolutely stupid stuff like outsourcing hosting a simple site to us, because client would rather hire 3rd party to buy domain and put a simple site there (some advertising), than to deal with their own security guys and host it on their own infrastructure.
"It's a cost center" "So is fucking HR, why you don't fire them ?" "Uh, I'll ignore that, pls just invoice anything you do to other teams" ... "Hey, they bought cloud solution that doesn't work/they can't figure it out, can you help them" "But we HAVE stuff doing that cheaper and easier, why they didn't come to us" "Oh they thought cloud will be cheaper and just work after 5 min setup"
There will be huge mistakes occasionally, but mostly it is death by a thousand cuts -- it's easy to commit a 0.1% regression here or there, and there are hundreds of other engineers per performance engineer. Clawing back those 0.1% losses a couple times per week over a large deployed fleet is worthwhile.
Because it's a recent 20TB HDD the read speeds approach 250MB/s and I've also specifically partitioned it at the beginning of the disk just for games so that it can sustain full transfer speeds without files falling into the slower tracks, the rest of the disk is then partitioned for media files that won't care much for the speed loss. It's honestly fine for the vast majority of games.
SSD sizes are still only equal to the HDD sizes available and common in 2010 (a couple TB~). SSD size increases (availability+price decreases) for consumers form factors have entirely stopped. There is no more progress for SSD because quad level cells are as far as the charge trap tech can be pushed and most people no longer own computers. They have tablets or phones or if they have a laptop it has 256GB of storage and everything is done in the cloud or with an octopus of (small) externals.
154GB was the product of massive asset duplication, as opposed 23GB being the product of an optimization miracle. :)
How did it get so bad on PC?
Well, it wasn't always so crazy. I remember it being reasonable closer to launch (almost 2 years ago) and more like ~40-60GB. Since then, the devs have been busy. There has been a LOT of reworking and a lot of new content, and the PC install size grew gradually rather than suddenly.
This was probably impacted to some extent by the discontinued game engine they're using. Bitsquid/Stingray was discontinued partway through HD2 development and they continued on with it rather than restarting production entirely.
https://en.wikipedia.org/wiki/Bitsquid
You should look at COD install sizes and almost weekly ridiculously huge "updates". 150gb for a first install is almost generous considering most AAA games.
Dead Comment
On phone, I bet you see some more effort.
Very fast decompression often means low compression or multicore. I have seen libjpgturbo vastly outperform raw disk reads though
Which is the primary problem: Computers are so varied and so ever changing that if you are optimizing without hard data from your target hardware, you aren't optimizing, you are just doing random shit.
Add to that, game devs sometimes are just dumb. Titanfall 1 came with tens of gigabytes of uncompressed audio, for "performance", which is horse shit. Also turns out they might have been lying entirely. Titanfall 1 was made on the source engine, which does not support the OGG audio format their audio files were in. So they decompressed them at install time. They could have just used a different audio file format.
Sure they may loose some sales but I have never seen many numbers on how much it really impacted sales.
Also on the disk side, I can't say I have ever looked at how much space is required for a game before buying it. If I need to clear out some stuff I will. Especially with it not being uncommon for a game to be in the 100gb realm already.
That all being said, I am actually surprised by the 11% using mechanical hard drives. I figured that NVME would be a lower percentage and many are using SSD's... but I figured the percent with machines capable of running modern games in the first place with mechanical drives would be far lower.
I do wonder how long it will be until we see games just saying they are not compatible with mechanical drives.
At any given point if it wasn't vital to shipping and not immediately breaking, then it can be backburnered.
Messing with asset loading is probably a sure fire way to risk bugs and crashes - so I suspect this mostly was waiting on proving the change didn't break everything (and Helldiver's has had a lot of seemingly small changes break other random things).
The latest Ratchet and Clank, the poster child used in part to advertise the SSD speed advantage, suffers on traditional hard drives as well in the PC port. Returnal is in the same boat. Both were originally PS5 exclusives.
By comparison a SATA III port caps out at 6Gbps (750 MB/s), and first generation NVMe drives ("gen 3") were limited to 3500 MB/s.
I believe this led to a huge wave of 'laziness' in game development, where framerate wasn't too high up in the list of requirements. And it ended up in some games where neither graphics fidelity or frame rate was a priority (one of the recent Pokemon games... which is really disappointing for one of the biggest multimedia franchises of all time).
Deleted Comment
I don’t think it’s always still the case but for more than ten years every release of OSX ran better on old hardware, not worse.
Some people think the problem was MS investing too eagerly into upgrading developer machines routinely, giving them a false sense of what “fast enough” looked like. But the public rhetoric was so dismissive that I find that pretty unlikely. They just didn’t care. Institutionally.
I’m not really into the idea of Helldivers in the first place but I’m not going to install a 150GB game this side of 2040. That’s just fucking stupid.
Also if goal was to improve things for small minority they could've just pawned it off to free DLC, like how some games do with 4k texture packs
> These loading time projections were based on industry data - comparing the loading times between SSD and HDD users where data duplication was and was not used. In the worst cases, a 5x difference was reported between instances that used duplication and those that did not. We were being very conservative and doubled that projection again to account for unknown unknowns
Unfortunately it's not only game development, all modern society seems operate like this.
Twenty years on, and somehow that's still 'big'.
Computing progress disappoints me.
Wait till you find out what engine this game is made in. https://80.lv/articles/helldivers-ii-was-built-on-an-archaic...
In other words, the game studio externalised an eight-figure hardware cost onto their users, to avoid a five-to-six-figure engineering cost on their side.
Data duplication can't just be banned by Steam, because it's a legitimate optimisation in some cases. The only safeguard against this sort of waste is a company culture which values software quality. I'm glad the developers fixed this bug, but it should never have been released to users in the first place.
Steam compresses games as much as possible, so in the case of Helldivers 2, you had to download between ~30 and ~40 GB, which was then unpacked to 150 GB (according to SteamDB[0])
[0] https://steamdb.info/app/553850/depots/
This is why the cost of the bug was so easy for the studio to ignore. An extra 131 GB of bandwidth per download would have cost Steam several million dollars over the last two years, so they might have asked the game studio to look into it.
> Originally, the game’s large install size was attributed to optimization for mechanical hard drives since duplicating data is used to reduce loading times on older storage media. However, it turns out that Arrowhead’s estimates for load times on HDDs, based on industry data, were incorrect.
It wasn't a bug. They made a decision on what to optimise which was based on incomplete / incorrect data and performed the wrong optimisation as a result.
As a player of the game, I didn't really care that it took up so much space on my PC. I have 2TB dedicated for gaming.
Questions, questions, questions.
Out of curiousity, how do you come up with a number for this? I would have zero idea of how to even start estimating such a thing, or even being able to tell you whether "marginal cost of wasted hard drive storage" is even a thing for consumers.
The lowest cost I could find to rent a server SSD was US$5 per TB-month, and it's often much higher. If we assume that markets are efficient (or inefficient in a way that disadvantages gaming PCs), we could stop thinking there and just use US$2.50 as a conservative lower bound.
I checked the cost of buying a machine with a 2 TB rather than 1 TB SSD; it varied a lot by manufacturer, but it seemed to line up with $2.50 to $5 per TB-month on a two-to-five-year upgrade cycle.
One reason I halved the number is because some users (say, a teenager who only plays one game) might have lots of unused space in their SSD, so wasting that space doesn't directly cost them anything. However, unused storage costs money, and the "default" or "safe" size of the SSD in a gaming PC is mostly determined by the size of games - so install size bloat may explain why that "free" space was purchased in the first place.
> whether "marginal cost of wasted hard drive storage" is even a thing for consumers
As long as storage has a price, use of storage will have a price :-)
The cheapest storage tier on s3 with instant retrieval is $.004 per GB-Month which implies AWS can still make money at $4 per TB-Month so $2.50 for consumer hardware sounds reasonable to me.
Where are you getting this number from? Not necessarily arguing with it, just curious.
from https://www.escapistmagazine.com/titanfall-dev-explains-the-...
Half life also did it with half the required minimum cores, and 1 ghz less clock speed on that CPU. It released a decade before Titanfall 1. Sure sure, it's got so much more going on, but uh, that much?
For a reference of how trivial it is for CPUs to decode MP3 files, software decoders take tens of MIPS. Remember that unit? Less than a percent of one of those minimum spec CPUs required.
You know what's funny? The source engine only supports MP3 compressed audio. Do you know what titanfall 1 downloads and decompresses to to create 30gb of audio data? Lossy compressed, 160kb/s OGG format audio.
[1] iPod Classic (1998 era ARM9) decodes 128 kbps stereo Opus at ~150% real time at stock cpu frequency. Opus is not the lightest choice either https://www.rockbox.org/wiki/CodecPerformanceComparison#ARM
It follows in the footsteps of trading in storage for less compute and/or better performance.
An opposite approach in the form of a mod for Monster Hunter: Wilds recently made it possible [0] for end-users to decompress all the game textures ahead of time. This was beneficial there, because GPU decompression was causing stalls, and the trading in of compute for less storage resulted in significantly worse performance.
[0] https://youtu.be/AOxLV2US4Ac
In this case, I don't think it was forgetfulness; unlike us, they have an excuse and they were trying to optimise for disk seek times.
Anyway, I've got a half-dozen cloud accounts I need to go check for unused resources waves.
Because offhand, I know you could do things like cute optimizations of redundant data to minimize seek time on optical media, but with HDDs, you get no promises about layout to optimize around...
The only thing I can think of is if it was literally something as inane as checking the "store deduplicated by hash" option in the build, on a tree with copies of assets scattered everywhere, and it was just nobody had ever checked if the fear around the option was based on outcomes.
(I know they said in the original blog post that it was based around fears of client performance impact, but the whole reason I'm staring at that is that if it's just a deduplication table at storage time, the client shouldn't...care? It's not writing to the game data archives, it's just looking stuff up either way...)
They realised, after a lot of players asking, that it wasn't necessary, and probably had less of an impact than they thought.
They removed the duplicates, and drastically cut the install size. I updated last night, and the update alone was larger than the entire game after this deduplication run, so I'll be opting in to the Beta ASAP.
It's been almost a decade since I ran spinning rust in a desktop, and while I admire their efforts to support shitty hardware, who's playing this on a machine good enough to play but can't afford £60 for a basic SSD for their game storage?
Deleted Comment
Let’s say you have UI textures that you always need, common player models and textures, the battle music, but world geometry and monsters change per stage. Create an archive file (pak, wad, …) for each stage, duplicating UI, player and battle music assets into each archive. This makes it so that you fully utilize HDD pages (some small config file won’t fill 4kb filesystem pages or even the smaller disk sectors). All the data of one stage will be read into disk cache in one fell swoop as well.
On optical media like CDs one would even put some data closer to the middle or on the outer edge of the disc because the reading speed is different due to the linear velocity.
This is an optimization for bandwidth at the cost of size (which often wasn’t a problem because the medium wasn’t filled anyway)
HDDs also have to real with fragmentation, I wonder what the odds that you get to write 150 GBs (and then regular updates in the 30GB range) without breaking it into fragments...
Even if you pack those, there's no guarantee they don't get fragmented by the filesystem.
CDs are different not because of media, but because of who owns the storage media layout.
Unfortunately there are still people on metered/slow connections that really do care still.
Dead Comment
It's also a title that shows you can have a really good game without the latest tech.
As others point out, several other good games were made on this engine, and they do not have such abysmal engineering quality. Vermintide 2 is on the same engine and very much doesn't have problems with constant bugs and regressions.
Arrowhead struggled to fix the lock on for the missile launcher for months, and then claimed that "raycasts are hard* and I just don't think they should be for a game dev. Several patches seem to show that they just do not test at all the release build, and that they don't seem to have functioning version control to keep from breaking things they've already fixed.
>and every month or two a new major bug relating to host client desyncing is found out by the community who then has to have big campaigns badgering the devs to notice and fix it.
This is not a game engine problem. Does fire damage properly work across multiplayer yet?
It would then proceed to download the update in 5 minutes and spend 8 HOURS UPDATING.
A full download of the game? 10 minutes.
Glad to see this update. I hope more games follow suit