There’s an implicit story here which I find quite sad that not a single developer at Rockstar in the prior eight years had the autonomy or bandwidth to discover and fix this themselves.
I’m starting to slowly realize how terrible it is working in environments where people simply aren’t curious. It’s so hard for me as an individual contributor to decouple this way of thinking. Curiosity is how we find better ways of doing things or finding out hidden flows that can be better. Incurious people are a contagion, but I want to believe you can make people curious. It can’t be innate.
Every company talks about wanting to have driven curious employees to solve complex issues or challenges but you just get stomped with a boot to your face if you do anything outside your lane or simply ask why.
It’s also worse because I’m starting to realize how lucky I was to be raised as a curious individual from my parents who never went to college but at least were curious themselves about different things (pottery, soldering, gardening).
The quote “Curiosity is insubordination in its purest form” suddenly has a different context when working for a company.
I'd be careful about painting the developers at Rockstar as "not curious," that feels like an incredibly unfair conclusion to draw from this. Those developers have other pressures, and deadlines to hit. There are a million reasons why this bug wasn't solved by the team there. When a user complains about a bug in software you wrote, is it because you just "weren't curious" enough? Or is the situation more complicated?
Lots of incredibly talented people worked on GTA Online, it's worth trying to think through why a team of smart developers might not have prioritized this, rather than assuming something negative about them.
+1 to the other replies here, I think it's very poor form to throw the engineers under the bus for not being curious. The most likely explanation is they were under insane time and resource constraints, especially in the games industry. They wrote something that worked, then were forced to move on to the other 5000 tasks in their backlog that had to get done before release. When you're already being told to crunch 80+ hour weeks for months or years at a time, you run out of gas.
I guess it's possible that the engineers who were talented enough to create this extremely impressive game in the first place somehow threw up their hands and said "guess it can never take less than 7 minutes to load, best we can do". But it seems much more likely that they were well aware, but never allowed by poor management to go back and fix it. The game was released and printing money, why bother. They're on to the next title.
People might start off curious but eventually they keep running into political roadblocks and other nonsense, which slowly beats the interest out of them.
If those are the kinds of people that continue surviving then those are the kinds of people which your environment cultivates.
Sadly at Rockstar it's likely that employees are curious, but are under so much pressure for deadlines, and working such long hours, that they don't have time to follow their curiosity.
Curiosity has to come from the top and fall all the way down. As others and you have noted in other comments, its the environment for the workers that sets up whether or not they will even be able to pursue curious things, and that's on the managers to create that environent. However, the managers themselves have bossess too, so that has to trickle up, to their bosses bosses all the way up the chain.
In short, every job needs to be able to have autonomy to enact on the ground solutions for what their immediate team needs. You shouldn't be penalized for going off the reservation and coming back with a creative solution. The problem is corporate structure for the past few decades has been about rooting out this autonomy and anything opaque about job responsibilities. There is a lot that's been done that would need undoing to change those environments, and imo you are better off finding another workplace than bringing about the monumental change required. A mature corporation is like a boulder tumbling down a mountainside; you can hardly change its course much less stop it short of where its heading.
In my experience it's the opposite. Most people are born innately curious. It takes intense systemic conditioning to suppress their natural curiosity. For many of us that conditioning begins in the traditional education system and is completed in the workplace.
It is not that developers arent curious. They probably are in their cages set by bad project managers who dont even play own game and dont notice it takes forever to load.
The project managers could put load times as a priority but they genuinely dont see it as a problem or dont care at all.
It happens all the time. There is a lot of unoptimized software.
Perhaps the issue comes from the top: the project managers are supposed to only do things that increase revenue and all other things are not rewarded at all? Perhaps executives only care if they hit quartely results? (Short term gain traded for long term loss - the game is made for whales who are so invested that they will play even when it is technically bad - so milk them now with new content for whales and ignore everththing else, who cares that new players will be discouraged by bad load times).
In my opinion if nobody plays own game / eats own dogfood, they dont even see its problems. Blizzard is at this stage now - some of their games barely work. You buy Starcraft 1 through the launcher... and it doesnt work(I think it works now, but it took them 1 month to notice it and another to fix it). Heroes of the storm - downloaded a 137mb patch every run - it took them half a year to notice it and do something with it. It is like nobody cares about the brand at all. Game company where your games dont work.
IMHO a problem with people - that comes from the top. They want only story points for things that bring money. The project managers either dont care, or cannot deal with other problems at all. I bet they dont care. This happens in many companies - there were few people who tried, they got burned for that: terminated or quit -> so now nobody cares. "The game is good enough to still sell some skins to the whales, so why bother".
Look at the results: they got an external comment what to fix. Did they launch their own internal project to optimize the game? Did they check for other things that make it load slow? It looka that someone else did a partial fix that is "good enough" so everything is back to usual and nobody cares about load times.
I'd never heard that Nabakov quote, but it's terrifically cynical and, I hope, false but, in my experience, not entirely untrue.
Designers' unnecessary desire to delight users staves off the more Kafkaesque user experiences, and we all live in the gap of cops' unnecessary (occasional) sympathy for (some) nominal outlaws (we're all nominal outlaws).[0]
When you mentioned curiosity I thought of a firm where colleagues scoffed at me as I rummaged through conference room drawers, saying, "you're not going to find anything interesting in there." I'm curious about cupboards as well as log traces -- an almost ubiquitously "eyes-glaze-over" boring artifact of modernity that, alongside popularly mind-numbing "code", I am equipped to find interesting.
Is it any wonder, then, in this workplace where such innocent, cheap, curiosity were scoffed at, I was not too long after threatened with the extraordinarily archaic (in our at-will state, at least) legal notion of "insubordination"?
It's a shame, because insofar as "all ambiguity is resolved by actions of practitioners at the sharp end of the system,"[1] the liberated curiosity of individual contributors is often the only "crack in everything" that's "how the light gets in."[2]
0. "With two lines of a man's handwriting, an accusation could be made against the most innocent." (Françoise Bertaut de Motteville)
In gamedev where crunch is the default and people are routinely working for 10, 12 or more hours a day, sometimes including weekends, sometimes for months on end - it's not very kind to assume it's curiosity that they lack.
>Every company talks about wanting to have driven curious employees to solve complex issues or challenges but you just get stomped with a boot to your face if you do anything outside your lane or simply ask why.
That's because "curiosity" translates to "more work" at #BigCo. And people work at #BigCo so that they can comfortably do as little work as necessary to maintain their lifestyles. This is a hard rule for any organization. So if you want what you're describing, it will only be found at small startups where everyone has skin in the game.
I strongly believe that R&D teams should have some time reserved to "do what you want" (limited to product improvement, obviously)
Perhaps this might not be appropriate in the gaming industry... and who knows if Rockstar don't already have this.
However, the GTA loading time is horrendous (and I had a spectrum sinclair when I was a kid) - surely there must have been a work item, ticket, bug reports, or whatever, for this.
I work in a company where that can happen. In fact we had just a case today. For the last 20 years, our ever growing C# monster application used across the world to make millions, literally, had random crashes. Comes with volume spike, usage spike, we never quite knew: everything would display a red cross, the gdi handlers would reach 10k and bam.
For 19 year we just looked at it in defeat. For the last year, bit on and off. Starting last monday it crashed every morning for a user sitting next to me, a big nasty german boss (Im French).
I sat down, read his logs, saw it failed on ImageList creation again for some fucking reason, looked how many of these things we used and what they did. 2 hours later this 20 yo mystery bug dozens of devs had noped out of was fixed. We fully get it now. Wasnt even a matter of priority, it took 2 hours of looking at it. A lunch break basically.
I dont have a sort of morale of the story or whatnot, but well, Rockstar would have figured it out eventually. Just need a nasty fucker next to you whining abt it everyday.
There's a attitude, arguably backed by sales, that load times don't really matter.
All of the Source games (HL2, Portal, Alyx, etc...) have arguably bad load times. They all sold great! GTA sold great even with it's previous load times.
It's just not a priority for most teams. In Rockstar's case it happened to be easy to fix but in most cases it's not so easy because the engineering team didn't want to deal with it.
AFAIK, Source, and maybe Unreal/Unity are built in such a way that by default, if you need to start the level over you have to re-load it entirely. They have mutable state spread all over the place and have no way to mark what's important (needs to be saved, what's not (can be zeroed), and/or they haven't bother to put all mutable state in a single places so either it can be restored from memory or loaded by just loading a tiny bit.
There's zero reason HL:Alyx should take 15-30 seconds to restart a level you're already in when clearly less than 1k of state has changed since you started the level. But, fixing that would require a major re-design of the Source engine. Slow load times don't affect sales. So almost no one does it. Easy just to let each gameplay programmer do whatever they want and to restart just reload the level.
I put up with it for some games (the ones mentioned above for example). But I've also quit a few recently. Games where you die a lot and to add to the frustration it takes 1-2 minutes to load again only to get killed in 5-20 seconds. That frustration means trial and erroring to figure out what to do gets a punch in the face and so I stop playing what might have been a good game otherwise.
where I work at, it is completely discouraged to do anything out of what was asked you to do. so I don't judge the workers/slaves. It's not allowed to be creative/careful in most places if it's not harming the bosse's wallets.
This was discussed a bit in the original post [1] but I believe this comes down to the problem being sufficiently deep in the stack that there was an assumption that this wasn't something that could be improved on. I'm sure the loading code looked good.
It took someone who didn't know what that code looked like being curious about what was actually happening.
Shows that dogfooding is one of the best thing you can do for your product. I remember reading a story about Valve when they were working on Dota 2, I think Gabe said that every day after work they would play for hours and couldn't stop.
If I recall correctly the Dev team didn’t notice this as they were using a test JSON which was not very complicated. Only when the game was in production did the complex JSON cause the long loading times
To be fair, the person that found the issue was persistent and did some serious digging to figure out what was going on.
I imagine that even if you were on the GTA team, you'd most likely think the long loading time was unavoidable due to some complexity you were not aware of.
Great work to solve it from outside of the company.
Serious digging was needed because he does not have the source. Any devs could easily profile the game during loading and that sscanf or strlen would be at top
Maybe they saw sscanf and assumed it is normal that it takes time, but more likely there were so much stuff to do and somehow no one checked
You would think that the more time people spend playing the game, the more money the game makes. I would think online loading metrics would be a similar problem for whomever is responsible for input latency.
That's why you stop asking permission to make quick fixes and just submit the PR. Obviously there is a balance here but if you can't "steal" a few hours in a week from the product driven roadmap to do so then there is something wrong in the engineering culture.
Absolutely. If something is oddly slow, I run the profiler to see what's going on. If the profiler shows me something boneheaded like this that is a big win to fix, I fix it. Obviously I make sure the tests still pass, but then I check it in. If someone is going to give me shit for improving the product, that is not somewhere I want to work.
Do any orgs explicitly ask the engineers themselves decide how to allocate a certain percentage of story points in a given sprint?
Product people are great for selecting which features to build out and which bugs to address, but are poorly equipped to decide how to prioritize things like: performance optimizations, bugfixes that haven't (yet) been reported by the end users, code refactoring, paying down tech debt, modernizing tooling, misc quality of life improvements for the engineering team, etc.
The solution is for ICs to stick their necks out, go over the product manager's head, and ignore the org chart? Not get a new product manager who cares about the product?
Think about how many human lifetimes worth of extra minutes were wasted from everyone waiting for this x number of players X years this has been a problem. Even if you can’t implement bubble sort on a whiteboard in your sleep, 5+ minute load times should have been setting off alarm bells in many engineers heads. I’m sure this did actually cost R* real money. Bounce rates for websites increase 32% for 1s vs 3s load times, according to Google: https://www.thinkwithgoogle.com/marketing-strategies/app-and...
Obviously we don’t have public data on it but I’m quite sure that some at least some people got frustrated with waiting and just stopped playing.
If you think about it, those lives actually were saved. People either didn't play the video game, or they spent a moment in the loading screen in quiet contemplation.
We are in the era where its impossible to be bored. In the elevator? Phone out on instagram. Loading screen? Same thing. Instant dopamine the minute you need a drip. The playerbase clearly doesn't care. To be honest, these loading screen times are as bad as they've always been for console games (especially considering it was only a generation ago that they ran off a disc).
Not much, the GTA5 online scene is alive and well. Considering they've sold this game over the course of three generations makes me believe they've made a mega flipton of money off this game.
> they've made a mega flipton of money off this game.
It can be simultaneously true that they made a lot off the game && with lower loading times they could have made even more. In-game microtransaction ARPU scales with playtime. >4 min per session spent waiting to load instead of playing == lost rev.
I know so many people that have stopped playing due to the loading times. Just because a scene is active and alive, doesn't mean it couldn't be even more active and alive with such a minor fix that could be implemented within a day.
I am pretty sure they lost >100.000.000$ by that. Just shows how shitty the management over there is that they never tried fixing that problem.
My exact same experience. As to what it cost, in the old world paradigm I'd say "nothing, we already bought the game", but I guess these days there's probably a decent argument for a cohort of people who are too impatient to wait to get into the game and who pay extravagant sums of real-world money to get in-game items immediately.
There's other costs: how many times per day to R* devs & testers have to go through the same loading? How many man-hours wasted? How many compute resources wasted? How many upgrades were justified by "I need better hardware to get my job done in a timely manner"?
Same. Asking around why players tolerate it, the anecdotal conclusion I’ve got is that what makes these frequent and extremely slow loads tolerable is playing with friends on audio chat.
You're probably wrong. I remember e-commerces trying to quantify how many millions of dollars they would lose when clicking would take too much time on their websites.
> Amazon’s calculated that a page load slowdown of just one second could cost it $1.6 billion in sales each year. Google has calculated that by slowing its search results by just four tenths of a second they could lose 8 million searches per day
I cannot fathom how dysfunctional Rockstar must be that this slipped through the cracks for so long given the fact that GTA Online is Rockstar’s cash cow.
6 minutes is insanity and must have turned off so many people from playing.. this person deserved far more than $10k
You are probably being sarcastic, or cynical about current corporate business world where nobody cares.
But user experience is important. With faster lpad timea they probably could get more players. More players probably means more money, since there are more chances to convert someone from a player to a paying customer.
Perhaps if someone can endure a 6 minute load time they are more likely to buy something in the game (a fan of the series?), but we could say that those who like fast load times also use their wallets fast. But it is pure speculation - we dont have the data. Also obviously Rockstar doesnt have this data - they couldnt A/B test if load time converts to more sales - their game only had bad load times.
Certainly there has to be some portion of the potential audience who would have become microtransaction purchasers who didn't because they were turned off by the long load times.
The problem wasn't strlen(), that just showed up at the bottom of the call stack when profiling.
IIRC the actual problem was an exceptionally dumb JSON parser combined with parsing a pretty big JSON file (which probably also grew much bigger than the original developer working on that feature anticipated).
Edit: the title has been edited in the meantime, originally it was something about strlen() being responsible for the slow loading time.
The JSON parser was not "exceptionally dumb": it just used sscanf(s, "%d", ...) for parsing numbers. But most implementations of C's standard library (yes, including BSD and Linux's) implement sscanf() by converting the source string into a dummy FILE* object (which involves calling strlen() to set the stream's size) and calling fscanf() on it. Apparently most people who use sscanf() are unaware of this "lovely" implementation quirk and I can't really blame them.
strlen() was a major component of the slowdown. Yes, the JSON file was large and strlen() was indirectly caused by sscanf() being used. But it's not accurate to say it's a clickbait title when a bulk of the slowdown was caused by constantly recalculating the length of the C string. The article shows in detail how he created a monkey-patched strlen() that cached the length to avoid recalculating it over and over which was attributed to the speedup.
The speedup gains were not caused by reducing the size of the JSON file or doing any exceptionally better JSON parsing. The speedups were a direct result of simply not calling strlen() over and over in an O(N^2) fashion.
All true, but replacing the strlen() with a caching version is just fixing the symptoms (of calling strlen too often... but the only realistic solution without the ability to recompile the game - so kudos to the author to come up with this hack).
I bet the "official" patch by Rockstar simply dropped in a less embarassing JSON parser library :)
It's funny though. Several years ago a coworker did some profiling on one of our server software components and found that it was spending up to 10% of the CPU time in strlen()... for very similar reasons.
Had something similar happen with low-level components in a PHP application I inherited repeatedly calling trim() and/or str_pad() on values that had already been processed. Refactoring those out led to a 10-15% performance boost in our test suites.
I've seen PHP applications use 30% of their CPU in the “realpath cache”, which at the time was a super-slow (O(n²) in this case) home-grown hash table for caching the result of realpath(), which was called all the time because someone decided include_once() needed to run its paths through realpath() in case someone included a file multiple time through symlinks. And I guess they couldn't trust inode numbers or Windows didn't have them or something…
I guess it's down to priorities and passing the buck.
Our priorities were always on profiling the running game to make sure we got every ounce of juice out of the hardware and the game ran smoothly on the lowest systems. I can't think of one time we would have had the spare time to look at loading times. In those days we were being pushed into 80+ hour weeks just to make the game work at all, never mind finding time for "luxuries" such as this.
And of course, on a big team, a problem like this would always be someone else's problem.
You are saying a game had a 6 minute load screen and everybody would throw their hands up and say “but look at the dynamic lighting once it loads… it’ll all be worthwhile!”
If that’s not group think gone bad, I don’t know what is.
Not to mention, unless they have a really clever system to do hot code reloading, EVERY SINGLE DEVELOPER AND TESTER would have been plagued by these load times.
I wonder if the watercooler rooms at Rockstar are exceptionally furnished as a result.
Readit News
Every company talks about wanting to have driven curious employees to solve complex issues or challenges but you just get stomped with a boot to your face if you do anything outside your lane or simply ask why.
It’s also worse because I’m starting to realize how lucky I was to be raised as a curious individual from my parents who never went to college but at least were curious themselves about different things (pottery, soldering, gardening).
The quote “Curiosity is insubordination in its purest form” suddenly has a different context when working for a company.
Lots of incredibly talented people worked on GTA Online, it's worth trying to think through why a team of smart developers might not have prioritized this, rather than assuming something negative about them.
I guess it's possible that the engineers who were talented enough to create this extremely impressive game in the first place somehow threw up their hands and said "guess it can never take less than 7 minutes to load, best we can do". But it seems much more likely that they were well aware, but never allowed by poor management to go back and fix it. The game was released and printing money, why bother. They're on to the next title.
People might start off curious but eventually they keep running into political roadblocks and other nonsense, which slowly beats the interest out of them.
If those are the kinds of people that continue surviving then those are the kinds of people which your environment cultivates.
In short, every job needs to be able to have autonomy to enact on the ground solutions for what their immediate team needs. You shouldn't be penalized for going off the reservation and coming back with a creative solution. The problem is corporate structure for the past few decades has been about rooting out this autonomy and anything opaque about job responsibilities. There is a lot that's been done that would need undoing to change those environments, and imo you are better off finding another workplace than bringing about the monumental change required. A mature corporation is like a boulder tumbling down a mountainside; you can hardly change its course much less stop it short of where its heading.
In my experience it's the opposite. Most people are born innately curious. It takes intense systemic conditioning to suppress their natural curiosity. For many of us that conditioning begins in the traditional education system and is completed in the workplace.
The project managers could put load times as a priority but they genuinely dont see it as a problem or dont care at all. It happens all the time. There is a lot of unoptimized software.
Perhaps the issue comes from the top: the project managers are supposed to only do things that increase revenue and all other things are not rewarded at all? Perhaps executives only care if they hit quartely results? (Short term gain traded for long term loss - the game is made for whales who are so invested that they will play even when it is technically bad - so milk them now with new content for whales and ignore everththing else, who cares that new players will be discouraged by bad load times).
In my opinion if nobody plays own game / eats own dogfood, they dont even see its problems. Blizzard is at this stage now - some of their games barely work. You buy Starcraft 1 through the launcher... and it doesnt work(I think it works now, but it took them 1 month to notice it and another to fix it). Heroes of the storm - downloaded a 137mb patch every run - it took them half a year to notice it and do something with it. It is like nobody cares about the brand at all. Game company where your games dont work.
IMHO a problem with people - that comes from the top. They want only story points for things that bring money. The project managers either dont care, or cannot deal with other problems at all. I bet they dont care. This happens in many companies - there were few people who tried, they got burned for that: terminated or quit -> so now nobody cares. "The game is good enough to still sell some skins to the whales, so why bother".
Look at the results: they got an external comment what to fix. Did they launch their own internal project to optimize the game? Did they check for other things that make it load slow? It looka that someone else did a partial fix that is "good enough" so everything is back to usual and nobody cares about load times.
Designers' unnecessary desire to delight users staves off the more Kafkaesque user experiences, and we all live in the gap of cops' unnecessary (occasional) sympathy for (some) nominal outlaws (we're all nominal outlaws).[0]
When you mentioned curiosity I thought of a firm where colleagues scoffed at me as I rummaged through conference room drawers, saying, "you're not going to find anything interesting in there." I'm curious about cupboards as well as log traces -- an almost ubiquitously "eyes-glaze-over" boring artifact of modernity that, alongside popularly mind-numbing "code", I am equipped to find interesting.
Is it any wonder, then, in this workplace where such innocent, cheap, curiosity were scoffed at, I was not too long after threatened with the extraordinarily archaic (in our at-will state, at least) legal notion of "insubordination"?
It's a shame, because insofar as "all ambiguity is resolved by actions of practitioners at the sharp end of the system,"[1] the liberated curiosity of individual contributors is often the only "crack in everything" that's "how the light gets in."[2]
0. "With two lines of a man's handwriting, an accusation could be made against the most innocent." (Françoise Bertaut de Motteville)
1. https://how.complexsystems.fail/
2. "Anthem" - Leonard Cohen
That's because "curiosity" translates to "more work" at #BigCo. And people work at #BigCo so that they can comfortably do as little work as necessary to maintain their lifestyles. This is a hard rule for any organization. So if you want what you're describing, it will only be found at small startups where everyone has skin in the game.
Deleted Comment
I ALWAYS ask questions like "How did this come to be, How much did this cost, who designed this and why" etc.
My brother gets annoyed that A) I ask and question everything, and B) Laments he doesnt know as much as me.
However, this is spurred by my ADHD... and I at times wish to be able to have a more precise control of my focus.
Perhaps this might not be appropriate in the gaming industry... and who knows if Rockstar don't already have this.
However, the GTA loading time is horrendous (and I had a spectrum sinclair when I was a kid) - surely there must have been a work item, ticket, bug reports, or whatever, for this.
For 19 year we just looked at it in defeat. For the last year, bit on and off. Starting last monday it crashed every morning for a user sitting next to me, a big nasty german boss (Im French).
I sat down, read his logs, saw it failed on ImageList creation again for some fucking reason, looked how many of these things we used and what they did. 2 hours later this 20 yo mystery bug dozens of devs had noped out of was fixed. We fully get it now. Wasnt even a matter of priority, it took 2 hours of looking at it. A lunch break basically.
I dont have a sort of morale of the story or whatnot, but well, Rockstar would have figured it out eventually. Just need a nasty fucker next to you whining abt it everyday.
All of the Source games (HL2, Portal, Alyx, etc...) have arguably bad load times. They all sold great! GTA sold great even with it's previous load times.
It's just not a priority for most teams. In Rockstar's case it happened to be easy to fix but in most cases it's not so easy because the engineering team didn't want to deal with it.
AFAIK, Source, and maybe Unreal/Unity are built in such a way that by default, if you need to start the level over you have to re-load it entirely. They have mutable state spread all over the place and have no way to mark what's important (needs to be saved, what's not (can be zeroed), and/or they haven't bother to put all mutable state in a single places so either it can be restored from memory or loaded by just loading a tiny bit.
There's zero reason HL:Alyx should take 15-30 seconds to restart a level you're already in when clearly less than 1k of state has changed since you started the level. But, fixing that would require a major re-design of the Source engine. Slow load times don't affect sales. So almost no one does it. Easy just to let each gameplay programmer do whatever they want and to restart just reload the level.
I put up with it for some games (the ones mentioned above for example). But I've also quit a few recently. Games where you die a lot and to add to the frustration it takes 1-2 minutes to load again only to get killed in 5-20 seconds. That frustration means trial and erroring to figure out what to do gets a punch in the face and so I stop playing what might have been a good game otherwise.
All I can do is prioritize and hope nobody gets too unhappy that their pet features is never going to happen.
1) It has to be alright to make miscellaneous changes like this, and
2) It has to contribute to getting promoted.
The second point is IMO why random QoL changes like this are rare in large corporations.
It took someone who didn't know what that code looked like being curious about what was actually happening.
1: https://news.ycombinator.com/item?id=26296339
I imagine that even if you were on the GTA team, you'd most likely think the long loading time was unavoidable due to some complexity you were not aware of.
Great work to solve it from outside of the company.
Maybe they saw sscanf and assumed it is normal that it takes time, but more likely there were so much stuff to do and somehow no one checked
Deleted Comment
Dev 1: Hey, I think we can improve the load time by fixing X, Y, and Z.
Dev 2: that's neat, create a ticket so we can keep track of that.
Product 1: hey, what's this ticket about?
Dev 1: oh, that's to speed up the load time. It's a quick fix.
Product 1: was that in the requirements?
Dev 1: No, but it will be a huge improvement.
Product 1: ok, I'll mark it as Tech debt, and put it in the back log.
~~ Many Month Later ~~
Dev 3: hey, did you know we can improve load time by updating X, Y, and Z?
Dev 2: Oh yeah, we even have a ticket for that. Let's let Product know it's still an issue and maybe they'll prioritize it.
Product 2: Was this part of the requirement?
~~ years later ~~
Product 3: hey, there's this old ticket here. Do you think it's still relevant?
Dev 35: I'm not familiar with that, but it's too old, most likely not relevant any more. We can delete it.
....
Dev N: Hey, I think we can improve the load time by fixing X, Y, and Z....
Product people are great for selecting which features to build out and which bugs to address, but are poorly equipped to decide how to prioritize things like: performance optimizations, bugfixes that haven't (yet) been reported by the end users, code refactoring, paying down tech debt, modernizing tooling, misc quality of life improvements for the engineering team, etc.
If the company pays me to close tickets and Jira stories, why should I care about improving its products?
If the company pays me to actually improve its products, then sure, I would do that.
I can't believe I'm alone, I wonder how much this cost rockstar
https://www.folklore.org/StoryView.py?project=Macintosh&stor...
It can be simultaneously true that they made a lot off the game && with lower loading times they could have made even more. In-game microtransaction ARPU scales with playtime. >4 min per session spent waiting to load instead of playing == lost rev.
I am pretty sure they lost >100.000.000$ by that. Just shows how shitty the management over there is that they never tried fixing that problem.
Deleted Comment
which is kinda hard if the game takes 30 minutes to load
EDIT: things like https://www.fastcompany.com/1825005/how-one-second-could-cos...
> Amazon’s calculated that a page load slowdown of just one second could cost it $1.6 billion in sales each year. Google has calculated that by slowing its search results by just four tenths of a second they could lose 8 million searches per day
6 minutes is insanity and must have turned off so many people from playing.. this person deserved far more than $10k
Enough still play it and buy a few micro tractions per month regardless of the load times Dev time is better spent reskinning a hat or something.
But user experience is important. With faster lpad timea they probably could get more players. More players probably means more money, since there are more chances to convert someone from a player to a paying customer.
Perhaps if someone can endure a 6 minute load time they are more likely to buy something in the game (a fan of the series?), but we could say that those who like fast load times also use their wallets fast. But it is pure speculation - we dont have the data. Also obviously Rockstar doesnt have this data - they couldnt A/B test if load time converts to more sales - their game only had bad load times.
The problem wasn't strlen(), that just showed up at the bottom of the call stack when profiling.
IIRC the actual problem was an exceptionally dumb JSON parser combined with parsing a pretty big JSON file (which probably also grew much bigger than the original developer working on that feature anticipated).
Edit: the title has been edited in the meantime, originally it was something about strlen() being responsible for the slow loading time.
The speedup gains were not caused by reducing the size of the JSON file or doing any exceptionally better JSON parsing. The speedups were a direct result of simply not calling strlen() over and over in an O(N^2) fashion.
I bet the "official" patch by Rockstar simply dropped in a less embarassing JSON parser library :)
I hope it's gone. I really do hope it's gone :-)
Two submissions an hour, 24 hours a day...
There's clearly a bot here but ddtaylor has commented on this thread, so there's also a person in there somewhere.
Should this be allowed? It's not breaking any "rules" but then 10k people could setup a bot to do the same thing and repost stuff twice an hour..
I guess it's down to priorities and passing the buck.
Our priorities were always on profiling the running game to make sure we got every ounce of juice out of the hardware and the game ran smoothly on the lowest systems. I can't think of one time we would have had the spare time to look at loading times. In those days we were being pushed into 80+ hour weeks just to make the game work at all, never mind finding time for "luxuries" such as this.
And of course, on a big team, a problem like this would always be someone else's problem.
If that’s not group think gone bad, I don’t know what is.
I wonder if the watercooler rooms at Rockstar are exceptionally furnished as a result.