I am 100% sold on developer experience being key for fast iteration and high quality software operations, but the actual research cited here seems fairly flimsy. It was based on a web survey by DX (love them by the way), that had 219 completed responses and asked likert scale (1-5) questions about how they felt.
Good developer experience increases self-reported productivity. I do suspect that it correlates with actual productivity and business outcomes, especially around reducing the productivity loss that comes from error remediation.
I'm one of the co-authors of the study. Your critique is valid though by research standards, for this type of study, our sample is sufficient. We are planning to replicate this study on a larger scale in the future, though!
Are there any plans to figure out objective ways to measure productivity and what distinguishes “good devex” from “bad devex”?
I’ve worked at a lot of big tech companies that do surveys about internal tooling and every year it’s rated as a weak spot, across years and companies this seemed like a consistent trend.
And yet everyone had teams dedicated to improving various aspects of devex so it’s unclear if these teams are just improving the wrong things or if productivity really is improving and it’s something else (eg the amount of code debt grows faster than devex improvements or people are asked to go faster than the devex improvements can keep up or the devex is being improved but the size of the survey means not enough people feel it because you optimize smaller subsets of engineering orgs).
That’s another thing to be mindful about large scale and small scale surveys - the latter might be sampling specific teams adopting the tool whereas the former might find there’s no way to make everyone happy and it all turns into a wash.
Having done internal developer & analyst tooling work (and used DX), this type of survey is great for internal prioritization when you have dedicated capacity for improvements.
I'd be curious to see more about organizational outcomes, as this is piece of DevOps/DevEx data that I feel is weakest and requires the most faith. DORA did some research here, but it's still not always enough to convince leadership.
For the uninitiated among us - can you share more context on the research standards and the reasoning behind it? I'm interested and would like this to influence some decisions I have but would like to understand the confidence here :).
I agree. I think this is just one of those things that we know is true but can't easily be proven via the scientific method.
I expect some HN readers will balk at the idea that we can know things but can't prove them scientifically but only because they haven't really thought about it. E.g. does giving variables sensible names reduce bug counts? Obviously. But it's very difficult to prove even something as obvious as that.
It's peculiar that "developer productivity" is even a term. It doesn't have a generally-agreed-upon definition. I don't think a definition is even possible.
So, we keep trying to come up with objective measures of a term that is impossible to objectively measure. Isn't that weird?
Why not "developer happiness", or, if that's too hippy-dippy for the suits, "developer satisfaction"? Surely we can reason that, all other things being equal, happier developers are more productive.
Then, surveys like this one make perfect sense. "Hey, how satisfied are you with the tools / processes / whathaveyou at your employer?" That's a valid question, and collecting a pile of answers does start to yield some useful insights into what works and what doesn't.
But no, we're trying to use "developer productivity" instead, as though we're all simple machines that provide value by converting coffee into code, and the winning business is the one that is able to extract the most SLOC out of the cheapest and most miniscule quantity of coffee.
> Good developer experience increases self-reported productivity.
So do ineffective DX, most times. There's "proper" way to assess productivity gains that don't rely on subjective self-reporting (eg the DORA metrics). The "improving feedback loops" item on this report makes sense on that light - the other two are questionable as a direct proxy
I generally agree with your point, with the caveat that metrics like DORA are the first to be gamed when applied to an organization.
It's the usual cycle of applying the metrics first in an informal way, then using it for team/process evaluations as it seems to be somewhat reliable, then teams start to reajust how they make the tickets and the deploy cycle in ways that dramatically increase the metrics while reality hasn't changed much.
Analyzing productivity with completely objective metrics will I think always bear these flaws (I'd assume even measuring things at the highest level as "wages vs revenue" would be gamed on how wages and revenue are calculated ?)
My role is DevOps but I find myself gravitating towards improving the developer experience, which I think is a worthy subcategory in the DevOps practice.
The team I'm on previously didn't have any automated way to set up developer workstations, now we've got an Ansible script that does most of everything they need in less than an hour. (Our VPN is slow as cold molasses) Previously, setup was ad-hoc, leading to troubleshooting questions like "did you follow instructions in this forwarded email, this one section from a wiki page, and a copy/paste comment in chat?". Now it's "did you run the script?".
It's kind of amazing to me that there are whole teams out there that struggle with awful environments that actively put up barriers to efficient coding.
I suppose I could be called DevOps, and I've always seen that as part of the job - making the infrastructure as easy for the devs to work with as possible. They've got their own specialities to worry about; infrastructure shouldn't be one of them. I've heard the role called somewhat derogatorily "helpdesk for devs", but that is part of the job, and I embrace it. I also embrace "yaml wrangler" - it's not my favorite thing to do, but it does sometimes need wrangling. I suppose I've always seen the role as a generalist one who has some understanding of the overall system, and how to glue (or duct tape) it all together.
I've been called a Systems Wrangler before. It was pointed out to me that I love to wrestle with a new unfamiliar system, "taming" it to work with our stuff.
I think I prefer that to some other choice descriptors that have been applied to me! ;) "Git Whisperer" is another one that is amusing to me, considering the difference in meaning between "Wrangler" and "Whisperer". Does that mean Git requires a subtle approach?
devops and dx share the same spirit of reducing friction
at my job i can't stop leaning toward them because I can count in blood the amount of energy, time and money wasted on the thousands papercuts stacked over time
one needs to find a balance of fluidifying the daily logistics involved, it amps up joy and reduce stress.. leaves more time for problem solving and creativity.
sadly most devs / managers couldn't care less about that
I've a similar background, and have worked recently w/ a handful of teams with atrocious, self-imposed developer experience. I think that applying concepts like SLOs and error budgets to relevant developer-centric activities can really help focus the team and other stakeholders well before the experience passes some threshold of terrible.
If you tolerate painful problems you are capable of fixing, you're a masochist. If you expect others to tolerate them, you're a sadist.
I often work with people who are more than happy to have someone fix problems. But I also work with people who don't just denigrate such work, but are obstructionist. You're expending energy to stop other people from making the system better. There's something deeply wrong with such people.
I completely agree. In my experience what, what works is:
1) Development containers. They're not for every situation, but if they suit your situation (prebuilt base images, private repo, long build times that can be split out into common layers, probably remote dev with a good IDE that natively supports them, i.e. a Jetbrains IDE), they're amazing. They completely remove the need for "works on my machine" as you deploy a multi stage built app from the same base container image.
2) Single "5 step" setup page. The setup guide should be: Install git/p4, install toolchain with versions, log in, clone, `make build`/`cmake --build` / `npm run dev`. Depending on what you work on, "run dependencies in a container" (database, grafana/prometheus, log agent) might be there too. Everyone who is onboarded follows these steps, and their first two tasks, in your teams issue tracker are "fix anything that didn't work in setup guide", and "make a PR", both of which should happen on day 1/2.
> developers who carve out significant time for deep work enjoy a 50% productivity boost.
Makes sense from personal experience. At my company we have 1 no-meeting day a week (Thursday). Especially since I've gotten into management I've found Thursday to be my favorite day and I probably get ~70% of the actual code work I do in a given week done on that day. If you don't have something like this, I really recommend it!
I often think the once-a-week-no-meeting-days should be reversed to once-a-week-meeting-days. Put all status, retrospective, planning and other recurring meetings into this one day so people can actually work the rest of the week.
During Covid my company had a 32 hour week for a year and we lost zero productivity because the first thing that was eliminated were the meetings that just fill time but achieve nothing.
Regular meetings and calls can't go over twice a week period. For me, it's one of the biggest reasons for unproductivity. I've worked on a lot of projects and the amount of meetings was directly correlated to how much everything was delayed.
In all cases, the people who pushed the most for meetings were the ones who contributed the least, and in fact had a negative impact. Nowadays it's a nonnegotiable rule from the start in our agency when taking on new clients.
Unfortunately, some companies approach culture and DevEx as an additive system where they keep piling more meetings, more engagement, more activities, and more process on top of everyone’s responsibilities.
When a company approaches DevEx and culture by subtracting things that aren’t working it’s like a breath of fresh air.
One of the things to be aware of is that two meeting free days won't double that output. You still do a lot of valuable "stuff" in that meeting time. But, a reliable period of uninterrupted focus time is rocket fuel for an IC. I get mornings for free (I live in the UK and work mostly with the US), and it's insane how efficient I am for it.
> We finally have data to back up the benefits of developer experience
Any manager who was waiting on the "data" (read: sponsored study) that you need to invest in proper tooling for your devs isn't going to do anything significantly different because of it.
You need to think like a desperate manager. Yes, I know, I know.
The job of a manager is to make good choices and decisions. In the absence of an obvious best choice, they have work backward from both success and failure scenarios, including the political and social impacts.
In the absence of external studies and data, if they make an investment in this area and it turns out not to help, they will be left telling their boss things like "everyone says this is right, not sure why my project didn't work."
But a flimsy fig leaf that comes from outside and smells objective is a perfect CYA tool that allows the manager to take a risk while anticipating the need to manage and contain failure.
Does that describe a healthy dynamic and a smart organization? No, but it's reality at many places.
Sometimes the manager knows it, but the manager have managers who need to have some external justification to cover their assess. It is basically the same principle that drives hiring consultants to say exactly the same thing the team have been saying for years.
If you read the introduction of the paper, you'll see that the aim of this paper is to give managers and developers concrete data to use to help get buy-in on investing in developer experience from business leaders.
Having worked as a software developer in several different areas of enterprise IT, the vast majority of the work is about "Improving {process} to increase {team}'s productivity." Businesses already know, or at least hope, that "Good {worker}Ex increases productivity" and push endlessly towards that objective. It's quite possibly the only reason non-software companies hire software developers.
It's just that IT departments routinely fail to be included in that. So the IT workers get burnt out working on projects they don't understand to make someone else's job easier, while their job gets continually worse because everything is an IT problem now. (And in many cases, nothing actually improves for the business either, because the project only considered the "happy path" so they are still spending most their time dealing with the special cases, but now have to go through IT who are too busy doing the same thing for everyone else.)
'Member when DevOps was about improving DX? I member.
Since every organization lost sight of the fact that DevOps should be serving the Developers as a customer - DevEx was a low priority and DevOps became about infrastructure purism.
GitHub Actions is predicated on push n' pray SDLC. Actions is also showing some of the warts that exist from being forked from the old Azure DevOps TFS pipelines.
I haven't used GitLab CI runner too often, but the carrot of being able to use a runner on my local machine is extremely appealing.
It's really amazing. Github Actions has been an absolute curse on my team, it's bundled with other Github and Azure charges so finance convinced us to abandon CircleCI and I have nothing but regrets.
Now imagine having to work with AWS CodePipeline+CodeBuild (because it's always one more service with AWS). The worst CI tool I've ever worked with, but it goes into the AWS bill and no one ever questions that so that's what we're using.
They are not 1:1 parity with the hosted version, they are hard to run, and they are slow. When I tried GA I often had to just test on the hosted service, and debugging on the hosted service is so slow and painful. I honestly am very surprised that GA is so popular, it's probably the worst CI I've used and I thought I was missing something.
The Source Code Control System, the first UNIX revision control system, written by Marc J. Rochkind
The remote job entry batch-submission system
The PWB shell, written by John R. Mashey, which preceded Steve Bourne's Bourne shell
The restricted shell (rsh), an option of the PWB shell, used to create widely-available logins for status-checking, trouble-reporting, but made safe by restricting commands
The troff -mm (memorandum) macro package, written by John R. Mashey and Dale W. Smith
Utilities like find, cpio, expr, all three written by Dick Haight, xargs, egrep and fgrep
yacc and lex, which, though not written specifically for PWB, were available outside of Bell Labs for the first time in the PWB distribution
Readit News
Good developer experience increases self-reported productivity. I do suspect that it correlates with actual productivity and business outcomes, especially around reducing the productivity loss that comes from error remediation.
I’ve worked at a lot of big tech companies that do surveys about internal tooling and every year it’s rated as a weak spot, across years and companies this seemed like a consistent trend.
And yet everyone had teams dedicated to improving various aspects of devex so it’s unclear if these teams are just improving the wrong things or if productivity really is improving and it’s something else (eg the amount of code debt grows faster than devex improvements or people are asked to go faster than the devex improvements can keep up or the devex is being improved but the size of the survey means not enough people feel it because you optimize smaller subsets of engineering orgs).
That’s another thing to be mindful about large scale and small scale surveys - the latter might be sampling specific teams adopting the tool whereas the former might find there’s no way to make everyone happy and it all turns into a wash.
Having done internal developer & analyst tooling work (and used DX), this type of survey is great for internal prioritization when you have dedicated capacity for improvements.
I'd be curious to see more about organizational outcomes, as this is piece of DevOps/DevEx data that I feel is weakest and requires the most faith. DORA did some research here, but it's still not always enough to convince leadership.
I expect some HN readers will balk at the idea that we can know things but can't prove them scientifically but only because they haven't really thought about it. E.g. does giving variables sensible names reduce bug counts? Obviously. But it's very difficult to prove even something as obvious as that.
So, we keep trying to come up with objective measures of a term that is impossible to objectively measure. Isn't that weird?
Why not "developer happiness", or, if that's too hippy-dippy for the suits, "developer satisfaction"? Surely we can reason that, all other things being equal, happier developers are more productive.
Then, surveys like this one make perfect sense. "Hey, how satisfied are you with the tools / processes / whathaveyou at your employer?" That's a valid question, and collecting a pile of answers does start to yield some useful insights into what works and what doesn't.
But no, we're trying to use "developer productivity" instead, as though we're all simple machines that provide value by converting coffee into code, and the winning business is the one that is able to extract the most SLOC out of the cheapest and most miniscule quantity of coffee.
So do ineffective DX, most times. There's "proper" way to assess productivity gains that don't rely on subjective self-reporting (eg the DORA metrics). The "improving feedback loops" item on this report makes sense on that light - the other two are questionable as a direct proxy
I generally agree with your point, with the caveat that metrics like DORA are the first to be gamed when applied to an organization.
It's the usual cycle of applying the metrics first in an informal way, then using it for team/process evaluations as it seems to be somewhat reliable, then teams start to reajust how they make the tickets and the deploy cycle in ways that dramatically increase the metrics while reality hasn't changed much.
Analyzing productivity with completely objective metrics will I think always bear these flaws (I'd assume even measuring things at the highest level as "wages vs revenue" would be gamed on how wages and revenue are calculated ?)
I don't find this hard to believe, but do you know of any research showing this?
The team I'm on previously didn't have any automated way to set up developer workstations, now we've got an Ansible script that does most of everything they need in less than an hour. (Our VPN is slow as cold molasses) Previously, setup was ad-hoc, leading to troubleshooting questions like "did you follow instructions in this forwarded email, this one section from a wiki page, and a copy/paste comment in chat?". Now it's "did you run the script?".
Similarly, the concept of the "Internal Developer Platform" might be interesting to some: https://internaldeveloperplatform.org/core-components/
It's kind of amazing to me that there are whole teams out there that struggle with awful environments that actively put up barriers to efficient coding.
I think I prefer that to some other choice descriptors that have been applied to me! ;) "Git Whisperer" is another one that is amusing to me, considering the difference in meaning between "Wrangler" and "Whisperer". Does that mean Git requires a subtle approach?
at my job i can't stop leaning toward them because I can count in blood the amount of energy, time and money wasted on the thousands papercuts stacked over time
one needs to find a balance of fluidifying the daily logistics involved, it amps up joy and reduce stress.. leaves more time for problem solving and creativity.
sadly most devs / managers couldn't care less about that
If you tolerate painful problems you are capable of fixing, you're a masochist. If you expect others to tolerate them, you're a sadist.
I often work with people who are more than happy to have someone fix problems. But I also work with people who don't just denigrate such work, but are obstructionist. You're expending energy to stop other people from making the system better. There's something deeply wrong with such people.
1) Development containers. They're not for every situation, but if they suit your situation (prebuilt base images, private repo, long build times that can be split out into common layers, probably remote dev with a good IDE that natively supports them, i.e. a Jetbrains IDE), they're amazing. They completely remove the need for "works on my machine" as you deploy a multi stage built app from the same base container image.
2) Single "5 step" setup page. The setup guide should be: Install git/p4, install toolchain with versions, log in, clone, `make build`/`cmake --build` / `npm run dev`. Depending on what you work on, "run dependencies in a container" (database, grafana/prometheus, log agent) might be there too. Everyone who is onboarded follows these steps, and their first two tasks, in your teams issue tracker are "fix anything that didn't work in setup guide", and "make a PR", both of which should happen on day 1/2.
Makes sense from personal experience. At my company we have 1 no-meeting day a week (Thursday). Especially since I've gotten into management I've found Thursday to be my favorite day and I probably get ~70% of the actual code work I do in a given week done on that day. If you don't have something like this, I really recommend it!
During Covid my company had a 32 hour week for a year and we lost zero productivity because the first thing that was eliminated were the meetings that just fill time but achieve nothing.
In all cases, the people who pushed the most for meetings were the ones who contributed the least, and in fact had a negative impact. Nowadays it's a nonnegotiable rule from the start in our agency when taking on new clients.
Unfortunately, some companies approach culture and DevEx as an additive system where they keep piling more meetings, more engagement, more activities, and more process on top of everyone’s responsibilities.
When a company approaches DevEx and culture by subtracting things that aren’t working it’s like a breath of fresh air.
Any manager who was waiting on the "data" (read: sponsored study) that you need to invest in proper tooling for your devs isn't going to do anything significantly different because of it.
The job of a manager is to make good choices and decisions. In the absence of an obvious best choice, they have work backward from both success and failure scenarios, including the political and social impacts.
In the absence of external studies and data, if they make an investment in this area and it turns out not to help, they will be left telling their boss things like "everyone says this is right, not sure why my project didn't work."
But a flimsy fig leaf that comes from outside and smells objective is a perfect CYA tool that allows the manager to take a risk while anticipating the need to manage and contain failure.
Does that describe a healthy dynamic and a smart organization? No, but it's reality at many places.
For example, what if an engineering manager needs to convince finance for headcount to hire a devex employee?
If you read the introduction of the paper, you'll see that the aim of this paper is to give managers and developers concrete data to use to help get buy-in on investing in developer experience from business leaders.
https://queue.acm.org/detail.cfm?id=3639443
It's just that IT departments routinely fail to be included in that. So the IT workers get burnt out working on projects they don't understand to make someone else's job easier, while their job gets continually worse because everything is an IT problem now. (And in many cases, nothing actually improves for the business either, because the project only considered the "happy path" so they are still spending most their time dealing with the special cases, but now have to go through IT who are too busy doing the same thing for everyone else.)
Since every organization lost sight of the fact that DevOps should be serving the Developers as a customer - DevEx was a low priority and DevOps became about infrastructure purism.
"One throat to choke." If there is a problem, you have operations, and development mixed together to produce one nexus of responsibility.
The moment someone says "I can't.... because I'm not a..." in those situation you've lost the point.
The whole point is... It IS your problem as a team. FIX IT. You have everything. Make it happen.
Now a days... the thread has been totally lost. I've gone back to development. I prefer tossing the dead cats over the fence over receiving them.
Either way... the poor cat is dead... we didn't work together to save it :(
I haven't used GitLab CI runner too often, but the carrot of being able to use a runner on my local machine is extremely appealing.
Lots of room for innovation for sure. But if you think actions is bad try iterating on Windows slip streaming.
I haven't used personally but curious if it has some shortcomings they don't mention.
Then to test locally you just do "make check".
The problem comes if you start using pre-made actions instead of simple commands.
Notable firsts in PWB include:
The Source Code Control System, the first UNIX revision control system, written by Marc J. Rochkind
The remote job entry batch-submission system
The PWB shell, written by John R. Mashey, which preceded Steve Bourne's Bourne shell
The restricted shell (rsh), an option of the PWB shell, used to create widely-available logins for status-checking, trouble-reporting, but made safe by restricting commands
The troff -mm (memorandum) macro package, written by John R. Mashey and Dale W. Smith
Utilities like find, cpio, expr, all three written by Dick Haight, xargs, egrep and fgrep
yacc and lex, which, though not written specifically for PWB, were available outside of Bell Labs for the first time in the PWB distribution
Amazing.
- An Introduction to the Programmer's Workbench, T. A. Dolotta and J. R. Mashey