Most engineering projects have standards. Not the cutting edge stuff, but most of it does. Many are from regulations and safety laws setting specifications for this and that, but some are simply there because they won by being the most popular.
Software Development by contrast has no standards. You can hire two different enterprise sized companies with this and that certification in the exact same technologies, who claim to adhere to the exact same principles for best practices and what not. They mean it too, often they’ve even gone through the same sort or internal re-education processes with the same consultant companies at the same time, yet they still manage to build software projects so different that transferring ownership from one to the other, or setting up a multiple supplier open source project, is sooooo much work you wouldn’t believe it.
If computer science wants to be engineering, I need to be capable of hiring an architect who makes plans that can be implemented by any supplier, and I need to be able to switch supplier/contractor 2-3 times during the process without issues because the plans can only be implemented one way. We’re probably a lot further away from that world today than we were 30 years ago. The only company that seems to have smelled the music on this stuff is Microsoft and their cloud.
I would say that plans sufficiently specific to be done by any supplier and ability to switch supplier would be equivalent to final code and the final step should be done by using appropriate frameworks instead of human contractors. Any repetetive software process with precise specification should be automated with better tools or other higher level abstractions. One reason it's not done more for physical engineering is that the robots or machines would be impractically complicated or expensive or other physical limitations. But even then manaual machining is increasingly getting replaced by CNCs and houses built from bigger prefabricated components. You might choose not having a house made from prefabricated blocks due to aesthetic reasons but in software hardly anyone except small fraction of programmers will complain about electron or too many levels of abstraction under the hood.
You’ve identified an aspect of other fields of engineering and are saying since you feel this aspect may be missing from software engineering that it therefore isn’t engineering. But if you look up what engineering means broadly this isn’t part of the definition at all. Which I guess goes on to say it depends on what you mean when you say “engineering”; a specific manifestation of engineering or the general practice.
Standards can only happen when an industry has matured and moves slow enough for these standards not to be obsolete by next year.
We have plenty of laws and standards regarding handling private information, payment systems, independent pen testing and so on. So it's incorrect to say we have "no standards" at all. Where it matters, we do.
But to try and standardize a process that's evolving faster than a standard can spread is pointless.
This doesn't make software engineering any less engineering. Engineering isn't about standards. It's about the design, construction and use of systems.
It happens with slow technologies that ought to have standards as well. I do work with OS2.eu (Danish public sector owned open source software organisation) and we recently brought the first project into multiple supplier territory. It’s a Drupal PHP project, and as I understand that means that everything that gets build should follow Drupal standards that are pretty much set in stone. After 6 months of governance going this and that way, and the countries second largest city (and biggest PHP developer) as a mediator, we are only now moving into the blissful world of compliance where everyone has agreed to develop Drupal modules in the same way.
Compare this to when we build a new city hall. A massive project involving almost 500 smaller local contractors, in a business that actually moves forward pretty fast. Some of the materials they ended up using weren’t even invented when they began. The project was finished a head of schedule and under budget.
I know construction isn’t always like this. We’ve had our share of catastrophic projects, one elder-care center in particular, but compared to buying software it’s almost child’s play, even when things go wrong.
So I think it’s perfectly fair to say that software engineering isn’t engineering. If you can even build a simple webpage, without two suppliers disagreeing so much on the “standards” that it takes 6 months of governance to achieve compliance, then there is just so far for the industry to go before it’s anything that even resembles engineering.
There's also the issue of certain huge companies deciding that they don't care about worldwide accepted standards used by literally everyone else besides them, and would rather just ignore the standards and do things their own way, or undermine/corrupt an existing standard, or create their own "de facto standard".
In medical, automotive, aerospace, military applications, the software that goes into the device is kind of considered part of the object. It needs this validation for obvious reasons, such as the long lifecycle and critical nature of these products.
But in cellphone handsets, PC software and firmware such as in IoT home devices, and even IT infrastructure like SoHo routers especially but also other devices, there is no agreed upon standards that I am aware of. Even in payment applications, there isn't much there. Large institutions and banks get by often by using "legacy" infrastructure like mainframes to keep the COBOL stuff running, as it has been vetted over decades.
Going further, the architecture of infrastructure, firmware and software should be examined more closely. Recently, IT architecture has been exploited over and over (Kaseya, SolarWinds, MTA) having quite asymmetric effects. In the past it was worms and drive-by malware from janky websites, today it is extremely costly disasters that make the news but roll on bye until the next turd is flipped over.
The problem space is very difficult to define, since "software" and "architecture" are such general terms encompassing basically anything. I rate the difficulty in analyzing these problems for sufficient solutions as very high. When I consider neural-net AI applications, it seems to get even more murky.
Is anybody aware of other software and infrastructure standards that exist or are in research which could apply to more general fields, or to corporate/business IT infrastructure, or to home devices, etc?
> If computer science wants to be engineering, I need to be capable of hiring an architect who makes plans that can be implemented by any supplier
You don’t though. This is just an artefact of the structure of your industry, being limited as it is by physical, geographic, and capital constraints. Software has none of those (capital used to be an issue but is no longer - this is perhaps why things are more flexible than 30yrs ago as you note).
You think you need these things because you see your role as an integrator - but software engineers already perform that function at a different level, via libraries and protocols.
So since the late stone age? When you understand "standard" as an agreed upon common practice, that is. Without that we wont have the pyramids (any of those regardless of continent).
I consider software development an engineering discipline.
In my weird, crazy world, engineering is all about discipline (sound of a riding crop slapping a leather-clad thigh), and follow-through. It's a lot less about slide rules and differential equations.
I believe that the Romans that built aqueducts (often using coerced labor -otherwise known as "slaves") were engineers. They had a lousy numbering system, no concept of trig (although a few had heard of Pythagorus), and not even a slide rule; yet they were able to build massive, well-designed structures that still stand, to this day, nearly three millenia later.
I can't use some Web sites that are just over a year old, because their components went belly-up.
But I also consider software development to be a craft. That's really my approach.
When I write software, I make a personal level of investment into the outcome. The project must complete. Even when I'm working with a test harness, on a project that is nothing more than an experimental platform, I treat it as if it were a shipping product, that I plan to display in the store window. In fact, I often revisit truncated experimental efforts, in order to scrounge snippets. Since they are already "ship" quality, those snippets tend to be quite useful.
A basic habit of mine, is that all of my code "ships," even when it doesn't. That's a craftsman attitude. Getting the project to "ship" state is an engineering effort.
I think the issue is misunderstanding what it means for something to be engineered. There is a correct answer. (Especially because you can always optimize for cost or a different quality metric)
You are so wrong. Software is both a craft and engineering. All software fits your definition of cost and quality optimization. It is a craft in that a software engineer isn’t taught how to write the software you want, they’ve built a toolbox from crafting different types of solutions because their existing tools didn’t fit.
The gist of the article is that software projects have more ad-hoc decisions and more flexibility. This is due to the nature of software. To claim this as a justification that software engineering is not engineering seems rather shallow to me.
You have less repetitive process in software because once software is built, copying it and reusing it as-is is free. In construction if you need to build 10 twin buildings, each of them is a project on its own. In software that'd be building one piece of software and deploying it 10 times.
Furthermore I question the notion that software projects can accommodate more "fundamental midcourse design changes". Can you spend 2 years writing a dedicated Windows desktop application and then midway suddenly switch it to being an Android application? That would be fundamental. Reskinning the UI or replacing a dependency with a similar one is not fundamental design change.
Software can't sustain fundamental design changes, unless we misinterpret what is fundamental and what isn't.
Basically we need to be careful with our analogies so we don't cross from trying to extract insight into flexing our useless cynicism about how software sucks because it's not like making a car in a superficial ways.
It is not just standards, you will find many engineering contracts these days refer to RAGAGEP - Recommended and Generally Accepted Good Engineering Practice.
My take on this is to allow for rapidly evolving practices in an industry where key standards may not be updated to over a decade in some cases, or even cease to exist at all due to emerging fields.
What this means is that a responsible engineer has to make a defendable decision, using experience and judgement to justify a particular course of action.
As my engineering manager used to say to me, "Would you feel comfortable saying those words again, just adding on Your Honour'at the end?"
I switched from engineering to programming because of pay, but I can tell you the word Software Engineering is not exactly real unless you are doing some safety critical C or assembly.
Whenever you have abstraction, you have lost what it means to be an engineer. There is really only 1 way to engineer something as you end up optimizing for various quality metrics, most often cost.
There's only 1 correct answer for a bracket. You want it cheap and as strong as it needs to be.
How many correct answers in my code? Do we optimize for clean code? Fast code? Do we change programming languages or go with a familiar highly supported code? There are 2 similar libraries, which do we choose?
Programming is a hybrid of science, art, tradition, and authority. I think it pays better because of this complexity.
That being said, I'd trust an engineer to program something before I'd trust a programmer to build something.
> Whenever you have abstraction, you have lost what it means to be an engineer.
That's an interesting observation. I'd say engineering does have abstractions, but they're much better. For example, treating an RSJ as a simple uniform piece of material is an abstraction. In reality it is a crazily complex mesh of crystals. But the abstraction is simple and well studied, so it can be used reliably.
Programming abstractions are less reliable. Even commonly used ones like "the filesystem" and "the heap" are vastly more complex. It is almost impossible to remember all the possible hazards when you use them.
Well, in Portugal it is illegal to call yourself Software Engineer like on US after 6 month US codecamps.
It is a professional title and while it isn't usually validated, thus only those that sign projects with legal liabilities do make the final exam, every single university in the country is only allowed to have certified engineering degrees.
I am aware of similar practices in other countries as well.
EE here. In electronics there are hardly liabilities but still the design work is way more based on evidence and understanding rather than fads, hype and opinions.
It might also be instructive to compare with medicine or law. These practitioners are also liable for their actions, have to be licensed by a professional organization, and can have their credentials revoked in the event of gross incompetence. Software designers have none of these burdens.
> The Economist magazine (Nov 27, 2004 p. 71) cites the Standish Group's estimates that
"...30% of all software projects are canceled, nearly half come in over budget, 60% are considered failures by the organizations that initiated them, and 9 out of 10 come in late."
In 2021 (as well as in 2004) it is meaningless to talk about "all software projects" without specifying domain, size, scope and project etymology. "Software" is too vast. It could be an app to tell the time with Mickey Mouses hands, or it could be the control software for a joint strike fighter.
Part of the problem is that software project decision makers view software projects as far more mutable than physical engineering projects.
In a billion dollar construction project, everyone understands that if you want to change the design or significantly expand the scope halfway through, you're going to have a heck of a lot of increased costs & time.
In software, it's very common to add scope or major design changes at any stage without understanding that the same cost and time increases will apply.
For example saying 90% through the project:
-Software: "we need a 2-way interface added between the complex software we're building and some other complex software"
-Building: "We need an enclosed concourse built between our new building and the next building over across the street."
It is obvious in the later case that there will be a lot of extra time & cost. While in the former, higher level decision makers on a software project don't understand the comparable difficulties.
If construction companies figured out they could outsource labor to countries with far lower currency values and thereby not-pay for extra labor, they'd do the same.... oh wait...
Not in my experience, which may not generalize. Your experience and my experience together won't really give a complete picture.
What I experience is this: Even on small projects I've had people say, "okay, can you just add this one thing in the next day or two?" And they don't understand until I walk them through in detail how that one thing is nearly as complex as the initial project, so they have to choose how important it is, and if it's important do they want to wait to release the whole thing or do a release in stages.
Of course my experience is not necessarily going to represent the whole.
Also when outside consulting groups are involved, I often see predatory behavior: they often agree to do any change asked for and don't bother to explain the costs and time involved until a deadline is missed and the budget is running out. At which point they say "the changes you requested have pushed us over the deadline and the extra time is going to cost $X".
Readit News
Software Development by contrast has no standards. You can hire two different enterprise sized companies with this and that certification in the exact same technologies, who claim to adhere to the exact same principles for best practices and what not. They mean it too, often they’ve even gone through the same sort or internal re-education processes with the same consultant companies at the same time, yet they still manage to build software projects so different that transferring ownership from one to the other, or setting up a multiple supplier open source project, is sooooo much work you wouldn’t believe it.
If computer science wants to be engineering, I need to be capable of hiring an architect who makes plans that can be implemented by any supplier, and I need to be able to switch supplier/contractor 2-3 times during the process without issues because the plans can only be implemented one way. We’re probably a lot further away from that world today than we were 30 years ago. The only company that seems to have smelled the music on this stuff is Microsoft and their cloud.
We have plenty of laws and standards regarding handling private information, payment systems, independent pen testing and so on. So it's incorrect to say we have "no standards" at all. Where it matters, we do.
But to try and standardize a process that's evolving faster than a standard can spread is pointless.
This doesn't make software engineering any less engineering. Engineering isn't about standards. It's about the design, construction and use of systems.
Compare this to when we build a new city hall. A massive project involving almost 500 smaller local contractors, in a business that actually moves forward pretty fast. Some of the materials they ended up using weren’t even invented when they began. The project was finished a head of schedule and under budget.
I know construction isn’t always like this. We’ve had our share of catastrophic projects, one elder-care center in particular, but compared to buying software it’s almost child’s play, even when things go wrong.
So I think it’s perfectly fair to say that software engineering isn’t engineering. If you can even build a simple webpage, without two suppliers disagreeing so much on the “standards” that it takes 6 months of governance to achieve compliance, then there is just so far for the industry to go before it’s anything that even resembles engineering.
That's not true in the context of medical and car software. See
https://en.wikipedia.org/wiki/IEC_62304
https://en.wikipedia.org/wiki/ISO_26262
In medical, automotive, aerospace, military applications, the software that goes into the device is kind of considered part of the object. It needs this validation for obvious reasons, such as the long lifecycle and critical nature of these products.
But in cellphone handsets, PC software and firmware such as in IoT home devices, and even IT infrastructure like SoHo routers especially but also other devices, there is no agreed upon standards that I am aware of. Even in payment applications, there isn't much there. Large institutions and banks get by often by using "legacy" infrastructure like mainframes to keep the COBOL stuff running, as it has been vetted over decades.
Going further, the architecture of infrastructure, firmware and software should be examined more closely. Recently, IT architecture has been exploited over and over (Kaseya, SolarWinds, MTA) having quite asymmetric effects. In the past it was worms and drive-by malware from janky websites, today it is extremely costly disasters that make the news but roll on bye until the next turd is flipped over.
The problem space is very difficult to define, since "software" and "architecture" are such general terms encompassing basically anything. I rate the difficulty in analyzing these problems for sufficient solutions as very high. When I consider neural-net AI applications, it seems to get even more murky.
Is anybody aware of other software and infrastructure standards that exist or are in research which could apply to more general fields, or to corporate/business IT infrastructure, or to home devices, etc?
You don’t though. This is just an artefact of the structure of your industry, being limited as it is by physical, geographic, and capital constraints. Software has none of those (capital used to be an issue but is no longer - this is perhaps why things are more flexible than 30yrs ago as you note).
You think you need these things because you see your role as an integrator - but software engineers already perform that function at a different level, via libraries and protocols.
In my weird, crazy world, engineering is all about discipline (sound of a riding crop slapping a leather-clad thigh), and follow-through. It's a lot less about slide rules and differential equations.
I believe that the Romans that built aqueducts (often using coerced labor -otherwise known as "slaves") were engineers. They had a lousy numbering system, no concept of trig (although a few had heard of Pythagorus), and not even a slide rule; yet they were able to build massive, well-designed structures that still stand, to this day, nearly three millenia later.
I can't use some Web sites that are just over a year old, because their components went belly-up.
But I also consider software development to be a craft. That's really my approach.
When I write software, I make a personal level of investment into the outcome. The project must complete. Even when I'm working with a test harness, on a project that is nothing more than an experimental platform, I treat it as if it were a shipping product, that I plan to display in the store window. In fact, I often revisit truncated experimental efforts, in order to scrounge snippets. Since they are already "ship" quality, those snippets tend to be quite useful.
A basic habit of mine, is that all of my code "ships," even when it doesn't. That's a craftsman attitude. Getting the project to "ship" state is an engineering effort.
But that's just me. YMMV.
I think the issue is misunderstanding what it means for something to be engineered. There is a correct answer. (Especially because you can always optimize for cost or a different quality metric)
I believe that they can both coexist. I think that the prior art is centuries old.
You have less repetitive process in software because once software is built, copying it and reusing it as-is is free. In construction if you need to build 10 twin buildings, each of them is a project on its own. In software that'd be building one piece of software and deploying it 10 times.
Furthermore I question the notion that software projects can accommodate more "fundamental midcourse design changes". Can you spend 2 years writing a dedicated Windows desktop application and then midway suddenly switch it to being an Android application? That would be fundamental. Reskinning the UI or replacing a dependency with a similar one is not fundamental design change.
Software can't sustain fundamental design changes, unless we misinterpret what is fundamental and what isn't.
Basically we need to be careful with our analogies so we don't cross from trying to extract insight into flexing our useless cynicism about how software sucks because it's not like making a car in a superficial ways.
My take on this is to allow for rapidly evolving practices in an industry where key standards may not be updated to over a decade in some cases, or even cease to exist at all due to emerging fields.
What this means is that a responsible engineer has to make a defendable decision, using experience and judgement to justify a particular course of action.
As my engineering manager used to say to me, "Would you feel comfortable saying those words again, just adding on Your Honour'at the end?"
Whenever you have abstraction, you have lost what it means to be an engineer. There is really only 1 way to engineer something as you end up optimizing for various quality metrics, most often cost.
There's only 1 correct answer for a bracket. You want it cheap and as strong as it needs to be.
How many correct answers in my code? Do we optimize for clean code? Fast code? Do we change programming languages or go with a familiar highly supported code? There are 2 similar libraries, which do we choose?
Programming is a hybrid of science, art, tradition, and authority. I think it pays better because of this complexity.
That being said, I'd trust an engineer to program something before I'd trust a programmer to build something.
That's an interesting observation. I'd say engineering does have abstractions, but they're much better. For example, treating an RSJ as a simple uniform piece of material is an abstraction. In reality it is a crazily complex mesh of crystals. But the abstraction is simple and well studied, so it can be used reliably.
Programming abstractions are less reliable. Even commonly used ones like "the filesystem" and "the heap" are vastly more complex. It is almost impossible to remember all the possible hazards when you use them.
If a civil engineer designs a bridge, and it crashes and people die due to a design flaw, he or she can be sued, and even put in jail.
On the other hand, software comes with a license that explicitly makes its authors not liable for design flaws.
When was the last time you had to agree to an end user license agreement before crossing a bridge?
It is a professional title and while it isn't usually validated, thus only those that sign projects with legal liabilities do make the final exam, every single university in the country is only allowed to have certified engineering degrees.
I am aware of similar practices in other countries as well.
"...30% of all software projects are canceled, nearly half come in over budget, 60% are considered failures by the organizations that initiated them, and 9 out of 10 come in late."
In 2021 (as well as in 2004) it is meaningless to talk about "all software projects" without specifying domain, size, scope and project etymology. "Software" is too vast. It could be an app to tell the time with Mickey Mouses hands, or it could be the control software for a joint strike fighter.
That's much better than I would've thought.
In a billion dollar construction project, everyone understands that if you want to change the design or significantly expand the scope halfway through, you're going to have a heck of a lot of increased costs & time.
In software, it's very common to add scope or major design changes at any stage without understanding that the same cost and time increases will apply.
For example saying 90% through the project:
-Software: "we need a 2-way interface added between the complex software we're building and some other complex software"
-Building: "We need an enclosed concourse built between our new building and the next building over across the street."
It is obvious in the later case that there will be a lot of extra time & cost. While in the former, higher level decision makers on a software project don't understand the comparable difficulties.
If construction companies figured out they could outsource labor to countries with far lower currency values and thereby not-pay for extra labor, they'd do the same.... oh wait...
Not in my experience, which may not generalize. Your experience and my experience together won't really give a complete picture.
What I experience is this: Even on small projects I've had people say, "okay, can you just add this one thing in the next day or two?" And they don't understand until I walk them through in detail how that one thing is nearly as complex as the initial project, so they have to choose how important it is, and if it's important do they want to wait to release the whole thing or do a release in stages.
Of course my experience is not necessarily going to represent the whole.
Also when outside consulting groups are involved, I often see predatory behavior: they often agree to do any change asked for and don't bother to explain the costs and time involved until a deadline is missed and the budget is running out. At which point they say "the changes you requested have pushed us over the deadline and the extra time is going to cost $X".