If you take money out of the equation, how will you convince the generals to select/help enhance your project when you cannot offer them a half million dollar a year VP job at your software project when they retire? I'm not joking or being cynical, this is how DoD acquisition works: the revolving door from General to VP to civilian DoD manager to Senior VP for a defense contractor to whatever political appointment can be wrangled. I would LOVE to see OSS succeed at the five-sided puzzle palace but unless it's a civilian DoD contractor pushing it (and charging a boatload in the process) I'm just skeptical this will change much.
In my opinion, the waves of change towards OSS are already being stirred by DoD military, civilian, and contractor personnel alike. I think this also aligns with industry adopting an open-source-first, provide services/hosting model as well.
Anecdata, but most DoD program managers (CIV, CTR, or MIL) I know who work on software programs are generally driven by a need to solve the problems that they face. They gladly embrace open source solutions when they're made available easily.
I work at a Navy research center that's 3000+ employee strong and feeds all of DoD (we're a working capital fund, meaning we take customers to fund ourselves). In four years here, I've worked projects sponsored by Air Force, DHS/CBP, Navy, Marines, and two three letter agencies. There was no general involved in any of them, and I don't know any projects by coworkers that involved acquisition of technology. As for my projects, all except one still still in R&D have transitioned into production systems. All involve OSS to some degree. One was a design for a COTS multi source information fusion system meant to be deployed in the cloud and has all the open source tech there you'd expect, NiFi, Kafka, Spark, etc. My current project is a combination of us (correlation) and two contractors (separately IT and predictive models). Again, it's all the OSS and tech you'd expect to find in a real time analytics/tipping system and CI/CD pipeline/orchestration.
I bring up my experience not to say you're wrong but to show there are other avenues it comes in. There are lots of research labs, and researchers aren't about buying software from contractors.
>There are lots of research labs, and researchers aren't about buying software from contractors.
In these DoD research environments (I've worked in them before), researchers typically aren't about building software, either. It's expensive, even if you hire the teams. Not always, but unless your research is technology driven, it tends to be about cutting corners on software and delivering research quality MVPs (semi-working prototypes) while all focus is on the research. You can forget overhead of best practice infrastructure for development/engineering, that's wasted overhead in some of those group's eyes. The more theoretical it is, the easier it is to get away with, the more applied, the less easy it is to handwave away reality.
If you're at a heavily funded technology focused org like NRL (which the description sounds a lot like--neat stuff comes out of NRL), then this can be the case. There are, however, a lot of DoD funded research labs that are dumpster fires in terms of the software--NRL is almost the golden child compared to most of them. Again, in some such labs, a project may only have sufficient budget for 0.05 to 0.1 FTEs worth of software engineers. In such cases, 0.4 FTEs of software engineers can be considered a significant investment for a project, which is beyond laughable.
I think the best future model for DoD software is to bring much more of the systems integration and product development in-house, mostly using open source, which will produce a higher quality and less expensive result for the DoD. Critical differentiated software technologies will be outsourced to boutique software companies that can offer software capabilities that are qualitatively superior to anything available in open source but can't be developed on DoD software engineering wages and/or budgets.
There are often large gaps between the mission requirements and the technical capabilities of open sources software, and research labs lack the expertise and budget to address those gaps in practice. Of course, the usual system integrators can't build this software either but they will charge the taxpayer a lot of money to fail at it.
> I'm not joking or being cynical, this is how DoD acquisition works
What is that based on? Have you worked at DoD? Is there research supporting it?
DoD acquisition depends much more on civilians, not uniformed officers, and then they must be part of the President's budget and, ultimately, Congress. Officers play a minor role.
Open source software needs support. They could use the old Cygnus model and have people who are paid well to fix everything the DoD needs fixed. The general in charge of the contract could move there on retirement.
Great example of this is Michael D. Griffin. He started the Space Development Agency for commercial launchers and then retired to join the Rocket Lab he enabled.
Real challenges to FOSS in the military (or other large organizations) would seem to be long-term stability of features, and implementation of enhancements and fixes:
A commercial vendor works to please large customers, such as the military. If the big customer needs a feature supported long term (remember that military systems commonly last decades), the vendor makes it happen. If the big customer needs a specific enhancement or fix, that is a priority for the vendor, and if it's needed ASAP, that can happen too.
FOSS communities are a bit more independent-minded (which I generally like) and some developers won't want to help the military for ideological reasons. Communication with FOSS communities is much different than communication with commercial vendors, and much less efficient.
I think DoD needs access to paid, professional developers who can provide those services for FOSS. That could be someone like Red Hat, it could be outside consultants, it could be internal developers, they could hire members of the FOSS community. Also, DoD will need to fork projects when the community is uninterested in a DoD priority.
And there's another major problem: Imagine bombs are falling, people are dying, national security priorities are being lost - and the military needs a fix or enhancement now. Submit a bug to bugzilla and wait? Email the dev list? IRC? They need someone to call, who can answer the phone and get things done.
I can't imagine that there's a FOSS project which directly provides mission-critical, "press this button to stop the war" functionality.
I think there are still one or two steps/precedents needed from this memo to a real pipeline of FOSS projects getting DoD funding, but these projects would be foundational things like video transports protocols, container orchestration tools, etc. (to name a few).
In addition, if there is a mission-critical fix which needs to be made to a project, this memo authorizes + encourages DoD personnel to contribute to FOSS projects!
> Employee participation in OSS projects used by DoD is often in the Government's
interest, and is typically a legitimate use of government resources when the Government uses the software in question, either directly, as a component of a larger government system, or as a component of underlying government infrastructure.
> (1) Government employees may contribute to existing OSS projects (including being the primary maintainer) as part of their official duties, so long as they consult with their supervisor first to ensure a common-sense approach for contributions that preserve Operations Security (OPSEC) and further the Government's interests.
> if there is a mission-critical fix which needs to be made to a project, this memo authorizes + encourages DoD personnel to contribute to FOSS projects!
Yes, but that is rarely a means to get anything done urgently.
> I think DoD needs access to paid, professional developers who can provide those services for FOSS. That could be someone like Red Hat, it could be outside consultants, it could be internal developers, they could hire members of the FOSS community. Also, DoD will need to fork projects when the community is uninterested in a DoD priority.
That much is a given, and it was already true today. Even if DoD would do nothing more than contract for outsiders to build its software, the people writing the contracts need to have access to government staff who understand software (i.e. actual developers).
> And there's another major problem: Imagine bombs are falling, people are dying, national security priorities are being lost - and the military needs a fix or enhancement now. Submit a bug to bugzilla and wait? Email the dev list? IRC? They need someone to call, who can answer the phone and get things done.
I regret to inform you that the military is already used to this taking months and years. The Navy was unable to field a COVID tracking software tool in time for it to matter when the aircraft carrier USS Theodore Roosevelt got sidelined by COVID, the needed tool ended up being fielded by Defense Digital Service who operate under different authorities.
But even if we pretend the wider military could do need this, that's the internal developers you already mentioned (like DDS), or developers already on contract to provide urgent support. But that's separate from FOSS vs. proprietary. If anything, the farther we get into an "everything-as-a-Service" model, the less likely it will be that a company whose product is not open source would be able to meet those kind of urgent timeframes, because making changes willy-nilly to their -aaS product may risk their FedRAMP certification and DISA P-ATO.
In the section on getting money from the DOD, this is the first step:
> If you have a large project that is getting significant usage, incorporate a 501(c)(3) organization which will officially manage the project. Funding for the effort will live in this 501(c)(3)
Not too long ago, the IRS was clamping down pretty hard on the creation of new 501(c)(3)s, especially ones that were functioning (as they saw it) as the equivalent of businesses:
This issue doesn't get much coverage today, but as far as I'm aware, this is still the status quo, with 501(c)(3) applications being subjected to additional scrutiny and denied regularly.
There are a number of fiscal sponsors for open source out there that you can use to sidestep the need to register a 501(c)(3), for example Conservancy or SPI:
Sure. That's been the standard workaround for over a decade, I think. My point was that the advice being given is misleading and setting up a lot of folks for a Sisyphean task.
And it is also probably setting up obstacles for getting money from the DOD, if they aren't expecting to bill a fiscal sponsor as an intermediary.
The problem I'm seeing with open source projects lately, is that they are essentially ran by companies who claim to support open source but yet almost every decision they make counters that. They are happy to have people work on their projects for free, but then you end up having to assign your work to them and they turn around and change their licensing or contributions later on where you no longer even own the open source efforts you put into the software. To them open source is just another business opportunity, but when it comes to values they could care less.
There are some open source companies that aren't like this; RedHat, Collabora, Igalia, Codethink and all the companies contributing to the Linux kernel are examples.
I work for the US EPA, speaking in my personal capacity.
We have a Github, but in a lot of ways it feels kind of like an archive. Does anyone have feedback on how it can be made more useful? We have all the authority to be good OSS community members.
Maybe if we were to prioritize like general purpose libraries instead of just our super domain specific projects?
Make sure every repository has an informative README. There are a ton of templates and generators out there, most are pretty good (if you find or create one that is especially useful for your agency, consider promoting it internally).
Now comes the 'next level' stuff.
If a project would be useful in an academic context, perhaps include a blurb on attributing a citation for it.
There are a lot of folks out there looking for projects that will accept their contributions so they can build up some cred. Populate projects' issues with stuff you can tag as a good-first-issue, and your projects will start showing up in various aggregators (be judicious about selecting projects to do this for, you don't want issues and PRs languishing without being incorporated for lack of maintainer interest or time, that just looks worse).
For projects that really are just being thrown over the transom, mark them clearly as such and invite friendly forks.
Try to imagine who might find a particular project useful. Write that up as an example and incorporate it into the README or other documentation (eg. I imagine that anything related to stormwater runoff may be useful to civic planning departments, or perhaps environmental advocacy groups). Blog/tweet about the use case (your agency's comms team may be helpful here).
I'm not sure what EPA can do to make it more useful. My personal example is the USEPA/USEEIO modeling framework [1]. My agency uses a pricey and rigid proprietary modeling framework. When I suggested USEEIO as an extensible cost-effective alternative I was met with resistance that it is free and mustn't be any good and definitely not defensible in court. Classic .gov thinking. I think DoD prioritizing open-source along with the work of USDS/GSA at digital.gov needs to diffuse to Senior Execs at other agencies to shift the paradigm. Unfortunately it feels like this is a decade out. Thank you for pushing OSS in .gov and to your peers at EPA for USEEIO.
Speaking from experience, convincing management is a long-haul process.
It is needs to be repeatedly brought (but gently).
Forward links to success stories, provide counter arguments to FUD (but again, gently!)
I say gently because it's easy to create enemies by making people look bad, and that's a no-no, you have to gradually wear down the old consensus and build a new one.
The usual stuff; Do everything in public in the community. Make your public GitHub the place you do internal deployments from, so that you ensure everything deployed internally is public. Contribute back to your dependencies' upstream projects with bug reports/triage, patches, test cases and reviews. Contribute back to the distros you use, with bug reports, packaging updates, patches, testing etc. As well as funding US EPA developers to do this work, fund the most important projects you use through donations or support contracts or whatever else is available.
Re general purpose libraries, definitely consider splitting projects up into separate libraries, and switch the domain specific projects to using those libraries.
The model is very simple, if a business uses open source, the the business has to pay an open source tax. It doesn't include OSS covered by must share source, like GNU , license.
Readit News
Anecdata, but most DoD program managers (CIV, CTR, or MIL) I know who work on software programs are generally driven by a need to solve the problems that they face. They gladly embrace open source solutions when they're made available easily.
I work at a Navy research center that's 3000+ employee strong and feeds all of DoD (we're a working capital fund, meaning we take customers to fund ourselves). In four years here, I've worked projects sponsored by Air Force, DHS/CBP, Navy, Marines, and two three letter agencies. There was no general involved in any of them, and I don't know any projects by coworkers that involved acquisition of technology. As for my projects, all except one still still in R&D have transitioned into production systems. All involve OSS to some degree. One was a design for a COTS multi source information fusion system meant to be deployed in the cloud and has all the open source tech there you'd expect, NiFi, Kafka, Spark, etc. My current project is a combination of us (correlation) and two contractors (separately IT and predictive models). Again, it's all the OSS and tech you'd expect to find in a real time analytics/tipping system and CI/CD pipeline/orchestration.
I bring up my experience not to say you're wrong but to show there are other avenues it comes in. There are lots of research labs, and researchers aren't about buying software from contractors.
In these DoD research environments (I've worked in them before), researchers typically aren't about building software, either. It's expensive, even if you hire the teams. Not always, but unless your research is technology driven, it tends to be about cutting corners on software and delivering research quality MVPs (semi-working prototypes) while all focus is on the research. You can forget overhead of best practice infrastructure for development/engineering, that's wasted overhead in some of those group's eyes. The more theoretical it is, the easier it is to get away with, the more applied, the less easy it is to handwave away reality.
If you're at a heavily funded technology focused org like NRL (which the description sounds a lot like--neat stuff comes out of NRL), then this can be the case. There are, however, a lot of DoD funded research labs that are dumpster fires in terms of the software--NRL is almost the golden child compared to most of them. Again, in some such labs, a project may only have sufficient budget for 0.05 to 0.1 FTEs worth of software engineers. In such cases, 0.4 FTEs of software engineers can be considered a significant investment for a project, which is beyond laughable.
There are often large gaps between the mission requirements and the technical capabilities of open sources software, and research labs lack the expertise and budget to address those gaps in practice. Of course, the usual system integrators can't build this software either but they will charge the taxpayer a lot of money to fail at it.
What is that based on? Have you worked at DoD? Is there research supporting it?
DoD acquisition depends much more on civilians, not uniformed officers, and then they must be part of the President's budget and, ultimately, Congress. Officers play a minor role.
License fees are a rounding error compared to the cost of personnel.
A commercial vendor works to please large customers, such as the military. If the big customer needs a feature supported long term (remember that military systems commonly last decades), the vendor makes it happen. If the big customer needs a specific enhancement or fix, that is a priority for the vendor, and if it's needed ASAP, that can happen too.
FOSS communities are a bit more independent-minded (which I generally like) and some developers won't want to help the military for ideological reasons. Communication with FOSS communities is much different than communication with commercial vendors, and much less efficient.
I think DoD needs access to paid, professional developers who can provide those services for FOSS. That could be someone like Red Hat, it could be outside consultants, it could be internal developers, they could hire members of the FOSS community. Also, DoD will need to fork projects when the community is uninterested in a DoD priority.
And there's another major problem: Imagine bombs are falling, people are dying, national security priorities are being lost - and the military needs a fix or enhancement now. Submit a bug to bugzilla and wait? Email the dev list? IRC? They need someone to call, who can answer the phone and get things done.
I think there are still one or two steps/precedents needed from this memo to a real pipeline of FOSS projects getting DoD funding, but these projects would be foundational things like video transports protocols, container orchestration tools, etc. (to name a few).
In addition, if there is a mission-critical fix which needs to be made to a project, this memo authorizes + encourages DoD personnel to contribute to FOSS projects!
> Employee participation in OSS projects used by DoD is often in the Government's interest, and is typically a legitimate use of government resources when the Government uses the software in question, either directly, as a component of a larger government system, or as a component of underlying government infrastructure.
> (1) Government employees may contribute to existing OSS projects (including being the primary maintainer) as part of their official duties, so long as they consult with their supervisor first to ensure a common-sense approach for contributions that preserve Operations Security (OPSEC) and further the Government's interests.
Yes, but that is rarely a means to get anything done urgently.
Of course. You have to do
sudo ./war.sh -f --command STOP
That much is a given, and it was already true today. Even if DoD would do nothing more than contract for outsiders to build its software, the people writing the contracts need to have access to government staff who understand software (i.e. actual developers).
> And there's another major problem: Imagine bombs are falling, people are dying, national security priorities are being lost - and the military needs a fix or enhancement now. Submit a bug to bugzilla and wait? Email the dev list? IRC? They need someone to call, who can answer the phone and get things done.
I regret to inform you that the military is already used to this taking months and years. The Navy was unable to field a COVID tracking software tool in time for it to matter when the aircraft carrier USS Theodore Roosevelt got sidelined by COVID, the needed tool ended up being fielded by Defense Digital Service who operate under different authorities.
But even if we pretend the wider military could do need this, that's the internal developers you already mentioned (like DDS), or developers already on contract to provide urgent support. But that's separate from FOSS vs. proprietary. If anything, the farther we get into an "everything-as-a-Service" model, the less likely it will be that a company whose product is not open source would be able to meet those kind of urgent timeframes, because making changes willy-nilly to their -aaS product may risk their FedRAMP certification and DISA P-ATO.
> If you have a large project that is getting significant usage, incorporate a 501(c)(3) organization which will officially manage the project. Funding for the effort will live in this 501(c)(3)
Not too long ago, the IRS was clamping down pretty hard on the creation of new 501(c)(3)s, especially ones that were functioning (as they saw it) as the equivalent of businesses:
https://www.techdirt.com/articles/20140701/11470827745/irs-r...
This issue doesn't get much coverage today, but as far as I'm aware, this is still the status quo, with 501(c)(3) applications being subjected to additional scrutiny and denied regularly.
https://sfconservancy.org/https://www.spi-inc.org/
And it is also probably setting up obstacles for getting money from the DOD, if they aren't expecting to bill a fiscal sponsor as an intermediary.
We have a Github, but in a lot of ways it feels kind of like an archive. Does anyone have feedback on how it can be made more useful? We have all the authority to be good OSS community members.
Maybe if we were to prioritize like general purpose libraries instead of just our super domain specific projects?
https://github.com/USEPA
Add a description to every repository.
Make sure every repository has an informative README. There are a ton of templates and generators out there, most are pretty good (if you find or create one that is especially useful for your agency, consider promoting it internally).
Now comes the 'next level' stuff.
If a project would be useful in an academic context, perhaps include a blurb on attributing a citation for it.
There are a lot of folks out there looking for projects that will accept their contributions so they can build up some cred. Populate projects' issues with stuff you can tag as a good-first-issue, and your projects will start showing up in various aggregators (be judicious about selecting projects to do this for, you don't want issues and PRs languishing without being incorporated for lack of maintainer interest or time, that just looks worse).
For projects that really are just being thrown over the transom, mark them clearly as such and invite friendly forks.
Try to imagine who might find a particular project useful. Write that up as an example and incorporate it into the README or other documentation (eg. I imagine that anything related to stormwater runoff may be useful to civic planning departments, or perhaps environmental advocacy groups). Blog/tweet about the use case (your agency's comms team may be helpful here).
I hope these suggestions help.
[1] https://github.com/USEPA/USEEIO
It is needs to be repeatedly brought (but gently).
Forward links to success stories, provide counter arguments to FUD (but again, gently!)
I say gently because it's easy to create enemies by making people look bad, and that's a no-no, you have to gradually wear down the old consensus and build a new one.
Deleted Comment