Readit News logoReadit News
duffpkg · 5 years ago
I was "ringleader" (many diverse groups have been involved over the years) of the OpenEMR project in 2003-2005 and went on to create the open source ClearHealth and HealthCloud EMR (electronic medical record) systems. OpenEMR has a lot of dedicated folks in it and has been a project of some sort of another for ~25 years at this point.

You can read quite a bit about open source in healthcare in my book, Hacking Healthcare. A bit dated but still in print.

Unfortunately there are massive headwinds against open source in US healthcare settings. Regulation requires certifications that cost upwards of $100K first time, $10K with each release, just in fees. Licensed data sets also make for real difficulties in licensing. Required 3rd parties like SureScripts are openly hostile to open source. Most largest buyers of systems are institutional and most current interpretations of law make it so that open source systems cannot be sold as "sole source" which makes life very hard to close and keep those deals. Finally, until a business model emerges that favors open source and patient health, everyone makes more money with lock-in and so that perpetuates.

Ask me anything.

Can confirm, a little concerningly, that code I wrote 17 years ago is still widely present in the OpenEMR codebase including my old office number for test patients, lol.

datahead · 5 years ago
Hi Duff! Happy to see you on HN. EDIT: not Fred.

I work for a large hosp. operations company and serve as the Dir. Engineering for our clinical operations group. Hacking Healthcare is required reading for new members of my team. It serves as an excellent introduction (with a healthy amount of critique) to the dynamics in the hc technology ecosystem. Thank you for providing this perspective on the industry and its challenges with tech.

We've been successful developing using open source technology internally. In fact, I take a fairly hard stance on disallowing proprietary healthcare specific "solutions" from working their way into our stack (aside from the EHR itself, it has staying power). We're lucky in that we are positioned as somewhat of a startup within a larger org, and are able to take that approach.

To avoid some of the issues you raise, we generally are working to reduce the surface area of the EHR to become simply the transactional backend which is then mirrored to a larger ecosystem of custom apps. This has the effect of boxing in the regulated entity. We focus on data integration (by spending $$$$ on custom HL7 interfaces, unfortunately not everyone can afford) to get outside of the walled garden. This means we can use the information/data for new and interesting purposes without worrying about the EHR vendor's roadblocks/tolls. More importantly to some people, we don't disrupt the billing cycle that originates from the EHR.

Do you notice any trends where healthcare operations/providers are starting to develop internal technology that integrates with the EHR to compliment vs. replace the core transactional system?

duffpkg · 5 years ago
It's Duff (David) instead of Fred but thanks. Fred is doing great too. Are you a former CHL/TXR managed or sub-owned group or facility?

Unfortunately I see the opposite trend right now, more silos, more lip service to interoperability, more tolls. I think driven by the burden of regulatory overhead. Moving forward there could be a shift to a "patient owned" record where providers and facilities feed standardized formats into a patient owned/managed "personal cloud". I hope that continues to pick up steam.

hcthrowaway111 · 5 years ago
Hey - I'm trying to get into health tech (EHR integrations, HL7, FHIR etc), do you have any additional "required reading" suggestions beyond Hacking Healthcare?
mixonic · 5 years ago
Howdy there! I was one of the original authors of OpenEMR back in high school. I'm still good friends with at least one of the other authors. We're always stunned to see OpenEMR in the news, and watching it creep up on HackerNew today has been fun.

I've always been curious why OpenEMR seemed to dominate in the OSS space after we walked away from it. I can only theorize that the code was more approachable than other projects (PHP), and that the GPL kept the work from being captured by any one business. I can't imagine that the code was the best, I'm painfully aware of how poor the security practices must have been in hindsight.

You've given me the chance to ask a question I never knew who to ask: Why, back in 2003 (just after we stopped giving the project attention), was OpenEMR the project you decided to spend time on? What made it the attractive thing to invest in?

If you can tell me I'll bottle that elixir and pour it into every OSS effort I work on today.

duffpkg · 5 years ago
Hi. James? I was CTO of Pennington Firm in that era and it was one of many industries where "internet modernization" was happening to a sort of sleepy status quo. OpenEMR with FreeB were the furthest along open source project at the time and so we started there. There were a lot of legacy type problems inherent in the OpenEMR codebase and I think the change to PHP 3 ultimately is what lead to starting fresh with ClearHealth. I'm dating myself but that's around the time that browser AJAX starting opening up a lot of UI possibilities.
tedjdziuba · 5 years ago
Second original co-author of OpenEMR here. He's telling the truth.
absorber · 5 years ago
> Unfortunately there are massive headwinds against open source in US healthcare settings.

This reminds me of one of the first articles I've read about Linux and open source in general. It was about a CEO (and largest shareholder) of Medsphere Systems Corp, who open sourced their tech stack (I believe called OpenVista) and was promptly sued by his own company (!)

Unfortunately it seems that the sands of time have eroded the original content (which was apparently hosted on linux-watch.com, which now redirects to a VPS provider), but I've still managed to find something [0] [1] [2]

0: https://70.42.23.9/servers/a-medical-open-source-legal-hell-...

1: https://medicalconnectivity.com/2007/10/25/medsphere-settles...

2: https://www.informationweek.com/medsphere-settles-lawsuit-wi...

duffpkg · 5 years ago
There is a whole bunch to the behind the scenes of Medsphere and OpenVista. I am not sure what I can say except that a lot of big personalities were involved. WebVista by ClearHealth is still in use by a few very large hospital chains.
russnewcomer · 5 years ago
I recently worked on a very small, very custom open source 'EMR' system for a friend involved in prenatal care in a developing country, and so what I wonder is are there attempts to get OpenEMR/Clear Health etc into countries and settings where there is not the same huge regulatory barriers? To me it seems like the U.S. is largely lost for a generation for open source/patient-centric EMR, but maybe there is hope for other countries?
duffpkg · 5 years ago
Excluding mexico where a ClearHealth derivative still, as far as I know, powers one of the large hospital chains there, no. In my personal experience the venn diagram of countries where there is not much regulation but yet it is advanced enough medically that EMR is a primary problem is pretty much zero. To put it another way most countries where regulation is low have much more pressing medical needs than software.
ethbr0 · 5 years ago
Why aren't more user-visible medical systems built as services: open source backend serving endpoints, closed front-ends?

Bespoke front-ends and UX have never been open source's forte, but shared serving technology running behind the scenes has been wildly successful.

Health care seems like a good fit for that.

(Said as someone with clients in insurance, and well aware of how quickly data interchange can embrittle an architecture)

duffpkg · 5 years ago
I think there are plenty of open source projects with great UI but that aside I'm not sure I understand what you mean? What type of service for example?

HIPAA greatly complicates a lot of data sharing because of appropriate data privacy issues.

anonymouse008 · 5 years ago
I've noticed a lot of practitioners use PhraseExpander or some other shortcut writing tool to write their patient notes - I'm curious to know how they get around the HIPAA certifications, especially since they are a dedicated key logger on top of any OS.

Do you have any insight in this arena? I wonder if it is because they are not 'the record,' but instead are 'tools to create' the record that is eventually uploaded and stored in another platform?

Might be a tangential question, thanks for the patience

duffpkg · 5 years ago
There is a lot of controversy in this area. Medicare rules are pretty clear that to the extent tools like that are used systematically to enhance bill-ability they are prohibited. Malpractice litigation is having a field day with computerized systems which is why so many states are being pressure to institute caps. Pretty much everyone tries to use templating tools to increase bill-ability to some extent. Healthcare is rife with conflicting goals.

The underlying problem is that we need an economical way for doctors to have more time to spend in the room with patients but no one, patients included, wants to pay for that.

I really hope "concierge" medicine, a lot of that now happening on the lower priced end not only for "luxury patients", continues to take off. You pay some cash out of pocket but get care that is dramatically better and more preventative.

manapause · 5 years ago
My experiences in healthcare/EMR were focused on Medicare part B (home health); physicians, NPs, and PAs that made house calls.

This is a great question; being thrust into hippa compliance from 2000 - 2012 was the easy part, and can perhaps be summed up as

“ fax machines behind a door, paper charts and/or laptops/tablets locked in trunks*, all-in Blackberry enterprise services, Exhange over ssl xml-rpc endpoints, and ensuring EMR portal uptime was so much of the fun parts.

The hard part: training home health medical groups how to VPN - and how to never mix work tech and personal tech.”

I am unaware of hippa changes since the blackberry-era, and while I can’t imagine specific apps being named in amendments to the law, I can say without a doubt that hippa at the time meant having total control over all technical Devices used by your clinicians, prepping all laptops and equipment to use encrypted storage, ssl-encrypted email, VPNs, etc was always made easier by focusing on training, troubleshooting, and being able to mitigate the raw chaos monkey power of a doctor with a gadget.

With the passage of the HITECH act, being able to facilitate the automated logging of a doctor’s conversation with a patient’s family or caregiver towards the CPO (care plan oversight) and making the fulfillment of PQRI initiatives automatic... “so easy a doctor could do it”... meant (at the time) a very hefty pay raise for part-B collections.

TLDR: hippa was easy, training doctors to change their ways is hard. producing the tech that makes their lives easier means assuaging them of their worries that they are maximizing Medicare collections.

It is fun to imagine these days that transcription, location, and barcode technology has “made it” to the point we dreamed about ~10 years ago.

kls · 5 years ago
Unfortunately there are massive headwinds against open source in US healthcare settings. Regulation requires certifications that cost upwards of $100K first time, $10K with each release, just in fees.

Seems to me like this could be overcome by licencing but not in the general sense but more in a you need accreditation, join this other pool of people utilizing the system and buy an accreditation licence. Seems like there would still be a value proposition there.

As for the 3rd party seems that could be hit or miss again if it is money, build those out as modules that cover the licence fees the third party is looking to recoup. Maybe with a little bone in it for the open source developers as well.

dman7 · 5 years ago
Thanks for AMA. My previous startup was in self-insured employer space so I only saw the issues you're describing from a distance.

> "Finally, until a business model emerges that favors open source and patient health, everyone makes more money with lock-in and so that perpetuates"

I'm curious to know if you've thought about what such a business model might look like. The closest to a viable business model I've seen gives patients control of their data & allows them to monetize it. But that feels like a pipe dream at the moment because a) EHR vendors don't have incentives to share data, and b) there is no marketplace of buyers for said data.

schoolornot · 5 years ago
I worked in the EMR space and don't remember the company having to pay $10k per release. Can you share more info on this?
nradov · 5 years ago
I assume that was to participate in the ONC Health IT certification program.

https://www.healthit.gov/topic/certification-ehrs/certificat...

BadInformatics · 5 years ago
See also https://gnuhealth.org, https://oscar-emr.com/ and https://hospitalrun.io/.

As with anything in the b2b healthcare space, most of these systems suffer from quite a bit of legacy and at-best-average code quality. Despite that, many doctors, clinics and even small hospitals use them because the private solutions (think Epic [1], but smaller) aren't necessarily better code-wise (don't ask me how I know). I wish more FAANG-calibre devs would look into contributing to and evangelizing these platforms rather than writing yet another note-taking/"productivity management" app. It has a direct impact on the quality of care delivery in certain parts of the world and a positive impact on tool-related clinician burnout [2].

[1] https://news.ycombinator.com/item?id=18735023 [2] https://news.ycombinator.com/item?id=24336039

safog · 5 years ago
I've worked in the space for a few years and I'd discourage anyone from trying to build a career as a software dev in the Healthcare IT / EMR space. It's extremely sales driven (devs aren't valued at all), code quality is terrible and the systems you write are mostly for the benefit of insurance companies / compliance than doctors or patients.

I think there was a YC funded iPad EMR startup that tried to be cool / hip / provider first until they got smacked in the face by reality.

floatrock · 5 years ago
> It's extremely sales driven... and the systems you write are mostly for the benefit of insurance companies / compliance than doctors or patients.

This.

Once I worked making healthcare software that would basically save costs by avoiding complications. We were bidding into a large hospital network. After the long sales cycle, we checked the most boxes, clinicians liked the feel of ours the best, they said ours was the most intuitive, helped them do their job the fastest, etc.

The clinicians weren't the ones writing the check though.

Our competitor approached the main insurance provider in the area and convinced them they could save x% in additional claims if the hospital network would adopt their software. Competitor's deal was to split the bill between the insurance company and the hospital network. To the admins writing the checks, they had the choice between two vendors that more or less did the same thing but one came in at half the cost. Clinicians' preferences didn't matter, it was a no brainer for them.

No amount of software engineering would have saved that deal.

BadInformatics · 5 years ago
Not all healthcare systems are beholden to private insurance (my own country's included) and that shows in where these systems are deployed.

Also, compliance doesn't fully explain why competitors are able to convince clinicians to switch systems. Word-of-mouth means that people will know your EMR is a flaming piece of garbage, but (as you noted) companies would much rather cut up-front prices so they can milk an extended contract than improving product quality. All that said, I think there's a bit of a chicken-and-egg thing going on here. Good people don't join the space because the culture sucks and the pay is bad, but those are because those with the talent and drive all self-selected out of it. I get it, health IT is a huge drag and not at all sexy. But just look at how often folks on HN ask about "doing social good" and how many complaints there are about healthcare delivery. Trying to run a "disruptive" VC-backed startup is IMO pretty crazy, but contributing to an OSS project is far less risky and more achievable.

erichurkman · 5 years ago
You're thinking of https://www.drchrono.com/
hanklazard · 5 years ago
That's depressing. I feel like the regulatory side needs to be changed so that options like OpenEMR can become viable.
toomuchtodo · 5 years ago
> I wish more FAANG-calibre devs would look into contributing to and evangelizing these platforms rather than writing yet another note-taking/"productivity management" app.

I would like to see the US Digital Service continue to task technologists with improving EMR systems at CMS (Centers for Medicare and Medicaid Services), but made free to use by all practitioners and citizens (and of course, open sourcing the resulting codebase). It seems sort of inefficient we keep reinventing the wheel (Epic and the like, which are crazy expensive, or self hosted solutions, when practitioners should not be spending time maintaining EMRs), when your records should be stored for your benefit by your government over the course of your life. This is where, imho, high calibre engineers provide the most leverage (one way ratchets on public goods at scale).

[1] https://www.usds.gov/resources/USDS-Impact-Report-2020.pdf

[2] https://www.va.gov/health-care/get-medical-records/

beh9540 · 5 years ago
The VA sort-of does this, at least when I worked for them. The issue they had when I was there was depending on where you went, you may or may not have access to the record, because the VA didn't use a central system, it used many systems across the country, each with their own records(basically their own mainframe). We once added a hospital to our system, and had to have dual workstations because the systems couldn't be easily merged, and they had to look up patients in both systems.

Also, with Veterans Choice, I don't know how much there was an effort to bring this data back. Same thing with the DoD, for a while there was an agreement to send medical records for active duty to the VA, but then that got pulled for a time.

I believe there was a huge undertaking to consolidate these to fewer systems in the last few years, but Vista[0] (the VA's EMR) is pretty scary. I wouldn't wish it on anyone.

https://en.wikipedia.org/wiki/VistA

trimbo · 5 years ago
The US government has a public domain EMR system which they are in the process of replacing with a commercial system.

https://news.ycombinator.com/item?id=25042125

sidlls · 5 years ago
I hope like hell “FAANG caliber devs” never take an interest in this space. Or at least not until the culture changes substantially.

There are just some things that “throw devs (of any quality) at it” just doesn’t work. The health care industry is one of them.

BadInformatics · 5 years ago
That's imprecise shorthand on my part. s/"FANG Calibre"/"objectively talented and used to/capable of negotiating good compensation"/. There's a degree of Stockholm Syndrome in healthcare tech where people don't know what a well-developed product or codebase looks like. It's unlikely to change from the top, so getting more technical folks with higher leverage into the field is IMO the next-best option.

And yes things are changing at a glacial pace, but they are changing. For example, my province is developing a new patient portal [1] out in the open. AFAICT, they seem to be doing everything aboveboard: CI, code quality standards, documentation and proper testing, etc. Yet if you look at another team in the same org (ministry of health), you'll find non-existent dev practices, oodles of VBA, or (even worse) some slow+buggy third party system put in place by one of the procurement vampires (IBM, CGI, Deloitte, you know the bunch). The biggest difference? The former project has a dedicated, US Digital Service-style team of skilled and hopefully better-compensated dev(ops) people who know how to deliver good software.

[1] https://github.com/bcgov/healthgateway

boston_sre87 · 5 years ago
Believe it or not, there are some engineers that have worked at faang companies that are not sloppy and I work with a few of them.
breck · 5 years ago
I love this space—mostly because I loathe the absolutely awful American EMR systems— and intermittently have been working on some ideas with a few people (https://github.com/treenotation/pau) (here are some fugly notes/links to EMS systems for anyone interested in the space: https://github.com/treenotation/pau/tree/master/paudb)
adzm · 5 years ago
One of the hurdles of EMR systems is that there is a pretty significant minimum viable product due to various standards that can't be ignored. Thankfully, this space is far more open now than it used to be; HL7 for example used to require payment just to see the standards. Pretty much everything you would need access to (HL7, CCDA, SNOMED, LOINC, ICD-10, etc) is freely available now!

Generally just having an EMR system is not enough; you also need practice management, scheduling, billing, insurance claims, etc. Interoperability between separate software for these things is... tenuous at best, though some practices do manage to handle it, it can be very fragile. Hence integrated solutions are pretty much the best way to go, and also prevent disruption from competitors which may be better in one space but not another, since it's so hard to get them to talk to each other well.

dragonwriter · 5 years ago
> Pretty much everything you would need access to (HL7, CCDA, SNOMED, LOINC, ICD-10, etc) is freely available now!

The AMA’s CPT-4, incorporated as a component of HCPCS, is not free, and is the mandated code set for most professional procedure coding.

And while otherwise that may be true for most of what you need for core EMR functionality, everyone wants EMR and billing/insurance transaction handling to be modules of the same core system (because you are going to need both, and they need to interface smoothly to avoid a whole lot of operational friction), and most of the mandatory billing/insurance standards are decidedly not gratis; older versions of at least the X-12 standards in this space were subsidized by CMS and available for free, but that hasn't been the case for the versions required since 2010. And that's just basic transaction standards, a lot of the code set standards are also proprietary.

(In addition to not being free, the standards in this space are exceptionally poorly written, ambiguous, self-contradictory, and incorporate vast quantities of external material, often also not free, by reference—and often not hyperlinks, but “here is the name of the document and the postal address from which you can contact the entity from which you can order it.”)

amelius · 5 years ago
I think what would help us the most is not writing software, but instead explaining the requirements in detail (like a specification). There are many people looking for a nice self-contained FOSS project to work on, but many don't know where to look and joining an existing codebase might be too daunting.
breck · 5 years ago
There will be an Uber of this space, someone who says "I understand the problems these laws are trying to solve, unfortunately they are a kludge and we can solve them much better with better technology". So complying with all these standards will be a second priority done for backwards compat for the person who comes in and disrupts this space.
ska · 5 years ago
I think that is an idiosyncratic definition of what uber did.

Either way though, the incentives that built and maintain the complexity in healthcare IT stacks goes much further than a few laws.

gowld · 5 years ago
That company won't have any customers. Hospitals aren't going to hire an Uber driver to run their IT>
bearjaws · 5 years ago
Yeah its a total mess, most health systems I work with have 30-50 different vendors all with some various forms of integration with the EMR... It's always a mess, with no end in sights.
punnerud · 5 years ago
I was release coordinator 5-7 years ago for an EMR software (DIPS) in Norway, to one of the bigger hospitals (Kalnes). It was said that there was a unwritten policy that they never use open source software, the only exception was for the ERP database running on Linux. There was more than 2000 systems in that portfolio to various hospitals under the same policy.

After checking with other employees if found the reason; they had to make sure the supplier did not have to many levels of sub contractors, and had to be close to the core development. So not open source in it self, but open source was seen as a big red flag.

subsaharancoder · 5 years ago
In this space it's worth mention OpenMRS https://openmrs.org/ which also aims to do the same thing. Most of its deployments are in developing countries (I was part of a team driving adoption of it in Kenya) but I'm yet to see a successful large scale deployment of the platform.
desktopninja · 5 years ago
Experimented with it and seemed to scale well. 3x load balancers (nginx or httpd or haproxy or iis) https only 3x apache tomcats to deploy openmrs war ... https only. azul jre with -Xmx=2g. Tomcats were also as close to default configuration as possible. 3x mysql clustered ... https only

No issues on any of these OS's: Windows Server 2008r2 Windows Server 2016 CentOS 7 CentOS 7.5 Raspian Lite MACOS

OSes ran on virtualized (hyper-v, virtualbox, xen) and baremetal. Commodity hardware (old laptops, desktops with at least 4G ram. and PIs) and real server hardware. Purposely used mixed computing resources to simulate what a team say in Ndola, Zambia might have at their disposal.

subsaharancoder · 5 years ago
By "scale" I'm not referring to OpenMRS capabilities t run on commodity hardware. I'm referring to a large deployment at a national level or at a large public hospital in Kenya. Most deployments have been pilots and at smaller establishments.
thinkmassive · 5 years ago
"OpenEMR is a Free and Open Source electronic health records and medical practice management application"
simonebrunozzi · 5 years ago
Thanks. It took me several seconds to realize what this was about.

My AWS-bias made me initially think of an open version of EMR (Elastic Map Reduce).

Bqhatevwr · 5 years ago
Ditto. I'm relatively new to HN and I am consistently frustrated by the "completely uninformative title" fetish link submitters seem to have.

I also wish that when people submit what is clearly a "new version of software/thing X" they would submit a title like:

"Salami 2.0 released, this popular meat product now has substantially more zest."

Instead people just submit:

"Salami 2.0"

That's no better than Freshmeat.

tprynn · 5 years ago
EMR is just an automated way of deploying open source components (Hadoop and co.) - there's some glue code there but the equivalent "open" version is probably the Hortonworks stuff (now owned by Cloudera): https://github.com/hortonworks
n_f · 5 years ago
Same lol, especially cuz it's on hacker news
ptx · 5 years ago
I clicked on "features", trying to find out what EMR means, and was greeted by a banner that really clarified things: "ONC Certified HIT! 2014 Complete Edition EHR!"
hanklazard · 5 years ago
I've been coming back to the project intermittently over the last few years and have been pleased to see the progress. The reason I came back this time was that our managing partner just told us we'll be paying next year in our small outpatient clinic--truly unreal. I dream of a day when we could just use our own system, maybe something that I could even manage myself (maybe not realistic?). Bravo to the contributors of this project!
WesBrownSQL · 5 years ago
1. Certification, in the USA if you want to get paid from the government you must be certified. This isn't a small undertaking at all and requires quite a bit of development and then the cost of actually doing the third party testing. It has driven companies out of business and forced consolidation even among the larger players.

2. Configurability. EMR's are crazy configurable to meet any hospitals requirements. This means lots of consultant hours to get things setup and running. Take a look at how much money Epic and Cerner make just from "consulting".

3. Interoperability. Again, there are standards like HL7 and FHIR are widely used but the data isn't always great. We are seeing more and more API endpoints all of this requires a level of customization.

All of this adds up to a ton of cost for a small-ish market with a large pool of no or low profit buyers and pretty much a replacement market.

Oh, and you are building software that could cause harm or death. I can't imagine why people don't want to come into this industry and really push the state of the art.