Not snarking. I'm honestly more confused reading this than I was before.
"Where do people get the confidence to say that their work is so uniquely novel that it's "something that has never been done before in the history of the universe", that the field is obviously totally different from another field they know nothing about, etc. ?" --Dan Luu
I think it is less that these things "have never been done before" and more "we haven't bothered to learn anything from people who've done things like this before". Same with parenting.
Every adult who becomes a parent seems to realize very quickly that their parents also had no clue WTF was going on.
So yeah, no different with what we do, since we're literally always building new things that haven't been done before.
- knowing the science enough to know they're contributing to others suffering
- knowing what might happen with reasonable certainty
- but compartmentalizing that awareness internally to avoid acting
- hiding it externally
- and keeping doing what they were doing
describes the reactions of most individuals about climate change.
It's like we never learn and then those who are in charge made rules to make it even harder to learn as we went forward.
Also that humans are terrible at existential threats.
So your options are either senior software engineers who have done some data work (that's how I got to be a Data Engineer) or people who've been doing analytical data work (either in the traditional warehousing space or via science/insurance/finance type spaces) that are semi-technical but have no formal engineering background.
The former are people who went to college in the late 90s/early 2000s (like myself) when things were different. The latter need to hyperfocus on coming up to speed in engineering.
I reviewed this guide a couple months ago for my employer to consider as the basis of an internal bootcamp, and I'd note that it's perfect for the audiences I mentioned. Also, even for people with more up to date academic experience, note that the transactional database schemas that software normally deals with often look wildly different than analytical structures.
I say this as someone who's both been a single point of failure at times (and learned my lesson) and someone who's had to clean up after piles of "clever" code that someone left behind when they got hired out somewhere else.