Tacit knowledge is inevitable... but it's also different from what this article is about. Tacit knowledge is the kind of knowledge (and skill!) that cannot be fully explained or taught in words—think mechanical skills you can only pick up through physical practice or very context-specific expertise you only get through experience. In programming good taste is crucial and entirely tacit: we can try to distill taste into words and write books about it, but it will never be enough.
Tacit knowledge is not the same as explicit knowledge that just happens to not be documented. Tacit knowledge is not documented because it is undocumentable. You cannot avoid tacit knowledge. Human language fundamentally cannot express all the knowledge that experts develop through years of practice and experience. Even if an expert can express their experience, there are simply ideas and skills that people cannot learn exclusively from words.
This is an important distinction. Trying to eliminate undocumented explicit knowledge is useful for teams. Trying to eliminate tacit knowledge is disastrous. I've seen attempts to make tacit knowledge legible to management—it is one of the most direct ways to sabotage your own experts because experts fundamentally cannot make all their knowledge explicit. Expertise is tacit knowledge and tacit knowledge is expertise.
People generally understand that design by committee or design by regulation has awful results; one of the main reasons for this is that committees require decisions to be fully explicit and explainable—which hamstrings creative problem-solving and pushes for poor design decisions. Exclusively explicit processes get you lowest-common-denominator results: not the lowest common denominator of the expertise present, but the lowest common denominator of what people can explicitly communicate and understand, which is even lower! If you don't let experts be experts, you won't get expert-level work.
> very context-specific expertise you only get through experience.
When giving advice, I'm often tempted to point to a given situation, and to explain what rule of thumb I used to solve it.
But then I would see a junior programmer applying the rule of thumb in a completely different situation and I'd want to say, "No, that will end in tears!" And I realized that a lot of my rules of thumb were really dumb when taken out of context.
As far as I can tell, a huge part of expertise is having a large set of mappings like:
- In contexts like "C1", use rule of thumb "R1" or "R8".
It's very tempting to talk about R1, R2, etc. You can write blog posts like, "How R2 will transform your team!"
But a lot of real expertise comes from being able to say, "Oh, this is a combination of C7 and C78, so I should consider rules of thumb R7, R49 and R102. Ah, wait, no, R49 is just gross here."
So I've learned that, when mentoring, I need to spend a lot more time talking about the context before proposing any rules of thumb.
My rule of thumb is to hesitate before giving advice if I can't come up with an anecdotal story to support it. Stories are more useful than generalizations because they come with the context, and can be dismissed in different contexts. At risk of presently breaking this rule of thumb, I'll give an example. When I notice code being structured with class inheritance, I'll point out a part of the codebase where I did similar, and how poorly it turned out, which someone can observe for themself. It started out nice, as it usually does, but devolved over time as complexity accumulated in parent classes to handle scenarios in child classes.
This is very insightful, thanks. I'm currently in the middle ground where I still want mentoring from the technical leads on some things, but I also have enough seniority to be mentoring new hires. You've articulated the way I think our tech leads are often not great at explaining themselves, while also giving me something to think about when passing on information to newbies.
I think almost all knowledge is tacit knowledge. Even stuff that would seem obviously explicit like mathematics and language.
In my experience tutoring high school students, I’ve encountered plenty of students who struggle with spelling, grammar, or mathematical mistakes and no matter how many times I correct them it doesn’t stick. Then I see other students who pick these things up so quickly without correction.
Mathematics, especially, seems to have a strong tacit component. If all you do is read the textbook you’re going to bomb the exam. You need to practice a lot to develop your mathematical intuition.
> I’ve encountered plenty of students who struggle with spelling, grammar, or mathematical mistakes and no matter how many times I correct them it doesn’t stick. Then I see other students who pick these things up so quickly without correction.
I often wonder about this. I believe that each person has a certain intellectual threshold beyond which they cannot progress farther in (memory, spatial reasoning, logical reasoning, learning speed, etc.) Logically there are genetic and natural components to this threshold. I also think part of what makes a lot of modern life so disastrously ineffective is the belief by those running society that everyone can reach a similar intellectual level given the right "system".
However, I think modern schooling is so brutally ineffective that barely anyone in school is operating near the real limits of their intellect. What do you think was holding those kids back? Lack of motivation, poor knowledge of fundamentals, or do you think basic high school concepts actually had maxed out their intellect?
This article confuses tacit knowledge for the phenomenon most eloquently described by a former (superb) sysadmin colleague “documentation is the enemy of job security”.
Is this true, though? After seeing enough bone-headed management decisions, I wonder if the bosses even know or care about documentation when making decisions to fire someone.
From my point of view, the lack of documentation might be a reason to fire someone.
This[1] article that was on HN front a few months ago and I could only agree with it.
I've seen teams trying to micro-document their way to inefficiency. Forcing doers to document every darned small thing is the surest way to get them to lose interest. By now I've realised that regardless of documentation it takes around 6 months for a s/w engineer to be fully productive at a new company and start creating value.
Thanks, this is the exactly correct critique of the article.
I'll just add that the author did not seriously grapple with the fact that leaving at least some explicit knowledge undocumented is inevitable. There is no sharp divide between "explicit knowledge every programmer should know" and "explicit knowledge that only makes sense in our organization". Documenting is costly to produce, and documentation is costly to maintain even after it is produced. (If nothing else, more documentation makes it harder to search.) This holds even if it's simultaneously true that large organizations usually document insufficiently because of laziness and bad incentives.
I agree that some is inevitable, but as that amount increases it will provide a drag in what the organization is able to accomplish. This becomes even worse as an organization is remote locally and globally spread out.
This is so meta. Here we are trying to verbalize our tacit knowledge of "tacit knowledge" :)
While I agree with the distinction you made (Last paragraph is excellent), there is this grey area : Because in the digital era almost everything is recorded (oral meetings/written chitchat...), there is a huge quantity of information that is neither in the "Guts" nor in the "docs". To join others in this thread, I guess LLMs will play a big part in making this grey area usefull and in a certain way bridge the gap (partially) between what is tacit and what is formally documented.
Exactly. However I think its a bit more complicated than a grey area. There is just no hard distinction between tacit and explicit knowledge, even though the distinction is useful.
It is not that any tacit bit of knowledge eludes articulation, it is the whole that is problematic. Nothing is truly undocumentable, but its like the specs for microsoft word: its just so much you already need to be an expert to navigate the whole of it.
Thats the distinction and LLM of course of very good at processing vast amounts of bits and pieces.
It's worth noting a tautology: tacit knowledge is undocmented because it is undocumentable.
The key insight (which I agree with!) is that some kinds of knowledge are undocumentable. The challenge - at least for those charged with ensuring processes are documented - is deciding what is and what isn't documentable.
I have a lot of habits because of lessons I've learned. There are 100 ways to do it, I chose 1. Documenting the 99 is unreasonable, but that's tacit knowledge, isn't it? I will document why I didn't choose the most obvious one, though.
It's a definition. I suppose definitions are tautologies by definition though :)
And yeah, the real insight is not the definition but the fact that tacit knowledge is inevitable and important, and that trying to eliminate it will be actively counterproductive.
Documentation makes the world go round. Definitely appropriate for
high-risk must-deliver projects, but not in all contexts. Is it maybe
a business fantasy that we can simply juice the talent of a team and
bottle it?
Where does this idea come from other than good engineering practice?
If you work with smart, talented people in a fast moving world it
seems, as you say, inevitable that you face the "problem" of tacit
knowledge and manage it positively.
In Designing Sound (and my lectures on sound design) I researched and
wrote about this as "ineffable knowledge". My observation in arts and
design through the 90s and early 00s was that we regularly get people
who are x100 superstars but they're unable to articulate exactly how
they do things and pass that knowledge to others (I was researching
this from a UI/UX point of view). That is /talent/ and it's why we
value people as more than just interchangeable cogs. Much process
knowledge, and crucially problem-solving knowledge in any project will
reside in the heads of individuals.
(TBF I think the article is more about tacit cultural knowledge within
the team as a whole)
Sure if we lose key individuals the project is threatened, but on the
other hand trying to get them to do a full knowledge dump, such that
any replacement would be frictionless, is a fools errand.
Treating "information" that way and ignoring people is one-dimensional
thinking.
So I don't think it's helpful to talk about tacit knowledge itself as
"dangerous". What is more dangerous is composing mediocre teams
only from replaceable workers and spending too much of their time on
documenting as a safety net when they should be pushing
forward. Meanwhile a danger is in treating valuable people so badly
their continuity is not assured. Far fewer people fall under the
proverbial bus than simply leave for better jobs.
So we could take the view that over-cautious self-reporting and
documenting is a sign of, and attempt to manage, low commitment
business attitudes, poor wages, conditions and low job-security.
Documentation is one of those things that once there's a lot of it, it's its own product.
First, you have to have someone constantly making sure it's up to date in the case of software. Really easy for it to go stale and just be wrong, and possibly actively harmful.
But then telling someone "RTFM" has a different quality if you're talking about 50 pages or 50,000 pages. Cool, I'll get back to you in a month.
Donald Schön’s work on this topic is really enlightening. It’s not just that there’s knowledge in the heads of people that can’t be linguistically expressed well, but also that expression of it requires interaction with a specific situation.
This is also represents a massive gap for AI systems to become actual in-the-world problems solvers.
Was just arguing with my father whether the difficulty in transplanting TSMC was due to tacit knowledge or missing documentation. I argued that an automated process failing to run would inherently be missing documentation but I'm curious to see what kind of tacit information would be involved here.
At a previous company where I had a lot of this "tacit knowledge", I had made a policy for myself. Whenever somebody asked me a question over IM or e-mail, I would look in the wiki. If the answer was not in the wiki, I would write a new wiki article or update an existing article with the answer, and I would respond with a link to the article, and a statement like "if this doesn't answer your question let me know what is missing or could be described better." The reasons for this were for me to dump my tribal knowledge out of my brain into something everybody could search and read, and to promote usage of the wiki as a place where people found answers. Unfortunately it did not appear to have another desired effect of getting everybody to contribute content. I think this was probably because people came to see the wiki as a place to find knowledge, not to put knowledge.
How do you keep your wiki from evolving into a kludge of random-ish info?
One example of tacit knowledge is knowing the Dell T420s Tower Servers have very poor cooling for the raid controllers and often, this can cook them over time, requiring replacement or, at least repasting the heatsink.
Now imagine 100s of those little tidbits. Useful, but hard to categorize.
That's what a wiki is good for: collecting snippets of hard to categorize information. If you've got a culture of recording those notes, then you can develop full text search habits.
A kluge of randomish info by people who make occasional attempts to organize and garden... sucks compared to a beautifully manicured knowledge base. But it's fantastic compared to a blank wall and a shrug.
> One example of tacit knowledge is knowing the Dell T420s Tower Servers have very poor cooling for the raid controllers and often, this can cook them over time, requiring replacement or, at least repasting the heatsink.
I don't even think that is tacit knowledge. Tacit knowledge is how do you know when the edge of your chisel is too dull and need re-honing. (It feels off.) Or knowing how much weight transfer you need to safely navigate a given turn with a motorcycle.
Or to grab a more similar example: Tacit knowledge is looking at a computer case you have never seen before and thinking "that doesn't look like enough cooling, we should calculate and check".
Put it in the runbook for the alarm you got when the server failed, or the maintainance guide that your team runs through every month. Or make heatsink repasting a regular documented process, or something similar.
Basically, as Toyota says, build quality _into_ each part of the product rather than layering it on top by documenting discrete pieces of info that an operator has to discover, understand and act upon
I don't want to get too much on the ai hype train - but it seems like this would be perfect for machine learning, trained on the wiki-snippets of the company and emails (initiator would have to set a flag saying this is going to be training data).
Use an internal Q&A / Quora-style tool with good search. Trying to keep little snippets of info like that under useful MECE categories only ever ends in pain & frequent refactoring.
Not OP, but you have to regularly review and adjust the ontology regularly as more articles are added.
In short, the categorization and organization is key, and you have to do it when the number of documents is not only increasing, but different enough.
Sometimes, I just write a longer document and once it's ready, it can be broken into smaller ones. Doing this helps save the organization tremendous amounts of time in not having to re-learn things, or dig it up.
That's what I did at my last job, now I mostly feel like that was a mistake. I was the guy contributing 90% of the wiki knowledge, for no real appreciation or incentive. And when something I posted inevitably become out of date and caused a problem for someone, guess who got blamed? I feel like I was enabling the team's bad habits.
At $NEW_JOB I'm just putting everything into my Obsidian repo. It's great because I have complete freedom to refactor my notes however and whenever I want. When someone has a question for me, I just look up the answer in my notes and send it over to them. Then they'll copy it into their OneNote, and everyone is satisfied.
This isn't my ideal world, but this seems to be the state of equilibrium when management doesn't lay down any expectations or incentives.
Interesting story about where blame is placed, and your move to Obsidian. Thanks for sharing.
I've actually done something similar to your Obsidian repo: I put all my notes into markdown and publish them with mkdocs[1]. This gives me the same editorial control and freedom from blame as your notes, but mine are public by default, so I can still simply reply with a link. I just can't post anything that is sensitive, which hasn't been a significant drawback yet. I have considered moving to an Obsidian based system, but have yet to find something compelling enough to make the switch. It looks like an awesome app though and is still on my radar.
I often create wiki pages to save myself the work of explaining it multiple times to different people. Ultimately, it's win-win. I don't have to put in more effort to explain something, and the requestor has the necessary information.
If other people don't contribute to the wiki then it's likely that there's nothing incentivizing it. It's not like management is handing out bonuses to people contributing to the wiki. Heck, even my contributions are self-serving.
Maintaining wiki requires a lot of time. Even just adding new content is much more work than a reply over IM/email because one needs to find a right place (if stricture is not completely flat), explain the context (which people exchanging messages have but would be missing for someone who would read wiki), organize content so it has some internal structure e. t. c. And there is a bigger problem - wiki to which new pages frequently added typically full of outdated or no longer relevant information. Updating or archiving old wiki pages requires a lot of time too (and this work sometimes can be done only by an author or someone who knows subject equally well).
It is nice to have knowledge documented, but there is a tradeoff between time spend on wiki and time saved thanks to the wiki.
I know many people love video, but I personally think this is only slightly better than not recording it anywhere. Video is much harder to search and nearly impossible to skim, so it's difficult to use for anything other than a classroom-style "let's sit down and learn" session.
Most often I'm looking for some particular piece of information. Finding a video that _may_ contain the answer _somewhere_ in the hour (or whatever) of information just means now I have to decide on the likelihood that it'll be worth trying to watch.
Yes, some things are better explained in video format. But not very many, especially in knowledge work.
ETA: Also video is much harder to update as the situation changes. For the most part you just need to re-record the whole thing, which probably isn't going to happen.
Per other comments here, this is not what tacit knowledge is. There is an entire body of research devoted to understanding tacit knowledge. Anyone interested in tacit knowledge can read the works of those who have spent years researching the topic. Here are some sample papers. Nonaka is perhaps the most well-known one in this space.
Hadjimichael, D., and Tsoukas, H. 2019. “Toward a Better Understanding of Tacit Knowledge in Organizations: Taking Stock and Moving Forward,” Academy of Management Annals (13:2), pp. 672–703. (https://doi.org/10.5465/annals.2017.0084).
Huang, X., Hsieh, J. P.-A., and He, W. 2014. “Expertise Dissimilarity and Creativity: The Contingent Roles of Tacit and Explicit Knowledge Sharing,” Journal of Applied Psychology (99:5), pp. 816–830.
Nonaka, I., and Von Krogh, G. 2009. “Perspective—Tacit Knowledge and Knowledge Conversion: Controversy and Advancement in Organizational Knowledge Creation Theory,” Organization Science (20:3), pp. 635–652. (https://doi.org/10.1287/orsc.1080.0412).
Nonaka, I. 1994. “A Dynamic Theory of Organizational Knowledge Creation,” Organization Science (5:1), pp. 14–37.
Thanks I appreciate this. I know I co-inflated different concepts. I should update my blog post once the comments here dry up and I can go through them.
> Tacit knowledge, often called “tribal knowledge” in tech, is prevalent in this industry.
Tacit knowledge is not the same as tribal knowledge. Tribal knowledge is undocumented stuff. Tacit knowledge is about things that cannot be learned from documentation.
Riding a bike is an example of tacit knowledge. No amount of reading about the theory of riding a bike will teach you how to ride a bike. You have to hop on the bike and fail a few times until you figure it out.
In software engineering, tacit knowledge are things like "How to factor a system so 50 engineers can work together without stepping on each other's toes" and "How to structure your files to naturally reduce architectural complexity" and "How do you write a useful test that isn't just writing tests for the sake of tests". You can read about these things all you want, but you can only learn by experience. You have to get it wrong a bunch of times, see how it was wrong, and then you eventually start doing it less wrong.
Learning by example also works to speed up the process.
I wrote a very similar comment and you beat me to it by one minute! Deleted and reposted below for compactness.
Their usage of “tacit knowledge” is different from the most common usage. Tacit knowledge ≠ tribal knowledge. Tacit knowledge isn’t knowledge that no one has bothered to write down. It’s knowledge that is inherently difficult to write down. Riding a bike is tacit knowledge. Knowing when a class is doing “too much” is tacit knowledge. Knowing that the internal account must be created before the external account is not, but it’s the sort of thing that they’re talking about.
I wouldn’t make such a big deal about semantics but it’s the title and the first few sentences of the blog post. I expected something very different than “tribal knowledge is bad”.
Thanks for the comments everyone. You're not the first ones to give me feedback about co-inflating the two. In my mind saying that something is "hard to" write down is still pretty similar to saying it is not written down.
For one, if you say it's hard to write something down, you're setting yourself up for a scenario where judging outcomes based on that lack of written down knowledge is going to be painful. For example: "It's hard to describe how to write good tests, so we didn't write it down. But we also want our tests to be more reliable." How do you solve the two without writing down some way to learn how to be better at tests?
Likewise, if you want to learn how to ride a bike, how do you get started? Back in the pre-covid days you could do a lot of experimentation and learning from others in person, but that doesn't scale as well in the post-covid, remote and global, workplace. That's one of the core concepts I was trying to capture.
This looks like a really good use case for LLMs. When you're small and starting out in person, record everything the dev team does -- meetings, whiteboads, pairing sessions, etc. (yeah, creepy, but I'm in brainstorming mode) and then as you start to scale beyond the ability to "ask someone who knows" trail up an LLM to serve as the 24x7 lore master.
Ah, nice, like feeding a custom GPT everything that goes on at the company. That seems dangerous to me, and not in the typical LLM weariness way.
Meetings can be messy, full of ambiguity, and "unsolved" at then end. If you feed a number of these experiences into an LLM, would its knowledge base be _that great_ at helping teach somebody about the company holding said meetings?
I'd argue that in order for your LLM suggestion to be effective, the input to it would need to be human-written content. I.E: the knowledge that the OP is discussing. In that case, the human effort is still required up front.
It would be interesting to see if you could train a LLM, given "<A> The foo is definitely xyfied" to exclusively answer "Well, at least two months ago, A thought that the foo was xyfied" rather than opine about foo directly.
This is very much like my panoptic computronium cathedral™. I only need $80B to build it to achieve AGI. The AGI requires 24/7 surveillance for making optimal decisions because it needs access to as much information as possible. The second stage will require worshipers to ingest nanosensors so that even their bowel movements can be tracked and analyzed by the AGI. This is all for the benefit of those who are under constant surveillance in the cathedral.
I used to work with a smart guy who had what I think is the real answer to this problem. All documentation is maintained by the newest member of the team. The new guys job is to solve all of his issues with the documentation and if he has to ask someone then his job is to update the documentation.
This is an odd definition for tacit knowledge. My understanding is that it's more caught up in the intuition of concepts/systems that's difficult to codify because it's very contextual. This "tribal knowledge" perspective seems more like processes and facts that aren't documented. Not because of the inherent difficulty but instead because of priorities. Not that there isn't some intersection between the two.
Well, as in all of these absolutist black/white, binary statements, "it depends."
In my experience, tribal knowledge, which also includes cultural inculcation, can be the difference between success and failure. Sometimes, on the success side, and, sometimes, on the failure side.
The main thing about "tribal knowledge," is that it relies on long-term association/employment, and that is not something we see much of, these days.
Surely there are tribes that transcend and cross corporations. That's why you can hire old colleagues and get a team up and running in a startup or new firm
Readit News
Tacit knowledge is not the same as explicit knowledge that just happens to not be documented. Tacit knowledge is not documented because it is undocumentable. You cannot avoid tacit knowledge. Human language fundamentally cannot express all the knowledge that experts develop through years of practice and experience. Even if an expert can express their experience, there are simply ideas and skills that people cannot learn exclusively from words.
This is an important distinction. Trying to eliminate undocumented explicit knowledge is useful for teams. Trying to eliminate tacit knowledge is disastrous. I've seen attempts to make tacit knowledge legible to management—it is one of the most direct ways to sabotage your own experts because experts fundamentally cannot make all their knowledge explicit. Expertise is tacit knowledge and tacit knowledge is expertise.
People generally understand that design by committee or design by regulation has awful results; one of the main reasons for this is that committees require decisions to be fully explicit and explainable—which hamstrings creative problem-solving and pushes for poor design decisions. Exclusively explicit processes get you lowest-common-denominator results: not the lowest common denominator of the expertise present, but the lowest common denominator of what people can explicitly communicate and understand, which is even lower! If you don't let experts be experts, you won't get expert-level work.
When giving advice, I'm often tempted to point to a given situation, and to explain what rule of thumb I used to solve it.
But then I would see a junior programmer applying the rule of thumb in a completely different situation and I'd want to say, "No, that will end in tears!" And I realized that a lot of my rules of thumb were really dumb when taken out of context.
As far as I can tell, a huge part of expertise is having a large set of mappings like:
- In contexts like "C1", use rule of thumb "R1" or "R8".
It's very tempting to talk about R1, R2, etc. You can write blog posts like, "How R2 will transform your team!"
But a lot of real expertise comes from being able to say, "Oh, this is a combination of C7 and C78, so I should consider rules of thumb R7, R49 and R102. Ah, wait, no, R49 is just gross here."
So I've learned that, when mentoring, I need to spend a lot more time talking about the context before proposing any rules of thumb.
In my experience tutoring high school students, I’ve encountered plenty of students who struggle with spelling, grammar, or mathematical mistakes and no matter how many times I correct them it doesn’t stick. Then I see other students who pick these things up so quickly without correction.
Mathematics, especially, seems to have a strong tacit component. If all you do is read the textbook you’re going to bomb the exam. You need to practice a lot to develop your mathematical intuition.
I often wonder about this. I believe that each person has a certain intellectual threshold beyond which they cannot progress farther in (memory, spatial reasoning, logical reasoning, learning speed, etc.) Logically there are genetic and natural components to this threshold. I also think part of what makes a lot of modern life so disastrously ineffective is the belief by those running society that everyone can reach a similar intellectual level given the right "system".
However, I think modern schooling is so brutally ineffective that barely anyone in school is operating near the real limits of their intellect. What do you think was holding those kids back? Lack of motivation, poor knowledge of fundamentals, or do you think basic high school concepts actually had maxed out their intellect?
> students who struggle with spelling
The ability to spell correctly is not tacit knowledge.
>> Tacit knowledge is not documented because it is undocumentable
Spelling is thoroughly documented.
It’s quantifiers, and bound and free variables.
This article confuses tacit knowledge for the phenomenon most eloquently described by a former (superb) sysadmin colleague “documentation is the enemy of job security”.
Only when you become replaceable at a certain task (or able to delegate it) you can progress to do new (more interesting) things.
From my point of view, the lack of documentation might be a reason to fire someone.
This[1] article that was on HN front a few months ago and I could only agree with it.
I've seen teams trying to micro-document their way to inefficiency. Forcing doers to document every darned small thing is the surest way to get them to lose interest. By now I've realised that regardless of documentation it takes around 6 months for a s/w engineer to be fully productive at a new company and start creating value.
[1] https://commoncog.com/tacit-knowledge-is-a-real-thing/
I'll just add that the author did not seriously grapple with the fact that leaving at least some explicit knowledge undocumented is inevitable. There is no sharp divide between "explicit knowledge every programmer should know" and "explicit knowledge that only makes sense in our organization". Documenting is costly to produce, and documentation is costly to maintain even after it is produced. (If nothing else, more documentation makes it harder to search.) This holds even if it's simultaneously true that large organizations usually document insufficiently because of laziness and bad incentives.
While I agree with the distinction you made (Last paragraph is excellent), there is this grey area : Because in the digital era almost everything is recorded (oral meetings/written chitchat...), there is a huge quantity of information that is neither in the "Guts" nor in the "docs". To join others in this thread, I guess LLMs will play a big part in making this grey area usefull and in a certain way bridge the gap (partially) between what is tacit and what is formally documented.
It is not that any tacit bit of knowledge eludes articulation, it is the whole that is problematic. Nothing is truly undocumentable, but its like the specs for microsoft word: its just so much you already need to be an expert to navigate the whole of it.
Thats the distinction and LLM of course of very good at processing vast amounts of bits and pieces.
The key insight (which I agree with!) is that some kinds of knowledge are undocumentable. The challenge - at least for those charged with ensuring processes are documented - is deciding what is and what isn't documentable.
And yeah, the real insight is not the definition but the fact that tacit knowledge is inevitable and important, and that trying to eliminate it will be actively counterproductive.
Where does this idea come from other than good engineering practice?
If you work with smart, talented people in a fast moving world it seems, as you say, inevitable that you face the "problem" of tacit knowledge and manage it positively.
In Designing Sound (and my lectures on sound design) I researched and wrote about this as "ineffable knowledge". My observation in arts and design through the 90s and early 00s was that we regularly get people who are x100 superstars but they're unable to articulate exactly how they do things and pass that knowledge to others (I was researching this from a UI/UX point of view). That is /talent/ and it's why we value people as more than just interchangeable cogs. Much process knowledge, and crucially problem-solving knowledge in any project will reside in the heads of individuals.
(TBF I think the article is more about tacit cultural knowledge within the team as a whole)
Sure if we lose key individuals the project is threatened, but on the other hand trying to get them to do a full knowledge dump, such that any replacement would be frictionless, is a fools errand.
Treating "information" that way and ignoring people is one-dimensional thinking.
So I don't think it's helpful to talk about tacit knowledge itself as "dangerous". What is more dangerous is composing mediocre teams only from replaceable workers and spending too much of their time on documenting as a safety net when they should be pushing forward. Meanwhile a danger is in treating valuable people so badly their continuity is not assured. Far fewer people fall under the proverbial bus than simply leave for better jobs.
So we could take the view that over-cautious self-reporting and documenting is a sign of, and attempt to manage, low commitment business attitudes, poor wages, conditions and low job-security.
First, you have to have someone constantly making sure it's up to date in the case of software. Really easy for it to go stale and just be wrong, and possibly actively harmful.
But then telling someone "RTFM" has a different quality if you're talking about 50 pages or 50,000 pages. Cool, I'll get back to you in a month.
Donald Schön’s work on this topic is really enlightening. It’s not just that there’s knowledge in the heads of people that can’t be linguistically expressed well, but also that expression of it requires interaction with a specific situation.
This is also represents a massive gap for AI systems to become actual in-the-world problems solvers.
One example of tacit knowledge is knowing the Dell T420s Tower Servers have very poor cooling for the raid controllers and often, this can cook them over time, requiring replacement or, at least repasting the heatsink.
Now imagine 100s of those little tidbits. Useful, but hard to categorize.
A kluge of randomish info by people who make occasional attempts to organize and garden... sucks compared to a beautifully manicured knowledge base. But it's fantastic compared to a blank wall and a shrug.
I don't even think that is tacit knowledge. Tacit knowledge is how do you know when the edge of your chisel is too dull and need re-honing. (It feels off.) Or knowing how much weight transfer you need to safely navigate a given turn with a motorcycle.
Or to grab a more similar example: Tacit knowledge is looking at a computer case you have never seen before and thinking "that doesn't look like enough cooling, we should calculate and check".
Basically, as Toyota says, build quality _into_ each part of the product rather than layering it on top by documenting discrete pieces of info that an operator has to discover, understand and act upon
In short, the categorization and organization is key, and you have to do it when the number of documents is not only increasing, but different enough.
Sometimes, I just write a longer document and once it's ready, it can be broken into smaller ones. Doing this helps save the organization tremendous amounts of time in not having to re-learn things, or dig it up.
At $NEW_JOB I'm just putting everything into my Obsidian repo. It's great because I have complete freedom to refactor my notes however and whenever I want. When someone has a question for me, I just look up the answer in my notes and send it over to them. Then they'll copy it into their OneNote, and everyone is satisfied.
This isn't my ideal world, but this seems to be the state of equilibrium when management doesn't lay down any expectations or incentives.
I've actually done something similar to your Obsidian repo: I put all my notes into markdown and publish them with mkdocs[1]. This gives me the same editorial control and freedom from blame as your notes, but mine are public by default, so I can still simply reply with a link. I just can't post anything that is sensitive, which hasn't been a significant drawback yet. I have considered moving to an Obsidian based system, but have yet to find something compelling enough to make the switch. It looks like an awesome app though and is still on my radar.
1. https://danielhoherd.com/tech-notes/exiftool/
If other people don't contribute to the wiki then it's likely that there's nothing incentivizing it. It's not like management is handing out bonuses to people contributing to the wiki. Heck, even my contributions are self-serving.
It is nice to have knowledge documented, but there is a tradeoff between time spend on wiki and time saved thanks to the wiki.
The explanation by IM/em can be troublesome because it could be out of date.
Helping your organization be creators is the way to go.
Most often I'm looking for some particular piece of information. Finding a video that _may_ contain the answer _somewhere_ in the hour (or whatever) of information just means now I have to decide on the likelihood that it'll be worth trying to watch.
Yes, some things are better explained in video format. But not very many, especially in knowledge work.
ETA: Also video is much harder to update as the situation changes. For the most part you just need to re-record the whole thing, which probably isn't going to happen.
Hadjimichael, D., and Tsoukas, H. 2019. “Toward a Better Understanding of Tacit Knowledge in Organizations: Taking Stock and Moving Forward,” Academy of Management Annals (13:2), pp. 672–703. (https://doi.org/10.5465/annals.2017.0084).
Huang, X., Hsieh, J. P.-A., and He, W. 2014. “Expertise Dissimilarity and Creativity: The Contingent Roles of Tacit and Explicit Knowledge Sharing,” Journal of Applied Psychology (99:5), pp. 816–830.
Nonaka, I., and Von Krogh, G. 2009. “Perspective—Tacit Knowledge and Knowledge Conversion: Controversy and Advancement in Organizational Knowledge Creation Theory,” Organization Science (20:3), pp. 635–652. (https://doi.org/10.1287/orsc.1080.0412).
Nonaka, I. 1994. “A Dynamic Theory of Organizational Knowledge Creation,” Organization Science (5:1), pp. 14–37.
Tacit knowledge is not the same as tribal knowledge. Tribal knowledge is undocumented stuff. Tacit knowledge is about things that cannot be learned from documentation.
Riding a bike is an example of tacit knowledge. No amount of reading about the theory of riding a bike will teach you how to ride a bike. You have to hop on the bike and fail a few times until you figure it out.
In software engineering, tacit knowledge are things like "How to factor a system so 50 engineers can work together without stepping on each other's toes" and "How to structure your files to naturally reduce architectural complexity" and "How do you write a useful test that isn't just writing tests for the sake of tests". You can read about these things all you want, but you can only learn by experience. You have to get it wrong a bunch of times, see how it was wrong, and then you eventually start doing it less wrong.
Learning by example also works to speed up the process.
But that's not the same as undocumented.
Their usage of “tacit knowledge” is different from the most common usage. Tacit knowledge ≠ tribal knowledge. Tacit knowledge isn’t knowledge that no one has bothered to write down. It’s knowledge that is inherently difficult to write down. Riding a bike is tacit knowledge. Knowing when a class is doing “too much” is tacit knowledge. Knowing that the internal account must be created before the external account is not, but it’s the sort of thing that they’re talking about.
I wouldn’t make such a big deal about semantics but it’s the title and the first few sentences of the blog post. I expected something very different than “tribal knowledge is bad”.
For one, if you say it's hard to write something down, you're setting yourself up for a scenario where judging outcomes based on that lack of written down knowledge is going to be painful. For example: "It's hard to describe how to write good tests, so we didn't write it down. But we also want our tests to be more reliable." How do you solve the two without writing down some way to learn how to be better at tests?
Likewise, if you want to learn how to ride a bike, how do you get started? Back in the pre-covid days you could do a lot of experimentation and learning from others in person, but that doesn't scale as well in the post-covid, remote and global, workplace. That's one of the core concepts I was trying to capture.
Meetings can be messy, full of ambiguity, and "unsolved" at then end. If you feed a number of these experiences into an LLM, would its knowledge base be _that great_ at helping teach somebody about the company holding said meetings?
I'd argue that in order for your LLM suggestion to be effective, the input to it would need to be human-written content. I.E: the knowledge that the OP is discussing. In that case, the human effort is still required up front.
In my experience, tribal knowledge, which also includes cultural inculcation, can be the difference between success and failure. Sometimes, on the success side, and, sometimes, on the failure side.
The main thing about "tribal knowledge," is that it relies on long-term association/employment, and that is not something we see much of, these days.
But there is also a great deal of extremely mercenary behavior, in today's workplace (not just tech).
I suspect that it would be hard to have contiguous relationships.
I worked at my last company for 27 years. I knew many folks for decades.
I believe that this kind of thing is ... uncommon ... these days.
PS. I loved the page you put up of your dad a few days ago and mentioned in the thread about diplomacy. Very interesting person.