Readit News logoReadit News
iancmceachern · 3 years ago
My parents owned anf ran a jewelry store for 35 years, nearly their entire working life. My Dad was a master jeweler, goldsmith, sculptor. Among many things, they would string and re-string pearls for folks. Usually family heirlooms that they wanted to keep, but also be able to wear and not fear they'd break and fall all.over the floor. After years of doing that, and also buying raw oearls from overseas, stranding them, and then selling them, my Dad asked the supplier, do yoi guys do stranding? Yes, yes they did, for free. He could have been ordering pre-strung pearls for the same price the whole time. Stranding pearls takes time.
bombcar · 3 years ago
One of the questions I like to ask contractors before beginning work is "what changes would make this cheaper/easier for you?" Often simple things like moving a planned wall an inch or switching the location of something can make it vastly easier; or there will be things they can get preassembled or done for free.

And sometimes the factory building whatever has a giant machine that can easily do what would take you quite a long time to do by hand.

thehappypm · 3 years ago
It’s funny. They seem to be two kinds of contractors out there. Type one will push as hard as possible for you to design things in a way that makes it easier for them. To the point where there are all sorts of design compromises like a beam in the middle of your kitchen. The second type is the one who wants to deliver exactly the vision of the client, to the point where they are adding a huge amount of complexity to perfectly meet unreasonable demands.
lkois · 3 years ago
I used to work in a place that assembled machines based on old designs that rarely got reviewed, since if it ain't broke don't fix it.

When working on one particular unit, I found that they would order a bunch of plastic panels, cut to size, and some with precut holes. And others with secondary drawings, instructing us how to drill out a bunch of holes ourselves. Sometimes not even all the holes, just a few, in addition to the precut.

When I asked why we didn't just ask the supplier to cut out ALL the holes, I basically got a shrug.

So I merged the secondary drawings and sent them to the supplier, and the assembly team never wasted time on that nonsense again.

HeyLaughingBoy · 3 years ago
It's funny how that works. I have a small electronics/firmware consulting side business. I used to make up wire harnesses for a customer, buying the wire, cutting to length and stripping insulation, crimping on connectors, etc. It was slow and my quality wasn't the greatest.

Then I looked around and found a company that would provide me the correct length of wire cut to size with the right terminals crimped in place all for less than I could buy just the bulk wire in the first place with no setup costs. Minimum order, IIRC was 100 pieces, which is peanuts.

Blew my mind!

At least I only struggled for a month or two before realizing that there had to be an easier way :-)

powersurge360 · 3 years ago
I am becoming increasingly skeptical that good internal documentation is even possible. I've been working in software development for around 12 years and have _never_ seen it done well and asking around the best I've heard offered up is the equivalent of an internal stack overflow.

For a while, I thought that maybe an exception would be having a technical writer on staff but after reading this post I'm significantly disheartened on that front too. I'd be interested to hear if anyone on hacker news has experienced good internal documentation and even more interested if any of you folks have experienced anything truly _great_.

For my contribution, I've found that documentation fails for a couple of reasons. The first is the burden of correctness. The people who most would like documentation are also the people who most need the documentation and they usually are also the people most likely to be reluctant to contribute to the documentation because they don't feel they can accurately represent the information. Imagine someone ramping into a feature and spending a few days reverse engineering how it works, collecting info, etc. Sometimes they'll put it up somewhere but a lot of the time contributing partial information feels 'wrong'.

And the second bit I find to be a big reason why documentation efforts fail is just the sheer friction of putting it into the documentation store to begin with. In confluence, for example, if you have a bit of information it can be tough to work out how to categorize it, where in the hierarchy it should go, etc, etc. Or if it's a GitHub wiki you want to put it somewhere that it is discoverable but also be careful that it's 'correct' because you don't want to break backlinks if it gets recategorized.

I've mostly given up on it at this point. Instead, I take detailed personal notes and make them publicly available. It doesn't have to be correct because being advertised as personal notes means that it's my opinion on the truth rather than objective fact. It isn't far away from my codebase, I can just tap a short keybinding in my editor to type my notes or search them and I can link directly from the notes to lines of code in files to jump back and forth. The particular system I use means that if I write a short snippet of code to solve a one off issue (like calling a path helper to derive a URL that I can't find in the interface) I can even drop it in a code block and execute it right from my note-taking tool. It isn't ideal for sure but I've gotten way farther in having a shareable knowledge base this way than I have in literal years of trying to get a shared, useful documentation store spun up.

P5fRxh5kUvp2th · 3 years ago
The issue is that people seem to equate "good documentation" with complete.

It's just not possible to do this. I think it would be better to talk about "effective documentation".

My opinion on the matter:

Three levels of documentation

1. high level documentation that describes the problems and the goals from the business perspective

2. mid-level documentation that describes the architecture that gets us there

3. low-level documentation, aka code.

high level gives you direction (where), mid-level gives you context (what), low-level gives you implementation (how)

You need the what to understand the how, and you need the where to understand the what.

The other piece is an acceptance that you can't document away the need for tacit knowledge.

To draw an analogy.

No amount of documentation is ever going to allow a kid to hop on a bike and ride it perfectly the first time. That is not what it means to teach a child to ride a bike. Instead the goal is to have enough documentation to minimize the amount of time it takes that child to gain the tacit knowledge necessary to ride a bike acceptably.

Once you accept the above and lower your bar, "effective documentation" becomes much more achievable.

froh · 3 years ago
yes. and the hard part is keeping the three levels in sync.
strken · 3 years ago
I'm convinced that the only good documentation is generated from code or config, and has a source which is necessary for the system to run.

A tool that tells you who commits most frequently to a service or file is better than an outdated yaml file with an owner. A firewall that won't let your service make outgoing requests unless you list its dependencies is better than a manual list of dependencies. An API doc compiled from the type definition of your API or from a schema which requests are validated against is better than a manual API definition.

pydry · 3 years ago
I've never seen it done properly either, but I think it's more a matter of incentives than structure or tools. The places where it was worst were places where people's reaction to wrong documentation tended to be a shrug.

Sometimes there were people who didn't document coz they kind of liked being the font of all knowledge.

Confluence also sucks - even harder than jira, if that's possible.

AtlasBarfed · 3 years ago
Well the real problem is: What's good documentation?

I'll settle for examples.

Documentation is a multi-headed hydra where you have different levels of technical competency and foreknowledge and familiarity with the programming environment patterns in use the organizations, particular technical stack organization, the people inside of it, what the product does, etc etc etc

Cthulhu_ · 3 years ago
This is why IMO in any nontrivial organization you need a dedicated technical writer - if only someone that sorts out the wiki and keeps it clean.
specialist · 3 years ago
In my future perfect world, every artifact, every service, every everything will have a TTL. Expired stuff gets culled, in stages.
austinwade · 3 years ago
I like this approach. What is this note-taking tool? I want to use it. Also linking to code from the docs sounds really nice.
stavros · 3 years ago
I don't know about the GP, but I wrote a simple script that will make a mdBook site out of Joplin notes:

https://gitlab.com/stavros/notes/

powersurge360 · 3 years ago
It's org mode in emacs with the org-roam plugin. If I wasn't using emacs I would probably have a folder full of markdown files instead and leverage a project-based grep to search it. The real power in the system is that it's close to your editor and personal so you can iteratively build up your knowledge.

In particular, every GitHub wiki belonging to an organization or a public project itself is a git repo of markdown files that you can clone and commit to. I probably would use that for per project documentation w/ a separate folder for managing broader notes. I think it's important to separate the markdown from a project's code because otherwise you can have important markdown updates trapped in a branch that hasn't been merged yet and because git works by line rather than by word you can get into complicated merge conflicts by features that touch similar business concepts.

Org mode happens to add some niceties on top in that you can have an example code block (roughly comparable to a triple back-tick code block in markdown) that also can be executed. The example code block can take some arguments like what interpreter to use and will paste the output into another code block underneath the source code block.

Because org mode is also used for organizing tasks it is easy to use it as a scratch pad as you grind through tasks and if something turns out to be useful you can promote it to a top level note (or org-roam node). And even if it's not obvious that it's a note candidate, just thinking through problems 'out loud' in org means you can search and find it later when the same or an adjacent problem comes up.

Another favorite feature I like in org mode is that in any code at any time I can tap out 'SPC n l' and it will capture a reference to the file and the line in the file and allow me to link to it in my notes. It doesn't capture by line number but by copying the literal line, which means that as feature updates push the line number up and down, it still will be able to find exactly the right file in exactly the right line as long as the line's content hasn't changed. It also serves as a rather nice canary because if the docs link to a line that no longer exists, then the documentation has probably gone stale.

The last major win with org mode vs markdown is with org-roam, which borrows the ideas from the standalone program Roam Research (which also is very similar to notion). Every 'node' you make initially is a file and you can fill it in as a wiki type structure. Everything is flat on the filesystem and has a unique id prepended to it so you can't overwrite it with another node of the same name. There's also a UUID associated with each node so if you move from one place to another, none of the links need to be updated. Contrast that with a regular wiki or markdown where when you move a file to a new folder you either have to leave a redirect behind or go through and update all of the back links.

You can also start a node as a subheading and later on promote it to a full file with all the backlinks still working correctly.

I started using org mode years ago with spacemacs, got frustrated with spacemacs, and gave up until earlier this year when I found doom emacs and gave it another spin. I wish I had a better recommendation for an alternative but after looking for 4-5 years for something as good as org mode, I could never find anything. Notion seems like a popular alternative but there's also a lot of shiny bells and whistles in it that can cause 'productivity procrastination' where you spend time configuring and turning knobs instead of actually using the system. Ironically, the second best tool I found was plain pen and paper using a minimal bullet journal technique and taking special care to do indexing. Of course, you can't share plain pen and paper in that way, but the name of the game for personal use is low friction and ready access. The better a system scores on those two in term of note-taking the more useful it will be.

grinnick · 3 years ago
My company - roadie.io - offers SaaS Backstage. We're the second largest contributor after Spotify and we're co-hosting the first ever BackstageCon at KubeCon NA this year.

Get in touch if you ever want to discuss Backstage. Email in bio.

0xbadcafebee · 3 years ago
FWIW, it would be great if you had a breakdown of the TCO. The price is at a point where many orgs would have some suit decide "we have a DevOps guy, we'll just run Backstage ourselves and save $27K/yr". I think it's worth the money but you need to show the suits where that money is going, what kind of value they're getting from having you run it, and consider the risks (what if it gets compromised, goes down, etc). Also, I don't see SSO mentioned; you really want to mention that, it's pretty important for enterprise.
grinnick · 3 years ago
Thanks for the feedback and fwiw I completely agree. We can do much better in this area. We're a team of engineers so we struggle with marketing and copy.

SSO is supported on all plans by default.

pforret · 3 years ago
No pricing on your page? Are you “contact our sales team” expensive?
grinnick · 3 years ago
? There is transparent pricing from 50 up to 150 developers: roadie.io/pricing/ It's linked directly in the header.

For deals bigger than 150 devs, we have found customers prefer to talk to sales.

BannedInSweden · 3 years ago
The main problem with Backstage (BS) is that while it does provide some canned functionality, any extension of that functionality requires the utilization of their plugins system and wholesale buy-in to their ecosystem. Welcome to Wordpress for enterprise developers folks!

The plugin system isn't bad - but its a false narrative to believe you plunk this down and get some sort of valuable developer portal at an enterprise level (small biz sure - fortune 500...yeah no).

Spotify does open-source a number of their plugins, but after having looked extensively at which ones they make available and which ones we would have to write (and deal with their plugin system for) we opted out. The cost/benefit ratio is definitely there for a technical writer, but IMO its not there for serious development. (Pls - Anyone who disagrees with that - begin by telling me why you don't use Joomla as a base for all your custom applications )

Team integration at a company level is another challenge with Backstage. GitHub (the basis and sort of harvested DB for BS) is largely disconnected from company structures and AD and all the things for most companies. Yes GitHub enterprise is a thing... no in general its still not connected to company structures. There are teams in Backstage, but they are generally managed in isolation from the rest of the company which presents its own challenges and integration problems with other systems. Don't even get me started on the fact that asking everyone to put their service defintions in specific files in specific places in GitHub is no better/worse than making them enter it into a ui or spreadsheet or confluence article or anything except now its scattered in 10,000 repos - good luck getting everyone to change that.

Ultimately this isn't software that you simply install and use. Its a base that will likely require extension, and doing so means going down a one way path of adoption. Nothing it does is compatible with anything else. Is that good for Spotify - sure. Lots of folks helping build software they want to use is great. Is it good for the community? I'm not so sure. This article felt like a paid add (not suggesting it was - just saying it read like one) more than an honest plus/minus of the software and is anyone really surprised a technical writer couldn't build a proper enterprise developer portal by themselves? In other news - water is wet, the sky is blue and my kid wants to watch youtube.

combustilin · 3 years ago
Indeed, Backstage is a framework for creating a developer portal, not a ready-made product.

I used to work on a consultancy (https://frontside.com) who's specializing in adopting Backstage at Fortune 500s like HP. When you adopt Backstage at scale, you rely on processors that scan your LDAP, GitHub, etc to discover, stitch, and feed cohesive data to the software catalog. HP is giving a talk next month on how they manage 100,000+ entities in Backstage.

Backstage does a lot of heavy lifting that I'd rather use than have to build from scratch.

Melatonic · 3 years ago
So what do you recommend then instead?
adra · 3 years ago
Our companies evaluators picked up Cortex for a service registry offering because it did more for them faster, but they may switch over to backstage in time if they find the integration straightforward enough
BannedInSweden · 3 years ago
How about just using github? or if you want more how about building a a node/vue app? Or for docs scrounge up any static doc creation system and thereyago. most have some really editing features and BEHOLD markdown with colors!

I'm being a little silly there but my issue isn't that this isn't valuable software in the right situation. My issues is that its presented as some sort of magic bullet for a specific documentation issue when the problems described are generally not software related or software solved.

Most of what is presented in the article isn't about BS doing what this person attempted to do for 2 years. Its about switching paradigms and letting folks document in their repos instead of in a centralized system. One could just as easily look over the repos in a github org and go to those files in a repo. Both solutions require standardizing where things are documented..... so does a centralized system.

The big callout should be that when trying to centralize docs that align to apps they become disconnected from the apps themselves and thus its a pita to get everyone to do everything in 2 places. I'd be very curious if they completely turned off BS if there would be any loss... if not why bother with the overhead. Templating and other things are well handled by MS Clis, cookie-cutter or other more portable concepts and who needs yet another team and resource mgmt system. Splunk and other off the shelf systems can easily provide more comprehensive and robust stats and generally speaking there is a more enterprise level solution for just about everything it does.

Yes BS is a swiss army knife but when was the last time you saw a chef use one of those in the kitchen?

qbasic_forever · 3 years ago
The real gem highlighted by backstage is the MkDocs Material theme: https://squidfunk.github.io/mkdocs-material/ It's seriously awesome and has a lot of nice customization options. Out of the box it makes very clean and professional technical docs from a pile of markdown files. Even if you don't need to go the whole backstage route, seriously consider using mkdocs and the material theme.

Deleted Comment

superdug · 3 years ago
This is great, and I think if it was fully adopted initially in greenfield, this would be a completely feasible solution. My question is though, these problems exist at large organizations and will be seen as "another tool to rule them all" synonymous with things like "sharepoint". Disclaimer: I'm not saying that backstage is anyway comparable to sharepoint except that it presents itself as this grandiose solution to all things on boarding and learning on your own.

With that said, what's the low hanging fruit to entice adoption of backstage into an organization that already has a "process" like mentioned in the article of whiteboard meetings with stakeholders. Obviously that's not a great use of time, but it's also no extra learning or work, that SME can talk about their function without any preparation and can interactively answer any questions the learner asks at the time.

That doesn't scale in practice, but it also doesn't require any more whole team buy in

royjacobs · 3 years ago
You are right, this is not going to be trivial and you definitely need buy-in on a management level. Backstage is pretty malleable but the only way to get it to work in an organisation if there is only a single source of truth for things like your software catalog.

If there is a lot of legacy in your org, or if there is not enough buy-in then it will definitely be "yet another" system. There's a lot of functionality to unlock once you have a bunch of data in Backstage, though, so that could help tip the scale. The cloud costs dashboard that helps to show how much every service costs is a good demo :)

One benefit to Backstage's approach is that you don't necessarily need to change your processes and tooling to work with Backstage. It can help, but it's not a strict requirement. Other tooling in the space requires you to buy in to their specific workflows and so on whereas Backstage is a bit more agnostic.

kenrose · 3 years ago
> So when I had questions, I needed to track down who owned the service, where their code lived, where their Jira ticket project was, which Slack channel they lived in, and where in the wiki their internal documentation was. Keeping track of this for 100+ services was a pain, so I ended up creating a spreadsheet. It turns out that everyone else in the org needed this information, so this spreadsheet that I created for myself became the document of record for services.

100%.

Solving this problem is why I started OpsLevel [1] a few years back. Everyone always starts with a spreadsheet because they're flexible.

But spreadsheets quickly fall apart because they're not complete / up-to-date, automated, or really scalable.

We figured we could automate a whole lot of collecting service info from different places (Git, k8s, other deployment streams) and keeping it up-to-date.

[1] - https://www.opslevel.com

chime · 3 years ago
I got interested in OpsLevel immediately upon reading your comment but was disappointed to find out that there is no public pricing and I have to contact for a demo. That goes from Cloud SaaS to Enterprise territory. Maybe some day we will grow enough to give you a call but your post made me realize we should definitely look beyond spreadsheets to document and manage our infrastructure. Thank you and hope you open up your pricing and trial accounts in future.
kenrose · 3 years ago
Thanks for that feedback re: pricing and demo.

It's always a tricky balance, but I definitely sympathize with the feeling of just getting started without having to talk to a salesperson.

corwinstephen · 3 years ago
Unrelated to the discussion, just wanted to comment on Backstage's tagline:

“Build an ecosystem, not a wilderness"

which is ironic because a wilderness is arguably the most complete and perfectly functioning ecosystem you're gonna get.

lgrebe · 3 years ago
i like that thought. What would be a more ideal pair then?

Build a wilderness, not a parking lot

codingdave · 3 years ago
"Build an ecosystem, not dystopian sprawl"
lazyasciiart · 3 years ago
Create a garden, not a wilderness. OR (depending on your audience) Build an ecosystem, not a garden.
georgeoliver · 3 years ago
An ecosystem, not a Superfund site.
SerCe · 3 years ago
"Build a universe, not a multiverse" could capture the meaning pretty well, in my opinion.
grangerg · 3 years ago
Build an ecosystem, not an experiment.

Build an ecosystem, not a science project.

Build an ecosystem, not a microbiome.