Ask HN: Is it hard to get into systems programming from back end engineer?

I do a lot of systems-type work academia (OSes, CPU sims, transient execution attacks, etc).

"Should not apply" is a very strong phrase. Would you be willing to humble yourself in a junior position so that you could learn? Are you willing to put in the time and effort to learn the ins and outs of the platform? If so, I think you'd be a great fit as a junior systems engineer at a 50-500 person org that has the self-awareness to put you on a team with more experienced engineers who can show you the ropes. If anything, you're a great candidate because you'll already have some programming experience under your belt which minimizes the barrier for learning syntax and language-agnostic programming concepts, allowing you focus more on actually learning how everything works under the hood.

I work professionally on computer vision software, and quite frankly one of the biggest barriers we have now is that the people who are deploying our ML code have the experience with all the experimental tooling (PyTorch/Python) but not as much with what's going on under the hood, and thus we're leaving lots of performance on the table.

My personal definition of "systems knowledge" is "can you answer 'how does this work under the hood?'" to the point where we're talking about assembly/bytecode and drivers. It's more tedious than learning the Java standard library, but I wouldn't necessarily say it requires more intellectual power. It just takes time.

mannymanman · 3 years ago

This is great insight. Do you have any recs for someone with python experience to get started with learning systems programming?

dwrodri · 3 years ago

Yes! But it depends what you want to get out of it. I'm a big proponent of project-based learning, so my question to you is: what would you want to build?

I'll lead with an example: I really like the idea of search indexes and search engines. I cobbled together a search engine on my own that indexes arXiv's metadata, very similar to Andrej Karpathy's Arxiv Sanity Preserver [1] in Python. I very quickly encountered that the "What can I rip off of medium blogs and StackOverflow" approach wasn't scaling well, so I had to dig into how each part of my tech stack worked to figure out where things were going slower, and how I could improve it.

You can "do systems engineering" in any language and just about any tech stack, but you'll often see it closely associated with compiled languages, because those are often chosen when we care about performance because extra abstractions often make it harder to reason about what's really going on in certain portions of your tech stack. It started in Python, but then dusted off my C skills, and then my C++ skills for different parts of the entire stack.

Before I get too carried away, here's the main point:

Try to find a problem where you have to deal with lots of data on a single computer/server/device. You'll quickly hit limitations on what your software can do. Figuring out why you hit those limits is gonna start you down a path to acquiring the "systems knowledge" that you seek.

It's actually quite difficult to hit performance limitations on your average server with just text. Video and 3d content are much more intensive mediums that very quickly let you hit those limits. It's one of the reasons why Twitch and YouTube don't have lots of competition, but we've hopped between link aggregator to link aggregator over the years.

1: https://arxiv-sanity-lite.com/ 2:

elipsey · 3 years ago

[deleted]

abc_lisper · 3 years ago

Each of those things are a self contained domain in themselves and will require a few years of effort for reasonably smart people. The q question thus is, how much time are you willing to grind?

I went from Python ML+backend work, to Linux kernel work. I wouldn't say it was a walk in the park, but it is doable, and the "shouldn't apply" attitude will obviously fail to get you anywhere.

For me, the first step was understanding that opportunity wouldn't fall into my lap. Sure, a recruiter might cold email you about your perfect job, but more often than not, it will be in the same backend roles you want to move away from. So you need to be actively searching, applying, and interviewing to make it happen. Define your goal: what work do you want to do? Then make all the steps centered around that.

I searched for "interim" roles which I thought might be easier to get into (for me, embedded development as a stepping stone into kernel development). I don't know if this actually helped in my case, but it could be a good strategy if you're not having luck getting your "goal role".

A big part of how I got the role I wanted was (and I know people love to hate this) side projects. If you can't talk about your systems development experience in prior roles, you need to be able to talk in depth about systems development problems you've solved on your own time. Make it targeted: if you want to work on an OS, make your own crappy one, etc. This won't work for every company, some have bad interview process and won't let you talk much about this (ahem, Google). You just have to accept that it's their loss, and move on.

Your first "goal role" will probably not be at the compensation or title level you're currently at, since you won't have the same level of relevant experience for your new role. Be clear with yourself about this and make sure it's ok. You should be able to make up the difference within a year or two hopefully, but you need to be honest with yourself. When I did get my goal role, I found that a great way to advance myself early on was to seek out ways that my python/data analysis experience could help my team solve problems quicker or easier. By taking a little initiative there, I could show my value while giving myself time to level up on the stuff I didn't have experience with. Maybe you can take a similar approach with your experience, and even use it to sell yourself in interviews.

Sorry for a long winded comment, but I hope my experience helps, or is at least interesting!