I'm currently working in a medium-sized AI startup (which is picking up steam). I'm currently a mid-level engineer in one of the product teams and I find myself in a bit of a search for meaning situation.
I've had a bit of an eclectic path: Degree in Economics -> Data Analyst -> Data Engineer -> Backend Engineer
As a data analyst, I got bored of making dashboards and went into data engineering since it was a growing field and, I believed, more challenging, after a while however I found myself a bit bored of making the Nth data pipeline and decided to try my hand at backend software engineering which I like a lot, but lately, I've been feeling that all I ever do is implement CRUD endpoints (data pipelines all over again) because I came in into an existing code base and there was nothing really left to do, but maintain it. I have an odd challenge here and there but life's become pretty dull, and looking at the senior devs their days are pretty much the same, except for a few more meetings.
Now my director has started asking me about what are my plans in the company and if I want to go the staff engineering route or the management route. I love engineering and would not want to take the management route, but also do not want to be making GET/POST API's for the next 15 years. Since the start of my journey into IT I've always known what the next step is and what I need to do to accomplish it (data analyst -> learn SQL, python; date engineer -> cloud, spark, kafka; backend-engineer: K8s, Go, CI/CD, DS&A, API's). But now I feel a bit stuck, I'm still reading a lot of books, but I feel like I never get to use the more advanced things I've learned.
I know that it might be a limitation of my company and I should probably try and find another position, but I am a bit afraid that going to another company would just be more of the same with a different flavored stack.
Has anyone else been in a similar position, and how did you deal with it? How did you find a way into something more interesting? Or is it just Airflow pipelines and CRUD endpoints all the way down?
Thank you
Overall what I've found is that the things I find interesting and enjoy working on in an engineering capacity almost have 0 overlap with things that will make money or turn into a business of some kind. Instead of continuing my search to find the perfect balance of company mission statement, interesting technology, competent and friendly coworkers (culture), and pay, I optimize for trying to get at least 2 out of 4 of these, if I can get 3 it means it's a great fit for me. Of these, company culture has made the biggest impact on my overall happiness.
There is some truth that joining a smaller company or a startup to wear multiple hats can fix this somewhat, but I would advise against doing this unless you either really strongly believe in whatever the company is doing, or really strongly want to learn something you feel is only possible to learn at that company. Otherwise you will end up working 60 hour weeks making CRUD endpoints for someone panicking about how to raise the next round of funding or get to the goal of the month that the investors have said is important now.
Until you find some field or area of technology that lights your hair on fire in excitement, keep CRUDing my friend. Realize that you are exchanging your time for money at your job and treat it as such. Save the interesting things for your personal time or if you believe strongly enough in it, start your own company and have someone else CRUD for you ;)
Anyway good luck!
I've had other software jobs where the work was 20% data formats, 80% business logic + algorithms related to the problem domain, and 0% APIs/integration projects. That wasn't working for a banking megacorp, that was working for a small software product/consulting shop in a weird niche.
Some ideas -- overfit from sample size of 1:
- trying searching for jobs in companies that are qualitatively different from the company where you've been working.
- if your current employer serves an industry that's relatively mature in its application of IT & software, you could try for a job in a company in an industry where use of computers and IT is much less widely adopted.
- if you've been working somewhere medium-sized, you could aim for some place smaller where you're more likely to be wearing multiple hats and having more customer or client contact and immersion in the industry's problem domain. maybe you get more chance to solve real world problems that way, not IT problems. You could look for roles where they need people to learn the domain and build a product containing some serious application-specific logic -- but I guess, not well-understood-by-academia-serious-logic, as you might be competing against a bunch of well-credentialled grads/grad students for those specialties.
There's potentially some career risk involved with these ideas -- you might find a job where work is much more interesting, but it might be in a smaller niche, or in smaller companies. In the extreme, if you end up specialized in a really interesting niche that's occupied by one employer, there's much less room to switch jobs if you end up with a difficult manager, or get a pay rise by switching jobs to somewhere else your expertise is valued, if negotiations with the current employer stall.