I was interviewing for a platform engineer role for a company that from the outside their application doesn't seem too complex. Through the interview process, I find out that you cannot run much of their app locally to develop against and their staging environment does not reflect their production enviroment well. I asked them how they develop their new features if they cannot run the whole app locally and if they intorduce many bugs without testing locally. They response was along the lines that they hired good engineers that could understand all the complexity before deploying new features.
I believe having a solid development environment where you can develop and test the whole stack is key to making reliable software. Is this common at other companies that you cannot run the whole app locally to develop and test against?
> Is this common at other companies that you cannot run the whole app locally to develop and test against?
Yes. I work in B2B banking software, so having an "authentic" environment to test against is extraordinarily complex. Even the big vendors in the space seem unable to maintain accurate facsimiles of production with their own decades-old core software, so smaller vendors stand no chance unless they adjust their entire strategy.
What we ultimately had to do was negotiate a way to test in the production environment in such a way that all the activity could be tracked and backed out afterwards. A small group of important customer stakeholders were given access to this pre production environment for purposes of validating things.
This is something that has never sat well with me. But, the way the business works the alternatives are usually worse (if you want to survive long-term, that is). We wasted 18 months waiting for a customer to have a test environment installed before we learned our lesson about how bad those environments are in reality.
My gf works in UX for a major bank and they simply issued her a bank account in her name (with her SSN etc -- it's a real account, with ATM and debit cards and the like). If she and some co workers want to test something out relating to accounts they actually use these accounts to minimize this kind of problem.
They must have some special code that says that these accounts can have their fees zeroed and probably are subject to some special out of band audits. I don't really know the details of course but I was actually kind of impressed.
(no she is not actually a customer of her employer)
I worked at banks too but mostly segregated from the core business. Today I see observability tools as the solution for problems like that. Specifically tools like Lightrun where you can observe production, mitigate and fix directly from the IDE.
I wonder if other developers are familiar with this approach?
Same here in e-commerce. We end up testing in production environments with an element somewhere that's in a development environment that can be tracked and actively watched. Amazon notoriously does not have a production environment, and eBay's is just unusable.
Worked with banking software before; it can be incredibly complicated. I worked on proof of concept infrastructure for a Websphere monolith that ran on AIX servers. It had at least 10-15 services it connected to and each one of those were huge Java monoliths, too. Some of those apps had 8+ copies of /each/ environment (8x dev, 8x test, 4-8x stage/uat, 1-2x prod) so you could do integrated tests without stepping on other people or something software (software B might have a dedicated test environment of software A)
Yep, work in b2b banking as well and there is a huge issue with banking ‘test’ environments. Our partners (100s of companies, mostly banks/brokers/insurers/etc) mostly don’t even have test envs and if they do, they don’t represent production at all (sometimes the api is a completely different version etc). Our software allows to spin up a test/dev env locally or where ever but nothing much works without the partners and to dev/test those, those, you will have to dev/test against their production systems with real money.
In the beginning (more than 20 years ago), I insisted that we implement test systems ourselves so we didn’t have to test in production; that was naive; it was expensive and there is so much complexity that this does not work well.
I'm in that boat. We have a "microservice" code base, and spinning up the whole "app" is a Herculean task.
Even if you do manage to get it all running, getting the multitude of database with useable data adds an odyssey to your day.
We have some parts automated, but ops cannot keep up with all the changes in the micro apps.
Back when I worked on a mono-rails app with a React front end, getting things up and running was as simple as running the JS server, rake db seed and then rails s. We used puppeteer to test FE, and rspec for the BE. Two test suites, but we knew if something was wrong.
That same place sometimes had a net outage, and I wouldn't even notice because I was running everything locally.
Now I can only get everything running locally with confidence, if I block out an hour or so.
Honestly... microservices are great in theory, but make dev a huge pain. Docker does not mitigate the pain.
That's the reason why I walk away as soon as I hear "microservices". It's cargo-cult mentality where complexity and the issues you speak of above are considered a feature.
Yep I've had the same thing happen - a company I worked for went from having a monolith that could run locally, then everyone drunk the microservice Kool-Aid and we ended up with no way to run the microservices locally in a way that integrated with the monolith. There were staging sites with everything but there was one staging site per team.
The worst cases are startups adopting microservices. I get where they are a solution to a large company with multiple teams hitting the pain of monolithic development, but a team of half a dozen adopting microservices is a big red flag.
Microservices are not even great in theory if you spend just a bit of time deeply thinking about the complexities that microservices introduce compared with simply using libraries/modules to organise your system.
Usually someone develops a remote or mixed local/remote development environment which is easier to use than configuring the full stack to run locally, and then the ability to ever do so again atrophies until nobody can figure out how anymore.
In some companies I've worked at this is fine and not a major setback. In others, it's been a mess. I don't think that "can run the stack locally" is a good litmus test for developer experience - the overall development experience is a lot more nuanced than that. "How long does it take to get going again if you replace your computer" and "what's the turnaround time to alter a single line of code" are usually better questions.
I work at a VOIP services provider and we operate multiple brands which are companies we’ve acquired over several years. One thing that has rang true of every company we’ve acquired is that there is typically barely even a way to run one piece locally, let alone the whole thing.
Some of these when I took over there wasn’t even version control, and they were being developed by FTPing files to production server or even editing them live with vim or something. Certainly no reproducible containerization or even VMs to speak of.
I agree completely with you about having a solid development environment. I’ve spent the better part of a year creating such a thing for some of the brands, and it has increased our time to ship features and fixes probably ten fold.
> they were being developed by FTPing files to production server or even editing them live with vim or something.
Many editors / IDEs (e.g. Notepad++) can directly connect to the FTP server, so working on a FTP servers looks almost same as working on a local folder. You just make changes to the files, and then refresh the web browser to see the changes.
I work at a company that has similar practices to what you describe (we don’t run our services locally). While it’s hard to get used to at first, I think the silver lining is that it forces everyone to write good unit tests, since it’s the only efficient way to run their code before publishing a PR. Having worked on the opposite end (at a team where we mostly tested locally and had almost no automated tests), I much prefer this environment.
My last job was working at a company where you could not bring up the app locally. It didn't start that way. When the company was smaller I could develop a whole feature from the backend to the frontend and see the whole flow. As we grew it became increasingly difficult to run the app locally. Then it became impossible. Bugs grew and development time increased honstly 4x to do the same feature I could have done 3 years ago. I now think a good clean developer experience is key to a stable app.
I'm at a company that works like this, and it's awful. There are also anti-patterns around git branching and container deployment. The staging environment is broken every other day and production deployment requires the full time effort of a very senior engineer who cherry-picks through git commits from one branch to the production branch (there are thousands of differences between these branches). I'm honestly done with them. They don't even know what sensible dev/test/deploy looks like.
I saw a project get themselves in that position by using a lot of AWS services. At some point it seems they had a dev environment but it quickly faded.
Their solution was to have an anonymized copy of prod to a few (like 3 for 20 engineers) environment where features could be tested. Engineers usually paired in order use them concurrently for unrelated features and they were very expensive.
I did not live enough to see the on-demand environments arrive - devops was always busy with operations tickets to continue building those.
A lot of it depends on the complexity and the technology stack.
Microservices based architecture with massive amounts of microservices and other service dependencies, its a pain to do more than local unit testing and mocks.
Are you using FaaS or a lot of other cloud features? Also difficult to fully test locally. Extract as much logic out of the cloud dependant code.
Often there will be sub-sections of large apps that can be run locally.
And if the only way to test is in production hopefully they practice some form of limited rollout so you can control who adopts the changes or similarly some form of optional feature flags that are used to enable the new behavior until it can become a new default. If their release model is "We are so good it goes live for everyone all at once" then that tends to make for a stressful release days.
> Microservices based architecture with massive amounts of microservices and other service dependencies, its a pain to do more than local unit testing and mocks.
For microservices if you use something like Consul for service discovery within those microservices then you can hit them with, eg, Postman and change your local Consul registrations so everything goes to a main environment except the handful of services you're working on - which you redirect to your local machine for debugging.
It is the case in my company. Production and staging is bunch of docker-composes scattered around several servers. We had to create another staging environment, I think it took several days to configure it. People run services locally, but not entire stack, that takes too much effort.
Right now I’m trying to understand and deploy Kubernetes. Scalability and availability is good, but one of the reasons that I want to make it possible to roll out entire stack locally.
Basically it’s lack of devops. We don’t have separate role and developers do the minimal job of keeping production running well enough, they got other tasks to do. There should be someone working on improving those things full time.
This: "Basically it's lack of devops". Companies think they can avoid hirig any infra/devops people and make all of that stuff the job of junior developers. They should contract someone it to set things up, at the very least.
You can find developer first roles where the company puts a lot of time and energy into making developers the most efficient workers, but most companies are a cluster-fuck of resource management and you'll be lucky to find a healthy company that also puts developers first in this way.
In my experience, recreating staging as "production junior with caveats" comes down to cost and living with bad architecture/design. I'm not defending the practice, but I do think it's incredibly common.
At Google, it is very team-specific but it is generally not possible to run servers locally for testing and there generally is no staging environment, but it might be possible to bring up some part of the stack on Google's clusters. Those severs may read production data or some test data. Other than that there are static checks, unit tests, code reviews and the release process to ensure reliability.
I've been at a place where the whole stack couldn't be run locally at all but everything is separated by well-defined API boundaries and there are mature mock servers where required.
The architecture was probably more complicated than it needed to be but it was also decades old.
Yeah, once you have more than a small handful of dependencies (SaaS, databases, microservices). Some of the stuff I worked on will have built in stubbing/mocking so you can run parts of the stack locally and the test suite also uses the mocks/stubs.
In my experience, good local development environments are relatively rare. We had them at a couple of small <10 person B2B, SaaS startups I was involved with. In larger companies, there were just too many pieces to run everything locally. Or in companies with software development, but not focused on software, local environments seemed more of an afterthought. With increased dependencies on cloud services, "serverless" environments, etc. it can also be painful to run stuff locally. Obviously you can build around this (and should...) but it requires thinking ahead a bit, not just uploading your lambdas to AWS and editing them there...
Yaknow I actually transitioned from doing dev stuff to ops at a smallish startup. I was brought in to help with a green field rewrite. Eventually it became clear just how much of a mess the deployment environments were so I cleaned everything up and parameterized things so that when we went to stand up the new production environment it was quick and painless.
I got hired on at megacorp to do pretty much that with a suite of green field apps. And so I did, across a few apps and a zillion different environments because all of a sudden spinning up a new integrated environment was easy. The devs ran a lot of stuff locally but still had cloud based playgrounds to use before they hit the staging environments. Perhaps the best part though was having the CI stuff hooked into appropriate integrations, so even if you couldn't run it locally you sure as shit tested it before it hit staging. SOC2 compliance was painful for the company but not so much for us.
If you're sensing a theme with "green field" you'd be on to something. Even in a largeish company it's not so much the complexity as it is the legacy stuff that'll get you. Some legacy stuff will just never be flexible (e.g. the 32-bit Windows crap we begged Amazon to let us keep running) and even the most flexible software will have enough inertia to make change painful.
Tangentially this is also why I get apprehensive when I see stuff like that linux-only redis competitor.
Yea, it’s common that one doesn’t exist. I work as a test engineer and am usually the one to build and support those environments, I’ve yet to find a company that has it all working well before I join. I actually made a killing building integration testing environments of blockchains and sticking them in CI too.
Local dev is great until production gets complex enough that you lose parity. That'll happen even earlier now that we run x86 on the server but ARM locally.
At my company we don't support local dev nor have any sort of staging environment. We have development linux servers connected to prod services and dbs. There are no pragmas or build flags to perform things differently in "dev". Things are gated with feature flags.
It scared me at first but now I think it makes sense: staging is always a pain and doing it this way we avoid parity problems between staging and prod. Local development would be impossible for a system at our scale but I think even a staging setup would result in more defects—not fewer.
Local dev isn't the end, it is the beginning. With it it allows you to get further than you would otherwise - and most importantly, when (not if) you do discover a problem on a lower environment (or prod) it gives you a place to try and replicate it. You run everything locally so hopefully you can make a change to mirror the problem... Not a guarantee but at least you have a good starting point.
Also your local dev should try and mirror prod as much as possible.
Often it's due to complexity. Systems accrete stuff (features, subsystems, dependencies, whatever) over time due to changing business requirements. The more this happens, the harder it becomes to integrate them properly without breaking other parts of the system. And the system is running the business, often 24 hours a day, which makes migrations hard to do. So you end up of something thats basically too complex to run locally.
And particularly in regulated industries like banking you have the security teams locking everything down hard. Problems accessing systems and getting realistic test data, etc.
>Is this common at other companies that you cannot run the whole app locally to develop and test against?
It depends what you mean by the whole app. To reflect locally 100% of the conditions the big microservice app I am working on is very hard as I have to replicate locally multiple Kubernetes environments, Kibana, Data Dog, Elasticsearch, Consul and several databases and data stores.
What I can do, is to run one or more microservices locally, connected to remote development databases and speaking to other microservices and apps from some development Kubernetes environments.
I'm sad to say this even happens at FAANG. So many engineers seem to either not want to or not know how to optimize their own workflow as they're building it. It's really frustrating when you want to contribute something to their project and the development workflow for verifying the application still starts after making any changes is "upload the 5GB deployment bundle to AWS". Like come on, you can't seriously expect people to be productive that way.
> Is this common at other companies that you cannot run the whole app locally to develop and test against?
Extremely. We can't replicate our entire infrastructure in development - it's simply too complex and expensive. We use a combination of dev, integration, staging, and canary environments plus pilot environments to gauge quality before a full worldwide rollout.
Simple is easy on a small product. Your simple sql setup doesn’t work so well when you have terabytes of data and need to be running tens of millions of jobs per day.
Why is it a big deal to run your app locally? You just give each developer access to a cloud account (either one account or one per developer) with wide enough guard rails.
Heck, when I am in a place with bad internet, I spin up a Cloud 9 (Linux) or Amazon Workspace account (Windows) and do everything remotely.
I’m not wasting my time trying to use LocalStack or SAM local.
It would be quicker for me to run things locally - in my team we all have access to an AWS account each but to test my code remotely I need to compile code, build my container, push my container to ECR then deploy the container to ECS - that last step takes minutes.
> "So unless your product legitimately only sells to the likes of Google, Facebook, and Apple, you probably don’t need to be 'enterprise ready'"
What in the actual fsck? Check out this list of companies ranked by the number of employees they have - https://companiesmarketcap.com/largest-companies-by-number-o.... Okay, Amazon is #2 - but notice the top 100 doesn't include Apple. There's a lot more to business than FAANG.
If you're selling enterprise software then the important number is not "number of employees" but "number of employees who would use your software". In large retail, manufacturing, oil etc companies they have a lot of software of course, but each person is typically accessing 4 or 5 apps at most (email, office suite, HR, etc). In FAANG companies each employee involved in software development is often using closer to 30 apps (project management, comms, source control, bug tracking, telemetry, server management, i18n, etc). It makes sense to focus on FAANG companies if you're producing enterprise software simply because they buy more of it. That obviously doesn't mean there isn't a huge market outside of that as well.
You’re better off focusing on banks, oil and gas, etc. Anything that uses a lot of technology to get its job done and operates at a massive scale. FAANG are a very small part of the ecosystem and tend to build vs buy.
A major bank employs between 50,000-10,000 software engineers and in the 250,000-50,000 employee range. Netflix has only 11,000 employees of all types.
>The buyer is typically a VP, a director, possibly C suite, depending on what you’re selling
If a company has VPs buying things surely they have some sort of internal vendor management that usually involves accounting/legal/compliance reviews. Being enterprise ready means you have the roles/documentation/process/procedure to answer those questions
Although each of those companies is made up of 5 to 50 mini-companies, each with more business units/products/services. You have to understand which of them you're selling to, what they need, how they will use your product. Are two completely different tiny departments gonna hem and haw over trying your product, or will the whole org use your software? You might end up supporting a dozen users or 500,000. Will you still have to support the same features, or can the tiny teams get away with less? Is it even worth supporting the enterprise features if your product fit is smaller teams and you wouldn't get a large-scale contract anyway?
right like any software deployed by Zebra is probably going to get used by every retail company and warehouse in the USA.
Now I don't know how much of the software on my TC26 actually comes from Zebra but, pretty sure, it's not zero.
ASSA ABLOY software is probably everywhere there's a door these days.
I don't know who the players in HVAC are but considering that a lot of HVAC/lighting control for like Target/Walmart are remotely controlled there's definitely somebody with tens of thousands of deployments.
Not to mention EdTech shit like BlackBoard, that gets deployed to every god damned school in the country
I'm working on a project where the backend devs decided to use Cassandra to manage a few hundred entities. Due to different access modes they had to denormalize the data into multiple tables and we're constantly running into consistency issues. A fucking sqlite db with a few tables and indexes would do just fine. Also, the tool has maybe a dozen DAUs. But it's web scale, so, yeah…
Why would you ever use SQL? The SQL transaction model doesn't make a lot of sense for most use cases (including essentially anything that uses the internet), and trivial usability things like being able to have collection columns without having to define a separate table all add up.
This completely tangential, but any time I see someone talking about normalizing a database, I briefly think they are talking about doing something like:
x/|x|
Ya know, like normalizing a vector. Of course this brief confusion doesn't matter, I sort it out quickly, and it is totally clear language to other database people -- it is just a funny quirk of overloaded lingo.
But I wonder if as big data and machine learning folks (since there's some overlap with Linear Algebra there) ever get their lingo mixed up.
"To predict where the ridesharing customers will want to go, we will apply the taxicab norm to our database."
While the overloading is annoying, database normalization actually has a strict definition that is not unlike the standard mathematical use; that is, to imply consistency (to make normal): https://en.wikipedia.org/wiki/Database_normalization
I would also point out that normalization within mathematics does not have a single concrete definition either, and it depends on your domain and desired property to stabilize: https://en.wikipedia.org/wiki/Normalization_(statistics)
You are absolutely right. 95% of problems in software development are created by the software developers themselves. The fastest/smartest developers I have worked with simply make less dumb mistakes. They keep it simple. That’s all they have to do to be 10x more productive.
I agree with article because you don't need to have it to "start talking" with Enterprise customers.
You probably have to prove that you have these on your roadmap and that you have engineering team that can implement these in not so distant future. That you know these things exist and have will to invest in it in future.
These capabilities are increasingly required just to get in the door, depending on your target customer.
The alternative is that your product becomes a security exception, which infosec teams are more and more unwilling to grant, for pretty good reasons.
The good news for product teams is that there are more and more off-the-shelf options that can be quickly integrated vs. having to start from scratch.
But the main point of this comment is: I think you’re right that “near term roadmap” was good enough a few years ago, but less so now, and continuing to trend towards “not acceptable” if selling to enterprise customers.
(Observations as a recent/former auth PM at a SaaS/PaaS that sold primarily to enterprise customers).
Yep, I'm working at a Series B startup and I can tell you, you absolutely need SOC 2 or ISO and/or GDPR compliance based on your market. It's not optional, you will lose business if you don't.
History is littered with dead startups that designed for scale before they had enough usage to justify it. Within reason, having users knock your site over resulting in failures like the Twitter Fail Whale is a good problem to have.
With that said, you need to be prepared to scale up quickly once you have this problem. There's a reason Facebook counts its users in the billions, and Friendster is a footnote in internet history.
Yes, but.. I am a software architect. ~90% of the job offers I get are from CEOs/CTOs who want me to join their small company to help them refactor because they can’t grow anymore. Tech debt kills as well.
Is that a "but" or is that how things should work? You solve a problem when you have it or can reasonably predict it (in technical/engineering terms). I also think it's sensible to look for experts when you need them right?
If they have the money to hire you and they have pent-up growth that refactoring will unlock, that really sounds like close to an ideal path for a startup to take. They're literally paying down their tech debt by paying to hire you. Maybe ideal would be a little earlier so it doesn't block growth, but yeah. In my experience way more startups have died from premature optimization
So how do you square they circle "Don't design to scale but scale up quickly when you need to"? You can't rewrite your stack just because you took on a new customer that is suddenly killing you.
I don't think that it's stupidly hard to get basic things set up for scaling at the start. These don't apply to everyone but:
- use a cloud provider, they let you grow
- write infrastructure as code (terraform, pulumi, cdk), it's not that hard and it makes building your 2nd, 3rd data center easier
- for the love of God think for 10 minutes about a scalable naming convention for data centers, infrastructure components, subdomains, IP ranges etc. This type of stuff causes amazing headaches if you get it wrong, and all you have to do is consider it at the start. Many AWS resources can't be renamed non-destructively, and are globally namespaced. Be aware of this. Don't paint yourself into a really stupid corner.
- even if you're not doing microservices, consider containers and an orchestrator that has scalability baked in (like Kubernetes). If you use a managed Kubernetes with managed node groups or fargate, you can basically forget about compute infrastructure from this point on
- deploy Prometheus, Grafana, Loki etc day one and build basic dashboards. With Kubernetes, you can get this installed quickly and within a day you'll at least be able to graph request/error count and parse logs.
- deploy an environment for developers to deploy to and use the product (even load test it)
The development cycle is now set up for building what you need to build. You can easily find developers at any level who have experience with containers, Kubernetes, Grafana so they can onboard quickly, be productive and troubleshoot their own stuff. They can refactor the monolith without having to invent service discovery, load balancing, ingress etc.
My personal experience with Kubernetes began at a start-up with a tiny team, none of whom had prior experience with it. We had started building a single monolithic app. But within a short time we figured out how to make a helm chart, add ingress and boom, we were "online". Every time someone was about to spend time and effort inventing something, we would quickly scan through the Kubernetes docs and CKAD material to see if we could avoid having to build it at all. We leverage cloud stuff too, where it was cheap and easy and scalable (like AWS SQS/SES/SNS/CodeArtifact/ECR/Parameter Store)
i didn't see the words "it depends" anywhere in this
thats the whole point of engineering, picking/building the right solutions for the problems and context
some stuff just needs a vps, a php file and an sqlite db and it will scale
other stuff might need container orchestration or anywhere in-between (or beyond). it should depend on the context though and we should be prepared to change strategy as needed
Is one of the problems though is that the early audiences appreciate this kind of over-engineering? If your investors/buy-in folks are being shown early demos and you mention "and this thing is gonna scale seamlessly up to 1,000,000 users because of tech words A, B, and C" then they get excited. They don't understand A, B, or C. But what they just heard is that you're planning for Big. And that stokes their aspirations.
Facebook is a great example. They did tons of things not traditionally known as highly scalable. Like one repo with the project coded in php which was largely a monolith.
And an imperative codebase which looked like something out of Matt's Script Archive. I often wonder whether a startup today could pull-off something similar.
Well it depends on what you're selling! Design for what you truly want to sell, and that will end up however it needs to end up, whether that's enterprise or not. This random blogger, and our random comments, can't tell you what you need. It has to emerge from the product development process like a sculpture from a block of marble.
I know, this is overly-general advice to the point of silliness.
It's a little like telling a construction worker "You don't need a super-powerful 'jackhammer' to do your work. A simple claw hammer is all you actually need."
Well... sure, that's definitely true if you're building a house.
But you might have a problem ripping up a sidewalk...
There are plenty of companies building houses, and plenty who are working with concrete! Neither one of them are "the big kids."
They're not even necessarily different-size markets, either -- you can make a lot of money building houses or making sidewalks -- it's just that different problems require different solutions, and different software companies are going to have different challenges to the problem of "get bigger."
Exactly this. It sounds easy but this type of creator who can do this is very rare but usually has a high success rate. This should be the defintion of the "10X dev"
I've delivered custom backend business servers to a various businesses of healthy size. Almost all were deployed on rented multicore servers with gobbles of RAM from Hetzner / OVH / etc. Some running for many years already. Not a single one required to be "enterprise scaled".
Granted all servers are C++ and talking to a local DB. No problems in processing thousands requests per second without breaking much sweat.
Postgresql. No framework (well there is rather low level one of my own but it is fluid). Using 4 libs for JSON, postgres, HTTP, logging. I expose basically JSON based RPC. Peers / Clients post JSON commands and receive JSON replies.
Upon startup all data (except couple of giant tables) is sucked from DB into RAM into highly efficient data structures hence most reading requests are handled in microseconds. Those 2 giant tables are loaded into the same structs but partially. This is usually enough as RAM keeps last few years worth of data and requests that need something older come just couple of times per month. Only writes touch the database and immediately reflected in RAM as well.
This is just an example. I write game like servers as well and those use different approach. Generally I treat each project / product individually.
I like the article, but overall I think it's the same advice as "start small, limit scope, be ready to pivot" with a dash of "don't make enterprises an early customer". I obviously paraphrased quite a bit.
What is glaringly obvious from this article is that "enterprise" in this industry has lost much of its meaning. It's almost ubiquitous with "medium+", which I think is a self-defeating definition that starts to make sense only when you look at what features are commonly gatekept behind enterprise distributions of software.
Readit News
I believe having a solid development environment where you can develop and test the whole stack is key to making reliable software. Is this common at other companies that you cannot run the whole app locally to develop and test against?
Yes. I work in B2B banking software, so having an "authentic" environment to test against is extraordinarily complex. Even the big vendors in the space seem unable to maintain accurate facsimiles of production with their own decades-old core software, so smaller vendors stand no chance unless they adjust their entire strategy.
What we ultimately had to do was negotiate a way to test in the production environment in such a way that all the activity could be tracked and backed out afterwards. A small group of important customer stakeholders were given access to this pre production environment for purposes of validating things.
This is something that has never sat well with me. But, the way the business works the alternatives are usually worse (if you want to survive long-term, that is). We wasted 18 months waiting for a customer to have a test environment installed before we learned our lesson about how bad those environments are in reality.
They must have some special code that says that these accounts can have their fees zeroed and probably are subject to some special out of band audits. I don't really know the details of course but I was actually kind of impressed.
(no she is not actually a customer of her employer)
I wonder if other developers are familiar with this approach?
In the beginning (more than 20 years ago), I insisted that we implement test systems ourselves so we didn’t have to test in production; that was naive; it was expensive and there is so much complexity that this does not work well.
Even if you do manage to get it all running, getting the multitude of database with useable data adds an odyssey to your day.
We have some parts automated, but ops cannot keep up with all the changes in the micro apps.
Back when I worked on a mono-rails app with a React front end, getting things up and running was as simple as running the JS server, rake db seed and then rails s. We used puppeteer to test FE, and rspec for the BE. Two test suites, but we knew if something was wrong.
That same place sometimes had a net outage, and I wouldn't even notice because I was running everything locally.
Now I can only get everything running locally with confidence, if I block out an hour or so.
Honestly... microservices are great in theory, but make dev a huge pain. Docker does not mitigate the pain.
Usually someone develops a remote or mixed local/remote development environment which is easier to use than configuring the full stack to run locally, and then the ability to ever do so again atrophies until nobody can figure out how anymore.
In some companies I've worked at this is fine and not a major setback. In others, it's been a mess. I don't think that "can run the stack locally" is a good litmus test for developer experience - the overall development experience is a lot more nuanced than that. "How long does it take to get going again if you replace your computer" and "what's the turnaround time to alter a single line of code" are usually better questions.
Some of these when I took over there wasn’t even version control, and they were being developed by FTPing files to production server or even editing them live with vim or something. Certainly no reproducible containerization or even VMs to speak of.
I agree completely with you about having a solid development environment. I’ve spent the better part of a year creating such a thing for some of the brands, and it has increased our time to ship features and fixes probably ten fold.
Many editors / IDEs (e.g. Notepad++) can directly connect to the FTP server, so working on a FTP servers looks almost same as working on a local folder. You just make changes to the files, and then refresh the web browser to see the changes.
This is not even sarcasm.
Their solution was to have an anonymized copy of prod to a few (like 3 for 20 engineers) environment where features could be tested. Engineers usually paired in order use them concurrently for unrelated features and they were very expensive.
I did not live enough to see the on-demand environments arrive - devops was always busy with operations tickets to continue building those.
Microservices based architecture with massive amounts of microservices and other service dependencies, its a pain to do more than local unit testing and mocks.
Are you using FaaS or a lot of other cloud features? Also difficult to fully test locally. Extract as much logic out of the cloud dependant code.
Often there will be sub-sections of large apps that can be run locally.
And if the only way to test is in production hopefully they practice some form of limited rollout so you can control who adopts the changes or similarly some form of optional feature flags that are used to enable the new behavior until it can become a new default. If their release model is "We are so good it goes live for everyone all at once" then that tends to make for a stressful release days.
For microservices if you use something like Consul for service discovery within those microservices then you can hit them with, eg, Postman and change your local Consul registrations so everything goes to a main environment except the handful of services you're working on - which you redirect to your local machine for debugging.
Right now I’m trying to understand and deploy Kubernetes. Scalability and availability is good, but one of the reasons that I want to make it possible to roll out entire stack locally.
Basically it’s lack of devops. We don’t have separate role and developers do the minimal job of keeping production running well enough, they got other tasks to do. There should be someone working on improving those things full time.
In my experience, recreating staging as "production junior with caveats" comes down to cost and living with bad architecture/design. I'm not defending the practice, but I do think it's incredibly common.
The architecture was probably more complicated than it needed to be but it was also decades old.
I got hired on at megacorp to do pretty much that with a suite of green field apps. And so I did, across a few apps and a zillion different environments because all of a sudden spinning up a new integrated environment was easy. The devs ran a lot of stuff locally but still had cloud based playgrounds to use before they hit the staging environments. Perhaps the best part though was having the CI stuff hooked into appropriate integrations, so even if you couldn't run it locally you sure as shit tested it before it hit staging. SOC2 compliance was painful for the company but not so much for us.
If you're sensing a theme with "green field" you'd be on to something. Even in a largeish company it's not so much the complexity as it is the legacy stuff that'll get you. Some legacy stuff will just never be flexible (e.g. the 32-bit Windows crap we begged Amazon to let us keep running) and even the most flexible software will have enough inertia to make change painful.
Tangentially this is also why I get apprehensive when I see stuff like that linux-only redis competitor.
At my company we don't support local dev nor have any sort of staging environment. We have development linux servers connected to prod services and dbs. There are no pragmas or build flags to perform things differently in "dev". Things are gated with feature flags.
It scared me at first but now I think it makes sense: staging is always a pain and doing it this way we avoid parity problems between staging and prod. Local development would be impossible for a system at our scale but I think even a staging setup would result in more defects—not fewer.
Also your local dev should try and mirror prod as much as possible.
Often it's due to complexity. Systems accrete stuff (features, subsystems, dependencies, whatever) over time due to changing business requirements. The more this happens, the harder it becomes to integrate them properly without breaking other parts of the system. And the system is running the business, often 24 hours a day, which makes migrations hard to do. So you end up of something thats basically too complex to run locally.
And particularly in regulated industries like banking you have the security teams locking everything down hard. Problems accessing systems and getting realistic test data, etc.
Deleted Comment
It depends what you mean by the whole app. To reflect locally 100% of the conditions the big microservice app I am working on is very hard as I have to replicate locally multiple Kubernetes environments, Kibana, Data Dog, Elasticsearch, Consul and several databases and data stores.
What I can do, is to run one or more microservices locally, connected to remote development databases and speaking to other microservices and apps from some development Kubernetes environments.
Extremely. We can't replicate our entire infrastructure in development - it's simply too complex and expensive. We use a combination of dev, integration, staging, and canary environments plus pilot environments to gauge quality before a full worldwide rollout.
Heck, when I am in a place with bad internet, I spin up a Cloud 9 (Linux) or Amazon Workspace account (Windows) and do everything remotely.
I’m not wasting my time trying to use LocalStack or SAM local.
What in the actual fsck? Check out this list of companies ranked by the number of employees they have - https://companiesmarketcap.com/largest-companies-by-number-o.... Okay, Amazon is #2 - but notice the top 100 doesn't include Apple. There's a lot more to business than FAANG.
A major bank employs between 50,000-10,000 software engineers and in the 250,000-50,000 employee range. Netflix has only 11,000 employees of all types.
I knew a guy who sold a $2M solution to track MSDS sheets in a prison system. Stupid simple application.
If a company has VPs buying things surely they have some sort of internal vendor management that usually involves accounting/legal/compliance reviews. Being enterprise ready means you have the roles/documentation/process/procedure to answer those questions
Now I don't know how much of the software on my TC26 actually comes from Zebra but, pretty sure, it's not zero.
ASSA ABLOY software is probably everywhere there's a door these days.
I don't know who the players in HVAC are but considering that a lot of HVAC/lighting control for like Target/Walmart are remotely controlled there's definitely somebody with tens of thousands of deployments.
Not to mention EdTech shit like BlackBoard, that gets deployed to every god damned school in the country
Nosql databases excel at simple lookups and gets by ID, rather than trying to sort and join pretty much ad hoc.
Ex Mongo sorts just fine, and can optimize sorting with indexes.
x/|x|
Ya know, like normalizing a vector. Of course this brief confusion doesn't matter, I sort it out quickly, and it is totally clear language to other database people -- it is just a funny quirk of overloaded lingo.
But I wonder if as big data and machine learning folks (since there's some overlap with Linear Algebra there) ever get their lingo mixed up.
"To predict where the ridesharing customers will want to go, we will apply the taxicab norm to our database."
I would also point out that normalization within mathematics does not have a single concrete definition either, and it depends on your domain and desired property to stabilize: https://en.wikipedia.org/wiki/Normalization_(statistics)
We are having a lot more nervous takes around legacy LDAP/AD auth mechanisms these days.
You probably have to prove that you have these on your roadmap and that you have engineering team that can implement these in not so distant future. That you know these things exist and have will to invest in it in future.
The alternative is that your product becomes a security exception, which infosec teams are more and more unwilling to grant, for pretty good reasons.
The good news for product teams is that there are more and more off-the-shelf options that can be quickly integrated vs. having to start from scratch.
But the main point of this comment is: I think you’re right that “near term roadmap” was good enough a few years ago, but less so now, and continuing to trend towards “not acceptable” if selling to enterprise customers.
(Observations as a recent/former auth PM at a SaaS/PaaS that sold primarily to enterprise customers).
History is littered with dead startups that designed for scale before they had enough usage to justify it. Within reason, having users knock your site over resulting in failures like the Twitter Fail Whale is a good problem to have.
With that said, you need to be prepared to scale up quickly once you have this problem. There's a reason Facebook counts its users in the billions, and Friendster is a footnote in internet history.
I don't think that it's stupidly hard to get basic things set up for scaling at the start. These don't apply to everyone but:
- use a cloud provider, they let you grow
- write infrastructure as code (terraform, pulumi, cdk), it's not that hard and it makes building your 2nd, 3rd data center easier
- for the love of God think for 10 minutes about a scalable naming convention for data centers, infrastructure components, subdomains, IP ranges etc. This type of stuff causes amazing headaches if you get it wrong, and all you have to do is consider it at the start. Many AWS resources can't be renamed non-destructively, and are globally namespaced. Be aware of this. Don't paint yourself into a really stupid corner.
- even if you're not doing microservices, consider containers and an orchestrator that has scalability baked in (like Kubernetes). If you use a managed Kubernetes with managed node groups or fargate, you can basically forget about compute infrastructure from this point on
- deploy Prometheus, Grafana, Loki etc day one and build basic dashboards. With Kubernetes, you can get this installed quickly and within a day you'll at least be able to graph request/error count and parse logs.
- deploy an environment for developers to deploy to and use the product (even load test it)
The development cycle is now set up for building what you need to build. You can easily find developers at any level who have experience with containers, Kubernetes, Grafana so they can onboard quickly, be productive and troubleshoot their own stuff. They can refactor the monolith without having to invent service discovery, load balancing, ingress etc.
My personal experience with Kubernetes began at a start-up with a tiny team, none of whom had prior experience with it. We had started building a single monolithic app. But within a short time we figured out how to make a helm chart, add ingress and boom, we were "online". Every time someone was about to spend time and effort inventing something, we would quickly scan through the Kubernetes docs and CKAD material to see if we could avoid having to build it at all. We leverage cloud stuff too, where it was cheap and easy and scalable (like AWS SQS/SES/SNS/CodeArtifact/ECR/Parameter Store)
thats the whole point of engineering, picking/building the right solutions for the problems and context
some stuff just needs a vps, a php file and an sqlite db and it will scale
other stuff might need container orchestration or anywhere in-between (or beyond). it should depend on the context though and we should be prepared to change strategy as needed
See I interpret that to mean that you should design for scale, but not implement or invest in scale until needed.
It's a little like telling a construction worker "You don't need a super-powerful 'jackhammer' to do your work. A simple claw hammer is all you actually need."
Well... sure, that's definitely true if you're building a house.
But you might have a problem ripping up a sidewalk...
There are plenty of companies building houses, and plenty who are working with concrete! Neither one of them are "the big kids."
They're not even necessarily different-size markets, either -- you can make a lot of money building houses or making sidewalks -- it's just that different problems require different solutions, and different software companies are going to have different challenges to the problem of "get bigger."
Granted all servers are C++ and talking to a local DB. No problems in processing thousands requests per second without breaking much sweat.
Upon startup all data (except couple of giant tables) is sucked from DB into RAM into highly efficient data structures hence most reading requests are handled in microseconds. Those 2 giant tables are loaded into the same structs but partially. This is usually enough as RAM keeps last few years worth of data and requests that need something older come just couple of times per month. Only writes touch the database and immediately reflected in RAM as well.
This is just an example. I write game like servers as well and those use different approach. Generally I treat each project / product individually.
What is glaringly obvious from this article is that "enterprise" in this industry has lost much of its meaning. It's almost ubiquitous with "medium+", which I think is a self-defeating definition that starts to make sense only when you look at what features are commonly gatekept behind enterprise distributions of software.