Ask HN: Best way to simultaneously run multiple projects locally?

Woah, so much overcomplication in the comments!

If you want to run multiple applications, how about ... just running them? It sounds like you already do that, so what is the real problem that you are trying to solve?

If it's annoying to start them by hand one by one, you could use Foreman or Overmind to start them with a single command, and interleave their output in a terminal.

zwnow · 6 months ago

I also dont know why OP wants to jump into kubernetes directly instead of just using docker...

tbrownaw · 6 months ago

> jump into kubernetes directly instead of just using docker

Option A: learn k8s, use it for anything containerized. Don't use the complicated parts if you don't need them.

Option B: learn docker (docker compose I guess?), use that... then also learn k8s because you'll eventually have to anyway

suralind · 6 months ago

It's not that I don't want to run them locally. It's more like I would like to

a) use HTTPS when developing services b) give each service a unique hostname c) run all my services on the same port

and there's simply no easy way to do that that I know of. Hence my Ask HN :)

ruuda · 6 months ago

Even Docker ... does that solve a problem OP has?

jinnko · 6 months ago

I hadn't come across overmind before and searching turned up several very different projects.

I think the parent comment is referring to this one: https://github.com/DarthSim/overmind

ruuda · 6 months ago

Yes, that’s the one I meant. One thing that annoyed me with Foreman is that sometimes it doesn’t terminate all processes when one of them crashes. Since I switched to Overmind/Hivemind I never had that problem again.

Deleted Comment

https://ddev.com/ has become standard in the circles I run in (most are web devs working in agencies touching multiple projects each week). You don't have to use DDEV specifically, but it works like a dream and may provide some inspiration.

Each project gets its own Docker Compose file. These allow you to set up whatever project specific services and configuration you need.

None of your projects need to expose a port. Instead each project gets a unique name like `fooproject` and `barproject` and the container listening to port 80 is named {project-name}-web.

It all gets tied together by a single global NGINX/Traefik/Caddy container (you choose) that exposes port 80 and 443 and reverse proxies to each project's web container using Docker's internal hostnames. In pseudo-code:

  https://fooproject.example.site 
  {
    reverse_proxy fooproject- web:80
  }

  https://barproject.example.site 
  {
    reverse_proxy barproject-web:80
  }

The final piece of the puzzle is that the maintainer of DDEV set up a DNS A record for

  127.0.0.1 *.ddev.site

You could do something similar yourself using your own domain or locally with DNSMasq.

It may seem overcomplicated (and it is complicated). But since it's a de-facto standard and well-maintained, that maintenance burden is amortized over all the users and projects. (To the naysayers, consider: PopOS/Ubuntu are quite complicated, but they're far easier to use for most people than a simpler hand-rolled OS with only the essentials.)

lelanthran · 6 months ago

What a bunch of over engineered solutions.

I run multiple side projects on my linux desktop using both postgresql and mysql, and using host entries work well enough.

For HTTP, using entries in hosts file locally allows the clients to all connect to port 80 on the local machine all using different domain names, and nginx can proxy the connection to whatever port the backend is running on.

jasonkester · 6 months ago

Everybody loves their random port numbers these days, but I still prefer custom hostnames.

Just chuck an entry into your hosts file and tell your web server to sniff for it and you’re done. Run your stuff on port 80 like nature intended and never have to teach either end of your app how to deal with colons and numbers in domain names.

And you get to skip the six paragraphs of pain the other commenters are describing, orchestrating their kubernetes and whatnot.

e.g.: http://dev.whatever/

necovek · 6 months ago

I like to combine both locally — bind to port 0 to get an arbitrary unprivileged port, and use domains in /etc/hosts for easier configuration ;-)

For those not familiar with what you are talking about, you can add things like the following to your /etc/hosts file:

  127.0.1.1  foo.local
  127.0.2.4  foo.bar.local
  ...

They will all resolve to local host, so make sure to bind your services to those IP addresses (eg. `nc -l 127.0.2.4 80` will bind only to 127.0.2.4 on port 80).

But running on unprivileged ports like 80 means you've got to run the server as root.

parasti · 6 months ago

At least on macOS you don't even need to add to /etc/hosts, just use .localhost as the TLD.

pacifika · 6 months ago

This is the right answer. But use the .test domain as this is meant for the usecase.

SkyPuncher · 6 months ago

Different ports.

Don't add unnecessary complexity unless it's strictly necessary.

vaylian · 6 months ago

Or, if you want to get a bit more fancy, use different IP addresses.

For IPv4 you have the entire 127.0-255.0-255.1-255 subnet available to you.

For IPv6 you can add additional addresses to your existing network interface.

unsnap_biceps · 6 months ago

I use the 127.xxx.xxx.xxx extensively for local development and add entries in /etc/hosts so I get dns resolution as well. It works great.

xlii · 6 months ago

That’s a huge trap.

Yes, it’s simple at the beginning but it takes a lot of effort to move to non-port based solution for anything.

Cuts are small at the beginning (oh, this service should use other PostgreSQL, so lets have two - oh but I my code doesn’t specify port in an config file, so let me put on direnv - oops IDE didn’t pick up env file) but they grow quickly.

Containers are standard nowadays and allow going for Kubernetes if one wants. With solutions like Justfile or Taskfile its reasonably ergonomic.

It’s a problem future me will always, happily solve.

If my system is large enough and complex enough, it _likely_ means the business behind it is successful. I will always solve “dumb problems from past me” for a successful business than the alternative of not having a business.

armarr · 6 months ago

Depends on which type of scale you are running. And it's not worth overengineering a project for a scale you hope to achieve some day. It will take away engineering resources from building something that provides value today

neilv · 6 months ago

Using Kubernetes can be good for your resume.

What I usually do is to use different ports on my workstation. So I can get the fastest iteration, by keeping things simple. Be careful to keep the ports straight, though.

You can put the port numbers and other installation-specific files in a `.env` file, application-specific config file, or a supplemental build system config file, that aren't checked into Git.

One way I did this was to have a very powerful `Makefile` that could not only build a complicated system and perform many administrative functions, of a tricky deployment, but also work in both development and production. That `Makefile` pulled in `Makefile-options`, which had all the installation-specific variables, including ports. Other development config files, including HTTPS certificates, were generated automatically based on that. Change `Makefile-options` and everything depending on any of those was rebuilt. When you ran `make` without a `Makefile-options` file, it generated one, with a template of the variables you could set.

Today I'd consider using Kubernetes for that last example, but we were crazy-productive without it, and production worked fine.

thierrydamiba · 6 months ago

What is your preferred method to deploy a python api to a frontend?

aprdm · 5 months ago

can use ansible (or ssh) and copy a tarball (or python package or pull docker container)

Containers.

I’ll also recommend commercial OrbStack for Mac, because it simplifies some configuration and is performant.

That was my focus over the last couple months (and right now with customer solution I’m running tens of isolated clusters of heterogenous services and custom network configurations).

I’ve tried nix, local Caddy/Haproxy, Direnvs, Devenvs, dozens of task file runners, DAG runners etc.

Kubernetes is fine but it’s a domain in its own and you’ll end up troubleshooting plenty of things instead of doing work itself.

The simplest solution I would recommend is a task runner (Justfile/Taskfile) with containers (either built or with volumes attached - this prevents secrets leakage). Pay special attention to artifact leakage and clone volumes instead of mutating them.

I don’t recommend Docker Compose because it has low entry and a high ceiling and it takes long time to back out.

For simple clusters (5-8 containers) it’s working well. If you need to level up my personal recommendation would be:

- Go for pure programmatic experience (I’ve tested bunch of API clients and IMO it’s less time learning Go than troubleshooting/backfilling missing control flow features) - there’s also Magefile for simplified flows

- Full Kubernetes with templating language (avoid YAMLs like a plague)

- Cuelang if you want to go for full reliability (but it’s difficult to understand and documentation is one of the worst I ever read through).

gabesullice · 6 months ago

I prefer setting up services that bind to port 0 ("get me an unprivileged port"), report that back, and use that to auto-configure dependent services.

This allows local development and debugging for fast iterations and quick feedback loop. But, it also allows for multiple branches of the same project to be tested locally at the same time!

Yeah, git does not make that seamless, so I'd have multiple shallow clones to allow me to review a branch without stopping work on my own branch(es).

tcoff91 · 6 months ago

Git worktrees make working with multiple branches a breeze

Nice, I wasn't aware of git worktrees at all!