I recently faced this same challenge and solved it using Amazon SQS, a simple Lambda function, and a corresponding SQS consumer/relay service running inside the private LAN.
Essentially GitHub's webhook is configured to hit the Lambda function, which authenticates the request and adds the data to an SQS queue. The internal relay service watches that queue and passes requests to the private Jenkins server.
The architecture is similar to this[1] project, but since the public-facing piece runs on Lambda, it costs almost nothing to run.
This is almost what we do as well. The difference being that we have a API Gateway in front that just invokes the lambda on an internal network, which validates the webhook data and only then does it forward it.
Takes the complexity of having to use a queue out of the equation, though at the expense of potentially lost webhook calls.
Looks nice. The key difference seems to be the OpenZiti solution enables you to manage by identities (instead of IP addresses), and close all the inbound firewall ports? However, believe the two solutions would work together?
Yes, if you also need users to log into Jenkins from outside the private network (without a VPN), it sounds like OpenZiti would be a good option. In my case, Jenkins is only used from within the LAN. The SQS solution authenticates GitHub webhooks using the sha256 hmac signature (not by IP), and no inbound ports need to be open.
Also embracing OpenZiti allows you to give users access to Jenkins securely (or any other app you want/need to keep off the public internet) without using a classic style VPN.
I was moving towards using userspace ZeroTier (via the Python bindings as a SSH proxy command) to allow Bitbucket build runners to access our private network repo server, for a similar result. The bitbucket runners run containers in such a way that the normal Zerotier wouldn't run (something about not having a capability flag for tun/tap IIRC).
In the end bitbucket released self-hosted runners and we're now running the builds in our network with normal firewall rules limiting access.
I'm still looking at overlay networks (probably Nebula), and want to take another look at OpenZiti (I peeked at it a couple months ago).
Author here, I want to learn more about ZeroTier too. I overlooked it for this project because it supports layer 2 and I needed tighter coupling of application+policy than layer 2 provides. Admittedly, I don't yet know enough about ZeroTier to know if I can disqualify all layer 2 points of contact with a particular policy. It sounds like the layer 2 DNA of ZeroTier tripped you up in this case too.
I know more about OpenZiti than Nebula and will be listening for topics in the Discourse in case you decide to take another stab at it. Nebula is definitely interesting. I got the impression that it's all about automating spin up and tear down of point-to-point VPN tunnels on demand, which has the potential to eliminate much of the headaches around maintaining VPNs. Essentially app and policy configuration driven over hand crafting. That's definitely the approach of OpenZiti too.
I haven't really used the layer 2 functionality of ZeroTier, I know it has it. I believe in the built-in firewalling rules in ZeroTier you can limit the protocols that it will pass. The rules for it are, in my limited use, pretty obscure, but I have had some luck at setting up rules.
When taking another look, you may find this brief set of 'superpowers' useful - https://www.youtube.com/playlist?list=PLMUj_5fklasKF1oisSSuL.... This includes closing all inbound ports (or FW rules) so that the apps become dark. I believe ZitiTV is going to have a comparison episode too soon incl. ZeroTier and Nebula.
As someone working on OpenZiti itself, I'd be interested in why choose Nebula over OpenZiti. I assume it'd just be due to the cache that Nebula has, since it comes from Slack? I am biased, I know, but OpenZiti offers a lot of interesting stuff over Nebula.
If it is a usability issue I'd really like to know since I'd want to make that experience better.
I remember my first look at OpenZiti now. Here's some feedback:
My first impression of OpenZiti was: Nebula but at the application level.
It took a fair bit of looking around for me to see the proxy component. I'm still not exactly sure how to get a test, but it looks like the getting started steps are:
- Build a network.
- Figure out the identities and services.
- Set up a proxy.
- Test out this proxy.
- Eventually migrate services from proxy to SDK.
There's a lot of stuff going on here I really need to wrap my mind around before I can even see how it would fit into my network.
Nebula on the other hand is an easier mapping for what I already have in my brain: It's a mesh VPN with a CA and built in firewall rules based on certificate roles.
OpenZiti looks very interesting now, but it honestly took me an hour to get to that point.
I can't recall where exactly I saw OpenZiti, it was probably from some discussion on HN. I went to look at it and thought it looked promising, but I'm already partway down the path of Nebula, and as you say it's partly because of Nebula having Slack experience behind it, partly because I have some experience with it, partly I like the architecture. I'll take a peek at OpenZiti again and refresh my understanding of it.
We wanted to stop external network attacks on our internal development environment while allowing tools like Jenkins to connect to other internet based tools such as GitHub. We do not want to have any inbound ports or deal with VPNs, ACLs, bastions, complex firewall rules etc. We dogfooded our opensource technology, OpenZiti to achieve this. We would love to know your thoughts.
>Everyone that needs to log in to Jenkins runs an OpenZiti tunneler on their computer. Each tunneler has an identity installed on the same computer. The OpenZiti network administrator issues these identities via email or chat in the form of one-time enrollment tokens and assigns role attributes that match up with a service policy. The service policies grant permission for those tunneler identities to use Jenkins. We control the service policies through a web console or CLI. As soon as the policy grants access to an enrolled identity, the tunneler automatically configures a private domain name and IP route to the Jenkins server on that user’s computer.
How do you get everyone to run the tunneler on all the machines that they could use?
For example our company uses subcontractor companies that have their own network admin team that manages their work computer and a strict VPN. Getting an employee to run a tunneler sounds like a non starter.
> How do you get everyone to run the tunneler on all the machines that they could use?
There's apps in the stores for Android/iOS/MacOS and installers for Windows. It's basically equivalent to running any other "VPN type of client". Wireguard does the same thing (in fact for Windows we used Wintun from Wireguard which was awesome). OpenVPN/Cisco/ZeroTier etc.
OpenZiti is also working on an "install-less" option for users that can't (or don't want to) install apps on their own. It's not ready/rolled out but hopefully it will be soon.
The tunneling apps provide some pretty cool functionality though, most notable is private, authenticated DNS. IF you were running the tunneller you could go to "jenkins.ziti" or whatever made up DNS name you like.
If you can't install a vpn client like that then yah, it'd be a non-starter for accessing jenkins like this.
You're absolutely right. Its an interesting solution to the problem but I'm not sure what additional benefit this solution bought them that isn't covered by standard paradigm of VPN w/ role based access controls. (at least for user access)
They did however correctly identify the fact Jenkins is pretty much the holy grail of targets for bad actors and hackers. Failing to identify this has caused more then fair share of known hacks (to not Jenkins specifically but any build/automation system that has the required insane levels of access).
A strict VPN locked down and for external ingress access of automated actions a restrictive proxy sitting on the edge significantly lowers the attack surface. Operationally also much cheaper to maintain.
can be difficult but one key is the openziti tunneler is the opposite of a traditional vpn client - it only intercepts apps by policy, e.g. it would protect the Jenkins traffic, but ignore whatever else the contractor is doing (assuming that is the policy).
there are also gateway models (contractor to gateway leg is left alone; the openziti magic then applies between gateway and Jenkins/your enviro).
are there other good options for this contractor scenario?
That's pretty nice, but won't work integrations that don't let you run custom code.
In those situations it's a good idea to run an authenticating reverse proxy in front of your Jenkins instance, and allow certain paths in case certain query parameters are set.
So for regular access, your users need to auth with your SSO.
Webhooks include something like `?apiToken=12345` in their request, which you check in the reverse proxy config, and then use to allow access to the relevant URLs.
To your point, not all SaaS or vendors with integrations let you run your own code. If they do allow it there's probably an OpenZiti-native way to build-in the connectivity you need.
One alternative is running a tunneler like our users did for their continued access to the Jenkins UI. This is like a VPN client that can run pretty much anywhere, and so there's an excellent chance that it's already in the app store or coming soon for whatever platform you have in mind.
As for an authenticated proxy, thanks for that idea. I do think that a hardened internet edge like that has utility in cases like this, like NGrok + SSO. Another reader suggested their solution involving a third-party queue, essentially offloading the internet exposure problem to a public cloud provider, which is basically what I did here but with different software.
It definitely was *the way* to do it before OpenZiti and other zero trust overlay networks started to take off. Being "in the VPN" is similar to participating on the OpenZiti overlay network. VPNs allow for horizontal movement much easier than being on an OpenZiti zero trust overlay network does. With OpenZiti you don't need any holes in your firewall open. That's a pretty good reason to consider moving away from VPNs to a zero trust overlay like OpenZiti in my book, but I'm biased, I work on the project. Embracing the zero trust networking mindset is definitely a change of pace but I think it's worth considering.
Good job avoiding public internet exposure, but the problem with being "on the VPN" is that it has all the problems of the internet at a smaller scale, so you're still exposed to an untrustworthy network. I'd say that's the core tenant of so-called "zero trust" philosophy.
In the past I have just restricted Jenkins access to our main static IP, and whitelisted GitHub's published Webhook IP ranges. Remote employees just need to login to the office VPN to access the UI & SSH.
Does the articles method provide further flexibility? It seems much more complex!
It's probably "similarly complex" but it's more complex at start because you won't have an OpenZiti overlay network already... So that's more complex for starters. Once you are familiar with OpenZiti it'll be easy. So there's an uphill swim for you since you're already familiar with SSH and firewall rules...
OpenZiti's big advantage here is that if those users pick up and move to starbucks, dunkin' or wherever - you don't need to whitelist an IP any more. it'll all "just work" because OpenZiti treats all networks as hostile/compromised following the zero trust networking principles. There's also a LOT of other stuff that OpenZiti will provide that is above and beyond SSH.
One other big factor is no open holes/firewall rules. That's a big benefit imo too.
So yes - there's a lot more flexibility with OpenZiti vs SSH/firewall rules. I don't want to just keep going on and on here (but i can :) )
I think the flexibility of operating independently of IP addresses is the key distinction of this approach. It also works for the web UI, not just the webhooks from GitHub.
Readit News
Essentially GitHub's webhook is configured to hit the Lambda function, which authenticates the request and adds the data to an SQS queue. The internal relay service watches that queue and passes requests to the private Jenkins server.
The architecture is similar to this[1] project, but since the public-facing piece runs on Lambda, it costs almost nothing to run.
[1] https://github.com/osbuild/webhook-relay
Takes the complexity of having to use a queue out of the equation, though at the expense of potentially lost webhook calls.
In the end bitbucket released self-hosted runners and we're now running the builds in our network with normal firewall rules limiting access.
I'm still looking at overlay networks (probably Nebula), and want to take another look at OpenZiti (I peeked at it a couple months ago).
I know more about OpenZiti than Nebula and will be listening for topics in the Discourse in case you decide to take another stab at it. Nebula is definitely interesting. I got the impression that it's all about automating spin up and tear down of point-to-point VPN tunnels on demand, which has the potential to eliminate much of the headaches around maintaining VPNs. Essentially app and policy configuration driven over hand crafting. That's definitely the approach of OpenZiti too.
If it is a usability issue I'd really like to know since I'd want to make that experience better.
Any input you wanted to share would be amazing!
My first impression of OpenZiti was: Nebula but at the application level.
It took a fair bit of looking around for me to see the proxy component. I'm still not exactly sure how to get a test, but it looks like the getting started steps are:
There's a lot of stuff going on here I really need to wrap my mind around before I can even see how it would fit into my network.Nebula on the other hand is an easier mapping for what I already have in my brain: It's a mesh VPN with a CA and built in firewall rules based on certificate roles.
OpenZiti looks very interesting now, but it honestly took me an hour to get to that point.
How do you get everyone to run the tunneler on all the machines that they could use?
For example our company uses subcontractor companies that have their own network admin team that manages their work computer and a strict VPN. Getting an employee to run a tunneler sounds like a non starter.
There's apps in the stores for Android/iOS/MacOS and installers for Windows. It's basically equivalent to running any other "VPN type of client". Wireguard does the same thing (in fact for Windows we used Wintun from Wireguard which was awesome). OpenVPN/Cisco/ZeroTier etc.
OpenZiti is also working on an "install-less" option for users that can't (or don't want to) install apps on their own. It's not ready/rolled out but hopefully it will be soon.
The tunneling apps provide some pretty cool functionality though, most notable is private, authenticated DNS. IF you were running the tunneller you could go to "jenkins.ziti" or whatever made up DNS name you like.
If you can't install a vpn client like that then yah, it'd be a non-starter for accessing jenkins like this.
They did however correctly identify the fact Jenkins is pretty much the holy grail of targets for bad actors and hackers. Failing to identify this has caused more then fair share of known hacks (to not Jenkins specifically but any build/automation system that has the required insane levels of access).
A strict VPN locked down and for external ingress access of automated actions a restrictive proxy sitting on the edge significantly lowers the attack surface. Operationally also much cheaper to maintain.
there are also gateway models (contractor to gateway leg is left alone; the openziti magic then applies between gateway and Jenkins/your enviro).
are there other good options for this contractor scenario?
In those situations it's a good idea to run an authenticating reverse proxy in front of your Jenkins instance, and allow certain paths in case certain query parameters are set.
So for regular access, your users need to auth with your SSO.
Webhooks include something like `?apiToken=12345` in their request, which you check in the reverse proxy config, and then use to allow access to the relevant URLs.
One alternative is running a tunneler like our users did for their continued access to the Jenkins UI. This is like a VPN client that can run pretty much anywhere, and so there's an excellent chance that it's already in the app store or coming soon for whatever platform you have in mind.
As for an authenticated proxy, thanks for that idea. I do think that a hardened internet edge like that has utility in cases like this, like NGrok + SSO. Another reader suggested their solution involving a third-party queue, essentially offloading the internet exposure problem to a public cloud provider, which is basically what I did here but with different software.
Does the articles method provide further flexibility? It seems much more complex!
OpenZiti's big advantage here is that if those users pick up and move to starbucks, dunkin' or wherever - you don't need to whitelist an IP any more. it'll all "just work" because OpenZiti treats all networks as hostile/compromised following the zero trust networking principles. There's also a LOT of other stuff that OpenZiti will provide that is above and beyond SSH.
One other big factor is no open holes/firewall rules. That's a big benefit imo too.
So yes - there's a lot more flexibility with OpenZiti vs SSH/firewall rules. I don't want to just keep going on and on here (but i can :) )