pyjug (u/pyjug) - Readit News

pyjug commented on Collection of best practices for providing and consuming webhooks webhooks.fyi/... · Posted by u/m3h

tasn · 4 years ago

Migrating to a new service: you can just update the webhook URL. On the other hand, there are at least a couple of problems with allowing even one redirect: it opens a pandora's box of security implications, and it's a performance penalty that is paid both by the sender and the receiver on every webhook sent. Realistically, 3xx are most likely to be a mis-configurations (e.g. including a trailing slash where one shouldn't) so I think being noisy about it is a great idea.

Not saying that there aren't valid use-cases (e.g. maybe some sort of dynamic webhook receiving), I'm just saying that we made this choice given the above, and we would be willing to change it if it's ever a barrier for someone.

pyjug · 4 years ago

>you can just update the webhook URL

Fair enough, but often consumers that use multiple vendors receive webhooks in the same service; think about going from /v1/webhooks -> /v2/webhooks, they'd have to change the URL for every vendor. Easier to redirect first then update the URLs later. I think it's a reasonable expectation that a HTTP client would honor redirects as long as the usage isn't malicious (like loops etc)

pyjug commented on Collection of best practices for providing and consuming webhooks webhooks.fyi/... · Posted by u/m3h

tasn · 4 years ago

Those are all great points! At Svix (we do webhooks as a service), we disallow redirects because of what you mentioned above, and because it's also just bad for performance (for both sides) and is most likely a configuration error anyway.

Resolution and timeouts: the aiohttp library for Python is slightly better in terms of letting you configure these things, though it's better to just use a sending proxy that does it all for you and is also located in an isolated VPC to make sure that you're protected.

pyjug · 4 years ago

Disallowing redirects altogether is probably too big of a hammer. There are legit reasons to use redirects (like migration to new versions). A limit to the number of redirects seems ideal -- that's what Twilio does, for example.

pyjug commented on Collection of best practices for providing and consuming webhooks webhooks.fyi/... · Posted by u/m3h

cuu508 · 4 years ago

There are some interesting attack vectors to be aware of if you run a service where users can define webhooks, and your service will will call the user-defined webhooks to notify about certain system events. In my case, a monitoring service which can send notifications by calling user-defined webhook.

* Timeouts: the user can set up a webhook receiver that takes very long to generate a response. Your service must be able to deal with that.

* Timeouts (slowloris): the webhook target could be sending back one byte at a time, with 1 second pauses inbetween. If you are using, say, the "requests" python library for making HTTP requests, the "timeout" parameter will not help here

* Private IPs and reserved IPs: you probably don't want users defining webhooks to http://127.0.0.1:<some-port> and probing your internal network. Remember about private IPv6 ranges too

* Domains that resolve to private IPs: attacker could set up foo.com which resolves to a private IP. It is not enough to just validate webhook URLs when users set them up.

* HTTP redirects to private IPs. If your HTTP client library follows HTTP redirects, the attacker can set up a webhook endpoint that redirects to a private IP. Again, it is not enough to validate the user-supplied URL.

* Excessive HTTP redirects. The attacker can set up a redirect loop - make sure this does not circumvent your timeout setting.

My current solution for all of the above is to use libcurl via pycurl. I wrote a wrapper that mimics requests API: https://github.com/healthchecks/healthchecks/blob/master/hc/... (may contain bugs, use at your own risk :-)

pyjug · 4 years ago

>Domains that resolve to private IPs: attacker could set up foo.com which resolves to a private IP

There's a clever extension to this attack; a naive way to mitigate it is to do a DNS resolution first to verify it's not a private IP and then do the actual request. An attacker can simply return a public IP on the first DNS resolution (with a 0 TTY) and then return a private IP on the second. This is called a "TOCTOU" (time-of-check time-of-use) vulnerability. I've written about this and other security best practices on my blog here - https://www.ameyalokare.com/technology/webhooks/2021/05/03/s...

I've also built an egress proxy that prevents such attacks here - https://github.com/juggernaut/webhook-sentry

Same caveat applies, use at your own risk :-)

pyjug commented on Security issue related to the NPM registry github.blog/2021-11-15-gi... · Posted by u/ManuelKiessling

pyjug · 4 years ago

> In this architecture, the authorization service was properly validating user authorization to packages based on data passed in request URL paths. However, the service that performs underlying updates to the registry data determined which package to publish based on the contents of the uploaded package file

I have a similar story . At my old job, we had a web socket gateway that authenticated using JWTs , then hit an internal service to request REST resources. The issue was that it didn’t actually validate the requested REST resource URL; a malicious user could authenticate as themselves but request a resource for any other account.

I found it as I was getting up to speed on the code base, having recently switched teams. Funnily enough, nobody on the team really understood the vulnerability - the EM marked it low priority and wanted the team to work on other things. I had to essentially go directly to the security team and convince them it was a sev 1. I sometimes wonder if it’s easier to just report security issues as an outsider through the bug bounty program; internal reports don’t seem to get taken as seriously.

pyjug commented on Apple still planning return to in-person work despite complaints macrumors.com/2021/06/30/... · Posted by u/geerlingguy

pyjug · 5 years ago

The FAANG have a lot more leverage over their employees than people seem to think.

Very few if any outside of these companies pay a mid level engineer 700k (yes, this number is real, can we please stop debating this fact?). So, if all FAANG force their companies back to the office, the employees have effectively no leverage.

This is similar to how the top cos colluded to keep engineer wages low in the 2000s, except they don’t need to explicitly make such a pact.

pyjug commented on AWS BugBust aws.amazon.com/about-aws/... · Posted by u/belter

jedberg · 5 years ago

> Sorry, I’m confused. This seems like a change in your position?

There is a benefit to both employees and employers. Being on the leaderboard benefits the employee because it raises their profile. Having an employee on the leaderboard, as an employer, also raises the company's profile. It's like how companies send people to conferences to talk. It benefits the employee by raising their profile but also benefits the company.

> I’m asking you in good faith. Why the contortions to defend such a product?

I'm not really making contortions. This product is aimed at large enterprises. It's not aimed at startups (so I would never suggest it for any company I invest in) and it's not aimed at companies that are already well known (FAANG). It's aimed at the 1000s of other large enterprises out there who want to incentivize their employees with some prizes to do a job they're hopefully already doing while also possibly raising the company profile a bit by having people show up on the leaderboard.

But it definitely solves a problem for it's target audience, which you don't seem to be a part of. I am not their target audience either. But I see the value to their target audience.

pyjug · 5 years ago

> I'm not really making contortions

I would politely disagree — in any case the thread is there for all to judge.

If you sincerely believe that this is a product that should exist and will positively impact everyone involved, I can only hope you’re right [0]. In the end it’s all up to your conscience, but I do wish our leaders in tech were a bit more sensitive to seeing the broader picture.

[0]: pour one out for all the naive devs that will no doubt sink 100s of hours to be on the leaderboard for little benefit to them.

pyjug commented on AWS BugBust aws.amazon.com/about-aws/... · Posted by u/belter

jedberg · 5 years ago

> would an engineer at Netflix ever “play this game” to win ?

I think you're missing the point. The leaderboard is an extra bonus. The main thing is squashing bugs you were already squashing. But yes, there are definitely engineers at Netflix who would play this to "win", just for the love of the game.

But let me relate another similar story -- when Amazon released the Deepracer, they added a leaderboard. All around the world, people got paid by their employers to compete. I think the first winner worked for Samsung and was paid her full time salary just to compete on Deepracer. They saw it as a point of pride that one of their employees was on the leaderboard, with enough value to pay her to work on it full time. I'm pretty sure that is what they are going for here with the global leaderboard. Everyone wins in that situation. The company gets to brag about having the best bug squasher, the employee gets to get paid for squashing bugs, and Amazon gets publicity.

pyjug · 5 years ago

You didn’t address whether engineers at Netflix would put up with this.

> I'm pretty sure that is what they are going for here with the global leaderboard

Sorry, I’m confused. This seems like a change in your position? Earlier you argued the intent is for employees to be able to showcase their skill to recruiters… you also responded to someone else who said they already have recruiter attention with “you are not their intended audience”

> paid her full time salary just to compete on Deepracer

Wait, are you seriously comparing a global competition in cutting edge tech to squashing bugs … in your own codebase? Come on.. there’s no comparison. If anything, a company would be semi embarrassed that they have so many bugs in their pre existing codebase.

I’m asking you in good faith. Why the contortions to defend such a product? I see from your profile that you are an investor and advisor. I would seriously try pushing this product to your companies and see how they react. If they do adopt it, run an anonymous survey of the devs .. then I would love to see you argue with a straight face that it “benefits developers”

pyjug commented on AWS BugBust aws.amazon.com/about-aws/... · Posted by u/belter

jedberg · 5 years ago

I'm not sure where the exploitation is. The developers are already getting paid to fix bugs. Now they have some structure and added incentives.

pyjug · 5 years ago

The structure already exists. It’s called your bug tracker. By building tools to gamify this process, it’s clearly incentivizing devs to put in more hours. I have no doubt that the global leaderboard will consist of devs that would’ve clocked 100s of hours above their regular hours. For very little benefit to the employee.

Let me put it you this way - would an engineer at Netflix ever “play this game” to win ? Would employees at Netflix even put up with something like this? If not, why would you think employees at random tech co should?

pyjug commented on AWS BugBust aws.amazon.com/about-aws/... · Posted by u/belter

jedberg · 5 years ago

> This invalidates your other point that it benefits developers.

Not at all. The company undervalues their employee and sees little risk in them showing up on the board, and the employee overvalues their skill, thinking that showing up on the leaderboard is a shoo-in. Maybe the employer is right, maybe the employee is right, or maybe both or neither is right.

Also, the employee does it because they get paid to hunt bugs regardless. It just helps the company by putting a structure around what they are already doing.

pyjug · 5 years ago

> the employee overvalues their skill, thinking that showing up on the leaderboard is a shoo-in. Maybe the employer is right, maybe the employee is right,

Come on, this is just being coy — you and I both know this is a terrible deal for employees — as you write earlier, there’s a very low chance that a random dev will appear on the global leaderboard. What’s more, the best devs won’t even play this game, so it’s left for juniors and people trying to “prove themselves” to trample over each other for peanuts.

Not to single you out, but I wish influential tech folks like you would speak up more about the cynical exploitation that’s going on here in plain sight.

pyjug commented on AWS BugBust aws.amazon.com/about-aws/... · Posted by u/belter

jedberg · 5 years ago

The risk to the company is pretty low that their employee would show up on a global leaderboard. And even if they do, they can give them a raise or other benefits. It helps them determine the market rate for their employee.

pyjug · 5 years ago

> The risk to the company is pretty low that their employee would show up on a global leaderboard.

This invalidates your other point that it benefits developers. Why would a dev sink countless hours for a “very low” chance at being on the global leaderboard and hence moving jobs.

You can’t simultaneously posit it’s good for both companies and developers.