I'm a Bryan Cantrill fan so I'm glad this is working out, I was extremely skeptical of them at the beginning(on HN too), I think because I've built DCs for many years and was stuck in a mindset that served my use case, I've come around to Oxide. My main concerns originally were 2 fold: "this seems bougie", is there actually a market for this, and, is there a good interoperability story with mix and match. From what I could tell the answers were "yes" and "don't care" - I had thought this wasn't a great answer but it seems I'm wrong. I was chatting with Boris Mann just last week about them and he said "actually John that isn't correct, think of how much quick compute needs to come online and how much discreet compute is going to be required with low management overhead, they're doing just fine and that market will grow" - After that I did some research and pondered on it for a day - I think my friend is right and I am wrong, I think at this point Oxide is going to be a really strong name and I wish them the best of luck.
I was skeptical as well, if only because just being a better product isn't enough to win the market. Everything we hear about Oxide sounds like an impressive green field implementation of a data center, but is that enough? Do the people making buying decisions at this scale care if their sysadmins have better tools?
> Do the people making buying decisions at this scale care if their sysadmins have better tools?
Look at who oxide is selling to and for what reasons.
It's about compute + software at rack scales. It does not matter if it is good it matters that it's integrated. Gear at this level is getting sold with a service contract and "good" means you dont have to field as many calls (keeping the margins up).
> Everything we hear about Oxide sounds like an impressive green field implementation of a data center, but is that enough?
Look at their CPU density and do the math on power. It's fairly low density. Look at the interconnects (100gb per system). Also fairly conservative. It's the perfect product to replace hardware that is aging out, as you wont have to re-plumb for more power/bandwidth, and you still get a massive upgrade.
I'm a fan just because they have such an incredibly good sounding product. like, it has no relevance to me, I'll never use it, but I get a deep sense of satisfaction just reading about how it works.
I must admit that I am much more unsophisticated than this, and yet I "invested" in Oxide (by running my own projects off Oxide servers), and it is gratifying to see them continue to grow. My (naive) assessment: (a) agreed with Cantrill's opinions on software, (b) liked his willingness to put himself out there, and (c) felt the eng blogs showed a high level of (socio-)technical ability.
I think for the internet to break out of walled gardens, high-quality independent datacenters need to exist -- nobody wants to manage their own datacenters, and nobody wants to rely on Google/Amazon/Microsoft's platforms or (even worse) business products. I hope this continues.
> If someone else's software is running the hardware
Our stack is open source.
> what difference does it make if its on-prem or offsite?
The difference is not where it runs, it's that you own our racks, rather than rent them. In the traditional cloud, you're renting. Other vendors who sell you hardware will still have you paying software licensing fees, so it never feels like you truly own it. We don't have any licensing fees.
Cloud computing is significantly more expensive than self-hosting for most large organizations. People are slowly figuring that out, and Oxide was a bet on the timing of that realization.
Banks care _a lot_ where the data is store, hence most banks are inclined to keep customer data on-prem.
Because data must be on-prem, banks are stuck in legacy infra paradigms. The whole org suffers, innovation is stiffled, yada yada…
An on-prem cloud product (hardware+software) is a game changer for these companies, IMO.
My question to oxide: how easy is to integrate external hardware into the cloud? For example: bunch of GPUs or a bunch of next-gen hardware like SambaNova.
Is the software open-source with reproducible builds of any runtime binaries?
Oxide has been remarkably transparent about the development and architecture of critical system components. We can only hope they succeed and inspire others to follow their transparency lead.
Even if it makes no sense as a technical thing, businesses will buy it. Look at all the huge companies that keep spending millions trying to DIY their own datacenter for the 3rd time. Enterprise loves to buy big iron and self-hosting crap, so I'm sure they will be successful selling this. However, I think they're going to need to branch out to more services in order to continue increasing their revenue every year (after 5+ years lets say).
> Does everyone at Oxide have the same equity grant?
I thought I saw this question answered in a previous thread and the answer was basically “no”, but the question has been avoided a lot.
Aspects of equity compensation would inherently need to be different over time due to valuation, fundraising stage, and so on. However I always thought it was strange that they made a big deal about paying everyone the same base salary but then were silent on the equity comp strategy. Everyone knows that in a job like this the total comp is important.
The old Oxide compensation discussions were interesting. There was discussion about how they thought candidates asking about compensation to be something of a negative signal because they wanted people who weren’t in it for the money, basically. I heard this from a now ex-employee of Oxide who was describing how to navigate their hiring process, so take with a grain of salt.
> Some will say that we should be talking about equity, not cash compensation. While it’s true that startup equity is important, it’s also true that startup equity doesn’t pay the orthodontist’s bill or get the basement repainted. We believe that every employee should have equity to give them a stake in the company’s future (and that an outsized return for investors should also be an outsized return for employees), but we also believe that the presence of equity can’t be used as an excuse for unsustainably low cash compensation. As for how equity is determined, it really deserves its own in-depth treatment, but in short, equity compensates for risk – and in a startup, risk reduces over time: the first employee takes much more risk than the hundredth.
> There was discussion about how they thought candidates asking about compensation to be something of a negative signal because they wanted people who weren’t in it for the money, basically.
I've heard this from multiple hiring managers and C levels. The cognitive dissonance is amazing.
Do you know why I show up and work? Because I am paid for it, and in this country, medical is also gatekept by employment.
If I wasn't paid, I wouldn't work for them.
But somehow, I'm supposed to not care about money at the same time caring about money.
> There was discussion about how they thought candidates asking about compensation to be something of a negative signal
It's not an unusual way of thinking, but every time I see it, it seems bizarre to me. If the candidate was to propose any project once hired, I'm sure these folks would want them to think about costs and benefits.
This policy selects either for people unable to reflect on their life with the same wit they apply to work; or people who will front about their motivations. Both seem like poor outcomes.
> The old Oxide compensation discussions were interesting. There was discussion about how they thought candidates asking about compensation to be something of a negative signal because they wanted people who weren’t in it for the money, basically.
I can't say anything of Oxide because I don't really know the people involved other than reading their writing.
This is an attitude a lot of industry veterans have. Most of them were in software pre-2015 and have long made their money and paid their debts. They saw the haydays of job hopping, massive salaries, massive equity, and "rockstars" (that never were). The people I know that joined software post-2015 and onward are not in the same financial position.
At this point I've accepted most equity comp in a non public company is worth less than toilet paper.
When it comes time to actually cash out or you notice some animals are more equal than others, the excuses start. The drama happens. No one knows what the equity is worth. If you try and advocate for yourself you'll be told the equity is actually worthless.
From here the 200k salary seems fair. Just don't live in SF. Chicago , Philly, Pittsburgh, Cleveland, quite a few cities are perfectly livable on 200k.
In fact, in any of the above metros you will be living nice.
> There was discussion about how they thought candidates asking about compensation to be something of a negative signal because they wanted people who weren’t in it for the money, basically.
I think companies avoiding discussions about compensation is a negative signal, because they’re only in it for my labor.
> Does everyone at Oxide have the same equity grant?
> equity compensates for risk – and in a startup, risk reduces over time: the first employee takes much more risk than the hundredth.
I think that paragraph answers the question pretty clearly. As an Oxide employee you will get equity. It will generally be less than the people that came before you, but more than people that come after you. So it's obviously not the same as everyone else
"total comp" (salary+equity) is really hard to quantify for a private company. In order to qualify as ISOs, the stock options need to be priced at the Fair Market Value (FMV), which makes them essentially worth ~$0 on paper on the day they are granted. In order to value them differently, you need to guess if/how the company will increase in value in the future. If the gains were guaranteed, then that should be factored into the current FMV, so options always have significant uncertainty.
This is unlike an RSU from a public company, where you can sell the value of your shares as they vest and add that to your income with minor risk of price volatility.
Exactly right. It doesn’t answer the question. It does some cartwheels to avoid to the reality: despite their philosophies and soap boxing on compensation, equity is a form of compensation and their equity is no doubt distributed very, very differently.
It’s intellectually dishonest. I look forward though to their “in depth” follow up post.
> Since originally writing this blog entry in 2021, we have increased our salary a few times, and it now stands at $207,264. We have also added some sales positions that have variable compensation, consisting of a lower base salary and a commission component.
This has been the biggest red flag to me about oxide since these blogs came out. You don’t join a startup for salary and every startup I’ve been at that emphasizes fair salaries is doing that to intentionally discourage people from pushing on total compensation.
Employees apart from the very early ones are likely really getting fucked on equity because oxide explicitly treats candidates asking about equity as a red flag.
Companies structured like this end up turning into a combination of rich people who don’t need money and are just interested in the problem space and then mediocre people who can’t get a better offer. (If you are good, remote work with high TC is definitely available.)
Unless Oxide is giving out huge equity bonuses for good perf (which would make their comp post hypocritical), it’s going to have a continuous talent decline as it grows. There aren’t enough independently wealthy high performers interested in what Oxide is doing to sustain a talent pool.
> (If you are good, remote work with high TC is definitely available.)
But how good do you have to be at things other than the work itself (edit: and how lucky as well) to land one of those jobs with higher compensation? For someone outside of high-cost-of-living tech hubs, Oxide's fixed salary could be life-changing all by itself, even with minimal equity. The fact that they take that deal doesn't necessarily mean that they're mediocre.
Out of curiosity, how would you imagine that working out, technically and specifically? You start out with founders having equity, and maybe some early investors. Then you employ some people and... what happens exactly? Are new shares created, is everyone who was already onboard slowly diluted, what happens when someone joins and leaves? Etc, I don't think there is an approach to this where all sides would be happy, so I'm interested to hear your thoughts.
Not your parent, and not suggesting in any way that Oxide is one, but the key term you want to search to learn more about that kind of arrangement is "worker owned cooperatives." There's a lot of variety in how it's handled.
It depends, there are certainly founders who come from sufficient money that the value of founding a unicorn is not material to their Net worth.
However I’d also suggest that the concentration of tech in the last decade is also partly due to startups chronically stiffing their employees on equity. The difference in compensation potential naturally forces a talent split where talent joins larger and much better compensated firms.
> Are you being snarky and suggesting that employees deserve the same upside that founders deserve
The founders are excluded from the employee compensation discussion. They own the company because they founded it. Nobody thinks they just put the equity into a structure that nobody owns.
The question is whether all employees are compensated equally, which is a very important detail. Giving everyone the same salary is very different than giving everyone the same total compensation.
That isn’t a snarky position; early employees in high output orgs like this generally work just as much as founders do.
The founders aren’t really taking on that much more risk than the rest of the early team; it’s the VC’s money, not theirs.
I absolutely don’t agree with the idea that employees deserve the same upside as founders (because I think initiative and persistence against adversity/inertia is insanely rare and valuable and should be rewarded immensely), but it is not an insane proposition.
It’s especially popular among people who think the actual work output is more important than the leadership initiative. Both are obviously essential, and founders do both, while employees do only the first (or they’d be founders themselves).
I actually think that having more cloud providers might deflate a lot of the pricing. If you think about it, companies like Amazon buy server hardware and then rent it out by the vcpu (with throttling if they can get away with it) per month. Add memory and IO and you are looking at bills that pay for the server in mere months/weeks several tenants carving up all the hardware and each paying tens/hundreds per month.
There are of course benefits to using cloud based VMs and I use them as well. But you are paying a very steep premium for what is a pitiful amount of compute and memory. There's a lot of wiggle room for price decreases and the only thing preventing that is a lack of competition. There's a reason Amazon is so rich: nobody seems to challenge them on AWS pricing. There's value in having them do all the faffing about with hardware of course. That's why companies use them. I'm in GCP; but same principle. I don't want to have to worry about replacing hard disks in the middle of the night, deal with network routers that are misbehaving, cooling issues, etc. That's why I pay them the big bucks. But I'm well aware that it's not that great of a deal.
I used Hetzner a decade ago and paid something like 50 euros per month for a quad core xeon with a raid 1 disks, 32 GB, etc. Bare metal of course. But also, 50 euro. We had five of those. Forget about getting anything close to that with modern cloud providers for anything resembling a reasonable price. Your first monthly bill might actually add up to enough to buy your own hardware. Very tempting. They have beefed up their specs since then. You now get more for less. And they also do VMs now.
I knew it was bad but I didn’t realize just how bad the pricing spread can be until I started dealing with the GPU instances (8x A100 or H100 pods). Last I checked the on-demand pricing was $40/hr and the 1-year reserved instances were $25/hr. That’s over $200k/yr for the reserved instances so within two years I’d spend enough to buy my own 8x H100 pod (based on LambdaLabs pricing) plus enough to pay an engineer to babysit five pods at a time. It’s insane.
With on-demand pricing the pod would pay for itself (and the cost to manage it) within a year.
In my experience, companies seem to want to pay the cloud provider tax in order to avoid capacity planning. Sometimes it makes sense because it is hard to predict when something is going to take off. I have also worked at companies with very predictable growth paying insane amounts. I didn’t understand the logic, but they still were profitable and paid well so whatever.
At the (very) low end it's pretty easy to build your own "cloud" with a NAS, containers, and reverse proxies and tunnels to the outside world. And this will get you suprisingly far.
But at the high end, I think the market is litterally infinite. Every large company should want this, and want it now. Cloud providers are extremely expensive and, outside of the 1-tier where prices are really outrageous, they perform poorly and often offer little support.
like us old assholes were saying when the cloud really started to take off: "this is nuts, it's just someone else's computer! and they're making a profit off of this service, meaning it's more expensive than what we were doing!"
Now a lot of the things that were done pre-cloud were done in bad ways, and I'm not saying that we were right about those things. Having APIs for provisioning and monitoring are far better than submitting a request to some queue and having your VM provisioned manually 1 week later by someone who gets a key detail wrong. APIs and granular permissions are how this should be done, and "the cloud" taught everyone that very early. But a lot of companies are really stuck in the cloud mindset now, and won't let go of it.
I think companies like Oxide and product lines like theirs are going to start becoming common. Microsoft, of course, completely fumbled the ball with Azure Stack, and I've never even heard of anyone deploying AWS Outpost, both for the same reason: the costs for these are absolutely insane for what they provide.
What most folks really want is their own infrastructure running their own stuff using APIs that are either written in-house or provided by some vendor. Oxide is betting that they can sell you a working scalable system for less money than it would take to hire a team to write the APIs that would allow a company to do the same with off-the-shelf hardware. I think that they're probably right about that.
> At the (very) low end it's pretty easy to build your own "cloud" with a NAS, containers, and reverse proxies and tunnels to the outside world. And this will get you suprisingly far.
Anyone can throw together a bunch of parts and software to run Internet-facing services from a closet. That doesn't mean that you're safe from issues that Oxide aims to solve, especially at that small scale.
My homelab (which hosts my blog and a couple of other things) runs off a Topton N17 micro-ATX motherboard ordered on AliExpress, featuring an AMD Ryzen 7 7840HS. Yes, that's a mobile CPU shoehorned onto a desktop platform with a funky mounting bracket to take AM4/AM5 coolers.
Anyways, I wanted to run SmartOS on it, but this system is so janky that the Illumos kernel couldn't find any PCIe devices at all. After spending an afternoon reconfiguring PCIe bridges by hand with the kernel debugger in an attempt to troubleshoot PCIe initialization, I gave up and installed Proxmox.
Admittedly, as far as janky hardware this takes the cake, but the point stands. To paraphrase Bryan, buggy firmware is the sysadmin's worst enemy.
I don’t disagree that there are some fat margins in the cloud, but how is vendor lock-in any different here? Companies could end up paying fat margins to oxide too while still managing physical gear and plant.
I've run my own servers for over 20 years now, but I guess I don't understand the pain point. Can you elaborate? They write:
> Our system delivers all the hardware and software you need to run cloud…
To "run cloud?" Does this mean treating your own servers like "serverless?" Does it mean running Kubernetes? Is this primarily for people who want to self-host LLMs?
I'm literally so old that I write programs that run on a server and never think about infrastructure.
I agree that's a bit awkwardly phrased, let me send in a patch for that.
> Does this mean treating your own servers like "serverless?" Does it mean running Kubernetes? Is this primarily for people who want to self-host LLMs?
Not exactly any of that. We let you treat an entire rack as a single pool of resources, spinning up virtual machines that our control plane manages for you. Think "VPS provider but you own it." There's an API, but if you want to see what our console looks like, you can poke around with a demo here: https://console-preview.oxide.computer/
> To "run cloud?" Does this mean treating your own servers like "serverless?" Does it mean running Kubernetes? Is this primarily for people who want to self-host LLMs?
Agree. But those fat margins can get starting to be shared with CIOs, CTOs and other managers with purchasing power. IMO the constant hectoring at workplace about migration to cloud or cloud native crap is not coming from some deep technical principles. It is more of Do it before you get fired for non-compliance
Oxide, at least for an outsider, looks like a company that channels some of the spirit of early Sun Microsystems (I'm aware of the connections of course). I'm quite envious of those who work there - I hope the demands of big money don't crush any of that spirit.
Sadly when I look at their jobs posted I don't see much that would line up with my skillset, but I keep an eye on them just on the offchance.
If I could, I'd invest. Sure, they might fail, but they're shooting their shot, and to me it has a clear differentiator that would improve the market for most of their users (if not the incumbents).
I wish there was a way for small investors ($25-50k) to get in. AFAIK, the only thing we can do is wait for an IPO and hope we can get in at a reasonable price.
Oxide's blogs talk about cooling _a lot_, apparently their stack runs very cool b/c they've reorganized the whole thing around efficiency, and written all the firmware to support that goal.
Skeptical of that. There's only so much you can do against the physics of moving electrons around at high speeds... "Bigger Fans" and "compute density" doesn't change that
Oxide is doing some great things, but there’s only so much you can do with firmware tweaks. A CPU running any load at all is going to completely eclipse the power usage of everything else in the system.
Incremental improvements from things like more efficient fans and reducing the number of power conversions is great, but the power drawn by the CPUs or GPUs is on another level.
If after that you're not satiated with data center cooling talk, Jane Street's Signals and Threads just did an episode about their cooling infrastructure a few days ago:
Always refreshing to see people who actually believe in software freedoms (and not just doing open source cosplay like many big corporations) forge a pathway to big success.
There are many things that suggest free software and the movement for software freedoms might be on its way to a historical footnote. This is absolutely not one of them.
Hey Bryan, one day when you’re very successful market-wise (you and your team have already obviously been massively successful from an engineering standpoint) and aren’t in crash-priority-override mode to get cash flow, please consider a project to build SME stuff that reaps the security and integration benefits of your big enterprise stuff that is affordable for end users like entrepreneurs and home hobbyists, like Ubiqiti does. I’d love a lil’ $5-10k homelab unit, and I bet a number of smaller universities and organizations would go for stuff in the low 5 figure 2-3kW range. Obviously your bread and butter comes from companies that size their orders by number of racks, but if you never go downmarket then thousands of us hackers that love what you’re doing will never get to touch Oxide stuff except at a job in a megacorp.
It would be a good onramp into their ecosystem. Compare this to the deep educational discounts and the school-targeted platforms from Microsoft, Google, Apple, and such.
The term "open source" was created by corporations in opposition to "free software", as a watered-down version of the latter. Open source itself is already free software cosplay.
There is something so calming and pleasant about a well-structured thesis statement:
> Our thesis was that cloud computing was the future of all computing; that running on-premises would remain (or become!) strategically important for many; that the entire stack — hardware and software — needed to be rethought from first principles to serve this market; and that a large, durable, public company could be built by whomever pulled it off.
Very clear and logical, stating from their first principle world view what the result could be if they succeed.
I would say it's very clear and logical ... but is it really "from first principles"?
I thought that Oxide was based on OpenCompute, which is basically the rack designs that Facebook open sourced, after hiring some Google employees to build their custom data centers. This project started in 2011:
Google definitely rebuilt the stack from first principles -- the data center was basically a huge embedded system, from power to racks to CPUs/memory/disk/network to kernel to user space to cluster software. (And no, they did not use Kubernetes.)
I have no idea how active the OpenCompute project is -- is Facebook still the main contributor, and are they still releasing their new rack designs?
And I also wonder how much Oxide has diverged from it? At least on the hardware side. On the software side, I guess the Illumos-derived parts and Rust parts are completely different.
Well, when they say they did their own:
- board designs
- microcontroller OS
- platform enablement software
- host hypervisor
- switch
- integrated storage service
- control plane
Then yeah it seems like maybe only board designs and the switch COULD have either come from or been influenced by OpenCompute, but maybe those didn't either. (I have no idea tbh)
Maybe they only got the mechanical and power stuff from OpenCompute? i.e. the parts that change more slowly
We didn't use anything from OCP. When we first started the company, we thought we might use the enclosure (and considered ourselves "OCP inspired"), but there ended up being little value in doing so (and there was a clear cost). And on the stuff that we really cared about (e.g., getting rid of the traditional BMC), we were completely at odds with OCP (where ASPEED BMCs abound!).
So in the end, even the mechanical and power didn't come from OCP. We clearly build on other components (we didn't build own rectifiers!), but we absolutely built the machine from first principles.
Good for them! I used to walk my dog past their (office? warehouse?) in Emeryville, and when the weather was warm they'd have the doors open and the giant server stacks just sitting there, looking awesome. I guess it's not really a concern that someone will steal something that looks like it'd take a forklift to move.
Honestly, not particularly. That area is really chill. It's literally across the street from Pixar's headquarters, and a block or so down from city hall. Utility outages are probably the biggest 'issue' in that neighborhood; we had power outages not infrequently, and this week they had their water main burst and shut down half the businesses nearby.
Readit News
Look at who oxide is selling to and for what reasons.
It's about compute + software at rack scales. It does not matter if it is good it matters that it's integrated. Gear at this level is getting sold with a service contract and "good" means you dont have to field as many calls (keeping the margins up).
> Everything we hear about Oxide sounds like an impressive green field implementation of a data center, but is that enough?
Look at their CPU density and do the math on power. It's fairly low density. Look at the interconnects (100gb per system). Also fairly conservative. It's the perfect product to replace hardware that is aging out, as you wont have to re-plumb for more power/bandwidth, and you still get a massive upgrade.
I think for the internet to break out of walled gardens, high-quality independent datacenters need to exist -- nobody wants to manage their own datacenters, and nobody wants to rely on Google/Amazon/Microsoft's platforms or (even worse) business products. I hope this continues.
Our stack is open source.
> what difference does it make if its on-prem or offsite?
The difference is not where it runs, it's that you own our racks, rather than rent them. In the traditional cloud, you're renting. Other vendors who sell you hardware will still have you paying software licensing fees, so it never feels like you truly own it. We don't have any licensing fees.
Because data must be on-prem, banks are stuck in legacy infra paradigms. The whole org suffers, innovation is stiffled, yada yada…
An on-prem cloud product (hardware+software) is a game changer for these companies, IMO.
My question to oxide: how easy is to integrate external hardware into the cloud? For example: bunch of GPUs or a bunch of next-gen hardware like SambaNova.
Oxide has been remarkably transparent about the development and architecture of critical system components. We can only hope they succeed and inspire others to follow their transparency lead.
Deleted Comment
Deleted Comment
You mean all the huge companies that ran multiple datacenters before the cloud was even a thing?
>We decided to do something outlandishly simple: take the salary that Steve, Jess, and I were going to pay ourselves, and pay that to everyone. [https://oxide.computer/blog/compensation-as-a-reflection-of-...]
Does everyone at Oxide have the same equity grant?
I thought I saw this question answered in a previous thread and the answer was basically “no”, but the question has been avoided a lot.
Aspects of equity compensation would inherently need to be different over time due to valuation, fundraising stage, and so on. However I always thought it was strange that they made a big deal about paying everyone the same base salary but then were silent on the equity comp strategy. Everyone knows that in a job like this the total comp is important.
The old Oxide compensation discussions were interesting. There was discussion about how they thought candidates asking about compensation to be something of a negative signal because they wanted people who weren’t in it for the money, basically. I heard this from a now ex-employee of Oxide who was describing how to navigate their hiring process, so take with a grain of salt.
EDIT: I checked their website again. The compensation link goes to a blog post ( https://oxide.computer/blog/compensation-as-a-reflection-of-... ) which has this section about equity:
> Some will say that we should be talking about equity, not cash compensation. While it’s true that startup equity is important, it’s also true that startup equity doesn’t pay the orthodontist’s bill or get the basement repainted. We believe that every employee should have equity to give them a stake in the company’s future (and that an outsized return for investors should also be an outsized return for employees), but we also believe that the presence of equity can’t be used as an excuse for unsustainably low cash compensation. As for how equity is determined, it really deserves its own in-depth treatment, but in short, equity compensates for risk – and in a startup, risk reduces over time: the first employee takes much more risk than the hundredth.
Which doesn't answer the question.
I've heard this from multiple hiring managers and C levels. The cognitive dissonance is amazing.
Do you know why I show up and work? Because I am paid for it, and in this country, medical is also gatekept by employment.
If I wasn't paid, I wouldn't work for them.
But somehow, I'm supposed to not care about money at the same time caring about money.
It's not an unusual way of thinking, but every time I see it, it seems bizarre to me. If the candidate was to propose any project once hired, I'm sure these folks would want them to think about costs and benefits.
This policy selects either for people unable to reflect on their life with the same wit they apply to work; or people who will front about their motivations. Both seem like poor outcomes.
I can't say anything of Oxide because I don't really know the people involved other than reading their writing.
This is an attitude a lot of industry veterans have. Most of them were in software pre-2015 and have long made their money and paid their debts. They saw the haydays of job hopping, massive salaries, massive equity, and "rockstars" (that never were). The people I know that joined software post-2015 and onward are not in the same financial position.
When it comes time to actually cash out or you notice some animals are more equal than others, the excuses start. The drama happens. No one knows what the equity is worth. If you try and advocate for yourself you'll be told the equity is actually worthless.
From here the 200k salary seems fair. Just don't live in SF. Chicago , Philly, Pittsburgh, Cleveland, quite a few cities are perfectly livable on 200k.
In fact, in any of the above metros you will be living nice.
Most valuable people know they’re valuable, and do (and should!) negotiate compensation.
I think companies avoiding discussions about compensation is a negative signal, because they’re only in it for my labor.
> equity compensates for risk – and in a startup, risk reduces over time: the first employee takes much more risk than the hundredth.
I think that paragraph answers the question pretty clearly. As an Oxide employee you will get equity. It will generally be less than the people that came before you, but more than people that come after you. So it's obviously not the same as everyone else
This is unlike an RSU from a public company, where you can sell the value of your shares as they vest and add that to your income with minor risk of price volatility.
It’s intellectually dishonest. I look forward though to their “in depth” follow up post.
> Since originally writing this blog entry in 2021, we have increased our salary a few times, and it now stands at $207,264. We have also added some sales positions that have variable compensation, consisting of a lower base salary and a commission component.
Employees apart from the very early ones are likely really getting fucked on equity because oxide explicitly treats candidates asking about equity as a red flag.
Companies structured like this end up turning into a combination of rich people who don’t need money and are just interested in the problem space and then mediocre people who can’t get a better offer. (If you are good, remote work with high TC is definitely available.)
Unless Oxide is giving out huge equity bonuses for good perf (which would make their comp post hypocritical), it’s going to have a continuous talent decline as it grows. There aren’t enough independently wealthy high performers interested in what Oxide is doing to sustain a talent pool.
But how good do you have to be at things other than the work itself (edit: and how lucky as well) to land one of those jobs with higher compensation? For someone outside of high-cost-of-living tech hubs, Oxide's fixed salary could be life-changing all by itself, even with minimal equity. The fact that they take that deal doesn't necessarily mean that they're mediocre.
However I’d also suggest that the concentration of tech in the last decade is also partly due to startups chronically stiffing their employees on equity. The difference in compensation potential naturally forces a talent split where talent joins larger and much better compensated firms.
Dead Comment
Are you being snarky and suggesting that employees deserve the same upside that founders deserve?
How is it silly if they already do that for salaries? Co-ops with equal ownership isn't unheard of, and isn't silly at all.
The founders are excluded from the employee compensation discussion. They own the company because they founded it. Nobody thinks they just put the equity into a structure that nobody owns.
The question is whether all employees are compensated equally, which is a very important detail. Giving everyone the same salary is very different than giving everyone the same total compensation.
The founders aren’t really taking on that much more risk than the rest of the early team; it’s the VC’s money, not theirs.
I absolutely don’t agree with the idea that employees deserve the same upside as founders (because I think initiative and persistence against adversity/inertia is insanely rare and valuable and should be rewarded immensely), but it is not an insane proposition.
It’s especially popular among people who think the actual work output is more important than the leadership initiative. Both are obviously essential, and founders do both, while employees do only the first (or they’d be founders themselves).
Plus I predict more companies will exit the cloud once they realize how thick the margins have become or want better guarantees over sovereignty.
There are of course benefits to using cloud based VMs and I use them as well. But you are paying a very steep premium for what is a pitiful amount of compute and memory. There's a lot of wiggle room for price decreases and the only thing preventing that is a lack of competition. There's a reason Amazon is so rich: nobody seems to challenge them on AWS pricing. There's value in having them do all the faffing about with hardware of course. That's why companies use them. I'm in GCP; but same principle. I don't want to have to worry about replacing hard disks in the middle of the night, deal with network routers that are misbehaving, cooling issues, etc. That's why I pay them the big bucks. But I'm well aware that it's not that great of a deal.
I used Hetzner a decade ago and paid something like 50 euros per month for a quad core xeon with a raid 1 disks, 32 GB, etc. Bare metal of course. But also, 50 euro. We had five of those. Forget about getting anything close to that with modern cloud providers for anything resembling a reasonable price. Your first monthly bill might actually add up to enough to buy your own hardware. Very tempting. They have beefed up their specs since then. You now get more for less. And they also do VMs now.
With on-demand pricing the pod would pay for itself (and the cost to manage it) within a year.
That’s the only thing I can tell is useful about “the cloud”
I can build racks and servers easily, but the challenge is availability and getting past everyone’s firewalls
So the real win is any service that allows for instant DNS table updates and availability of DNS whitelisting.
This is why Google, Msft etc win in email because they have trusted endpoints
Alternative routes with self signed DKIM etc is more or less blocked by default forcing you onto a provider
We need more cloud flare tunnel and local hosting via commercial ISP routes and less new centralized data centers
I've also been doing web-related work for a living for over 25y and yours is the most spot-on take I've seen in this discussion.
Dead Comment
But at the high end, I think the market is litterally infinite. Every large company should want this, and want it now. Cloud providers are extremely expensive and, outside of the 1-tier where prices are really outrageous, they perform poorly and often offer little support.
This really feels like the future.
Now a lot of the things that were done pre-cloud were done in bad ways, and I'm not saying that we were right about those things. Having APIs for provisioning and monitoring are far better than submitting a request to some queue and having your VM provisioned manually 1 week later by someone who gets a key detail wrong. APIs and granular permissions are how this should be done, and "the cloud" taught everyone that very early. But a lot of companies are really stuck in the cloud mindset now, and won't let go of it.
I think companies like Oxide and product lines like theirs are going to start becoming common. Microsoft, of course, completely fumbled the ball with Azure Stack, and I've never even heard of anyone deploying AWS Outpost, both for the same reason: the costs for these are absolutely insane for what they provide.
What most folks really want is their own infrastructure running their own stuff using APIs that are either written in-house or provided by some vendor. Oxide is betting that they can sell you a working scalable system for less money than it would take to hire a team to write the APIs that would allow a company to do the same with off-the-shelf hardware. I think that they're probably right about that.
Anyone can throw together a bunch of parts and software to run Internet-facing services from a closet. That doesn't mean that you're safe from issues that Oxide aims to solve, especially at that small scale.
My homelab (which hosts my blog and a couple of other things) runs off a Topton N17 micro-ATX motherboard ordered on AliExpress, featuring an AMD Ryzen 7 7840HS. Yes, that's a mobile CPU shoehorned onto a desktop platform with a funky mounting bracket to take AM4/AM5 coolers.
Anyways, I wanted to run SmartOS on it, but this system is so janky that the Illumos kernel couldn't find any PCIe devices at all. After spending an afternoon reconfiguring PCIe bridges by hand with the kernel debugger in an attempt to troubleshoot PCIe initialization, I gave up and installed Proxmox.
Admittedly, as far as janky hardware this takes the cake, but the point stands. To paraphrase Bryan, buggy firmware is the sysadmin's worst enemy.
You can "just" migrate by exporting or importing the vms.
I'm literally so old that I write programs that run on a server and never think about infrastructure.
I agree that's a bit awkwardly phrased, let me send in a patch for that.
> Does this mean treating your own servers like "serverless?" Does it mean running Kubernetes? Is this primarily for people who want to self-host LLMs?
Not exactly any of that. We let you treat an entire rack as a single pool of resources, spinning up virtual machines that our control plane manages for you. Think "VPS provider but you own it." There's an API, but if you want to see what our console looks like, you can poke around with a demo here: https://console-preview.oxide.computer/
Machine/system/service deployment via API.
See §Essential Characteristics
> On-demand self-service; Broad network access; Resource pooling; Rapid elasticity; Measured service
* https://en.wikipedia.org/wiki/Cloud_computing
None are generally applicable to standalone pizza boxes as managed individual (as opposed to being herded by (e.g.) OpenStack).
Sadly when I look at their jobs posted I don't see much that would line up with my skillset, but I keep an eye on them just on the offchance.
And such a clear value statement.
If I could, I'd invest. Sure, they might fail, but they're shooting their shot, and to me it has a clear differentiator that would improve the market for most of their users (if not the incumbents).
Folks seem to be moving toward liquid cooling[4] either to the rack/chassis[5] or even to the chip[6].
[1] https://oxide.computer/blog/how-oxide-cuts-data-center-power...
[2] https://www.youtube.com/watch?v=4vVXClXVuzE
[3] https://www.youtube.com/shorts/hTJYY_Y1H9Q
[4] https://blogs.nvidia.com/blog/blackwell-platform-water-effic...
[5] https://datacentremagazine.com/data-centres/top-10-liquid-co...
[6] https://zutacore.com
Incremental improvements from things like more efficient fans and reducing the number of power conversions is great, but the power drawn by the CPUs or GPUs is on another level.
https://signalsandthreads.com/the-thermodynamics-of-trading/
There are many things that suggest free software and the movement for software freedoms might be on its way to a historical footnote. This is absolutely not one of them.
Hey Bryan, one day when you’re very successful market-wise (you and your team have already obviously been massively successful from an engineering standpoint) and aren’t in crash-priority-override mode to get cash flow, please consider a project to build SME stuff that reaps the security and integration benefits of your big enterprise stuff that is affordable for end users like entrepreneurs and home hobbyists, like Ubiqiti does. I’d love a lil’ $5-10k homelab unit, and I bet a number of smaller universities and organizations would go for stuff in the low 5 figure 2-3kW range. Obviously your bread and butter comes from companies that size their orders by number of racks, but if you never go downmarket then thousands of us hackers that love what you’re doing will never get to touch Oxide stuff except at a job in a megacorp.
> Our thesis was that cloud computing was the future of all computing; that running on-premises would remain (or become!) strategically important for many; that the entire stack — hardware and software — needed to be rethought from first principles to serve this market; and that a large, durable, public company could be built by whomever pulled it off.
Very clear and logical, stating from their first principle world view what the result could be if they succeed.
I thought that Oxide was based on OpenCompute, which is basically the rack designs that Facebook open sourced, after hiring some Google employees to build their custom data centers. This project started in 2011:
https://en.wikipedia.org/wiki/Open_Compute_Project
https://github.com/opencomputeproject
Google definitely rebuilt the stack from first principles -- the data center was basically a huge embedded system, from power to racks to CPUs/memory/disk/network to kernel to user space to cluster software. (And no, they did not use Kubernetes.)
I have no idea how active the OpenCompute project is -- is Facebook still the main contributor, and are they still releasing their new rack designs?
And I also wonder how much Oxide has diverged from it? At least on the hardware side. On the software side, I guess the Illumos-derived parts and Rust parts are completely different.
Well, when they say they did their own:
Then yeah it seems like maybe only board designs and the switch COULD have either come from or been influenced by OpenCompute, but maybe those didn't either. (I have no idea tbh)Maybe they only got the mechanical and power stuff from OpenCompute? i.e. the parts that change more slowly
So in the end, even the mechanical and power didn't come from OCP. We clearly build on other components (we didn't build own rectifiers!), but we absolutely built the machine from first principles.
Why does it have to be a public company though?
The ultimate aspirational homelab setup, ngl.