I think Meta's tooling is inferior to industry standard. I actually took a survey while I was there, and that wasn't the majority opinion, but frankly I think most outsiders would absolutely agree.
Things like Eden were a great idea, but tools had all sorts of issues they gloss over (a virtual file system can be really slow if you have a ton of small files), dev environments would randomly fail a lot, really the only tool they had that nobody disliked was their log-searching thing (can't remember what it's called) but it was still lightyears behind something likes splunk.
It was my conclusion that "Wow you have 15 people working on a dev-tool compared to a public company with 100 building the industry-standard version over 10 years with actual product managers and UI experts, no wonder ours looks like crap... wouldn't it be cheaper just to take .01% of your salary and buy standard dev tools"
I guess "Clunky" is the word I'm looking for. "Blow it away and make a new one" was a phrase that happened with some regularity for dev-envs, repo-checkouts, etc. And iirc restarting your dev box took like >30min.
---
Side note - the other strangest thing was some of these tools people agreed were terrible (restart takes 30min). So you'd expect thousands of engineers to be swarming any system with any UI bug, edge-case, or whatever. But it just didn't work that way.
For what it's worth, as a Meta employee, I generally disagree with this comment. I mean, calling it 'inferior' is just laughable. It's way better than what open source projects have access to and it's way better than Amazon internal tooling (is that 'industry standard' enough?).
The sapling workflow is better than git. I was skeptical when I joined, but between no branches and excellent stack and merge support, it's just a better, more intuitive workflow.
Eden is fast. Crazy fast, in fact, if you look at the size of the monorepo. The comment about small files is odd to me. Source code are small files. The entire monorepo is just small files.
Now if we are going to complain about something, it would be the custom Android environment we've got. Android Studio integration is actually clunky, bordering on non-existent. Compared to writing vanilla Android code, which I have done for years before joining Meta, Meta folks wrapped almost every API, so all your pre-existing Android knowledge is useless. I've never been so unproductive when writing Android code.
Fastmail had great developer tooling. Give yourself about two minutes from a issued command via slack or CLI and you'd have a brand new tagged developer box the live system that was a self contained infrastructure in a box of everything from the imap server to the user interface. Another command you could get as many copies for whatever development branch, etc.
The entire system replicated the real system (not user data but the software) and you could spin up test addresses and accounts and all sorts trivially. Beautiful!
> Eden is fast. Crazy fast, in fact, if you look at the size of the monorepo.
You’re measuring the wrong thing. The user actions are slow and take several seconds to complete usually. No one gives a shit how big the repo is. They care how long their operation takes.
>> Amazon internal tooling (is that 'industry standard' enough?).
I'm not sure, maybe?
When I say industry-standard tools I mean "Best you can purchase." So I mean a UI as good as github, a platform as good as AWS (meta's doesn't hold a candle to AWS), full-text log searching as good as splunk, chat as fast/searchable as slack (workplace doesn't hold a candle), video chat as fast/clear/clean as Zoom (workplace doesn't hold a candle)
Some problems are harder at scale (feature toggles interactions just get harder with size), but some of them are just a mess of their own making (restarting taking 30m, butterfly rules having >20m+ delays, eden not being optimized to work with buck, which needs to read tons of files out of tons of directories)
No company I've worked at after FB/Meta ships or works even at a non-eng level with the same velocity. I always attributed that to their internal tools, since most of the other companies seem to be using the same crap. Slack is straight up painful compared to their chat system. And don't get me started on the how good the task tool is compared to literally any other ticketing system out there. Everything is behind their intern tool, and usually built to all just work together without friction.
>Slack is straight up painful compared to their chat system.
As someone who vastly prefers Slack over all its competitors, I'm very interested in hearing how Meta's compares. What makes it better than Slack for you?
Its not the tools, its the culture. Meta cut red tape in its early years especially. I know there is more "process" now, but they empowered teams and individuals to just make decisions and move on and the ecosystem around everything supports this.
They'd be just as fast with Slack or whatever, for the most part.
EDIT: as someone posted below:
> The open-ness of code, visibility, diffs. It was perfection. Something broke in my env suddenly? OH, I just checked recently pushed diffs that affect my realm. Hey there it is, security pushed something weird. I'll just revert the part that affects me and tag them. No meeting, maybe a SEV for visibility and review, maybe not. Easy peasy.
This is the kind of empowerment developers are given and expected to handle (both the explicitly ability to revert previously committed code and the implicit responsibility that the teams code that was overridden must deal with it once they're notified rather than push back)
Shipping faster should not be the only metric, the amount of SEVs should be given much higher weight. The attitude of most engineers is to show impact and if there is a SEV, it is better as it will show even more impact. This slows other people down and causes a gradual decline is actually shipping things that matter - quality over quantity.
> No company I've worked at after FB/Meta ships or works even at a non-eng level with the same velocity.
This is a curious thing to see.
As a mere user, and speaking purely about FB not any other Meta IP…
Other than unbreaking things that break when the OS and browsers change under you, has Facebook shipped even one thing in the last five years?
Don't get me wrong, I know it takes a lot of effort to stand still — forgetting the Red Queen effect was Musk's obvious mistake with Twitter even before stuff broke — but Facebook seems completely unchanged.
I can't really compare to Meta, but I will confess that "industry standard" tooling has been very disappointing for me. Trying to get a build setup that automatically pulls in dependency changes once a week is something I'm still not entirely sure how to do in a way that everyone agrees with. Seems most tools bake that into a code commit with a lock file nowadays. But even that is amusingly recent. Years ago, you had to mirror any repository system locally and learn how to set that up. Probably getting it wrong in the process.
Mentioning splunk, I'm assuming you are using more paid industry tools. I suspect that opens things up a bit more, but realize that most of the industry doesn't use those due to prohibitive pricing. And I don't even know what the appropriate paid tool for building would be.
We seem remarkably primed to just hate whatever tooling we have at our disposal in ways that baffles me.
I remember when Cisco mentioned potential acquisition of Splunk for $20B, the jokes about whether it was to buy the company or to pay the running Splunk bill wrote themselves.
I know splunk is expensive, especially if you don't have somebody who is actively monitoring your spend, but sumo is a pretty good alternative. It's actually something I look for in the interview.
Regardless, many companies underspend by an order of magnitude on dev tools. If a tool makes you 1% more efficient, it's worth them putting 1% of your annual salary into it (multiply this by engineers at the company who need the tool). Soon you'll see that a tool costing hundreds of thousands to license a year is often a steal.
So you may say "oh splunk is overkill and unfair expectation" but in their CI/CD system, if your job failed (frequent occurrence) and you wanted to search for something in there you'd manually DOWNLOAD a 300 meg file to your machine and grep it for errors.
Basically a fang engineer's time costs $3 a minute minimum, so if a query could save 1 minute then the break-even cost is paying up to $3 to run that query.
FB's log shipping and parsing is much better than "industry" however we ship billions of logs to get a metric, because the internal graphite service isn't easy to use. and unidash really sucks compared to grafana for non-SQL based metric discovery and prototyping.
As a Meta employee as well, and working in the data analytics / engineering space, I'm finding tooling to be pretty high standards actually.
Though we arguably rely on a lot of Apache products, whatever we use that's internal only is great to work with. Daiq** recently started supporting notebooks, which has been a game changer for us as well as for the teams we work with. Phabricator is great as well, and makes shipping stuff super easy. Only Ben**, the internal notebook solution, I find meh. Especially compared to Google Colab. But the rest has simply been a joy to work with.
I would say it’s quite uneven: most tools were better than the state of the art when they were introduced.
For many, the world has moved on a lot since, and those tools feel obsolete but so embedded in practices that it’s unthinkable to apply better approaches. One example of that is tracking data sources: legacy is slow, dysfunctional, and even fairly straightforward questions time out because you have to load so much meta-data. That’s because no one really uses it: most of that information is carried by the Data engineers who have built the system, and they are over-whelmed with questions that could be answer with a good system, but derive power over analysts. They benefit from it through comments they copy in their review, so fixing it isn’t anyone’s priority.
Others have evolved because the internal demand pushed things forward. PMs want Deltoid to work, so that system has moved to be state-of-the-art or interestingly unique in many ways: scaling, not implementing MCC, integration with the metric definitions.
My current role for the first time exposed me to amounts of data and variety of data (in type and how/where it's stored) that's difficult to learn organically. I wonder how a FAANG does that at "scale".
> most of that information is carried by the Data engineers who have built the system, and they are over-whelmed with questions that could be answer with a good system, but derive power over analysts.
Is that the kind of question you're talking about here?
Strongly not my experience. The tooling at meta was so good compared to my previous 5 jobs that I left with changed opinions about the tradeoffs of investing in tooling.
Yeah, that's fair. I think another thing is that a lot of these internal tooling projects date back 5-10 years when open source alternatives may not have been viable yet. For example, my company uses an in-house written time series database, which probably made sense at the time because when the project started, Prometheus wasn't 1.0 yet. Now it's reaching growing pains as we've scaled, and at this point it's a bit of a sunk cost fallacy to not migrate off of it.
Another thing is that sometimes it's truly impossible to use one-size fits all for some tools if your company scale is large enough. I've heard from friends at Amazon that the dev experience and tools are totally different in the hardware space than it is for those working in the retail or AWS orgs.
> For example, my company uses an in-house written time series database, which probably made sense at the time because when the project started, Prometheus wasn't 1.0 yet.
That's why big orgs are now opensourcing their tools. But it's not to clean up all "too internal" stuff from a tool.
It's interesting to hear the contrast between Meta and my experience talking with ex-Googlers who complain that open-source or industry standard infrastructure is far inferior to what they were used to at Google. Why such a big chasm between the internal tool quality at Meta and Google?
I think GP's opinion is fairly contrarian. I would say the in-house tools at Meta are substantially better than typical open source or commercial tools used at smaller companies.
FWIW I am a Meta-to-Google transplant and I feel the opposite way. There’s a lot of internal tooling and functionality I miss dearly. Most of all, Workplace.
I REALLY miss Phabricator, Scuba and (gasp) Tasks. Would love to have those back in industry. Far better than Github, Superset (?) and of course the dreadful Jira I have to use these days.
I was very impressed by Scuba's ability to ingest and search data... and very unimpressed by the interface. I found it terribly unintuitive, and often didn't get you what you wanted without having to run multiple queries.
Phabricator was pretty sweet... and the internal version of mercurial was a dream!
I don't know anything about tooling at Meta, but I have appreciated custom-built tooling at all the jobs I have worked at so far -- much smaller companies ranging from about 100 to 5000 people.
Some of this is just having worked in the industry for a while. Guess what, 15 years ago "Put all your code in a web service running on AWS" was not nearly as slam-dunk a proposition as it is now; and tools optimized for on-prem were faster and more reliable.
The other thing is that you don't always have to reinvent things to the point that Meta does. You can just wrap around an Open Source project, or even contribute to it. This is what all of my previous jobs did -- make custom tooling from industry-standard OSS building blocks; and contribute back when it made sense for the larger community.
I actually think Meta was actually trying to do this with Mercurial a while ago [1], you probably have a much better idea than us as to why that didn't work out.
Yeah, I've worked at another large tech company who had accumulated a lot of in house tooling with a lot of invested SMEs who had been there a long time and missed that the state of the art out in the world had passed them by.
I left FB in the early-mid 2010s and can attest to this. It took a year or so to adjust to the "real world." Things like, no, you don't have a fancy scheduler; you get SSH, ansible, and systemd timers. Docker, Jenkins, and Kubernetes (RIP Mesos) were all new to me when other devs/devops folks had years of experience.
Like a few others already commented, I also disagree. Was at FB for 7 years, and now almost 2 years at a place with the “industry standard” things like slack / datadog / github / sourcegraph / (20+ other tools all disconnected from the rest and all behind SSO).
As a backend engineer, the dev experience was just incomparably better at FB. Some things I most most are probably phabricator / stacked diffs workflows, buck dep management with everything in a monorepo, deep integration across all the the tooling etc.
As an ex-meta mobile engineer, I 100% agree that I found Eden to be impossible to use productively. I always had to pre-warm the cache, pretty much defeating its stated goals. Maybe it has gotten better since I left a year ago, but I had 2 TB of local storage, and mercurial's sparse checkouts were more than enough for me and far more reliable.
However, all the other tooling around code were just superior in every way. I miss Phabricator, and mercurial and the way it seamlessly integrates into a team. I also miss all of the command line helpers that let me manage all of it.
It used to be the case that legitimate teams could form around and focus just on building internal tools for other teams if it improved engineering velocity. Not sure if that's still the case in this new era of layoff-happy Meta, but it was definitely true when I was there.
Eden definitely works better on a dev server and sucks on a laptop, especially arm based mac's (it's slowly getting better but still requires occasional reboots). +1 on the coding tools being great
as a former employee who has worked both in many other companies over a decade + starting a few startups, Meta tooling is one of the best if not the best, it's so good to a point where Saas companies came out of facebook simply by replicating their internal tooling, like Asana, Scuba, etc
The problem with metas approach to tooling from a very high level viewpoint is that they virtualize everything. They abstract all common tasks like building, testing or running by building an entirely new system that builds an abstraction layer.
Abstraction layers tend to be slow.
What they should focus on is plumbing. Take what exists and connect it in a smart way.
Take React: the idea of applying functional programming and spitting out HTML was a great idea. HTML existed before react. Then they also implemented a virtual-DOM, which was the unnecessary part.
Or react-native: rendering UIs in a functional way is great, but stuffing JavaScript into everything is not necessary.
Every tool Meta has, Google also has. Google open source theirs so the industry gravitates towards that, but Meta keeps theirs internal. So is Meta's tooling better than Industry/Google open source tools? probably not. The only tool I was impressed by was scuba. But Meta basically build it out of gold (thousands of machines with TB's of ram)
Sapling looks quite cool! I've used git extensively in my career and consider myself as having a slightly-more-advanced-than-typical understanding of how to use it just based on conversations with colleagues. However, one thing that's always been very limiting with git has been stack-based PR reviews, and as they mentioned amending deep commits. It's not impossible, but it makes it awkward enough that I usually avoid it if possible.
Curious if anyone has used Sapling after lots of time using git. Is it the future?
Some of the recent tool started to come out to fix Git unfriendly UX.
Meta's Sapling (1) is definitely one of them. But there is also `jj` (2) and `git-branchless` (3). These tools target a smaller set of workflow where there is 1 main branch inside a big repo and everything else are short-lived branch/topic that could be treated as ephemeral stack of patches, constantly being uproot / rebase on top of the main branch to derive final result.
If that's the workflow you use daily, then you should give these tools a try.
When we first switched from git to hg internally I was real cranky. Git was the clear industry winner, and hg was adding a whole bunch of churn for no good reason.
Now, hg is amazing and when I leave I will be extremely sad to go back to git. The UX is well thought out, with commands mapping to operations (how do you undo a commit? hg uncommit vs git playing around with the reflog).
Amending deep commits is pretty good -- it's still tricky and absorb works on some pretty limited heuristics. But the merge tooling is pretty good around moving around in a stack, and the general UX over interacting with the stack is way better: hg histedit edits history, while hg rebase moves the stack around.
I'm one of the earliest GitHub users, have run FOSS projects, etc. Sapling is absolutely excellent. If anything, the 'sl web' UI alone, which can do rebases/commits, is worth giving it a shot. 'sl web' makes Git rebase look like the dark ages.
The UX just has lots and lots of polish in small ways and it has a nice amount of good features. It has fewer verbs than Git, but it's still a bit different from Mercurial. Having a built in 'undo' command that basically always works is nice.
I have replaced Git with Sapling (and a similar-but-not-the-same system, Jujutsu) in most of my own personal workflows at this time. If that's a good enough endorsement for you, then I suggest trying it out. You might be surprised.
Probably not, if only for reasons of inertia. Git plus third-party review tools (like GitHub) is more than "good enough" for most purposes.
I used to work at FB, and while sapling is quite nice to use in practice, without the internal version of Phabricator to do code review (and, in all likelihood, mononoke), I don't think I'd pick it up again.
reviewstack adds the missing stack and versioning phab UX on top of github apis, been enjoying using it outside meta for some oss stuff, and fixed a limitation of it recently in https://reviewstack.dev/facebook/sapling/pull/656
This kind of stuff puts me off from wanting to work at FB. If I got really good at working with these tools (or fell in love with this tooling), I wouldn't really be able to go back to working "in the real world"
For me the barrier to working at Facebook is the fact their product is causing so much harm in our world, from elections to the environment to mental health to the breakdown of social capital.
I know I’ll be downvoted for taking an ethical rather than financial or technological position, but ethics matter. Especially in technology and finances.
It's true. Exiting FB's incredible in-house information & developer tools culture is for many people one of the hardest things about leaving. Ex-fb groups maintain lists of potential "in the wild" replacements for each tool but very few are up to the task (pun intended).
> If I got really good at working with these tools (or fell in love with this tooling), I wouldn't really be able to go back to working "in the real world"
SWEs need to learn new tools all the time. This shouldn't be a concern. Besides, it's not like "the real world" only use a single tool. Each company has their own tweaks, even those using open source stuff.
Big tech has _way_ better tools than what most companies use. At Amazon so many concerns were taken care of for you like:
* How do I setup a new project?
* How do I build code?
* How do I pull internal dependencies?
* How do I publish artifacts?
Every company makes a horrible copy of these systems with off-the-shelf tools. At Amazon there were certainly flaws in the internal tools, but overall it was much better than the "real world".
Isn't this the Google curse? Companies hire an engineer who was at Google and suddenly find the Google engineer is building out all the Google internal infrastructure within the new company and because You Aren't Google, it sucks.
No see, what happens is you leave then go to some other company and complain about how shitty their tools are then build a crappy half-baked version of whatever you had at $BigTechCorp and when shit hits the fan you boomerang back to $BigTechCorp for a sweet promo and raise.
I avoid hiring people coming from places like this for reasons similar. Extends past tools into libraries and frameworks, databases and other middleware, developer and business workflows. Only knowing proprietary stuff is a handicap. Assuming that proprietary stuff is superior because the big ad companies prefer it is even worse.
I know git is complex, and the UX is sometimes messy, but after really, really learning it (shoutout to the Github training folks), I've never had any problems that couldn't be solved. I understand the desire to simplify some things, and their log looks way better than gits, but I wish they'd contribute back instead of rolling their own entire VCS.
The one thing that is super exciting to me is the stacked pull request support. Using Github for this kind of workflow is enormously painful. Conversations constantly get outdated, and its nearly impossible to track whether comments have been addressed.
I know they're working on an improved UX/experience there, but it seems like it'll be a good long while, especially for enterprise server customers.
> I've never had any problems that couldn't be solved.
Stockholm syndrome. I've used Git for 15 years, early GitHub user, etc. Yes, you can solve many of these things, but until recently even things like "I am changing patch 2 in a series of 5 and need to rebase the following 3" were ridiculously painful. This is a common workflow many people like (including the Linux kernel devs) and Git was bad at it.
Git submodules. I'm not even going to go into this, they're so bad. That's a problem I wish Git had never "Solved" to spare us the burden.
There are tons of minor nits in Git all over the place. "Solving" something is completely different from actually having something that can be easily used for your team. There's no amount of contributing Facebook could have done to fix Git, because they'd be turning Git into something else that it fundamentally is not. And it doesn't matter if you have a trillion dollars, it's often not practical to just overhaul someone else's whole project when these goals don't align.
Is an intellectually dishonest fantasy invented for the sole purpose of using it to discredit and distract from criticism of the actions of the inventor of the phrase, so should never be ascribed as the source of a position you want to argue against unless your intent is to signal that your own position lacks a reasonable argument and you are just choosing to character-assassinate the opposition to cover for that.
I know git makes you shove toothpicks under your fingernails in order to let you use the keyboard, but after really, really learning to do this, i've never found it a blocker for my daily work. I understand the desire to simplify things, but i wish they'd contribute back ways of making people more comfortable with the toothpicks rather than just removing them and starting from scratch.
I am not a fan of FB, but they tried - you can find them on the git mailing list where they got told they are doing it wrong for things like "scaling" or "productivity". Which is always ironic since basically nobody in open source generates or uses any real data about productivity, it's all just gut feelings about users.
I think the biggest issue is that back around 2011/2012 ish, when Facebook devs went to git core devs and asked how they could get git to scale to the size of their predicted monorepo, the response was roughly "no, shard it".
git falls over and dies really, really badly when the repo gets stupidly large.
"Contribute back to our piece of crap that is 99% antagonistic to your use case" is not realistic. No amount of third party contributions to git will relieve git of its opinions about how development workflow should be done, and those opinions are not shared with every organization.
Their internal Phabricator is quite different from the open source version. When I worked there (around 2017), it was definitely miles ahead of anything I'd used before. A lot of how it works is probably... controversial, I guess, but it suited my mental model quite well.
I used Phab at a job and it was a complete mess. Stacked Diffs only sort-of worked, CI integration was bad, notifications were so noisy everyone tuned them out, etc.
Talking with a Meta person, it sounds like Phab really needs Mercurial to work well, at least for Stacked Diffs because you need to be able to identify commits independent of their location in history to properly maintain the Stacked Diff associations.
I wonder whether they'll continue using their in-house Phrabricator or choose to support Phorge (https://we.phorge.it/) now that open-source Phabricator is no longer supported.
Internal Phabricator isn't even really Phabricator anymore. In fact, the name changed to just Diffs. I'm pretty sure its entire codebase has been rewritten, at the very least into Hack from PHP. They have a custom API; not Conduit. Etc.
So, no. They wont support Phorge. They really never supported open-source Phabricator after Evan left and made it his own with Phacility.
reviewstack is the thing now, adds some parts of phab UX ontop of github apis. Been enjoying using it for oss stuff oursite meta and fixed a limitation of it recently, https://reviewstack.dev/facebook/sapling/pull/656 shows what it looks like with the versioning available
I am genuinely curious about wasabi, the python LSP they announced on Meta open source but is not available anywhere. Would love to try that out, there is not enough competition in the LSP space in Python and it would foster new development https://developers.facebook.com/blog/post/2022/07/18/enablin...
In the “Offline + Online Processing” section, are they talking about an external service that needs to run alongside the processing that occurs on your machine? Or am I misunderstanding that?
The linked article has a section about IDE that this one does not touch. I was surprised they are locking them selves in to one tool with their workflow.
I'm a bit worried about the dominance of VS Code. I can't stand the editor and it's popularity just grows and grows.
They aren't locking themselves in. It just makes sense to support less IDEs than more IDEs. You can move faster if you don't have to duplicate your work between vscode, intellij, android studio, emacs, vim, etc. Nothing is stop developers from using ed, the standard editor, if they wanted to.
Readit News
Things like Eden were a great idea, but tools had all sorts of issues they gloss over (a virtual file system can be really slow if you have a ton of small files), dev environments would randomly fail a lot, really the only tool they had that nobody disliked was their log-searching thing (can't remember what it's called) but it was still lightyears behind something likes splunk.
It was my conclusion that "Wow you have 15 people working on a dev-tool compared to a public company with 100 building the industry-standard version over 10 years with actual product managers and UI experts, no wonder ours looks like crap... wouldn't it be cheaper just to take .01% of your salary and buy standard dev tools"
I guess "Clunky" is the word I'm looking for. "Blow it away and make a new one" was a phrase that happened with some regularity for dev-envs, repo-checkouts, etc. And iirc restarting your dev box took like >30min.
---
Side note - the other strangest thing was some of these tools people agreed were terrible (restart takes 30min). So you'd expect thousands of engineers to be swarming any system with any UI bug, edge-case, or whatever. But it just didn't work that way.
The sapling workflow is better than git. I was skeptical when I joined, but between no branches and excellent stack and merge support, it's just a better, more intuitive workflow.
Eden is fast. Crazy fast, in fact, if you look at the size of the monorepo. The comment about small files is odd to me. Source code are small files. The entire monorepo is just small files.
Now if we are going to complain about something, it would be the custom Android environment we've got. Android Studio integration is actually clunky, bordering on non-existent. Compared to writing vanilla Android code, which I have done for years before joining Meta, Meta folks wrapped almost every API, so all your pre-existing Android knowledge is useless. I've never been so unproductive when writing Android code.
The entire system replicated the real system (not user data but the software) and you could spin up test addresses and accounts and all sorts trivially. Beautiful!
You’re measuring the wrong thing. The user actions are slow and take several seconds to complete usually. No one gives a shit how big the repo is. They care how long their operation takes.
It's slowly changing but I wouldn't consider Amazon's tooling to be industry standard by any definition.
I'm not sure, maybe?
When I say industry-standard tools I mean "Best you can purchase." So I mean a UI as good as github, a platform as good as AWS (meta's doesn't hold a candle to AWS), full-text log searching as good as splunk, chat as fast/searchable as slack (workplace doesn't hold a candle), video chat as fast/clear/clean as Zoom (workplace doesn't hold a candle)
Some problems are harder at scale (feature toggles interactions just get harder with size), but some of them are just a mess of their own making (restarting taking 30m, butterfly rules having >20m+ delays, eden not being optimized to work with buck, which needs to read tons of files out of tons of directories)
As someone who vastly prefers Slack over all its competitors, I'm very interested in hearing how Meta's compares. What makes it better than Slack for you?
They'd be just as fast with Slack or whatever, for the most part.
EDIT: as someone posted below:
> The open-ness of code, visibility, diffs. It was perfection. Something broke in my env suddenly? OH, I just checked recently pushed diffs that affect my realm. Hey there it is, security pushed something weird. I'll just revert the part that affects me and tag them. No meeting, maybe a SEV for visibility and review, maybe not. Easy peasy.
This is the kind of empowerment developers are given and expected to handle (both the explicitly ability to revert previously committed code and the implicit responsibility that the teams code that was overridden must deal with it once they're notified rather than push back)
This is a curious thing to see.
As a mere user, and speaking purely about FB not any other Meta IP…
Other than unbreaking things that break when the OS and browsers change under you, has Facebook shipped even one thing in the last five years?
Don't get me wrong, I know it takes a lot of effort to stand still — forgetting the Red Queen effect was Musk's obvious mistake with Twitter even before stuff broke — but Facebook seems completely unchanged.
Mentioning splunk, I'm assuming you are using more paid industry tools. I suspect that opens things up a bit more, but realize that most of the industry doesn't use those due to prohibitive pricing. And I don't even know what the appropriate paid tool for building would be.
We seem remarkably primed to just hate whatever tooling we have at our disposal in ways that baffles me.
Regardless, many companies underspend by an order of magnitude on dev tools. If a tool makes you 1% more efficient, it's worth them putting 1% of your annual salary into it (multiply this by engineers at the company who need the tool). Soon you'll see that a tool costing hundreds of thousands to license a year is often a steal.
So you may say "oh splunk is overkill and unfair expectation" but in their CI/CD system, if your job failed (frequent occurrence) and you wanted to search for something in there you'd manually DOWNLOAD a 300 meg file to your machine and grep it for errors.
Basically a fang engineer's time costs $3 a minute minimum, so if a query could save 1 minute then the break-even cost is paying up to $3 to run that query.
Though we arguably rely on a lot of Apache products, whatever we use that's internal only is great to work with. Daiq** recently started supporting notebooks, which has been a game changer for us as well as for the teams we work with. Phabricator is great as well, and makes shipping stuff super easy. Only Ben**, the internal notebook solution, I find meh. Especially compared to Google Colab. But the rest has simply been a joy to work with.
For those interested, a former DE made this nice repo that maps internal tools against "real world" products: https://github.com/thijsessens/xmeta2external
For many, the world has moved on a lot since, and those tools feel obsolete but so embedded in practices that it’s unthinkable to apply better approaches. One example of that is tracking data sources: legacy is slow, dysfunctional, and even fairly straightforward questions time out because you have to load so much meta-data. That’s because no one really uses it: most of that information is carried by the Data engineers who have built the system, and they are over-whelmed with questions that could be answer with a good system, but derive power over analysts. They benefit from it through comments they copy in their review, so fixing it isn’t anyone’s priority.
Others have evolved because the internal demand pushed things forward. PMs want Deltoid to work, so that system has moved to be state-of-the-art or interestingly unique in many ways: scaling, not implementing MCC, integration with the metric definitions.
> most of that information is carried by the Data engineers who have built the system, and they are over-whelmed with questions that could be answer with a good system, but derive power over analysts.
Is that the kind of question you're talking about here?
Another thing is that sometimes it's truly impossible to use one-size fits all for some tools if your company scale is large enough. I've heard from friends at Amazon that the dev experience and tools are totally different in the hardware space than it is for those working in the retail or AWS orgs.
That's why big orgs are now opensourcing their tools. But it's not to clean up all "too internal" stuff from a tool.
Phabricator was pretty sweet... and the internal version of mercurial was a dream!
Some of this is just having worked in the industry for a while. Guess what, 15 years ago "Put all your code in a web service running on AWS" was not nearly as slam-dunk a proposition as it is now; and tools optimized for on-prem were faster and more reliable.
The other thing is that you don't always have to reinvent things to the point that Meta does. You can just wrap around an Open Source project, or even contribute to it. This is what all of my previous jobs did -- make custom tooling from industry-standard OSS building blocks; and contribute back when it made sense for the larger community.
I actually think Meta was actually trying to do this with Mercurial a while ago [1], you probably have a much better idea than us as to why that didn't work out.
----------------------------------------
[1] https://engineering.fb.com/2014/01/07/core-data/scaling-merc...
It did work out.
As a backend engineer, the dev experience was just incomparably better at FB. Some things I most most are probably phabricator / stacked diffs workflows, buck dep management with everything in a monorepo, deep integration across all the the tooling etc.
However, all the other tooling around code were just superior in every way. I miss Phabricator, and mercurial and the way it seamlessly integrates into a team. I also miss all of the command line helpers that let me manage all of it.
It used to be the case that legitimate teams could form around and focus just on building internal tools for other teams if it improved engineering velocity. Not sure if that's still the case in this new era of layoff-happy Meta, but it was definitely true when I was there.
Abstraction layers tend to be slow.
What they should focus on is plumbing. Take what exists and connect it in a smart way.
Take React: the idea of applying functional programming and spitting out HTML was a great idea. HTML existed before react. Then they also implemented a virtual-DOM, which was the unnecessary part.
Or react-native: rendering UIs in a functional way is great, but stuffing JavaScript into everything is not necessary.
s/Meta's tooling/Meta/g
s/inferior/very inferior/g
Curious if anyone has used Sapling after lots of time using git. Is it the future?
Meta's Sapling (1) is definitely one of them. But there is also `jj` (2) and `git-branchless` (3). These tools target a smaller set of workflow where there is 1 main branch inside a big repo and everything else are short-lived branch/topic that could be treated as ephemeral stack of patches, constantly being uproot / rebase on top of the main branch to derive final result.
If that's the workflow you use daily, then you should give these tools a try.
(1): https://github.com/facebook/sapling (2): https://github.com/martinvonz/jj/ (3): https://github.com/arxanas/git-branchless
Now, hg is amazing and when I leave I will be extremely sad to go back to git. The UX is well thought out, with commands mapping to operations (how do you undo a commit? hg uncommit vs git playing around with the reflog).
Amending deep commits is pretty good -- it's still tricky and absorb works on some pretty limited heuristics. But the merge tooling is pretty good around moving around in a stack, and the general UX over interacting with the stack is way better: hg histedit edits history, while hg rebase moves the stack around.
The UX just has lots and lots of polish in small ways and it has a nice amount of good features. It has fewer verbs than Git, but it's still a bit different from Mercurial. Having a built in 'undo' command that basically always works is nice.
I have replaced Git with Sapling (and a similar-but-not-the-same system, Jujutsu) in most of my own personal workflows at this time. If that's a good enough endorsement for you, then I suggest trying it out. You might be surprised.
I used to work at FB, and while sapling is quite nice to use in practice, without the internal version of Phabricator to do code review (and, in all likelihood, mononoke), I don't think I'd pick it up again.
I know I’ll be downvoted for taking an ethical rather than financial or technological position, but ethics matter. Especially in technology and finances.
SWEs need to learn new tools all the time. This shouldn't be a concern. Besides, it's not like "the real world" only use a single tool. Each company has their own tweaks, even those using open source stuff.
* How do I setup a new project? * How do I build code? * How do I pull internal dependencies? * How do I publish artifacts?
Every company makes a horrible copy of these systems with off-the-shelf tools. At Amazon there were certainly flaws in the internal tools, but overall it was much better than the "real world".
> While it is used in production within Meta, it currently does not build in an
> open source context and is not yet supported for external usage.
May be this is a feature not a bug? They want to filter out some Resume-Driven developers.
A bit cynical, but maybe not so unpopular opinion.
Meta as business is all about locking users to the ecosystem.
The one thing that is super exciting to me is the stacked pull request support. Using Github for this kind of workflow is enormously painful. Conversations constantly get outdated, and its nearly impossible to track whether comments have been addressed.
I know they're working on an improved UX/experience there, but it seems like it'll be a good long while, especially for enterprise server customers.
Stockholm syndrome. I've used Git for 15 years, early GitHub user, etc. Yes, you can solve many of these things, but until recently even things like "I am changing patch 2 in a series of 5 and need to rebase the following 3" were ridiculously painful. This is a common workflow many people like (including the Linux kernel devs) and Git was bad at it.
Git submodules. I'm not even going to go into this, they're so bad. That's a problem I wish Git had never "Solved" to spare us the burden.
There are tons of minor nits in Git all over the place. "Solving" something is completely different from actually having something that can be easily used for your team. There's no amount of contributing Facebook could have done to fix Git, because they'd be turning Git into something else that it fundamentally is not. And it doesn't matter if you have a trillion dollars, it's often not practical to just overhaul someone else's whole project when these goals don't align.
Is an intellectually dishonest fantasy invented for the sole purpose of using it to discredit and distract from criticism of the actions of the inventor of the phrase, so should never be ascribed as the source of a position you want to argue against unless your intent is to signal that your own position lacks a reasonable argument and you are just choosing to character-assassinate the opposition to cover for that.
I am not a fan of FB, but they tried - you can find them on the git mailing list where they got told they are doing it wrong for things like "scaling" or "productivity". Which is always ironic since basically nobody in open source generates or uses any real data about productivity, it's all just gut feelings about users.
git falls over and dies really, really badly when the repo gets stupidly large.
There's an article alluding to the discussion here: https://engineering.fb.com/2014/01/07/core-data/scaling-merc..., but I can't find the original thread on the git mailing list.
[0] https://web.archive.org/web/20210119051414/http://git.661346...
I used it briefly but couldn't get most people to adopt it widely enough.
Talking with a Meta person, it sounds like Phab really needs Mercurial to work well, at least for Stacked Diffs because you need to be able to identify commits independent of their location in history to properly maintain the Stacked Diff associations.
Deleted Comment
So, no. They wont support Phorge. They really never supported open-source Phabricator after Evan left and made it his own with Phacility.
I'm a bit worried about the dominance of VS Code. I can't stand the editor and it's popularity just grows and grows.
- refactoring (extracting functions and variables, renaming across entire project etc.)
- context aware selection (expand selection from cursor logically)
- stack trace / error parsing
- comparing anything (diff selection with clipboard for example)
- DB schema support even in SQL formatted as strings (e.g. when using psycopg) directly from DB by just connecting to a DB
- find in files (I've never seen a VS code user have an easy time finding stuff)
Etc.