Readit News logoReadit News
cletus commented on Try to take my position: The best promotion advice I ever got   andrew.grahamyooll.com/bl... · Posted by u/yuppiepuppie
cletus · a month ago
So there's some survivor bias here but it's generally not bad advice. You should be focusing on outcomes like improving SLAs, top line metrics and so on. You should be solving user and business problems. That's all good advice. But still this article presumes a lot.

In my experience, managers will naturally partition their reports into three buckets: their stars, their problems and their worker bees. The worker bees tend to be ignored. They're doing fine. They're getting on with whatever they've been told to do or possibly what they've found to do. They're not going to create any problems. The problems are the underperformers. These are people who create problems and/or are at risk of getting a subpar performance rating.

Now there are lots of reasons that someone can be a problem. I tend to believe that any problem just hasn't found the right fit yet and, until proven otherwise, problems are a failure in management. That tends to be a minority view in practice. It's more common to simply throw people in the deep end and sink or swim because that takes much less overhead. You will see this as teams who have a lot of churn but only in part of the team. In particularly toxic environments, savvy managers will game the system by having a sacrificial anode position. They hire someone to take the bad rating they have to give to protect the rest of the team.

And then there are the stars. These are the people you expect to grow and be promoted. More often than not however they are chosen rather than demonstrating their potential. I've seen someone shine when their director is actively trying to sabotage them but that's rare.

Your stars will get the better projects. Your problems will get the worse ones. If a given project is a success or not will largely come down to perception not reality.

The point I'm getting to is that despite all the process put around this at large companies like performance ratings, feedback, calibration, promo committees, etc the majority of all this is vibes based.

So back to the "take my job" advice. If someone is viewed as a star, that's great advice. For anyone else, you might get negative feedback about not doing your actual job, not being a team player and so on. I've seen it happen a million times.

And here's the dirty little secret of it all: this is where the racism, sexism and ableism sneaks in. It's usually not that direct but Stanford grads (as just one example) will tend to vibe with other Stanford grads. They have common experience, probably common professors and so on. Same for MIT. Or CMU. Or UW. Or Waterloo. And so on.

So all of the biases that go into the selection process for those institutions will bleed into the tech space.

And this kind of environment is much worse for anyone on the spectrum because allistic people will be inclined to dislike from the start for no reason and that's going to hurt how they're viewed (ie as a star, a worker bee or a problem) and their performance ratings.

Because all of this is ultimately just a popularity contest with very few exceptions. I've seen multiple people finagle their way to Senior STaff SWE on just vibes.

And all of this gets worse since the tech sector has joined Corporate America in being in permanet layoff mode. The Welchian "up or out" philosophy has taken hold in Big Tech where there are quotas of 5-10% of the workforce have to get subpar ratings every year and that tends to kill their careers at that company. This turns the entire workplace even more into an exercise in social engineering.

cletus commented on Lessons from 14 years at Google   addyosmani.com/blog/21-le... · Posted by u/cdrnsf
cletus · a month ago
I'm going to pick out 3 points:

> 2. Being right is cheap. Getting to right together is the real work

> 6. Your code doesn’t advocate for you. People do

> 14. If you win every debate, you’re probably accumulating silent resistance

The common thread here is that in large organizations, your impact is largely measured by how much you're liked. It's completely vibes-based. Stack ranking (which Google used to have; not sure if it still does) just codifies popularity.

What's the issue with that? People who are autistic tend to do really badly through no fault of their own. These systems are basically a selection filter for allistic people.

This comes up in PSC ("perf" at Meta, "calibration" elsewhere) where the exact same set of facts can be constructed as a win or a loss and the only difference is vibes. I've seen this time and time again.

In one case I saw a team of 6 go away and do nothing for 6 months then come back and shut down. If they're liked, "we learned a lot". If they're not, "they had no impact".

Years ago Google studied the elements of a successful team and a key element was psychological safety. This [1] seems related but more recent. This was originally done 10-15 years ago. I agree with that. The problem? Permanent layoffs culture, designed entirely to suppress wages, kills pyschological safety and turns survival into a game of being liked and manufacturing impact.

> 18. Most performance wins come from removing work, not adding cleverness

One thing I really appreciated about Google was that it has a very strict style guide and the subset of C++ in particular that you can use is (was?) very limited. At the time, this included "no exceptions", no mutable function arguments and adding templtes had an extremely high bar to be allowed.

Why? To avoid arguments about style issues. That's huge. But also because C++ in particular seemed to attract people who were in love with thier own cleverness. I've seem some horrific uses of templates (not at Google) that made code incredibly difficult to test for very little gain.

> 9. Most “slow” teams are actually misaligned teams

I think this is the most important point but I would generalize it and restate it as: most problems are organizational problems.

At Meta, for example, product teams were incentivized to ship and their impact was measured in metric bumps. But there was no incentive to support what you've already shipped beyond it not blowing up. So in many teams there was a fire and forget approach to filing a bug and forgetting about it, to the point where it became a company priority to have SLAs on old bugs, which caused the inevitable: people just downgrading bug priorities to avoid SLAs.

That's an organizational problem where the participants have figured out that shiping is the only thing they get rewarded for. Things like documentation, code quality and bug fixes were paid lip service to only.

Disclaimer: Xoogler, ex-Facebooker.

[1]: https://www.aristotleperformance.com/post/project-aristotle-...

cletus commented on Was it a billion dollar mistake?   gingerbill.org/article/20... · Posted by u/signa11
cletus · a month ago
The mistake was not having nullability be expressed in the type system.

At Facebook I used their PHP fork Hack a lot and Hack has a really expressive type system where PHP does not. You can express nullability of a type and it defaults to a type being non-nullable, which is the correct default. The type checker was aware of changes too, so:

    function foo(?A $a): void {
      $a->bar(); // compile error, $a could be null
      if ($a is null) {
        return;
      }
      $a->bar(); // not a compiler error because $a is now A not ?A
      if ($a is ChildOfA) {
        $a->childBar(); // not an error, in this scope $a is ChildOfA
      }
    }
Now Hack like Java used type erasure so you could force a null into something non-nullable if you really wanted to but, in practice, this almost never happened. A far bigger problem was dealing with legacy code that was converted with a tool and returned or used the type "mixed", which could be literally anything.

The real problem with Java in particular is you'd end up chaining calls then get the dreaded NullPointerException and have no idea from the error or the logs what was broken from:

   a.b.c.d();
I'm fine with things like Option/Maybe types but to me they solve different problems. They're a way of expressing that you don't want to specify a value or that a value is missing and that's different to something being null (IMHO).

cletus commented on Total monthly number of StackOverflow questions over time   data.stackexchange.com/st... · Posted by u/maartin0
johnfn · a month ago
Some comments:

- This is a really remarkable graph. I just didn't realize how thoroughly it was over for SO. It stuns me as much as when Encyclopædia Britannica stopped selling print versions a mere 9 years after the publication of Wikipedia, but at an even faster timescale.

- I disagree with most comments that the brusque moderation is the cause of SO's problems, though it certainly didn't help. SO has had poor moderation from the beginning. The fundamental value proposition of SO is getting an answer to a question; if you can the same answer faster, you don't need SO. I suspect that the gradual decline, beginning around 2016, is due to growth in a number of other sources of answers. Reddit is kind of a dark horse here, as I began seeing answers on Google to more modern technical questions link to a Reddit thread frequently along with SO from 2016 onwards. I also suspect Discord played a part, though this is harder to gauge; I certainly got a number of answers to questions for, e.g., Bun, by asking around in the Bun Discord, etc. The final nail in the coffin is of course LLMs, which can offer a SO-level answer to a decent percentage of questions instantly. (The fact that the LLM doesn't insult you is just the cherry on top.)

- I know I'm beating a dead horse here, but what happens now? Despite stratification I mentioned above, SO was by far the leading source of high quality answers to technical questions. What do LLMs train off of now? I wonder if, 10 years from now, LLMs will still be answering questions that were answered in the halcyon 2014-2020 days of SO better than anything that came after? Or will we find new, better ways to find answers to technical questions?

cletus · a month ago
As an early user of SO [1], I feel reasonably qualified to discuss this issue. Note that I barely posted after 2011 or so so I can't really speak to the current state.

But what I can say is that even back in 2010 it was obvious to me that moderation was a problem, specifically a cultural problem. I'm really talking about the rise of the administrative/bureaucratic class that, if left unchecked, can become absolute poison.

I'm constantly reminded of the Leonard Nimoy voiced line from Civ4: "the bureaucracy is expanding to meet the needs of the expanding bureaucracy". That sums it up exactly. There is a certain type of person who doesn't become a creator of content but rather a moderator of content. These are people who end up as Reddit mods, for example.

Rules and standards are good up to a point but some people forget that those rules and standards serve a purpose and should never become a goal unto themselves. So if the moderators run wild, they'll start creating work for themselves and having debates about what's a repeated question, how questions and answers should be structured, etc.

This manifested as the war of "closed, non-constructive" on SO. Some really good questions were killed this way because the moderators decided on their own that a question had to have a provable answer to avoid flame wars. And this goes back to the rules and standards being a tool not a goal. My stance was (and is) that shouldn't we solve flame wars when they happen rather than going around and "solving" imaginary problems?

I lost that battle. You can argue taht questions like "should I use Javascript or Typescript?" don't belong on SO (as the moderators did). My position was that even though there's no definite answer, somebody can give you a list of strengths and weaknesses and things to consider.

Even something that does have a definite answer like "how do I efficiently code a factorial function?" has multiple but different defensible answers. Even in one language you can have multiple implementations that might, say, be compile-time or runtime.

Another commenter here talked about finding the nearest point on an ellipse and came up with a method they're proud of where there are other methods that would also do the job.

Anyway, I'd occasionally login and see a constant churn on my answers from moderators doing pointless busywork as this month they'd decided something needed to be capitalized or not capitalized.

A perfect example of this kind of thing is Bryan Henderson's war on "comprised of" on Wikipedia [2].

Anyway, I think the core issue of SO was that there was a lot of low-hanging fruit and I got a lot of accepted answers on questions that could never be asked today. You'll also read many anecdotes about people having a negative experience asking questions on SO in later years where their question was immediately closed as, say, a duplicate when the question wasn't a duplicate. The moderator just didn't understand the difference. That sort of thing.

But any mature site ultimately ends with an impossible barrier to entry as newcomers don't know all the cultural rules that have been put in place and they tend to have a negative experience as they get yelled at for not knowing that Rule 11.6.2.7 forbids the kind of question they asked.

[1]: https://stackoverflow.com/users/18393/cletus

[2]: https://www.npr.org/2015/03/12/392568604/dont-you-dare-use-c...

cletus commented on Stepping down as Mockito maintainer after ten years   github.com/mockito/mockit... · Posted by u/saikatsg
throwaway7783 · a month ago
I'm not sure I understand. "The solution is to have dummy versions of services and interfaces that have minimal correct behavior".

That's mocks in a nutshell. What other way would you use mocks?

cletus · a month ago
Imagine you're testing a service to creates, queries and deletes users. A fake version of that service might just be a wrapper on a HashMap keyed by ID. It might have several fields like some personal info, a hashed password, an email address, whether you're verified and so on.

Imagine one of your tests is if the user deletes their account. What pattern of calls should it make? You don't really care other than the record being deleted (or marked as deleted, depending on retention policy) after you're done.

In the mock world you might mock out calls like deleteUserByID and make suer it's called.

In the fake world, you simply check that the user record is deleted (or marked as such) after the test. You don't really care about what sequence of calls made that happen.

That may sound trivial but it gets less trivial the more complex your example is. Imagine instead you want to clear out all users who are marked for deletion. If you think about the SQL for that you might do a DELETE ... WHERE call so your API call might look like that. But if the logic is more complicated? Where if there's a change where EU and NA users have different retention periods or logging requirements so they're suddenly handled differently?

In a mokcing world you would have to change all your expected mocks. In fact, implementing this change might require fixing a ton of tests you don't care about at all and aren't really being broken by the change regardless.

In a fake world, you're testing what the data looks like after you're done, not the specific steps it took to get there.

Now those are pretty simple examples because there's not much to do the arguments used and no return values to speak of. Your code might branch differently based on those values, which then changes what calls to expects and with what values.

You're testing implementation details in a really time-consuming yet brittle way.

cletus commented on Stepping down as Mockito maintainer after ten years   github.com/mockito/mockit... · Posted by u/saikatsg
cletus · a month ago
My second project at Google basically killed mocking for me and I've basically never done it since. Two things happened.

The first was that I worked on a rewrite of something (using GWT no less; it was more than a decade ago) and they decided to have a lot of test coverage and test requirements. That's fine but they way it was mandated and implemented, everybody just testing their service and DIed a bunch of mocks in.

The results were entirely predictable. The entire system was incredibly brittle and a service that existed for only 8 weeks behaved like legacy code. You could spend half a day fixing mocks in tests for a 30 minute change just because you switched backend services, changed the order of calls or just ended up calling a given service more times than expected. It was horrible and a complete waste of time.

Even the DI aspect of this was horrible because everything used Guice andd there wer emodules that installed modules that installed modules and modifying those to return mocks in a test environment was a massive effort that typically resulted in having a different environment (and injector) for test code vs production code so what are you actually testing?

The second was that about this time the Java engineers at the company went on a massive boondoggle to decide on whether to use (and mandate) EasyMock vs Mockito. This was additionally a waste of time. Regardless of the relative merits of either, there's really not that much difference. At no point is it worth completely changing your mocking framework in existing code. Who knows how many engineering man-yars were wasted on this.

Mocking encourages bad habits and a false sense of security. The solution is to have dummy versions of services and interfaces that have minimal correct behavior. So you might have a dummy Identity service that does simple lookups on an ID for permissions or metadata. If that's not what you're testing and you just need it to run a test, doing that with a mock is just wrong on so many levels.

I've basically never used mocks since, so much so that I find anyone who is strongly in favor of mocks or has strong opinions on mocking frameworks to be a huge red flag.

cletus commented on Mattermost restricted access to old messages after 10000 limit is reached   github.com/mattermost/mat... · Posted by u/xvilka
cletus · 2 months ago
Story time. This has basically nothing to do with this post other than it involves a limit of 10,000 but hey, it's Christmas and I want to tell a story.

I used to work for Facebook and many years ago people noticed you couldn't block certain people but the one that was most public was Mark Zuckerberg. It would just say it failed or something like that. And people would assign malice or just intent to it. But the truth was much funnier.

Most data on Facebook is stored in a custom graph database that basically only has 2 tables that are sharded across thousands of MySQL instances but most almost always accessed via an in-memory write-through cache, also custom. It's not quite a cache because it has functionality built on top of the database that accessing directly wouldn't have.

So a person is an object and following them is an edge. Importantly, many such edges were one-way so it was easy to query if person A followed B but much more difficult to query all the followers of B. This was by design to avoid hot shards.

So I lied when I said there were 2 tables. There was a third that was an optimization that counted certain edges. So if you see "10.7M people follow X" or "136K people like this", it's reading a count, not doing a query.

Now there was another optimization here: only the last 10,000 of (object ID,edge type) were in memory. You generally wanted to avoid dealing with anything older than that because you'd start hitting the database and that was generally a huge problem on a large, live query or update. As an example, it was easy to query the last 10,000 people or pages you've followed.

You should be able to see where this is going. All that had happened was 10,000 people had blocked Mark Zuckerberg. Blocks were another kind of edge that was bidirectional (IIRC). The system just wasn't designed for a situation where more than 10,000 people wanted to block someone.

This got fixed many years ago because somebody came along and build a separate system to handle blocking that didn't have the 10,000 limit. I don't know the implementation details but I can guess. There was a separate piece of reverse-indexing infrastructure for doing queries on one-way edges. I suspect that was used.

Anyway, I love this story because it's funny how a series of technical decisions can lead to behavior and a perception nobody intended.

cletus commented on At Least 13 People Died by Suicide Amid U.K. Post Office Scandal, Report Says   nytimes.com/2025/07/10/wo... · Posted by u/xbryanx
cletus · 7 months ago
People should go to jail for this.

Anyone who has worked on a large migration eventually lands on a pattern that goes something like this:

1. Double-write to the old system and the new system. Nothing uses the new system;

2. Verify the output in the new system vs the old system with appropriate scripts. If there are issues, which there will be for awhile, go back to (1);

3. Start reading from the new system with a small group of users and then an increasingly large group. Still use the old system as the source of truth. Log whenever the output differs. Keep making changes until it always matches;

4. Once you're at 100% rollout you can start decomissioning the old system.

This approach is incremental, verifiable and reversible. You need all of these things. If you engage in a massive rewrite in a silo for a year or two you're going to have a bad time. If you have no way of verifying your new system's output, you're going to have a bad time. In fact, people are going to die, as is the case here.

If you're going to accuse someone of a criminal act, a system just saying it happened should NEVER be sufficient. It should be able to show its work. The person or people who are ultimately responsible for turning a fraud detection into a criminal complaint should themselves be criminally liable if they make a false complaint.

We had a famous example of this with Hertz mistakenly reporting cars stolen, something they ultimately had to pay for in a lawsuit [1] but that's woefully insufficient. It is expensive, stressful and time-consuming to have to criminally defend yourself against a felony charge. People will often be forced to take a plea because absolutely everything is stacked in the prosecution's favor despite the theoretical presumption of innocence.

As such, an erroneous or false criminal complaint by a company should itself be a criminal charge.

In Hertz's case, a human should eyeball the alleged theft and look for records like "do we have the car?", "do we know where it is?" and "is there a record of them checking it in?"

In the UK post office scandal, a detection of fraud from accounting records should be verified by comparison to the existing system in a transition period AND, moreso in the beginning, double checking results with forensic accountants (actual humans) before any criminal complaint is filed.

[1]: https://www.npr.org/2022/12/06/1140998674/hertz-false-accusa...

cletus commented on Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix   netflixtechblog.com/uda-u... · Posted by u/Bogdanp
cletus · 8 months ago
I realize scale makes everything more difficult but at the end of the day, Netflix is encoding and serving several thousand videos via a CDN. It can't be this hard. There are a few statements in this that gave me pause.

The core problem seems to be development in isolation. Put another way: microservices. This post hints at microservices having complete autonomy over their data storage and developing their own GraphQL models. The first is normal for microservices (but an indictment at the same time). The second is... weird.

The whole point of GraphQL is to create a unified view of something, not to have 23 different versions of "Movie". Attributes are optional. Pull what you need. Common subsets of data can be organized in fragments. If you're not doing that, why are you using GraphQL?

So I worked at Facebook and may be a bit biased here because I encountered a couple of ex-Netflix engineers in my time who basically wanted to throw away FB's internal infrastructure and reinvent Netflix microservices.

Anyway, at FB there a Video GraphQL object. There aren't 23 or 7 or even 2.

Data storage for most things was via write-through in-memory graph database called TAO that persisted things to sharded MySQL servers. On top of this, you'd use EntQL to add a bunch of behavior to TAO like permissions, privacy policies, observers and such. And again, there was one Video entity. There were offline data pipelines that would generally process logging data (ie outside TAO).

Maybe someone more experienced with microservices can speak to this: does UDA make sense? Is it solving an actual problem? Or just a self-created problem?

cletus commented on The world could run on older hardware if software optimization was a priority   twitter.com/ID_AA_Carmack... · Posted by u/turrini
mike_hearn · 9 months ago
I worked there too and you're talking about performance in terms of optimal usage of CPU on a per-project basis.

Google DID put a ton of effort into two other aspects of performance: latency, and overall machine utilization. Both of these were top-down directives that absorbed a lot of time and attention from thousands of engineers. The salary costs were huge. But, if you're machine constrained you really don't want a lot of cores idling for no reason even if they're individually cheap (because the opportunity cost of waiting on new DC builds is high). And if your usage is very sensitive to latency then it makes sense to shave milliseconds off because of business metrics, not hardware $ savings.

cletus · 9 months ago
The key part here is "machine utilization" and absolutely there was a ton of effort put into this. I think before my time servers were allocated to projects but even early on in my time at Google Borg had already adopted shared machine usage and therew was a whole system of resource quota implemented via cgroups.

Likewise there have been many optimization projects and they used to call these out at TGIF. No idea if they still do. One I remember was reducing the health checks via UDP for Stubby and given that every single Google product extensively uses Stubby then even a small (5%? I forget) reduction in UDP traffic amounted to 50,000+ cores, which is (and was) absolutely worth doing.

I wouldn't even put latency in the same category as "performance optimization" because often you decrease latency by increasing resource usage. For example, you may send duplicate RPCs and wait for the fastest to reply. That could be double or tripling effort.

u/cletus

KarmaCake day35207July 2, 2009
About
I am a Java, C++, JavaScript and Python software engineer.

I am from Perth, Western Australia but currently live in New York City. Xoogler and Ex-Facebooker.

I am a contributor on Stackoverflow as user Cletus.

I can be contacted on wcshields at the big G's service.

View Original