Readit NewsThat's one of the best ads I've ever seen ;)
What about the excessive typing - in practice do you get used to it? Are you allowing Copilot and such inside the business?
If by excessive typing you mean complex trait bounds and the like, it's really something isolated to the most generic libraries, and reasonably rare. Most application code is simply typed - take a struct or an enum, return another. So it's not been a problem.
We don't use Copilot or any tool that sends our source code because of our security posture, defense in depth, etc. If and when there will be an offering that lets us run the server component and provides significant productivity improvements, we will subscribe.
Huge fan of Kraken.
Looking back, is there anything that you would have done differently? I find that half or more of the rewrites that I have dealt with have been driven by all the wrong motivations. You get inevitable turnover and at some point people dislike code that they didn't write themselves and push for a rewrite, maybe changing the stack to something trendy, justifying it with thin arguments. Once the rewrite starts the company ends up treading water for years while incurring a ton of costs. For me, I think only 1 rewrite that I was part of was a good decision in my 15 years in tech. If I could go back in time, I think I would kill all rewrite discussions the moment that someone first whispers the idea.
How did you guys enjoy switching to Rust? I assume the safety and performance benefits for the trading system are a huge plus (didn't Kraken trading go down for an entire week a few years ago?). Did you also rewrite the webapp backend in Rust as well? How has staffing and budgeting been affected? I would assume that the supply of Rust developers is much lower unless you train them in house. Rust sounds fun, but I can't imagine trying to justify a rewrite of a legacy system, a major tech stack change, and training/building a new team all at the same time.
Sorry for the onslaught on questions. The "rewrite it in rust" fever has spread to my work and I'm fighting myself on how to respond.
With hindsight, considering the cards we were dealt, there's not much I would have done differently. If I had known better before, I would have ensured stronger buy-in because after a while our internal stakeholders were often pushing back on the effort, and that led to concessions where throw away code in the legacy systems was built even for weak business outcomes.
Overall I share your concerns. Having the right reasons to rewrite is key. I believe this blog[1] about software as theory building does a great job at describing the challenge with software gardening, and the times where a rewrite is the solution are few. Even then, it's critical to handle the rewrite in ways that can work - in our case, we chose to progressively eat the legacy software without making major changes when we could avoid them. The legacy software we had was mostly the results of one man heroics and traded off performance and availability for correctness and security. It also was designed to be maintained by a small group. Solid choices if you are early Kraken - but as many successful startups, we were victim of our success and we needed it all.
When it became clear that we had to rewrite the stack (the 2017 3-days shutdown happened just before that realization), those in charge at the time decided to experiment with Rust. It was a crazy bet in early 2018, it was still Rust 2015, no NLL, no async, far rougher ecosystem. The fact that it became successful enough to warrant pursuing a full rewrite is to credit on some lucky hires who made it a success.
In that regard, Rust was a very strong talent magnet. In my experience, having hired 200+ Rust engineers over the last 5 years, there are a few kind of engineers attracted to Rust: (1) some just like shiny things/hype, (2) some are perfectionists and never complete a project, (3) some just are doers who have found that Rust is a particularly effective language.
Overall, Rust has been a great to hire for. Many engineers out there want to use Rust, even if it's their 1st Rust professional experience. We were also known in the Rust community for hiring for full time Rust, probably also the place currently with the highest density of Rust talent (there are massive companies with more Rust devs, but smaller % overall). Budget wise, our Rust engineers are not paid particularly better than other engineers in the company, but the compensations at Kraken are generally in the higher tier.
At the risk of sounding boastful, in my experience Rust is reasonably easy to learn for experienced/strong developers (we have some very young outstanding Rust devs as well, most of the time they learned before joining). Average developers struggle and may never become productive. Again, we have an engineering excellence culture so it is okay for us, YMMV.
Re scope, yes we use Rust for everything in the backend, including CRUD type of work like Web APIs. We've found we're at least as productive than other languages (Go / Java+Spring / Ruby / PHP) while having far fewer incidents, and easier maintenance / cheaper KTLO. Rust's ability for reuse is excellent which means that there are very strong network effects when having more services in Rust, including the Web layer.
A nice "side effect" of a full Rust stack is that our p99.9 latency internally is usually stable around 3-4ms for most operations, even though multiple services are involved. That's coming from a much higher baseline with much more deviation across operations providing the same functionality (60-100ms).
Regarding your own rewrite discussions, you're not going to be convinced by a post on HN, I'll just say that I am very reluctant to even think working at a workplace that doesn't predominantly uses Rust. I've been in the industry for 20+ years, across many stacks and there's a before and an after Rust for me. It has been a super power and made our life easier. It makes it easier to model business problems thanks to algebraic data types and their usage for error handling (versus inheritance), traits allow to abstract behavior better than OOP-style interfaces, the absence of data races is a game changer for multi-threaded code, dependency management is trivial, the ecosystem is rich and things work well. A lot of these are properties found in other languages but no other has the same full package and is on its way to become mainstream.
Completed a full rewrite of many components of the Kraken.com backend in about 4 years.
The new system is around 1.5M loc of Rust. There was no serious alternative to rewriting, sometimes you find yourself in a corner and need to fix issues, and pay the price.
I wrote about it 3 years ago here https://blog.kraken.com/product/engineering/oxidizing-kraken...
Everything in that blog post still rings true and hindsight is that it were were right. But it was a massive grind and required extreme dedication to get it done, for a variety of reasons that work was very taxing.
We also didn't stop feature development and kept the two systems running concurrently (which explains why it took so long, also growing and training a new team 10x the size took time, so there are many factors).
I'm also against rewrites if I can help it, but reality is complex and sometimes we can't help it. Now however, since we removed the last pieces of legacy that were preventing larger DB schema changes (or required massive, unreasonable changes to the legacy systems), we've been shipping faster and easier than ever and caught-up on a lot of the accumulated backlog, including some of the more ambitious projects that were unthinkable in the legacy systems due to limitations.
Kraken | Full remote, globally distributed | Full-time
DESCRIPTION: Kraken is one of the top cryptocurrency exchanges worldwide (and becoming more than that!). We build a lot of our modern backend code in Rust in various teams, and the Frontend in React (and React Native) and Angular. There are many more roles as we're hoping to hire 100-200 engineers (developers, SREs/DevOps, data...) in 2021. Make sure to check our careers page.
Web/Mobile roles: React, NextJs, TailwindUi, Angular
Rust roles (backend or desktop): Core Backend (the core services of the exchange), Bank (the neo-bank we are building), Futures (the Futures/derivatives trading platform), Trading Technology (trading related services). We have a trading desktop app written in Rust with the Iced framework (https://cryptowat.ch/apps/desktop). We hire backend, test and site reliability engineers starting at 2 years of experience. For dev roles, Rust experience is mandatory, but significant pet projects counts. I wrote a blog to provide more context on what the Core Backend team does and the profiles we are hiring: https://blog.kraken.com/post/7964/oxidizing-kraken-improving.... Please read the blog before applying!
LOCATION: We are fully remote+global and have been for years. We also have some offices in around the world for those who like these.
ESTIMATED COMPENSATION: Depends on the role - we base our compensation on the top percentiles of US compensations and apply the same ranges wherever you live. All our offers have option packages. For an idea of the compensation, head to levels.fyi and check Google, Amazon or Apple comps per levels.
REMOTE: Some teams have timezone restrictions: Futures are centered around UTC, other teams are mostly Eastern Europe to US West Coast.
The Core Backend team is planning to expand its presence in APAC and is hiring outstanding engineers who are senior + self-driven + advanced in Rust, who agree to adapt their working hours to maximize overlap with US or EU at least until the group gets big enough to become fully autonomous.
VISA: No visa sponsorship
CONTACT: You can apply on our career page at https://jobs.lever.co/kraken or you can DM me on Twitter (@simach) though I may take a few days to reply.
Kraken Digital Asset Exchange | Software & Site Reliability Engineers | Remote | https://www.kraken.com/en-us/careers
Kraken was founded in 2011, being a pioneer exchange in the crypto market. Kraken is in rapid expansion of its engineering teams. At Kraken we have a remote global workforce, allowing our employees to have more freedom in choosing how to organize their routine and balance work and personal life.
March Highlight: (Senior) Backend Engineer - Rust
Check out our blogpost for further insights into Rust at Kraken: https://blog.kraken.com/post/7964/oxidizing-kraken/ - I'm the author of that post so feel free to reach directly for any question.
We're also hiring software engineers with the following skillsets: Backend (Rust/Go/Typescript/PHP), Frontend (React/Flux/Typescript/Sass/Less/NextJS) and many more.
Please apply directly on our careers page at https://www.kraken.com/en-us/careers.
> Ideally, each fallible function would have its own error enum to precisely capture its errors and handle them, but in practice it is too verbose and leads to using the less precise Error trait or one enum per module. The language could support this better: there are several initiatives and macros exploring this.
Definitely something I can agree with! Has your team explored any initatives / macros that seem promising? We're currently using the crate 'thiserror' but it's made our compilation times painful.
We're using thiserror, and haven't explored macros that generate an enum from the Result type. I remember there are crates exploring this, but this is something that would really benefit from compiler support. Ocaml's polymorphic variants appear to provide an interesting solution here.
Author of the post here - feel free to shoot any questions.
Also see thread on r/rust: https://www.reddit.com/r/rust/comments/lnkx9h/oxidizing_krak...
simag commented on Richard M. Stallman resigns fsf.org/news/richard-m-st... · Posted by u/maxdeviant
The pedophilia advocating is much earlier. From his website (the last sentence is his opinion)
Dutch pedophiles have formed a political party to campaign for legalization. [Reference updated on 2018-04-25 because the old link was broken.]
I am skeptical of the claim that voluntarily pedophilia harms children
I disagree with Stallman's skepticism (I do think it harms children and I don't think children can make that choice). But that one liner does not make him a pedophilia advocate. That makes him someone who is wrong on that topic.
We originally introduced Rust (back in 2018) as a small experiment alongside existing systems, mostly to validate safety and performance assumptions under real production load.
Over time, the reduction in memory-related incidents and clearer failure modes led us to expand its use into increasingly critical paths. This post focuses less on “Rust is great” and more on the tradeoffs, mistakes, and organizational changes required to make that transition work in practice.
Happy to answer questions about what did not work, where Rust was a poor fit, or how we handled interop with existing systems.