I love Zod, but recently I've been converting to Effect's Data and Schema modules.
Previously I liked a combination of Zod and ts-pattern to create safe, pattern matching-oriented logic around my data. I find Effect is designed far better for this, so far. I'm enjoying it a lot. The Schema module has a nice convention for expressing validators, and it's very composable and flexible: https://effect.website/docs/guides/schema/introduction
There are also really nice aspects like the interoperability between Schema and Data, allowing you to safely parse data from outside your application boundary then perform safe operations like exhaustively matching on tagged types (essentially discriminated unions): https://effect.website/docs/other/data-types/data#is-and-mat...
It feels extremely productive and intuitive once you get the hang of it. I didn't expect to like it so much.
I think the real power here is that these modules also have full interop with the rest of Effect. Effects are like little lazy loaded logical bits that are all consistent in how they resolve, making it trivial to compose and execute logic. Data and Schema fit into the ecosystem perfectly, making it really easy to compose very resilient, scalable, reliable data pipelines for example. I'm a convert.
Zod is awesome if you don't want to adopt Effect wholesale, though.
I've had a very similar experience, and have been slowly moving from zod and ts-rest to @effect/schema and @effect/platform/HttpApi as well as migration to Effect Match from ts-pattern. There is a learning curve but its a pretty incredible ecosystem once you are in it for a bit.
I think the real turning point was typescript 5.5 (May 2024). The creator of typescript personally fixed a bug that unlocked a more natural generator syntax for Effect, which I think unlocks mainstream adoption potential.
I feel like Effect is today's Ramda. So cool but it's going to be regretted by you and people coming after you in few years.
Me and my team reverted to more stupid code and we are happier.
How did you find learning Effect? The sales pitch sounds great, but when I went through the docs it seemed pretty confusing to me. I’m sure there are reasons for the everything but I couldn’t grok it. In particular, I’m thinking of the Express integration example.[0] I look at that and think, I need all that just to create a server and a route? What’s the benefit there? I’m hesitant to buy into the ecosystem after looking at that. I want to like it, though.
I agree about the turning point. Things have improved dramatically. And I know it probably doesn't feel the same for tons of people, but I love to see generators being used in every day code.
The learning curve just about turned me away from it at the start, but I'm glad I stuck with it.
I think learning Effect would actually teach a lot of people some very useful concepts and patterns for programming in general. It's very well thought out.
Why not write your code in F# and compile it to TypeScript using Fable [1]?
This way you can use native language features for discriminated unions, functional pipelines, and exhaustive pattern matching to model your domain instead of shoe-horning such functionality into a non-ML language!
Model your domain in F#, consume it in Python or C# backends and TypeScript frontends. The downside is needing to know all of these languages and run times but I think I'd rather know F# and the quirks with interacting with TypeScript than a library like Effect!
Which is why our principle with Effect is NOT to port Haskell. For example we don't use Typeclasses which are big in haskell, we lean heavily on variance which is not in Haskell. It's an ecosystem though to write production-grade TypeScript, not Haskell in TypeScript
I’d love to hear more stories of people using Effect in production codebases.
It looks very similar in its ideas to fp-ts (in the “let’s bring monads, algebraic types etc to typescript” sense).
But I did hear from teams that embraced fp-ts that things kinda ground to a halt for them. And those were category theory enthusiasts that were very good scala devs so I’m sure they knew what they were doing with fp-ts.
What happened was that the typescript compile time just shot into minutes, for a moderately sized micro-service, without anything externally heavy being introduced like you could on the frontend.
It just turned out that Typescript compiler was not so great at tracking all the inferred types throughout the codebase.
So wonder if things have improved or effect uses types more intelligently so that this is not an issue.
Effect reminds me more of something like NestJS - essentially an end to end application framework that takes over your whole application.
Rather disappoint to see something like this being plugged as an alternative to something like zod which is a nice library that stays in its corner and has a nice fixed scope to it's functionality.
This is just a link to the front page of possibly the #1 most popular type validation library in the ecosystem? Anyways, ya'll might want to check out up-and-coming Valibot, which has a really nice pipe API.
Valibot is really nice, particularly for avoiding bundle size bloat. Because Zod uses a "fluent builder" API, all of Zod's functionality is implemented in classes with many methods. Importing something like `z.string` also imports validators to check if the string is a UUID, email address, has a minimum or maximum length, matches a regex, and so on - even if none of those validators are used. Valibot makes these independent functions that are composed using the "pipe" function, which means that only the functions which are actually used need to be included in your JavaScript bundle. Since most apps use only a small percentage of the available validators, the bundle size reduction can be quite significant relative to Zod.
Is there a reason why tree shaking algorithms don't prune unused class members? My IDE can tell me when a method is unused, it seems odd that the tree shaker can't.
Valibot also has much, much more efficient type inference, which sounds unimportant right up until you have 50 schemas referencing each other and all your Typescript stuff slows to a molasses crawl.
Not only that, runtime schema validation was also way faster in my last project for larger arrays of complex objects with lots of union types. Went from 400ms to 24ms all else being equal. Since our validation was running on every request for certain reasons this made a huge difference in perceived performance for our users and less load on our servers.
Agreed. I recently moved a production app from Zod to Valibot for exactly this reason. I still slightly prefer Zod’s syntax but the performance issues were absolutely killing us.
I'm currently using valibot and the author's associated modular form library in a production app and can vouch that it's been very pleasant to work with.
I recently stumbled onto this when looking for alternatives, and though it might sound a bit extreme: the style of documentation means I'll never consider it for any project. It's strange and honestly somehow unnerving to read documentation written from a first-person perspective. Is there some good reason for this that I'm missing? Or am I just crazy and this isn't an issue for anyone else?
I only see second-person "you" and not first-person "I" in the linked documentation. Am I missing what you intended to point out?
In any case, this might actually be a good use for an LLM to post-process it into whatever style you want. I bet there's even a browser extension that could do it on-demand and in-place.
I don't think you are crazy at all. The concept of professionalism is about doing things a certain way to indicate to others that you are, of course, a professional. Writing technical documents in the 3rd person is a basic part of what I would consider professionalism in the broader engineering field.
In the software field you get a large portion of people that don't buy into the concept of professionalism. For various reasons - chiefly the hacker culture and the easy of contributing to the "field" means the gauntlet one runs to become a "professional" isn't inherently a given.
As a whole this is a good thing but it does mean if you operate as a "professional" maybe sometimes you have to realize that something doesn't exactly gel with your ethos (case in point). It doesn't mean it is bad; just maybe not for you and yours.
A whole lot of yapping and ZERO code on the homepage. Authors should remove 90% of the stuff on that landing page, jesus!
also since zod is the de facto validation lib, might be worth a specific page talking about why this vs zod. even their migration from zod page looks nearly identical between the two packages.
I wonder about a Typescript with alternate design decisions.
The type system is cool and you can do a lot of compile time metaprogramming.
However, when there's a type error in computed types, it's a nightmare to visually parse
type { foo: ... , bar: ..., ... } can't be assigned to type { foo: ..., bar: ... }. type { foo: ... bar: ..., ... } is missing the following properties from { foo: ..., bar: ..., ... }
Apart from repeating roughly the same type 4 times (which is a UX issue that's fixable), debugging computed types requires having a typescript interpreter in your head.
I wonder if we had nominal runtime checked types, it could work better than the current design in terms of DX.
Basically, the : Type would always be a runtime checked assertion. Zod certainly fills that gap, but it would be nicer to have it baked in.
The type system would not be as powerful, but when I'm writing Kotlin, I really don't miss the advanced features of Typescript at all. I just define a data class structure and add @Serializable to generate conversions from/to JSON.
TypeScript has a crazy powerful type system because it has to be able to describe any crazy behavior that was implemented in JavaScript. I mean, just take a look at @types/express-serve-static-core [0] or @types/lodash [1] to see the lengths TS will let you go.
If you write in TS to start with, you can use a more sane subset.
Right. I'm aware. My point was that even though the type system is powerful, somehow, I'm able to represent everything I need to in Kotlin's type system just fine and it feels a lot more type safe because it will throw a type error at runtime in the right place if I do a bad cast.
Typescript `as Foo` will not do anything at runtime, and it will just keep on going, then throw a type error somewhere else later (possibly across an async boundary).
You can, in theory use very strong lint rules (disallow `as` operator in favour of Zod, disallow postfix ! operator), but no actual codebase that I've worked on has these checks. Even the ones with the strictest checks enabled have gaps.
Not to mention, there's intentional unsoundness in the type system, so even if you wanted, you couldn't really create a save subset of TS.
Then there's the issue of reading the library types of some generic heavy code. When I "go to definition" in my fastify codebase, I see stuff like this
> = (
this: FastifyInstance<RawServer, RawRequest, RawReply, Logger, TypeProvider>,
request: FastifyRequest<RouteGeneric, RawServer, RawRequest, SchemaCompiler, TypeProvider, ContextConfig, Logger>,
reply: FastifyReply<RouteGeneric, RawServer, RawRequest, RawReply, ContextConfig, SchemaCompiler, TypeProvider>
// This return type used to be a generic type argument. Due to TypeScript's inference of return types, this rendered returns unchecked.
) => ResolveFastifyReplyReturnType<TypeProvider, SchemaCompiler, RouteGeneric>
Other languages somehow don't need types this complicated and they're still safer at runtime :shrug:
This is my exact experience. TypeScript seemed to hit a complexity sweet spot about 5 years ago, then they just kept adding more obscure stuff to the type system. I find that understanding TypeScript compiler errors can be almost as difficult as understanding C++ template errors at times.
Yes, I did mention that the UX of the type errors could be improved and probably should, but once you start getting into conditional types, and nested types (which may not be fully expanded when you see the error), it gets hairy.
ts-morph provides an easy way to use the TypeScript Compiler API to view and edit the AST before compile. Once you get your head around the API, which has good examples but isn't thoroughly documented on the web.
I'd recommend TypeBox[1] as an alternative, which has a runtime “compiler” for generating optimized JS functions from the type objects. It also produces a JSON schema, which can be useful for generating API docs and API clients if needed.
It also has a companion library[2] for generating TypeBox validators from TypeScript definitions, which I'm currently using in an RPC library I'm working on.
I'm a rust,zig and elixir developer and I had to work on a large typescript project that use zod.
I felt like I was being punished for everything. (Maybe some things were project specific so I am not saying this as an absolute) It's slow, syntax is horrible, error are obscure super long lines, it can compile and explode later (which is what elixir does, except elixir will happily restart and recover)...
Elixir is based on duck typing mostly, but it works very well because you just pattern match your data when you use it. Rust is very strict and can have cryptic errors, but as everything is baked in the language it is way easier to manage.
I am not saying this to be snob about typescript and JS, but I really felt pain when working with that ecosystem, and I wonder if I'm old and stupid or if those tools are really half baked and over complicated.
My company uses zod to validate data for HTML requests, and I'd say it works pretty well for that. It lets us detail our data structure in zod, validate the completely untyped HTML data, and be confident that once we reach our actual code, the data is well typed.
Zod feels like a crutch for limitations in Javascript and Typescript. But I've found it to be a very useful crutch, and I wouldn't want to write a Typescript API without it.
Thats not true. It just crashes the “sub” process and if the parent process spawns the sub process again with the same inputs its just going to crash again.
Are you aware you can also try/catch your errors in typescript?
The whole point of the library is to validate something at runtime so of course it is going to blow up. There are also API methods that simply return a boolean instead of crashing if it fails validation. You can then use type guarding and narrowing of the type.
The fragmentation around runtime validation libraries is pretty crazy. The fact that half these comments mention some alternative library that mimics almost the exact API of Zod illustrates that.
It is filling a necessary shortcoming in the gradual typing of TypeScript, and using validator schema types to power other APIs generic inference is powerful. I am optimistic about an obvious leader emerging, or at least a better story about swapping between them more easily, but a bit annoying when trying to pick one to settle on for work that I have confidence in. That being said, Zod seems like the community favorite at the moment.
Yes, it's annoying. I share your optimism. This is how the JavaScript (and now TypeScript) community figures things out.
Note that TypeScript had competitors, too. It got better. Zod has an early lead and is good enough in a lot of ways, but I'm not sure it will be the one.
Perhaps someday there will be a bun/deno-like platform with TypeScript++ that has validation merged in, but it's probably good that it's not standardized yet.
For those of you from a Python background - this is basically "Pydantic for Typescript". See also trpc - cross client/server typing using zod under the hood - https://trpc.io/
Looking around on Twitter and repos in the OSS community, it appears that Zod is now almost always favored over yup, despite an almost identical API. Curious to hear what people think if they've worked with both. We went with Yup early on at my company, and now that's what we use for consistency in our codebase. I haven't personally found it to be lacking, but some of the logic around nulls and undefined always lead me back to the docs.
My company used Yup initially but we moved to Zod to be able to infer types from schemas. For example, API response payloads are Zod schemas. OpenAPI components are also generated from Zod schemas.
There are some performance issues, and WebStorm is struggling, which forced me over to VS Code.
Previously I liked a combination of Zod and ts-pattern to create safe, pattern matching-oriented logic around my data. I find Effect is designed far better for this, so far. I'm enjoying it a lot. The Schema module has a nice convention for expressing validators, and it's very composable and flexible: https://effect.website/docs/guides/schema/introduction
There are also really nice aspects like the interoperability between Schema and Data, allowing you to safely parse data from outside your application boundary then perform safe operations like exhaustively matching on tagged types (essentially discriminated unions): https://effect.website/docs/other/data-types/data#is-and-mat...
It feels extremely productive and intuitive once you get the hang of it. I didn't expect to like it so much.
I think the real power here is that these modules also have full interop with the rest of Effect. Effects are like little lazy loaded logical bits that are all consistent in how they resolve, making it trivial to compose and execute logic. Data and Schema fit into the ecosystem perfectly, making it really easy to compose very resilient, scalable, reliable data pipelines for example. I'm a convert.
Zod is awesome if you don't want to adopt Effect wholesale, though.
I think the real turning point was typescript 5.5 (May 2024). The creator of typescript personally fixed a bug that unlocked a more natural generator syntax for Effect, which I think unlocks mainstream adoption potential.
https://twitter.com/MichaelArnaldi/status/178506160889445172...https://github.com/microsoft/TypeScript/pull/58337
[0] https://effect.website/docs/integrations/express
The learning curve just about turned me away from it at the start, but I'm glad I stuck with it.
I think learning Effect would actually teach a lot of people some very useful concepts and patterns for programming in general. It's very well thought out.
This way you can use native language features for discriminated unions, functional pipelines, and exhaustive pattern matching to model your domain instead of shoe-horning such functionality into a non-ML language!
Model your domain in F#, consume it in Python or C# backends and TypeScript frontends. The downside is needing to know all of these languages and run times but I think I'd rather know F# and the quirks with interacting with TypeScript than a library like Effect!
[1] https://fable.io
1. Debugging can become quite a pain. Nobody likes debugging generated code.
2. You don't get to use libraries and tools from the enormous JavaScript ecosystem.
3. Eventually you'll find some web feature that they haven't wrapped in your language and then you're in for FFI pain.
In the end I found Typescript was good enough that it wasn't worth dealing with those issues.
It looks very similar in its ideas to fp-ts (in the “let’s bring monads, algebraic types etc to typescript” sense).
But I did hear from teams that embraced fp-ts that things kinda ground to a halt for them. And those were category theory enthusiasts that were very good scala devs so I’m sure they knew what they were doing with fp-ts.
What happened was that the typescript compile time just shot into minutes, for a moderately sized micro-service, without anything externally heavy being introduced like you could on the frontend.
It just turned out that Typescript compiler was not so great at tracking all the inferred types throughout the codebase.
So wonder if things have improved or effect uses types more intelligently so that this is not an issue.
It's the next version of fp-ts, developed by the same people, AFAIK
Rather disappoint to see something like this being plugged as an alternative to something like zod which is a nice library that stays in its corner and has a nice fixed scope to it's functionality.
https://valibot.dev
Zod doesn't (yet[0]) and it's been a pain point for me.
[0] https://github.com/colinhacks/zod/issues/635
[1]: https://github.com/sinclairzx81/typebox
In any case, this might actually be a good use for an LLM to post-process it into whatever style you want. I bet there's even a browser extension that could do it on-demand and in-place.
In the software field you get a large portion of people that don't buy into the concept of professionalism. For various reasons - chiefly the hacker culture and the easy of contributing to the "field" means the gauntlet one runs to become a "professional" isn't inherently a given.
As a whole this is a good thing but it does mean if you operate as a "professional" maybe sometimes you have to realize that something doesn't exactly gel with your ethos (case in point). It doesn't mean it is bad; just maybe not for you and yours.
Not an issue for me, to be honest. Why does it bother you at all?
I think a more important issue is that Valibot hasn't reached 1.0 yet. But it looks like it's very close.
also since zod is the de facto validation lib, might be worth a specific page talking about why this vs zod. even their migration from zod page looks nearly identical between the two packages.
Deleted Comment
I wonder if we had nominal runtime checked types, it could work better than the current design in terms of DX. Basically, the : Type would always be a runtime checked assertion. Zod certainly fills that gap, but it would be nicer to have it baked in.
The type system would not be as powerful, but when I'm writing Kotlin, I really don't miss the advanced features of Typescript at all. I just define a data class structure and add @Serializable to generate conversions from/to JSON.
If you write in TS to start with, you can use a more sane subset.
[0] https://github.com/DefinitelyTyped/DefinitelyTyped/blob/mast...
[1] https://github.com/DefinitelyTyped/DefinitelyTyped/blob/mast...
Typescript `as Foo` will not do anything at runtime, and it will just keep on going, then throw a type error somewhere else later (possibly across an async boundary).
You can, in theory use very strong lint rules (disallow `as` operator in favour of Zod, disallow postfix ! operator), but no actual codebase that I've worked on has these checks. Even the ones with the strictest checks enabled have gaps.
Not to mention, there's intentional unsoundness in the type system, so even if you wanted, you couldn't really create a save subset of TS.
Then there's the issue of reading the library types of some generic heavy code. When I "go to definition" in my fastify codebase, I see stuff like this
Which expands to this > = ( this: FastifyInstance<RawServer, RawRequest, RawReply, Logger, TypeProvider>, request: FastifyRequest<RouteGeneric, RawServer, RawRequest, SchemaCompiler, TypeProvider, ContextConfig, Logger>, reply: FastifyReply<RouteGeneric, RawServer, RawRequest, RawReply, ContextConfig, SchemaCompiler, TypeProvider> // This return type used to be a generic type argument. Due to TypeScript's inference of return types, this rendered returns unchecked. ) => ResolveFastifyReplyReturnType<TypeProvider, SchemaCompiler, RouteGeneric>Other languages somehow don't need types this complicated and they're still safer at runtime :shrug:
Also, there are many ways to make types opaque (not show their entire verbose structure).
ts-morph provides an easy way to use the TypeScript Compiler API to view and edit the AST before compile. Once you get your head around the API, which has good examples but isn't thoroughly documented on the web.
https://github.com/dsherret/ts-morph
It also has a companion library[2] for generating TypeBox validators from TypeScript definitions, which I'm currently using in an RPC library I'm working on.
[1]: https://github.com/sinclairzx81/typebox [2]: https://github.com/sinclairzx81/typebox-codegen
After doing a deep-dive comparison, I’m left wondering why to ever choose Zod over TypeBox.
https://github.com/dittofeed/dittofeed/blob/main/packages/is...
Man why doesn’t it have a cli tool?? Wouldn’t it be nice to :'<'>!typeboxify a selection.
I felt like I was being punished for everything. (Maybe some things were project specific so I am not saying this as an absolute) It's slow, syntax is horrible, error are obscure super long lines, it can compile and explode later (which is what elixir does, except elixir will happily restart and recover)...
Elixir is based on duck typing mostly, but it works very well because you just pattern match your data when you use it. Rust is very strict and can have cryptic errors, but as everything is baked in the language it is way easier to manage.
I am not saying this to be snob about typescript and JS, but I really felt pain when working with that ecosystem, and I wonder if I'm old and stupid or if those tools are really half baked and over complicated.
Zod feels like a crutch for limitations in Javascript and Typescript. But I've found it to be a very useful crutch, and I wouldn't want to write a Typescript API without it.
Thats not true. It just crashes the “sub” process and if the parent process spawns the sub process again with the same inputs its just going to crash again.
Are you aware you can also try/catch your errors in typescript?
The whole point of the library is to validate something at runtime so of course it is going to blow up. There are also API methods that simply return a boolean instead of crashing if it fails validation. You can then use type guarding and narrowing of the type.
It is filling a necessary shortcoming in the gradual typing of TypeScript, and using validator schema types to power other APIs generic inference is powerful. I am optimistic about an obvious leader emerging, or at least a better story about swapping between them more easily, but a bit annoying when trying to pick one to settle on for work that I have confidence in. That being said, Zod seems like the community favorite at the moment.
Note that TypeScript had competitors, too. It got better. Zod has an early lead and is good enough in a lot of ways, but I'm not sure it will be the one.
Perhaps someday there will be a bun/deno-like platform with TypeScript++ that has validation merged in, but it's probably good that it's not standardized yet.
There are some performance issues, and WebStorm is struggling, which forced me over to VS Code.
But overall pretty happy.