I think the instrumenting and generator stuff gets disproportionate attention. For me by far the biggest win from spec has been with parsing. This completely changes how you'd write a library that takes a data structure and parses it into something meaningful (for example, what hiccup does for html or what honeysql does for sql).
In the past, this required a lot of very ugly parsing code and manual error-checking. With spec, you write specs and call s/conform. If it failed, you get a nice error, especially if you pair it with expound. If it succeeded, you get a destructured value that is really easy to pull data out of. I've done this in a half dozen different libraries and i'm pretty sure i wouldn't have even written them without spec.
I started playing with spec because of the idea of automated test generation, but the reality of it is that I use it as a super-charged validation library.
I think this emphasis actually does the library a disservice in that I see new users ask questions along the lines of "Should I use s/valid? to manually check inputs to my API"? The answer to that, in my usage, is "Yes! Of course!", but many people seem to think that they are using Spec wrong if they use it for something other than instrumentation and generation.
I remember doing just that - writing some ugly parsing code, thinking that I should be a good team member and add some specs for what I was doing, and when I tried calling conform... Oh, it did the parsing for me!
You could call it on every input change, though I'd say Spec conforming is not the most performant parsing library in the world, so not sure if it be fast enough for running it on each input.
As a very heavy user of spec, I’ve since switched to using Malli [0], which is similar but uses plain data to model specs (and doesn’t use Clojure Spec at all).
Also, Malli is being funded / supported by Clojurists Together [1], which is a wonderful initiative that’s also worth a look.
I found spec very useful and use it more and more. I'm looking forward to newer revisions, with support for optionality, it's been a big problem area in my case.
Here's a quick list of gotchas (well, they got me, so perhaps other people will find this list useful):
* `s/valid?` does not actually tell you that the data is valid
The naming of `s/valid?` suggests that you can call it on your data and find out if the data is valid according to the spec. This isn't true. What it actually tells you is if the data, when conformed, will be valid according to the spec.
If you pass the data as-is to your functions (without passing it through `s/conform`), you might find that they will be surprised at what they get.
* Conformers are likely not what you think. They are not intended for coercion and have many pitfalls (for example, multi-specs dispatch on unconformed value).
* s/merge doesn't necessarily do what you wanted if you're using conformers, only the last spec passed to merge will be used for s/conform (but you're not using conformers, right?)
* specs are checked eagerly. If you think that (s/valid? ::my-spec x) will only check ::my-spec, that is not the case. It will check any key of x found in the spec registry.
I settled on a subset of spec, because of the pitfalls.
I think all of those problems are only true when you try to do coercion with custom conformers. Which is not the intended use of conformers.
Conformers are meant to parse the data when there are multiple possibility of what something can validate against, the conformer will disambiguate and return a result that tells you which path was chosen.
Coercion is not supported as part of Spec, you're expected to do that seperatly either before or after validating/conforming.
Can you elaborate with an example for s/valid? on needing to preconfrom your data. I have not had this issue.
Also note for others with regard to eagerness, this is only for maps. "When conformance is checked on a map, it does two things - checking that the required attributes are included, and checking that every registered key has a conforming value. We’ll see later where optional attributes can be useful. Also note that ALL attributes are checked via keys, not just those listed in the :req and :opt keys. Thus a bare (s/keys) is valid and will check all attributes of a map without checking which keys are required or optional."
Can you explain the point of :opt with s/keys if it will always check any registered spec in if present?
> Can you elaborate with an example for s/valid? on needing to preconfrom your data. I have not had this issue.
If you have any conformers, s/valid? will use them before validating.
So, if you have a 'set' conformer, for example, s/valid? will tell you that the data is valid even if the value is not a set, but a vector, for example.
Your code must explicitly call 'conform', checking with s/valid? is not enough.
That is exactly my point. Optionality is complex and causes a lot of bugs in applications (certainly in mine, I would say it's the #1 root cause of bugs).
Which is why I'd like spec to help me with managing it. It's not obvious, because whether certain data is required or optional depends on context. But from what I've heard, bright minds at Cognitect are thinking about it, and given their track record so far, I'm pretty confident I will like the solution.
We use spec A LOT to validate a huge config file/DSL thing for our internal ETL application.
My general feelings are:
- spec is great, you should use it, the composeability and flexibility are awesome features.
- I've never once used conformers - maybe I just don't "get it" (which if so, speaks badly of them I think since I've been heavily using spec for years), but the use cases for them seem strange to me and I feel they cause more confusion than they're worth. I wish they were separated out into more separate/optional functionality.
- It's SO MUCH more powerful than things like JSON schema, but that comes at the cost of portability - there's no way we could send our spec over the wire and have someone else in a different environment use it. But also, there's no way we could implement some of our features in a tool like JSON Schema [and have it be portable] ("Is the graph represented by this data-structure free of cycles?" "When parsed by the spark SQL query parser, is this string a syntactically valid SQL query?").
- Being able to spec the whole data input up front has saved hundreds of lines of error-checking code and allows us to give much better errors up-front to our users and devs
- Spec has a lot of really cool features for generative testing, but we rarely use them since we've implemented lots of complex specs where it's not really practical to implement a generator (i.e. "strings which are valid sql queries" or "maps of maps of maps which when loaded in a particular way meet all the other requirements and are also valid DAGs"). I feel torn about this because the test features are great, but the extreme extensibility of spec is what I love most about it. I haven't often found a scenario where I actually have a use for the generative features (either the data is so simple I don't need them, or so complex that they don't work).
At a previous job I streamed Clojure over a trusted wire and executed it remotely in some circumstances. We weren't using Spec (Prismatic Schema at the time), but it worked pretty well and we even validated our schemas in javascript via cljs-wrapped library. I don't see why this wouldn't be possible with Spec, although you're a bit limited by code that will execute everywhere.
> I've never once used conformers - maybe I just don't "get it" (which if so, speaks badly of them I think since I've been heavily using spec for years), but the use cases for them seem strange to me and I feel they cause more confusion than they're worth. I wish they were separated out into more separate/optional functionality
That's because you don't Spec your functions and macros.
A lot of people have only used Spec to validate data that enters and leaves the boundary of their application. Which is a great use of Spec, and I use Spec mostly for that as well.
But there is a whole other world where Spec was designed to validate your functions and macros as well.
That's where conformers make sense.
For macros, you can use conformers to help you with writing a macro, by using Spec to define a DSL and conform to parse it out for you. It both validates the macro DSL and makes it easier for you to parse it.
For functions, conform can be useful to assert the output is what you expect for some given input. Often times, the output might depend on what kind of input you got. Conform basically tells you the kind of input it was, so in your validation you can validate differently based on each kind conform tells you it received.
Your feeling on there being no way to transport this over the wire is puzzling to me but I admit I don't have all the details. My feeling is why not? Surely if we have wire formats for self describing binary objects that can then be serialized into an in memory structure, transporting a spec shouldn't be harder than that?
Not that there's _no_ way to transport it over the wire, it just requires the full environment (a JVM on the other end - because we have java-specific stuff like spark calls). I'd put it at an order-of-magnitude more complex than something declarative like a JSON schema which is pretty safe to execute anywhere.
I don't think this is a really big failing of spec - I don't know of ANY validation tools that don't have to compromise between power/extensibility/ease-and-safety-of-execution-somewhere-else.
Maybe if you implemented some kind of uber-validator in purely functional prolog or something?
In fact we do very similar things - that is to say, of course spec cannot reverse-engineer my "this is valid sql" predicate to do it automatically and I end up having to hand-code a lot of generators anyways. The generation features of spec [for us, not for everyone] end up being a relatively minor value add compared to plain-ole test.check
I strongly recommend that anyone using spec for validation should check out Orchestra, to instrument the functions and have them automatically validated on each and every call: https://github.com/jeaye/orchestra
For my team, generators and parsing are basically useless with spec. We just don't use them. But describing the shape of data and instrumenting our functions, using defn-spec, to ensure that the data is correct as it flows through the system is exactly what we want and nothing I've seen in Clojure land does it like spec + Orchestra can.
I think part of this may boil down to different types of testing. We primarily use functional testing, especially for our back-end, so we're starting from HTTP and hitting each endpoint as the client would. Then we ensure that the response is correct and any effects we wanted to happen did happen. This is much closer to running production code, but we do it with full instrumentation. Being able to see an error describing exactly how the data is malformed, which function was called, and what the callstack was is such a relief in Clojure.
Call to #'com.okletsplay.back-end.challenge.lol.util/provider-url did not conform to spec.
-- Spec failed --------------------
Function arguments
(nil)
^^^
should satisfy
(->
com.okletsplay.common.transit.game.lol/region->info
keys
set)
-- Relevant specs -------
:com.okletsplay.common.transit.game.lol/region:
(clojure.core/->
com.okletsplay.common.transit.game.lol/region->info
clojure.core/keys
clojure.core/set)
-------------------------
Detected 1 error
I'm onboard with orchestra because I tend to be lazy. But I also want to explain the rational for why Spec's instrumentation only instrument the input.
This has to do with the philosophy. If you want to write bug free programs, and I mean, if you care A LOT about software correctness.
The idea in that case will be that all your functions will have a set of unit tests and generative tests over them that asserts that for most possible inputs they return the correct output.
Once you know that, you know that if provided valid input, your functions are going to return valid output. Because you know your function works without any bugs.
Thus, you no longer need to validate the output, only the input. Because as I just said, you now know that any valid input will result in your code returning valid output as well. So re-validating the output would be redundant.
And this goes one further. After you've thoroughly tested each functions, now you want to test the integration of your functions together. So you'd instrument your app, and now you'd write a bunch of integration tests (some maybe even using generative testing), to make sure that all possible input from the user (or external systems if intended for machine use) will result in correct program behavior and an arrangement of functions that all call each other with valid input.
Once you've tested that, you now also know that the interaction/integration of all your functions work.
At this point you are confident that given any valid user input, your program will behave as expected in output and side-effect.
You can thus now disable instrumentation.
But before you go to prod, you need one more thing, you have to protect yourself against invalid user input, because you haven't tested that and don't know how your program would behave for it. Thus with Spec, you add explicit validation over your user input which reject at the boundary the input from the user of invalid.
You now know there things:
1. All your individual functions given valid input behave as expected in output and side-effect.
2. Your integration of those functions into a program works for all given valid user input.
3. Your program rejects all invalid user input, and will only process valid user input.
Thus you can go to prod with high confidence that everything will work without any defect.
---
Now back to orchestra. Orchestra assumes that you weren't as vigilant as I just described, and that you might have not tested each and every function, or that you only wrote a small amount of tests for them which only tested a small range of inputs. Thus it assumed because of that, probably when you go towards running functional/integ tests, you want to continue to assert the output of each function is still valid, as you anticipate those will probably create inputs to functions that your tests over that function did not test.
Something like Haskell, or even Rust, requires similar vigilance in order to get the program even into a working state. With thorough, strong, static type checkers, novel borrow checkers, and more, a lot of development time is spent up front, dealing with compiler/type errors. Thus you can go to prod with high confidence that everything will work without any defect.
Now, back to Clojure. Clojure assumes that you weren't as vigilant as I just described, and that you don't have static type checking for each function, or that you don't have a fixed domain for all of your enums. Thus it is assumed because of that, probably when you go running toward testing (unit, functional, or otherwise), you want to assert the validity of all of this data.
My point in re-painting your words is that we all trade certain guarantees in correctness for ease of development, maintainability, or whatever other reasons. Developers may choose Clojure over Haskell, for example, because maintaining all of that extra vigilance is undesirable overhead. Similarly, developers may reasonably choose not to unit test every single function in the code base, but instead functionally test the public endpoints and unit test only certain systems (such as the one which validates input for the public endpoints), because maintaining all of that extra vigilance is undesirable overhead.
My pet project is a partial evaluator for Clojure code that uses data generated from spec to fuzz code and optimize it. The coverage is accepted as complete, so there are no guards and deoptimizations like you'd have in a JIT, the programs are just wrong.
It seems like a fairly powerful technique, although you couldn't ever rely on it with production code. After several years of tinkering I managed to get a Forth interpreter written in Clojure executing a specific input string partially evaluating down to OpenGL shader code, to hardware accelerate my friend's Stackie experiment (link to his version below).
(nth (sort [0 n 5]) 1) where sort is a merge sort also successfully compiles down to just the branches you'd hand optimize it to, which is Graal's party trick. Although they're solving the problem in a bulletproof general way, so the difficulty is incomparable.
The eventual goal is to write Clojure in Clojure without it being horrendously inefficient.
At Hyperfiddle we are using spec to specify UIs, so e.g. (s/fdef foo :args (s/cat :search-needle string? :selection keyword?) :ret (s/coll-of (s/keys ...))) describes exactly what you need to render a masterlist table with some query parameters. It's being used with pilot customers, now in production. This stuff works!
Readit News
In the past, this required a lot of very ugly parsing code and manual error-checking. With spec, you write specs and call s/conform. If it failed, you get a nice error, especially if you pair it with expound. If it succeeded, you get a destructured value that is really easy to pull data out of. I've done this in a half dozen different libraries and i'm pretty sure i wouldn't have even written them without spec.
I started playing with spec because of the idea of automated test generation, but the reality of it is that I use it as a super-charged validation library.
I think this emphasis actually does the library a disservice in that I see new users ask questions along the lines of "Should I use s/valid? to manually check inputs to my API"? The answer to that, in my usage, is "Yes! Of course!", but many people seem to think that they are using Spec wrong if they use it for something other than instrumentation and generation.
When writing complicated macros, Spec conforming is so useful!
so that "john mcallister" yields { :name "john" :lastname "mcallister" } but "john " would yield { :name "john" :lastname nil } (or :todo even)
Also, Malli is being funded / supported by Clojurists Together [1], which is a wonderful initiative that’s also worth a look.
[0]: https://github.com/metosin/malli
[1]: https://www.clojuriststogether.org/news/q3-2020-funding-anno...
Here's a quick list of gotchas (well, they got me, so perhaps other people will find this list useful):
* `s/valid?` does not actually tell you that the data is valid
The naming of `s/valid?` suggests that you can call it on your data and find out if the data is valid according to the spec. This isn't true. What it actually tells you is if the data, when conformed, will be valid according to the spec. If you pass the data as-is to your functions (without passing it through `s/conform`), you might find that they will be surprised at what they get.
* Conformers are likely not what you think. They are not intended for coercion and have many pitfalls (for example, multi-specs dispatch on unconformed value).
* s/merge doesn't necessarily do what you wanted if you're using conformers, only the last spec passed to merge will be used for s/conform (but you're not using conformers, right?)
* specs are checked eagerly. If you think that (s/valid? ::my-spec x) will only check ::my-spec, that is not the case. It will check any key of x found in the spec registry.
I settled on a subset of spec, because of the pitfalls.
Conformers are meant to parse the data when there are multiple possibility of what something can validate against, the conformer will disambiguate and return a result that tells you which path was chosen.
Coercion is not supported as part of Spec, you're expected to do that seperatly either before or after validating/conforming.
Not all of the above problems are due to coercion, but the majority are.
To be clear: I'm not complaining here, I find spec to be very useful and I like it, just pointing out traps for the unwary.
Also note for others with regard to eagerness, this is only for maps. "When conformance is checked on a map, it does two things - checking that the required attributes are included, and checking that every registered key has a conforming value. We’ll see later where optional attributes can be useful. Also note that ALL attributes are checked via keys, not just those listed in the :req and :opt keys. Thus a bare (s/keys) is valid and will check all attributes of a map without checking which keys are required or optional."
Can you explain the point of :opt with s/keys if it will always check any registered spec in if present?
If you have any conformers, s/valid? will use them before validating.
So, if you have a 'set' conformer, for example, s/valid? will tell you that the data is valid even if the value is not a set, but a vector, for example.
Your code must explicitly call 'conform', checking with s/valid? is not enough.
You should be careful asking for this.
Both Protobufs and Cap'n Proto eventually decided that "optional" creates more grief that it saves.
It is also a bit of a religious flamewar.
Caution is advised.
Which is why I'd like spec to help me with managing it. It's not obvious, because whether certain data is required or optional depends on context. But from what I've heard, bright minds at Cognitect are thinking about it, and given their track record so far, I'm pretty confident I will like the solution.
My general feelings are:
- spec is great, you should use it, the composeability and flexibility are awesome features.
- I've never once used conformers - maybe I just don't "get it" (which if so, speaks badly of them I think since I've been heavily using spec for years), but the use cases for them seem strange to me and I feel they cause more confusion than they're worth. I wish they were separated out into more separate/optional functionality.
- It's SO MUCH more powerful than things like JSON schema, but that comes at the cost of portability - there's no way we could send our spec over the wire and have someone else in a different environment use it. But also, there's no way we could implement some of our features in a tool like JSON Schema [and have it be portable] ("Is the graph represented by this data-structure free of cycles?" "When parsed by the spark SQL query parser, is this string a syntactically valid SQL query?").
- Being able to spec the whole data input up front has saved hundreds of lines of error-checking code and allows us to give much better errors up-front to our users and devs
- Spec has a lot of really cool features for generative testing, but we rarely use them since we've implemented lots of complex specs where it's not really practical to implement a generator (i.e. "strings which are valid sql queries" or "maps of maps of maps which when loaded in a particular way meet all the other requirements and are also valid DAGs"). I feel torn about this because the test features are great, but the extreme extensibility of spec is what I love most about it. I haven't often found a scenario where I actually have a use for the generative features (either the data is so simple I don't need them, or so complex that they don't work).
That's because you don't Spec your functions and macros.
A lot of people have only used Spec to validate data that enters and leaves the boundary of their application. Which is a great use of Spec, and I use Spec mostly for that as well.
But there is a whole other world where Spec was designed to validate your functions and macros as well.
That's where conformers make sense.
For macros, you can use conformers to help you with writing a macro, by using Spec to define a DSL and conform to parse it out for you. It both validates the macro DSL and makes it easier for you to parse it.
For functions, conform can be useful to assert the output is what you expect for some given input. Often times, the output might depend on what kind of input you got. Conform basically tells you the kind of input it was, so in your validation you can validate differently based on each kind conform tells you it received.
I don't think this is a really big failing of spec - I don't know of ANY validation tools that don't have to compromise between power/extensibility/ease-and-safety-of-execution-somewhere-else. Maybe if you implemented some kind of uber-validator in purely functional prolog or something?
For my team, generators and parsing are basically useless with spec. We just don't use them. But describing the shape of data and instrumenting our functions, using defn-spec, to ensure that the data is correct as it flows through the system is exactly what we want and nothing I've seen in Clojure land does it like spec + Orchestra can.
I think part of this may boil down to different types of testing. We primarily use functional testing, especially for our back-end, so we're starting from HTTP and hitting each endpoint as the client would. Then we ensure that the response is correct and any effects we wanted to happen did happen. This is much closer to running production code, but we do it with full instrumentation. Being able to see an error describing exactly how the data is malformed, which function was called, and what the callstack was is such a relief in Clojure.
This has to do with the philosophy. If you want to write bug free programs, and I mean, if you care A LOT about software correctness.
The idea in that case will be that all your functions will have a set of unit tests and generative tests over them that asserts that for most possible inputs they return the correct output.
Once you know that, you know that if provided valid input, your functions are going to return valid output. Because you know your function works without any bugs.
Thus, you no longer need to validate the output, only the input. Because as I just said, you now know that any valid input will result in your code returning valid output as well. So re-validating the output would be redundant.
And this goes one further. After you've thoroughly tested each functions, now you want to test the integration of your functions together. So you'd instrument your app, and now you'd write a bunch of integration tests (some maybe even using generative testing), to make sure that all possible input from the user (or external systems if intended for machine use) will result in correct program behavior and an arrangement of functions that all call each other with valid input.
Once you've tested that, you now also know that the interaction/integration of all your functions work.
At this point you are confident that given any valid user input, your program will behave as expected in output and side-effect.
You can thus now disable instrumentation.
But before you go to prod, you need one more thing, you have to protect yourself against invalid user input, because you haven't tested that and don't know how your program would behave for it. Thus with Spec, you add explicit validation over your user input which reject at the boundary the input from the user of invalid.
You now know there things:
1. All your individual functions given valid input behave as expected in output and side-effect.
2. Your integration of those functions into a program works for all given valid user input.
3. Your program rejects all invalid user input, and will only process valid user input.
Thus you can go to prod with high confidence that everything will work without any defect.
---
Now back to orchestra. Orchestra assumes that you weren't as vigilant as I just described, and that you might have not tested each and every function, or that you only wrote a small amount of tests for them which only tested a small range of inputs. Thus it assumed because of that, probably when you go towards running functional/integ tests, you want to continue to assert the output of each function is still valid, as you anticipate those will probably create inputs to functions that your tests over that function did not test.
Now, back to Clojure. Clojure assumes that you weren't as vigilant as I just described, and that you don't have static type checking for each function, or that you don't have a fixed domain for all of your enums. Thus it is assumed because of that, probably when you go running toward testing (unit, functional, or otherwise), you want to assert the validity of all of this data.
My point in re-painting your words is that we all trade certain guarantees in correctness for ease of development, maintainability, or whatever other reasons. Developers may choose Clojure over Haskell, for example, because maintaining all of that extra vigilance is undesirable overhead. Similarly, developers may reasonably choose not to unit test every single function in the code base, but instead functionally test the public endpoints and unit test only certain systems (such as the one which validates input for the public endpoints), because maintaining all of that extra vigilance is undesirable overhead.
It seems like a fairly powerful technique, although you couldn't ever rely on it with production code. After several years of tinkering I managed to get a Forth interpreter written in Clojure executing a specific input string partially evaluating down to OpenGL shader code, to hardware accelerate my friend's Stackie experiment (link to his version below).
(nth (sort [0 n 5]) 1) where sort is a merge sort also successfully compiles down to just the branches you'd hand optimize it to, which is Graal's party trick. Although they're solving the problem in a bulletproof general way, so the difficulty is incomparable.
The eventual goal is to write Clojure in Clojure without it being horrendously inefficient.
http://blag.fingswotidun.com/2015_09_16_archive.html
[1] https://github.com/metosin/malli
[2] https://github.com/reifyhealth/specmonstah