This is really great! Maybe I'll incorporate this into my own software (scratchdata/scratchdb)

Question: it looks like you wrote the parser by hand. How did you decide that that was the right approach? I myself am new to parsers and am working on implementing the PostgREST syntax in go using PEG to translate to Clickhouse, which is to say, a similar mission as this project. Would love to learn how you approached this problem!

It looks a lot like Kusto query language. Here is a kusto query:

    StormEvents
    | where StartTime between (datetime(2007-01-01) .. datetime(2007-12-31)) 
    and DamageCrops > 0
    | summarize EventCount = count() by bin(StartTime, 7d)

edit... yes it indeed was inspired by Kusto as they mention on the github Readme https://github.com/runreveal/pql

This is actually pretty awesome! I use KQL every few days for reading some logs from Azure App Insight. The syntax is pretty nice and you can make pretty complex stuff out of it. But that's it, I can't use KQL anywhere else outside Azure. With this, I can show off my KQL-fu to my teammates and surprise them with how fast you can write KQL-like syntax compared to SQL.

What's the reason to go for this over PRQL?

Looks similar to PRQL[0].

Neither PRQL nor Pql seem to be able to do anything outside of SELECT like Preql[1] can.

I propose we call all attempts at transpiling to SQL "quels".

[0] https://prql-lang.org/ [1] https://github.com/erezsh/Preql

We're developing TQL (Tenzir Query Language, "tea-quel") that is very similar to PQL: https://docs.tenzir.com/pipelines

Also a pipeline language, PRQL-inspired, but differing in that (i) TQL supports multiple data types between operators, both unstructured blocks of bytes and structured data frames as Arrow record batches, (ii) TQL is multi-schema, i.e., a single pipeline can have different "tables", as if you're processing semi-structured JSON, and (iii) TQL has support for batch and stream processing, with a light-weight indexed storage layer on top of Parquet/Feather files for historical workloads and a streaming executor. We're in the middle of getting TQL v2 [@] out of the door with support for expressions and more advanced control flow, e.g., match-case statements. There's a blog post [#] about the core design of the engine as well.

While it's a general-purpose ETL tool, we're targeting primary operational security use case where people today use Splunk, Sentinel/ADX, Elastic, etc. So some operators are very security'ish, like Sigma, YARA, or Velociraptor.

Comparison:

    users
    | where eventTime > minus(now(), toIntervalDay(1))
    | project user_id, user_email

vs TQL:

    export
    where eventTime > now() - 1d
    select user_id, user_email

[@] https://github.com/tenzir/tenzir/blob/64ef997d736e9416e859bf...

[#] https://docs.tenzir.com/blog/five-design-principles-for-buil...

InfluxDB tried to do this with InfluxQL but abandoned it, and are now back to SQL. The biggest problem I had with it when I tried it, was that is was simply too slow, queries were on average 6x slower than their SQL equivalents. I think a language like this is just too hard to optimize well.

Wow, so many query languages, right? Do we really need another one? What's the story behind that decision? Cheers.