This is the beauty of open source. I love that you’re being so collaborative here instead of seeing another tool as competition.
What's better than 1 nifty tool for querying semistructured data?
2 nifty tools for querying semistructured data!
Readit Newscharlesdaniels commented on Show HN: Qq: like jq, but can transcode between many formats github.com/JFryy/qq... · Posted by u/arandomhuman
charlesdaniels commented on Show HN: Qq: like jq, but can transcode between many formats github.com/JFryy/qq... · Posted by u/arandomhuman
I have been working on a project in a similar vein: rq[0]. Mine started out as an attempt to make a jq-like frontend for the Rego[1] language. However, I do find myself using it to simply convert from one format to another, and for pretty printing, quite often.
The interactive mode that qq has is really slick. I didn't torture test it, but it worked pretty smoothly with a quick test.
I see that the XML support is implemented using the same library as rq. This definitely has some problems, because the data model of XML does not map very cleanly onto JSON. In particular, I recall that I had problems using mxj to marshal arrays, and had to use this[2] ugly hack instead. qq seems to have done this a little more cleanly[3]. I may just have to lift this particular trick.
I definitely found CSV output to be challenging. Converting arbitrary JSON style data into something table shaped is pretty tricky. This[4] is my attempt. I have found it works well enough, though it isn't perfect.
I can also see that qq hasn't yet run into the byte order marker problem with CSV inputs yet. Might want to check out spkg/bom[5].
One final breadcrumb I'll drop - I drew a lot of inspiration for rq's input parsers from conftest[6]. That may be a good resource to find more formats and see specific usage examples of them.
Thanks for sharing! It's really interesting to see some of the convergent evolution between rq and qq.
0 - https://git.sr.ht/~charles/rq
1 - https://www.openpolicyagent.org/docs/latest/policy-language/
2 - https://git.sr.ht/~charles/rq/tree/c67df633c0438763956ff8646...
3 - https://github.com/JFryy/qq/blob/2f750f04def47bec9be100b7c89...
4 - https://git.sr.ht/~charles/rq/tree/c67df633c0438763956ff8646...
5 - github.com/spkg/bom
This looks neat, but as someone who doesn't use Linux on the desktop, I'm typically fine with
macOS (zsh/bash): command | pbcopy
windows (powershell) command | Set-Clipboard
I wrote a wrapper around these some years ago which I’ve been using since. It’s been working pretty well for me.
https://git.sr.ht/~charles/dotfiles/tree/171c95a20394552e02a...
charlesdaniels commented on This is How Civilisations Collapse unherd.com/2021/11/this-i... · Posted by u/rbanffy
Sea people? Like pirates? Vikings?
There isn’t consensus on who the sea people were.
You might be interested in the book “1177 B.C.: The Year Civilization Collapsed” by Eric H. Cline, which covers what we know about the pre-collapse civilizations as well as the collapse itself. It’s very fascinating, though it seems a lot of the details are simply lost to time.
charlesdaniels commented on IRS Will Soon Require Selfies for Online Access krebsonsecurity.com/2022/... · Posted by u/todsacerdoti
I feel like requiring users to enter into a legal agreement with a private third party in order to perform legally required obligations for a government entity has some troubling edge cases.
What happens if a user has a dispute with ID.me, who decides to terminate their account for violating their ToS? Something tells me they will still be on the hook to pay their taxes (presumably via paper, for however long that's still supported).
charlesdaniels commented on Ask HN: What are these low quality “code snippet” sites? · Posted by u/endofreach
For years now I've ran a programming site (stackabuse.com) and have closely followed the state of Google SERPs when it comes to programming content. A few thoughts/ramblings:
- The search results for programming content has been very volatile the last year or so. Google has released a lot of core algorithm updates in the last year, which has caused a lot of high-quality sites to either lose traffic or stagnate.
- These low-quality code snippet sites have always been around, but their traffic has exploded this year after the algorithm changes. Just look at traffic estimates for one of the worst offenders - they get an estimated 18M views each month now, which has grown almost 10x in 12 months. Compare that to SO, which has stayed flat or even dropped in the same time-frame
- The new algorithm updates seem to actually hurt a lot of high-quality sites as it seemingly favors code snippets, exact-match phrases, and lots of internal linking. Great sites with well-written long-form content, like RealPython.com, don't get as much attention as they deserve, IMO. We try to publish useful content, but consistently have our traffic slashed by Google's updates, which end up favoring copy-pasted code from SO, GitHub, and even our own articles.
- The programming content "industry" is highly fragmented (outside of SO) and difficult to monetize, which is why so many sites are covered in ads. Because of this, it's a land grab for traffic and increasing RPMs with more ads, hence these low-quality snippet sites. Admittedly, we monetize with ads but are actively trying to move away from it with paid content. It's a difficult task as it's hard to convince programmers to pay for anything, so the barrier to entry is high unless you monetize with ads.
- I'll admit that this is likely a difficult problem because of how programmer's use Google. My guess is that because we often search for obscure errors/problems/code, their algorithm favors exact-match phrases to better find the solution. They might then give higher priority to pages that seem like they're dedicated to whatever you searched for (i.e. the low-quality snippet sites) over a GitHub repo that contains that snippet _and_ a bunch of other unrelated code.
Just my two cents. Interested to hear your thoughts :)
I wonder if we'll see a comeback of hand-curated directories of content? I feel like the "awesome list" trend is maybe the start of something there.
I would be willing to pay an annual fee to have access to well-curated search results with all the clickbait, blogspam, etc. filtered out.
Until then, I recommend uBlacklist[0], which allows you to hide sites by domain in the search results page for common search engines.
charlesdaniels commented on Using ASCII waveforms to test real-time audio code goq2q.net/blog/tech/using... · Posted by u/jwosty
That's a protocol that's a good forty years old, and even that is not supported. And I can see why, why on earth would you want to be adding support for that in 2021? What a ridiculous state of affairs.
> That's a protocol that's a good forty years old
34 years old, actually. I guess we can go ahead and deprecate the x86 instruction set, tcp/ip, ASCII, C, tar, and many other tools and standards that are old.
> and even that is not supported.
xterm supports vt430 emulation. I use this semi regularly. I believe mintty also supports sixels, plus a handful of others. The libsixel website has a full list.
> And I can see why, why on earth would you want to be adding support for that in 2021?
You might want to read your own post ( https://news.ycombinator.com/item?id=28856005 ).
What’s your great idea as opposed to sixels?
charlesdaniels commented on Using ASCII waveforms to test real-time audio code goq2q.net/blog/tech/using... · Posted by u/jwosty
That’s interesting. Do you think sixels could work for the baseline tests? Would it be feasible to have them display nicely in an IDE, like VS Code or Visual Studio?
I don’t see why sixels couldn’t work. You’d probably want a tool to decode them, diff the images, and then output another sixel image. I’m admittedly not sure of such a tool existing off the shelf though.
I’m not aware of text editors supporting sixels, which could make preparing the tests a challenge. Certainly, you could imagine a text editor supporting them, but I’m not aware of one that does personally.
I will concede that for your specific use case, an off the shelf ASCII plotting library probably involves less custom tooling.
charlesdaniels commented on Using ASCII waveforms to test real-time audio code goq2q.net/blog/tech/using... · Posted by u/jwosty
Imagine if we had terminals that could handle graphical data. We wouldn't have to do weird kludges like this, we could just plot the waveforms in the output of our tools.
But it's 2021, and not only is this not possible, there is not even a path forward to a world where this would be possible. It's just not an option. Nobody is working on this, nobody is trying to make this happen. We're just sitting here with our text terminals, and we can't even for a second imagine that there could be anything else.
It's sad, is what it is.
I would point out that sixels[0] exist. There is a nice library, libsixel[1] for working with it, which includes bindings into many languages. If the author of sixel-tmux[2][3] is to be believed[4], the relative lack of adoption is a result of unwillingness on the part of maintainers of some popular open source terminal libraries to implement sixel support.
I can't comment on that directly, but I will say, it's pretty damn cool to see GnuPlot generating output right into one's terminal. lsix[5] is also pretty handy as well.
But yeah, I agree, I'm not a fan of all the work that has gone into "terminal graphics" that are based on unicode. It's a dead-end, as was clear to DEC even back in '87 (and that's setting aside that the VT220[6] had it's own drawing capabilities, though they were more limited). Maybe sixel isn't the best possible way of handling this, but it does have the benefit of 34 years of backwards-compatibility, and with the right software, you can already use it _now_.
0 - https://en.wikipedia.org/wiki/Sixel
1 - https://saitoha.github.io/libsixel/
2 - https://github.com/csdvrx/sixel-tmux
3 - https://news.ycombinator.com/item?id=28756701
4 - https://github.com/csdvrx/sixel-tmux/blob/main/RANTS.md
rq was shared with me yesterday and just wanted to say it's very impressive, I had heard of OPA/Gatekeeper and have looked into Rego before for policy assertions w/ terraform but I was not aware the language was so expressive until I saw rq. Also the amount of codecs rq supports and quality of them is really great.
It is really neat seeing a lot tools solve a similar problem in such unique ways (especially the case with rq) and has been a lot of fun reading your experiences here. Thanks for sharing your experiences and expertise with the de-serializing/serializing content - It is really cathartic to hear you mention the challenges you solved with xml and csv. I really like how you solved for CSV output/input and the conditions on the input data you chose for evaluating it makes a lot of sense and is really comprehensive, it bothered me too since the content would either need to be a matrix or a slice of maps but seeing as jq has string formatting that can convert things to csv and @tsv - I was at a bit of a standstill of how to approach.
Thanks so much for the bread crumbs I look forward to reading this in more detail over the week/weekend :)
Rego is “for” those authz cases like the ones you mentioned in the sense that it’s definitely designed with those in mind, and I do think it does a good job for those needs. OPA itself is definitely geared for use as a microservice or container sidecar, talking over the wire. That’s kinda hard to use in a shell script though.
Once I learned it I found myself using opa eval for searching and transforming data, eventually so much so that I made a shell script called “rq” that was basically opa eval -I -f pretty… the rest is history.