rwiggins (u/rwiggins)

rwiggins commented on Jqfmt like gofmt, but for jq github.com/noperator/jqfm... · Posted by u/Bluestein

jzelinskie · 5 months ago

This is an incredibly useful one-liner. Thank you for sharing!

I'm a big fan of jq, having written my own jq wrapper that supports multiple formats (github.com/jzelinskie/faq), but these days I find myself more quickly reaching for Python when I get any amount of complexity. Being able to use uv scripts in Python has considerably lowered the bar for me to use it for scripting.

Where are you drawing the line?

rwiggins · 5 months ago

Hmm. I stick to jq for basically any JSON -> JSON transformation or summarization (field extraction, renaming, etc.). Perhaps I should switch to scripts more. uv is... such a game changer for Python, I don't think I've internalized it yet!

But as an example of about where I'd stop using jq/shell scripting and switch to an actual program... we have a service that has task queues. The number of queues for an endpoint is variable, but enumerable via `GET /queues` (I'm simplifying here of course), which returns e.g. `[0, 1, 2]`. There was a bug where certain tasks would get stuck in a non-terminal state, blocking one of those queues. So, I wanted a simple little snippet to find, for each queue, (1) which task is currently executing and (2) how many tasks are enqueued. It ended up vaguely looking like:

    for q in $(curl -s "$endpoint/queues" | jq -r '.[]'); do
        curl -s "$endpoint/queues/$q" \
        | jq --arg q "$q" '
            {
                "queue": $q,
                "executing": .currently_executing_tasks,
                "num_enqueued": (.enqueued_tasks | length)
            }'
    done | jq -s

which ends up producing output like (assuming queue 0 was blocked)

    [
        {
            "queue": 0,
            "executing": [],
            "num_enqueued": 100
        },
        ...
    ]

I think this is roughly where I'd start to consider "hmm, maybe a proper script would do this better". I bet the equivalent Python is much easier to read and probably not much longer.

Although, I think this example demonstrates how I typically use jq, which is like a little multitool. I don't usually write really complicated jq.

rwiggins commented on Jqfmt like gofmt, but for jq github.com/noperator/jqfm... · Posted by u/Bluestein

petercooper · 5 months ago

Not anywhere near as sophisticated as yours but I have something vaguely similar for simplifying JSON documents (while maintaining what the data also looks like) for feeding to LLMs to help them code against:

    jq 'walk(if type == "array" then (if length > 0 then [.[0]] else . end) else . end)'

So that 70,000+ line Amazon example of yours would boil down to:

    {
      "syncToken": "1753114994",
      "createDate": "2025-07-21-16-23-14",
      "prefixes": [
        {
          "ip_prefix": "3.4.12.4/32",
          "region": "eu-west-1",
          "service": "AMAZON",
          "network_border_group": "eu-west-1"
        }
      ],
      "ipv6_prefixes": [
        {
          "ipv6_prefix": "2600:1f69:7400::/40",
          "region": "mx-central-1",
          "service": "AMAZON",
          "network_border_group": "mx-central-1"
        }
      ]
    }

.. which is easier/cheaper to feed to an LLM for getting it to write code to process, etc. than the multi-megabyte original.

rwiggins · 5 months ago

Oh wow, that's fantastic. I love that it includes real values while still summarizing the doc's structure. I'm going to steal that. I'll probably keep jq-structure around because it's so easy to copy/paste paths I'm looking for, but yours is definitely better for understanding what the JSON doc actually contains.

rwiggins commented on Jqfmt like gofmt, but for jq github.com/noperator/jqfm... · Posted by u/Bluestein

rwiggins · 5 months ago

Oh, fantastic. jq has become an integral part of work for me.

I'll use this opportunity to plug the one-liner I use all the time, which summarizes the "structure" of a doc in a jq-able way: https://github.com/stedolan/jq/issues/243#issuecomment-48470... (I didn't write it, I'm just a happy user)

For example:

    $ curl -s 'https://ip-ranges.amazonaws.com/ip-ranges.json' | jq -r '[path(..)|map(if type=="number" then "[]" else tostring end)|join(".")|split(".[]")|join("[]")]|unique|map("."+.)|.[]'
    .
    .createDate
    .ipv6_prefixes
    .ipv6_prefixes[]
    .ipv6_prefixes[].ipv6_prefix
    .ipv6_prefixes[].network_border_group
    .ipv6_prefixes[].region
    .ipv6_prefixes[].service
    .prefixes
    .prefixes[]
    .prefixes[].ip_prefix
    .prefixes[].network_border_group
    .prefixes[].region
    .prefixes[].service
    .syncToken

(except I have it aliased to "jq-structure" locally of course. also, if there's a new fancy way to do this, I'm all ears; I've been using this alias for like... almost a decade now :/)

In the spirit of trying out jqfmt, let's see how it formats that one-liner...

    ~  echo '[path(..)|map(if type=="number" then "[]" else tostring end)|join(".")|split(".[]")|join("[]")]|unique|map("."+.)|.[]' | ~/go/bin/jqfmt -ob -ar -op pipe
    [
        path(..) | 
        map(if type == "number" then "[]" else tostring end) | 
        join(".") | 
        split(".[]") | 
        join("[]")
    ] | 
        unique | 
        map("." + .) | 
        .[]%
    ~

Not bad! Shame that jqfmt doesn't output a newline at the end, though. The errant `%` is zsh's partial line marker. Also, `-ob -ar -op pipe` seems like a pretty good set of defaults to me - I would prefer that over it (seemingly?) not doing anything with no flags. (At least for this sample snippet.)

rwiggins commented on (On | No) Syntactic Support for Error Handling go.dev/blog/error-syntax... · Posted by u/henrikhorluck

teeray · 7 months ago

It's funny to me when people see screenfuls of stack traces and remark how clear and useful it is. Perhaps, but do you really need all that? At what cost to your logs? I'd much rather have a one-liner wrapped error that cuts through all the framework and runtime noise. Yes, I can trace just as effectively (usually better)--the wrapping is very greppable when done well. No, in over a decade of writing Go full time, I have never cared about a runtime function or the other usual verbose garbage in my call stack.

rwiggins · 7 months ago

> how clear and useful it is. Perhaps, but do you really need all that?

Do I need clear and useful things? Maybe not. Would I like to have them anyway? Yes.

Years ago, I configured a Java project's logging framework to automatically exclude all the "uninteresting" frames in stack traces. It was beautiful. Every stack trace showed just the path taken through our application. And we could see the stack of "caused-by" exceptions, and common frames (across exceptions) were automatically cut out, too.

Granted, I'm pretty sure logback's complexity is anathema to Go. But my goodness, it had some nice features...

And then you just throw the stack trace in IntelliJ's "analyze stacktrace" box and you get clickable links to each line in every relevant file... I can dream.

> the wrapping is very greppable when done well

Yeah, that's my other problem with it. _When done well._ Every time I write an `if err != nil {}` block, I need to decide whether to return the error as is (`return err`) or decorate it with further context (`return fmt.Errorf("stuff broke: %w", err)`). (Or use `%v` if I don't want to wrap. Yet another little nuance I find myself needing to explain to junior devs over and over. And don't get me started about putting that in the `fmt` package.)

So anyway, I've seen monstrosities of errors where there were 6+ "statement: statement: statement: statement: statement: final error" that felt like a dark comedy. I've also seen very high-level errors where I dearly wished for some intermediate context, but instead just had "failed to do a thing: EOF".

That all being said, stack traces are really expensive. So, you end up with some "fun" optimizations: https://stackoverflow.com/questions/58696093/when-does-jvm-s...

rwiggins commented on GPT-4.5: "Not a frontier model"? interconnects.ai/p/gpt-45... · Posted by u/pama

TZubiri · 9 months ago

That's badass.

Was it the stick on the earth and measuring the right triangle casted by the shadow?

IIRC you also had to do the same thing on another spot far apart and measure the difference between both triangles. At the same time.

rwiggins · 9 months ago

It's been so long that I barely remember the details, but yes, we used a gnomon to calculate several things... the earth's circumference, our local solar noon, latitude and longitude, etc.

I specifically remember that we measured how far the shadow moved over time using chalk. I think this was in lieu of having a second triangle somewhere else, although, for all I know we might have used a second reference distance.

Our campus had this elaborate outdoor "observatory" that had all sorts of interesting features. The gnomon was one of them, but there were other cool things, like IIRC there was a metal sculpture where Polaris would be seen through a central hole (and a guide inscribed in the concrete to figure out where to stand for that to happen -- I think it was based on viewing height).

rwiggins commented on GPT-4.5: "Not a frontier model"? interconnects.ai/p/gpt-45... · Posted by u/pama

LPisGood · 10 months ago

There is something to be said for trusting people’s (or systems of people’s) authority.

For example, have you ever personally verified that humans went to the moon? Have you ever done the experiments to prove the Earth is round?

rwiggins · 10 months ago

> Have you ever done the experiments to prove the Earth is round?

I have, actually! Thanks, astronomy class!

I've even estimated the earth's diameter, and I was only like 30% off (iirc). Pretty good for the simplistic method and rough measurements we used.

Sometimes authorities are actually authoritative, though, particularly for technical, factual material. If I'm reading a published release date for a video game, directly from the publisher -- what is there to contest? Meanwhile, ask an LLM and you may have... mixed results, even if the date is within its knowledge cutoff.

rwiggins commented on What Is Vim? blog.jonas.foo/whats_vim.... · Posted by u/sauercrowd

stevebmark · a year ago

Agree that Vim is a language to talk to your computer, but it's not necessarily efficient. Vim is famously an imperative editor: You have to tell Vim the sequence of steps to perform. Clicking your mouse on a position on the screen is declarative: You declare you want the cursor here. With Vim you have to imperatively walk the cursor to where you want to go, or god forbid use something like easymotion.

rwiggins · a year ago

Setting aside that you can enable mouse mode in vim and use the mouse to your heart's desire --

and setting aside the "declarative" vs "imperative" nomenclature debate --

vim is pretty dang efficient. Typing `gg` to go to the beginning of a file, or `G` to go the end. Or all the other variety of motions. There are a lot.

Also, there is not necessarily any need to move the cursor anywhere. For example, if I'm in the middle of writing some string and I've decided to change the whole thing, `<ESC>ci"` does that, with no cursor movement required.

rwiggins commented on Honeycrisp apples went from marvel to mediocre seriouseats.com/how-honey... · Posted by u/haunter

sigilis · a year ago

That was it, yes. Long term storage under refrigeration is harming the quality of the apples at market in the interest of making the apple available year round.

The actionable information is in the beginning: honey crisp apples are not worth it anymore buy whatever instead. For me this keeps it from being insufferable. The stuff in the middle demonstrates what the conditions were like before it went mass market, which supports the conclusion that something changed and that it was likely related to the logistics of growing and distributing the apple.

If you don't like reading these articles, you could try using an LLM to extract a summary before choosing to dive in. I'm sure there are browser extensions for that.

rwiggins · a year ago

Yeah, having the read the article, my conclusion was that you should avoid honeycrisps from like February through ~August. i.e. only buy them when they're vaguely in season, or not too long afterward.

Anecdotally, I had some pretty delicious honeycrisps last night (in WA).

rwiggins commented on How good are American roads? construction-physics.com/... · Posted by u/chmaynard

bluGill · a year ago

I'm not sure we disagree. You use the gravel rural roads to get to the nearest paved road. So rarely are you going more than a few miles on gravel, then you hit a paved road which you travel for the many miles to where you are going. Most of the roads are still unpaved, but you spend most of your driving time on the paved roads.

rwiggins · a year ago

Errr, not in the rural area I grew up in. Gravel driveways are super common, gravel roads not so much.

To give some specifics: I only remember driving down an actual gravel road (like, for public use) a single time. In 18 years. Even my friends who lived >30min from the nearest "city" (~10k population) had paved roads all the way.

But that is just my own experience. Areas with a different climate or geography might be a totally different story. My hometown area is relatively flat, lots of farmland, and rarely gets severe winter weather.