Readit News logoReadit News
quilombodigital · a month ago
This really reminds me of what Plan 9 was aiming for — breaking out of the 'box' by making everything a file, using per-process namespaces, and cleanly exposing system and network resources with proper permissions. It had that same idea: your environment shouldn't be a prison, it should be a flexible, composable space. (https://en.wikipedia.org/wiki/Plan_9_from_Bell_Labs) (https://fqa.9front.org/fqa0.html)
CGMthrowaway · a month ago
Cool idea. Seems like it would require an entirely new philosophy vs our present one on security.
packetlost · a month ago
Yes, but it also removes a lot of footguns. Access to resources (ie. paths mostly) is controlled almost entirely by the parent process, which makes access controls highly pluggable and flexible.

The real problem is Plan9 never really had a lot of attention put on the things that make having a sane security policy good. Factotum seems, at best, to be bolted on after the fact.

quilombodigital · a month ago
Yes, you would eventually be capable of sharing GPU power, devices, audio, anything. Imagine all your machine´s idle power available to others. Right now your GPU is barely being used.
no_wizard · a month ago
I think OpenDoc was meant to be this kind of thing as well. I mean the breaking out of the box part, you can read what other programs write kinda thing.
cryptonector · a month ago
What an awesome username!
fud101 · a month ago
plan9 sucks and people who think it's cool just admit their own bad taste. Unix haters handbook made the case against Plan9 even before anyone thought of the stupid idea.
MisterTea · a month ago
Nice take. You sound like the kind of person who only ever reads other peoples opinions and then parrots them creating the illusion that you have knowledge. Meanwhile you never bothered to boot the OS or see why people like it. It's actually pretty fucking amazing but that's okay, you are obviously too smart to use it since you read Unix Haters. Have fun with creaky old Unix.
williamcotton · a month ago
Just this past week I've been working on a toy/experimental web DSL [0] that uses dynamically loaded shared libraries as middleware that pass per request arena-allocated Jansson json objects between steps in a pipeline. It's extensible in that new middleware can be created. Influenced by bash and OCaml/F#, here is some kind of demo of the syntax:

  POST /api/users
    |> validate: `
      name: string(3..50)
      email: email
      age?: number(18..120)
      team_id?: number
    `
    |> jq: `{ sqlParams: [.body.name, .body.email, .body.age] }`
    |> pg: `INSERT INTO users (name, email, age) VALUES ($1, $2, $3) RETURNING *`
    |> result
      ok(201):
        |> jq: `{ success: true, user: .data.rows[0] }`
      validationError(400):
        |> jq: `{
          error: "Validation failed",
          field: .errors[0].field,
          rule: .errors[0].rule,
          message: .errors[0].message
        }`
I'm generally curious as to how jyn thinks this would fit in to their box-based framework.

[0] https://github.com/williamcotton/webpipe

jynelson · a month ago
oh hey Will! long time no see lol, it's been ages. small world.

i think this is on a good track! i like that it's designed to be extensible while still keeping some amount of structure, and that the DSL makes things very compact. how are those filters implemented? are you spawning a shell to run jq or are you interpreting in-process or something like that?

in general i'd love to see a bunch more DSLs, i think using general-purpose languages for everything is most of the time not actually helpful (cc https://jyn.dev/constrained-languages-are-easier-to-optimize...). i have some vague thoughts about how to make interop between DSLs and the host language easier but i need to think about them some more.

williamcotton · a month ago
> how are those filters implemented? are you spawning a shell to run jq or are you interpreting in-process or something like that?

Interpreting in-process. There's a jq C lib, libjq-dev.

I hope you're doing well! I've very much enjoyed your recent blog posts.

cryptonector · a month ago
I've wanted to do just this. I've used libmicrohttpd, and... it's not my favorite. But if you're going to do it in C then libmicrohttpd is probably the best API to use. You get bonus points for using jq :)

If you're also using jq you might as well ditch Jansson and use jq's `jv` API. I highly recommend it.

williamcotton · a month ago
One thing that sold me on Jansson is that you can set custom allocators which work very well with the request arena. Once those allocators are set the json objects continue to use the arena even when used in middleware. This makes memory management a cinch!

From what I can tell jq’s C lib doesn’t yet expose a way to set custom allocators.

freedomben · a month ago
This looks really neat! I typically don't like these DSLs, but this is one I would actually use.
wwarner · a month ago
The key point of the article is "your data is trapped inside your program", i.e. data models can't generally be shared between programs. One thing that has improved my life has been using apache arrow as a way to decrease the friction of sharing data between different executables. With arrow (and it's file based compressed cousin parquet), the idea is that once data is produced it never needs to be deserialized again as you would with json or avro.
PaulDavisThe1st · a month ago
Data and data models are not the same.

Sharing data is just totally undefined for the overwhelming majority of all data in the world, because there just isn't any standard for the format the data should be in.

Data models are even harder, because whereas data is produced by the world, and data formats are produced to intentionally be somewhat generalized, data models are generally produced in the context of a piece of software.

bsder · a month ago
How are you handling data update? Last I checked, Arrow and similar systems had extremely poor performance if you needed to mutate data at even modest rates.
wwarner · a month ago
you create an output arrow table and populate it with rows. but w/r/t the original idea, arrow data always comes with a schema and is efficient and compact, so it makes it easier to share data between different programs.
simpaticoder · a month ago
Computers are boxes, therefore all software is literally (and figuratively) "in a box", are they not? This might seem like a frivolous jest, but it is not. For example, the author points out that clojure, java, kotlin can interoperate, but notes they are stuck in the same jvm 'box'. This generalizes and recurses, so you must find a specific place to stop, and then motivate that.

One likely place to stop is at "processes". But this must be motivated since ultimately processes are as synthetic a convention as a language thread - it's just that the runtime is called a "kernel" instead of a "runtime".

Ultimately I think what the author is getting at is a data problem, not a code problem. Or rather, it's yearning toward a world where data and code are strongly decoupled, conventions around data are standardized, so that processes written in disparate tooling can successfully interoperate locally. However I doubt there is much appetite for a "model of everything" registry (such things have been tried, and failed, in the past). That said we might take another stab at this, since LLMs make likely that software will become more dynamic in terms of translating data structures at runtime such that one program can examine a peer program and its data structures, and how to map them to local data structures, thus achieving interoperability without a centralized agreement on representation.

xnorswap · a month ago
It's not long now until we re-invent SOAP and pretend it's a productivity breakthrough.
singpolyma3 · a month ago
This is called GraphQL
dnpp123 · a month ago
The vocabulary you speak/write every day is a box.

Your brain is a box.

Your body is a box.

/s

singpolyma3 · a month ago
I like this post but the whole thing is a tease for an unwritten next article
jynelson · a month ago
lol yeah it absolutely is

originally i had them both in one article but it was getting to be really quite long and i am still thinking through what i want to say in the follow-up

rustyminnow · a month ago
A bit off-topic, but in a shell pipeline like that, if you put your pipe chars at the end of the line you don't need backslashes and you can comment out bits of the pipe for devving.

This little change was mind-blowing for me so I always try to share when I can :)

PaulHoule · a month ago
I feel this the most on mobile platforms where the phone really should be acting as your agent but instead we're stuck with all these apps.
pjc50 · a month ago
There's an additional factor on the phone and increasingly the computer: mutual distrust.

All the apps are carefully sandboxed, because left unattended they will steal your data. The new category of AI largely works by sending your data to a server in the US where it can be surveilled. It would be great to have interoperability but first the user has to have confidence that it's not going to be weaponized against them.

jdauriemma · a month ago
The "in a box" phenomenon is very tangible to me when I am using the iOS Shortcuts feature. Its capabilities are so powerful, but its utility will always have a ceiling because app publishers' interests are generally not aligned with exposing a Shortcuts API to users. The more easily a user can automate and script the tasks that they use your app for, the less engagement their metrics will show.
idle_zealot · a month ago
It is worse on phones, but most desktop computing feels like this too, at least when you're not at a command line. I've been trying to puzzle out what I'd like computing to look like instead, but I don't get far beyond a concept of "objects" and "actions" as fundamental building blocks. How to actually expose this... yeah, it's tough.
coldpie · a month ago
COM, buddy! Publish your interface with a known UUID, anyone can claim support for your interface in the system registry, there's a standard way to initialize libraries and pull objects supporting the interface out of it, so now you can pull other peoples' applications into yours, without knowing anything about their software. This is used _all over the place_ on Windows, for things like arbitrary cross-application embedding and context menu support... at least before we realized we miiiight want to have some notion of "computer security".

https://learn.microsoft.com/en-us/windows/win32/com/com-tech...

jynelson · a month ago
have you seen https://pharo.org/ by chance? it's a smalltalk IDE built in smalltalk, which means that the whole thing is editable at runtime. it's hard to describe before you see it, https://pharo.org/features has some demos.
PaulHoule · a month ago
There's a tension between the bash economy which is too simple but pleasantly terse and the powershell economy which has a richer data structure but feels painfully verbose.
nikolayasdf123 · a month ago
interesting. how would you make it better?
PaulHoule · a month ago
Clear APIs and better semantics. Another post points out how gross mistrust gets in the way but there are alao little mistrusts. For instance if there was an API to compare restaurant menus and order things through an agent that moves power up to the agent who can influence who gets the business.

That is, I'm not afraid of being branded subversive because I like to eat strange foreign foods, I'm afraid that I'm going to get the worst pizza in town instead of the best pizza in town because somebody paid off Apple or because Google or Facebook can put up a tollbooth in front of new entrants or that they might not be interested in working with or being fair with independent restaurants because private equity has bought most of them up.

BwackNinja · a month ago
Zawinski's Law, when taken literally, argues that programs all eventually need to be communicated with by people and other programs using a generic protocol and without using program-specific or domain-specific libraries to do so.

Unix shells (local), I'll add in HTTP (remote), and Email (remote and asynchronous) are the only forms that are ubiquitous, precisely because they enforce no structure for the payload. The structured alternatives are only popular in specific domains because they create their own ecosystem. As long as input can clearly be parsed, which goes hand in hand with being human-readable as long as you're not trying to be too efficient, you get schema implicitly transmitted out of band (by having output that you can visually inspect) and interoperability for anything that can read and write text.

I'd be interested in other directions this might go, but I remain skeptical of anything that will introduce and enforce a new paradigm that requires adding complexity to programs to accommodate it.

gjvc · a month ago
every object should have its own url
chubot · a month ago
> there is no interop between powershell and nushell

FWIW I wrote a post about this design issue:

Oils Is Exterior-First (Code, Text, and Structured Data) - https://www.oilshell.org/blog/2023/06/ysh-design.html#survey...

That is

- Powershell and nushell have an "interior" design (within a process/VM)

- while POSIX shell, bash, OSH, and YSH have an "exterior" design (between processes)

And I'll claim that the exterior design is the glue you need in large, heterogeneous systems. Making the shell "interior" and "easy to use" is at odds with the role as essential glue -- it creates pressure for something else to be used instead.

---

Maybe the more pithy statement is here:

A Sketch of the Biggest Idea in Software Architecture - https://www.oilshell.org/blog/2022/03/backlog-arch.html

The lowest common denominator between a PowerShell, Elvish, Rash, and nushell script is a Bourne shell script (and eventually a YSH script)

I also claim this isn't theoretical -- there are probably a non-trivial number of bash scripts gluing together PowerShell and other shells. IMO it's better to have 1 type of glue, than 2 or more types, which I call "Unix sludge / Cloud sludge".

---

And I disagree with this part, which references protocol buffers:

> how do you get a schema? well, you establish in-band communication. RPC is ...

Protocol buffers transmit schemas OUT of band, usually via a monorepo. The data sent over the wire can't be interpreted without the schema information compiled into the binary.

The monorepo works well enough within Google, but even there it failed to scale (probably around the time of "Alphabet", e.g. Waymo and other companies)

Also, protobufs are biased toward C++; users of other languages feel this friction to varying degrees. In general, they'd rather use .gob for Go, pickle for Python, JSON for JS, Java serialization, etc.

zzo38computer · a month ago
> And I'll claim that the exterior design is the glue you need in large, heterogeneous systems.

Yes, but the specific format (and the way that the data transmission between processes and other pieces of the system is working) for the "exterior" design is an effect of the operating system. A different operating system might have a different way.

Whether it is interior or exterior, there is going to be different data types and other differences between structures, used in different ways. (One example of a difference which does not merely involve the interpretation of byte sequences, is transmissions of capabilities in systems that use capability-based security.)

I think that it is helpful to have a combination of interior and exterior functions, although most things will be exterior. However, for some things it is more helpful to do them internally within a process, for various reasons (including efficiency, requirements according to the specific operating system in use, security, and others).

> IMO it's better to have 1 type of glue, than 2 or more types

It might depend on the specific use. For some uses, it might be necessary to do it differently.

I also think that it is a mistake to assume one character set or one operating system for everything, though. However, even then, you probably wouldn't need more than one type of glue for many things, even if this "type of glue" is not the same as you would use in a different system.

> The data sent over the wire can't be interpreted without the schema information compiled into the binary.

A program is unlikely to be able to do much with data that the program is not designed to handle, regardless of how the schema is tramsmitted, so in-band communication of the schema probably wouldn't help much.

However, there can be multiple levels; e.g. with DER format, a program that does not understand the specific schema would still be able to read most of the values (with some exceptions if implicit types are used), sequences, etc, but does not know what any of that stuff means.

(I use DER format in some programs and formats that I do, because I think it is better than JSON in many ways.)

These issues are also things that my own ideas of operating system design would hopefully improve some of this. The command shell has a combination and data format would hopefully be more helpful.

chubot · a month ago
the specific format ... for the "exterior" design is an effect of the operating system. A different operating system might have a different way.

I'd say it's both the operating system and the hardware. The OS tends to follow the hardware (network, disk), and there's only one kind of hardware that I know of: that which traffics in bytes, or blocks of bytes.

You can (and should) build formats on top of that, like UTF-8 and JSON, but at bottom it's bytes. Which is of course not true of the "interior" designs -- those are more like in-memory data structures created by the .NET VM, and invisible to the outside world, without serialization.

It might depend on the specific use.

I'll have to write this out fully later, but the argument is that using 2 types of glue is strictly worse than using 1, because you create a combinatorial explosions of code. You also create more software interfaces, and interfaces are where bugs tend to happen.

More code and more interfaces leads to more bugs. One type of glue is better!

Also, the glue should be dynamically typed. Because shell is for gluing together say Rust and Go, which have wildly different type systems. It's better to have 2 type systems than 3. If you have 3, then you need more glue.

"I don't glue on top of my glue" - https://oils.pub/ysh.html