It's a clever format, especially if the focus is on machines generating it and humans or machines reading it. It might even work for humans occasionally making minor edits without having to load the file in the spreadsheet.

I think it can encode anything except for something matching the regex `(\t+\|)+` at the end of cells (*Update:* Maybe `\n?(\t+\|)+`, but that doesn't change my point much) including newlines and even newlines followed by `\` (with the newline extension, of course).

For a cell containing `cell<newline>\`, you'd have:

    |cell<tab>|
    \\<tab   >|

(where `<tab >` represents a single tab character regardless of the number of spaces)

Moreover, if you really needed it, you could add another extension to specify tabs or pipes at the end of cells. For a POC, two cells with contents `a<tab>|` and `b<tab>|` could be represented as:

    |a<tab  ><tab>|b
    ~tab pipe<tab>|tab pipe

(with literal words "tab" and "pipe"). Something nicer might also be possible.

*Update:* Though, if the focus is on humans reading it, it might also make sense to allow a single row of the table to wrap and span multiple lines in the file, perhaps as another extension.

If only ASCII had a field separator character, then we could just use that instead.

          Dec      Octal  Hex   Binary

          028      034    01C   00011100       FS    (File Separator)
          029      035    01D   00011101       GS    (Group Separator)
          030      036    01E   00011110       RS    (Request to Send)(Record Separator)
          031      037    01F   00011111       US    (Unit Separator)

Is there a text format like TSV/CSV that can represent nested/repeating sub-structures?

We have YAML but it's too complex. JSON is rather verbose with all the repeated keys and quoting, XML even moreso. I'd also like to see a 'schema tree' corresponding to a header row in TSV/CSV. I'd even be fine with a binary format with standard decoding to see the plain-text contents. Something for XML like what MessagePack does for JSON would work, since we already have schema specifications.

  {
    "headers": ["name", "habitat", "food"],
    "data": [
      ["Acorn Woodpecker", "forest", "grain"],
      ["American Goldfinch", "grassland", "grain"],
      ["Anhinga", "wetland", "fish"],
      ["Australian Reed Warbler", "wetland", "grub"],
      ["Black Vulture", "forest", null]
    ]
  }

I have been using TSV a LOT lately for batch inputs and outputs for LLMs. Imagine categorizing 100 items. Give it a 100 row tsv with an empty category column, and have it emit a 100 row tsv with the category column filled in.

It has some nice properties: 1) it’s many fewer tokens than JSON. 2) it’s easier to edit prompts and examples in something like Google sheets, where the default format of a copied group of cells is in TSV. 3) have I mentioned how many fewer tokens it is? It’s faster, cheaper, and less brittle than a format that requires the redefinition of every column name for every row.

Obviously this breaks down for nested object hierarchies or other data that is not easily represented as a 2d table, but otherwise we’ve been quite happy. I think this format solves some other things I’ve wanted, including header comments, inline comments, better alignment, and markdown support.

The poor delimiter special characters in the ascii table never get any love.

Good on you to leverage EditorConfig settings. Almost every modern IDE or editor supports it either out of the box or with a plug-in.

Or we could use the actual characters for this purpose - the FS (file separator), GS (group separator), RS (record separator), and US (unit separator).

ASCII (and through it, Unicode) has these values specifically for this purpose.

I like that there is plenty of room for comments, and the multiline extension is also cool. The backslash almost looks like what I would write on paper if I wanted to sneak something into the previous line :)