Show HN: Apple Notes Liberator – Extract Notes.app Data and Save It as JSON

I am excited to see more solutions that work with Apple Notes database directly. This is my go-to application to capture unstructured memos while researching various topics. Last year I got carried away and built a simple site generator for Notes [0]. As a fun exercise I implemented ORM models to query Sqlite database in Ruby:

  User::Account.server(database: "tmp/test.sqlite").first.
    folders_dataset.offset(1).first
    notes.
    map(&:attachments).flatten.
    find { |el| el.is_a?(User::ObjectTypes::EmbeddedPublicJpeg) &&
      el.height > 700 &&
      el.summary[/animal/i] }.
    media_filepath
 => "Accounts/<uuid>/Media/F669B83A-0C47-408E-9645-015737F3B11F/Pasted Graphic.jpg"

Apple Notes' database structure is a fascinating product story. Notes started as a simple note-taking application that stored its content as raw HTML in a few simple tables. However, as the product grew and needed more features like collaboration, they had to make changes and add new columns and tables. They also implemented two flavors of CRDT-like algorithms that evolved over time: one for plaintext chunks and the other one for embedded objects like tables. On top of that, every change that Apple makes must be backward compatible to support legacy devices and systems; but backward-compatibility was not always possible and users had to go through “Migrate your Notes” steps, but it is again a product story.

I wish that one day, someone from the development team would write an article about it.

[0] https://devlog.notespub.com/2022/08/site-generator-for-apple...

dunham · 3 years ago

Yeah, I'd like to know the story behind it. topotext is weird and unlike anything I've found in the literature. It is kind of like RGA, but with more links. And I haven't seen anyone else layer attributes on top of text.

It looked like the other stuff came later and was somewhat orthogonal. (Although their ordered set construct was abusing topotext to represent ordered lists of ids.)

I don't know how much has changed since 2018. I think my table decoding may be broken now.

oneearedrabbit · 3 years ago

Thanks for notesutils, I enjoyed reading your protobuf implementation!

polyrand · 3 years ago

Hey! I remember reading your blog post a few months ago, and it inspired me to try to create a CMS using Apple Notes + the IMAP syncing features[0]. But everything felt like a hack all the time. Many features are not supported when syncing over IMAP. I wanted to have access to the SQLite database, so I can't have a non-macOS VM syncing and processing notes continuously (or at least I couldn't find a tool that can do that). Parsing the data seems like a lot of work, I was also a bit worried that Apple could introduce a change in the format that would break the current tools. Linking between notes is not well-supported, etc.

At some point, I also had some issues syncing notes between different iCloud accounts when the folder had 100s of notes, and that was probably the tipping point. As I was struggling to make it work, I kept thinking that if I just had a bunch of Markdown files I could move around, the whole process would have been easier.

I eventually realized I didn't need most features of Apple Notes, and moved to Obsidian[1]. It feels quite snappy too, and now I can easily publish some of my notes. It can also use iCloud Drive for syncing between Apple devices. It's missing some collaborative features, but I also didn't need those.

[0]: https://ricardoanderegg.com/posts/apple-notes-blog-cms/ [1]: https://ricardoanderegg.com/posts/apple-notes-to-obsidian/

oneearedrabbit · 3 years ago

I second your sentiment that it is a hack, and here be dragons. At the same time, I had an honest conversation with myself about the dilemma, and whether I am worried about changes to the database or not. I ended up answering that it is a low-risk factor especially for the project of that kind.

I am sure that Apple won’t be making any substation breaking changes to a database structure unless they have to, there is a price to pay. Such changes will always be tied to a major macOS/iPhone release, and it doesn't happen very often.

That said, I had to put this project on the back burner, it was not immediately clear what is the ROI especially considering there is a long tail of databases to parse. It starts with Mountain Lion and then adds up quickly.

I need to release my source code to Github--ORM approach was worth exploring, it is a concise declaration that is relatively easy to update and manage.

techdragon · 3 years ago

I’m trying to consolidate my note taking with Obsidian but I’m with you about Notes being a go-to tool… it’s just so much better integrated into all the apple devices, I can scribble quick notes with the pencil, ask Siri and all sorts of shortcuts can easily make or add to notes… but I’m absolutely sick of being held hostage by my software and for me my notes are too important not to protect. I’m glad to see anyone work on this, because I wasn’t relishing the idea of trying to force Apple to open up using the EU digital markets directive. Its my nuclear option if the situation gets worse, the various scripts and fixes stop working and I can’t work around it, because I’m pretty blown away by how complicated the notes format is and I won’t be surprised if they add something new and the format develops another layer of complexity.

azeirah · 3 years ago

For what it's worth, I am working on a tool that syncs files from a ReMarkable drawing tablet to Obsidian.

I was planning on looking into support Samsung notes and Apple notes next. An open-source project like in the OP will help a lot here!

You can subscribe to a newsletter here if you're interested (in the top right of the page)

https://streamsoft.gumroad.com/l/remarkable-to-obsidian?layo...

some_guy_in_ca · 3 years ago

For Mac - I use Noteplan. Obsidian has some amazing feature but it is soo complicated. I find Noteplan has a more simple user interface and most of the features. They can work together because they both use local markdown files.

wslh · 3 years ago

I would love to have a tool to retrieve the hyperlink of the current note so you can organize the information as a wiki. I know it uses URIs.

oneearedrabbit · 3 years ago

I was toying with this idea, and the best (and also the worst) workaround I found is to share a note with yourself; then link a note with ⌘-k. I tried doing this a couple of times--nah, it is a workaround.

pps · 3 years ago

Hookmark app.

sillywalk · 3 years ago

Interesting.. Have you discovered any limitations to Notes e.g. how many total or largest size?

codetrotter · 3 years ago

That is awesome! Thank you for sharing your notes on this :)

I wrote a similar tool in Python a few weeks ago, which uses AppleScript to liberate the data and saves it to a SQLite database: https://datasette.io/tools/apple-notes-to-sqlite

I didn't actually know AppleScript before writing this tool... but it turned out ChatGPT knew it well enough to unblock me and provide me with exactly the code I needed to build the rest of the project! https://til.simonwillison.net/gpt3/chatgpt-applescript

Apple Notes Liberator creates a copy of the SQLite database and then runs queries against that directly to extract the data.

I chose not to do that (despite being all-in on SQLite for everything) because I worry about future changes to the software baking my script - I figured the AppleScript interface was less likely to have breaking changes in the future.

tzs · 3 years ago

From your article on how you wrote that:

> I've been stubbornly refusing to learn AppleScript for nearly twenty years at this point. Thanks to ChatGPT I can continue not to learn it, but I can now use it to solve problems

The Apple scripting stuff can also use JavaScript. It might be better to continue not to learn in JavaScript instead of AppleScript, if ChatGPT is as adept at Apple scripting in JavaScript as it seems to be in AppleScript.

That way if you want to tweak what ChatGPT gives you it will probably be easier. With AppleScript both interacting with other things and control flow or computation can be weird and confusing. With JavaScript at least the control flow and computation will likely be more normal.

simonw · 3 years ago

Huh, that's a great tip! ChatGPT knew how to convert my AppleScript to the equivalent JavaScript and showed me how to run it too: https://gist.github.com/simonw/86c2153b7882747122f1e3d501260...

Added a section about this to my TIL: https://til.simonwillison.net/gpt3/chatgpt-applescript#user-...

IggleSniggle · 3 years ago

I was trying to do a thing with AppleScript, and the barriers I ran into were not with the language, but with discovery of the APIs that Apple surfaces to the end user. In that regard, you’re better off with AppleScript. I can’t believe just how bad the AppleScript documentation is, but the js documentation is even worse; using js is just using AppleScript but with an additional layer of indirection that doesn’t provide any productivity gain.

mcint · 3 years ago

IggleSniggle expresses it well. As ugly as the AppleScript expressions are, they force you to write in a way that's sympathetic to the API, so, in addition to returning errors at all in more cases, they're easier to diagnose or minimize.

https://gist.github.com/mcint/561f72e6baa3e5c68c2577b04dad78...

I've linked some of my scripts, bash calling applescript, that I use extensively to export tabs from browsers.

aidos · 3 years ago

I did a bit of AppleScript recently (scripted changing between draw and erase in zoom annotations from the buttons on my tablet pen). In my reading I got the impression that not everything is implemented in js so you might get stuck.

whatch · 3 years ago

"Unblock" is the perfect term for the effect of all these GPT things on me. Thank you for coining that!

Recently I wanted to have something like that but for Safari Reading list. The main difference is I tried to find a solution not for one time liberation, but for continuous 2-way exchange with some web service. It appears you HAVE to run everything locally for that. I like "privacy", but I like having options even more.

By the way it was ChatGPT that helped me to remove this several month old item from my TODO list.

pjot · 3 years ago

I loved your post about the pelican! https://simonwillison.net/2020/May/21/dogsheep-photos/

ivansavz · 3 years ago

Here is a Python library that can also create notes: https://github.com/RhetTbull/macnotesapp

It is also based on Action Script commands. I haven't played with it much yet, but it's on my TODO list (to add automatic headings for every day + archive DONE tasks automatically).

jasonriddle · 3 years ago

Interesting. I was curious and wanted to achieve something similar with voice memos. I used the prompt "Write applescript to loop through all of my Apple Voice Memos" and the code it produced couldn't run in applescript.

simonw · 3 years ago

I just tried that prompt against GPT-4 and got "Note that the Voice Memos app does not have an AppleScript dictionary, so the script will use UI Scripting to interact with the app" followed by a very complicated recipe.

Zufriedenheit · 3 years ago

Nice tool! Just tried it, works great. Interesting that Applescript exports the images as Base64, or does Apple notes store them like that as well? Maybe converting them to binary when exporting would save some space.

frizlab · 3 years ago

A bit out of context, but did you know you can actually write OSA scripts using JS? In Script Editor there’s a menu where you can change the language.

ecliptik · 3 years ago

A feature that a lot of Apple apps have that doesn't seem talked about much is they can use standard protocols and not require iCloud.

Mail.app/Reminders.app/Notes.app can use IMAP, Cal.app with CalDAV, and Contacts.app with CardDAV [1].

It doesn't offer a lot of the features iCloud has like advanced formatting or collaboration, but is a useful for simple standards based syncing and portability.

I've done this with my iPhone/Macs and Fastmail for the last few years and it works great [2].

1. https://support.apple.com/guide/iphone/set-up-mail-contacts-...

2. https://www.fastmail.help/hc/en-us/articles/1500000279941

I…can’t believe I didn’t know this about Notes or Reminders. Thank you for mentioning this.

Yes, the IMAP sync protocol was one of my discoveries when trying to turn Apple Notes into a CMS[0]. But like you said, many features are missing. I could do without the collaboration features, but I found that tables didn't work either. A lot of the unsupported features just disappeared from the IMAP-synced note, so you can't recover them later if you move the note back to an iCloud account.

[0]: https://ricardoanderegg.com/posts/apple-notes-blog-cms/

donatj · 3 years ago

Hmm… I wonder how hard it would be to throw together a little IMAP server just for personal note synchronization.

Is note handling a standard part of the protocol or an extension?

kello · 3 years ago

Wow! this is so cool! I had no idea about this!

samwillis · 3 years ago

Super interesting, particularly the link through to the blog with all the research into how Notes works.

A particularly interesting thing that I don't see mentioned anywhere is that Apple Notes uses CRDTs (conflict free replicated data types) to enable both shared notes and multiple concurrent edits. With the CRDTs resolving the conflicts.

It looks like the info on the page may be a good place to start looking to how that works: https://www.ciofecaforensics.com/2020/10/20/apple-notes-clou...

eddyg · 3 years ago

CRDTs are definitely an “unsung hero” of Apple Notes… and a big reason why it is one Apple’s best “built-in” apps as of late.

starbugs · 3 years ago

If they could now implement table resizing and tables that do not render with graphical errors, that would be so nice.

howinteresting · 3 years ago

A notes app that only works with one 2-trillion-dollar company's hardware and does not have builtin functionality to export notes (e.g. a menu option to do so) cannot reasonably be described as good. One's expectations have to be through the floor for that to be an honest appraisal.

Compare it to Obsidian, which just stores data in a bunch of markdown files on disk.

user_7832 · 3 years ago

Thank you for posting about CRDTs! I remembered Apple Notes used some type of versioning that was good and I’d wanted to use in my own app design but I had forgotten the name.

ChintanGhate · 3 years ago

I have published an app [Exporter](https://apps.apple.com/us/app/exporter/id1099120373?mt=12) that does something similar - exports Apple Notes to HTML and markdown with the attachments. Give it a try, let me know if it solves your issue.

Also, why did you select JSON as the output format?

daydream · 3 years ago

Thanks for making your app. I’ve used it more than once and it works well. Many export solutions don’t handle attachments but yours does. So kudos for that.

A few years ago I tried to write my own exporter using AppleScript but got stuck by a showstopping bug in Notes AppleScript implementation. Or so it seemed. I hope that bug is fixed now. I just moved on to use your app instead.

I am glad it's being useful. :)

I used AppleScript for the first two years of this app as well (Can't believe it's been 7 years since I started it). But it's hard to handle complex programs, and handing off data to Swift or Objective-C from AppleScript. So I switched to [ScriptingBridge](https://developer.apple.com/documentation/scriptingbridge). It provides better Objective-C APIs to handle Apple Events.

This is so cool and way cooler than my tool!! I chose JSON because it was the easiest option. And also because it is the format I wanted to support first, but mostly because it was the easiest.

Deleted Comment

mark_l_watson · 3 years ago

Worked well for me, thanks.

dangoor · 3 years ago

Cool! Thanks for sharing. Also worth noting that there's a Mac app in the app store to export to markdown or html:

https://apps.apple.com/us/app/exporter/id1099120373?mt=12

(I haven't had a need yet, so I haven't tried it)

voltaireodactyl · 3 years ago

I have used it several times and it’s fantastic. Exports attachments into properly named media folders and everything.

css · 3 years ago

If anyone ever wanted to do something similar for iMessage data, I built a similar tool for that: https://github.com/ReagentX/imessage-exporter

n8henrie · 3 years ago

Oh man, excited to see this. Was brainstorming doing something this in rust last week. Beat me to it!

mkmk · 3 years ago

I consistently struggle with Notes and iMessage sync issues, and I suspect it’s due to being a customer for so many years and making my way through many devices over those years.

The UX of the Apple ecosystem is so nice but when it goes sideways you’re kind of out of luck, even if you have applecare.

I wish they had some sort of premium “fix my esoteric appleID account issues” service you could pay them for once a decade or so.

bwbmr · 3 years ago

I agree- long standing issues that have popped up and are never fixed over the years:

1) My iPhone has never properly synced screen time with my other devices (presently my other devices: 2x iPads, MBP) sync fine together. This has persisted across multiple iPhones and major releases of iOS.

2) Apple News has a permanently saved “story” of a specific magazine issue’s table of contents. No idea how it got saved, but since it isn’t actually a story (it’s a ToC instead) the UI option to “unsave” it is greyed out / unavailable. I reported it to a CSR about four years ago, he took down bug reports, I even sent a screen capture demonstrating the bug… it still persists and Apple provides no mechanism to manually clear all saved story data. I can remove it locally from the device from disabling iCloud sync for Apple News -> selecting yes when prompted to remove local data, but it will still persist in iCloud and there is no way to wipe it there… it’s back as soon as I reenable sync.

amunster · 3 years ago

I had a sync related issue with Messages.app on macOS recently. Escalated through apple support. The solution was to reinstall macOS from scratch. Not pretty but a small price to pay.

Now my lingering issue is with search: Calendar events and Reminders prior to the reinstall date do not appear in search results. This has been an issue for years on both macOS and iOS! Much Google searching was no help.

I wish that premium fix service included useful forums and…search.