Readit News logoReadit News
ran3000 commented on Content-Aware Spaced Repetition   giacomoran.com/blog/conte... · Posted by u/ran3000
gwd · 22 days ago
It started out as a side project just for myself to study Mandarin in 2019.

There's an open beta of the system ported to Biblical Greek here:

https://www.laleolanguage.com

I've got several active users without really having done any advertising; working on revamping the UI and redesigning the website before I do a big push and start advertising. Most of the people using the site have learned Biblical Greek entirely through the system.

There are experimental ports to Korean and Japanese as well, but those (along with the Mandarin port) aren't public yet. The primary missing pieces are:

1. Content -- the system relies on having large amounts of high-quality content. Finding it, tagging it, and dealing with copyright will take some time

2. On-ramp: It works best to help people at the intermediate level to advance. But if you start at an intermediate level, it doesn't know what you know.

Another thread I'm pursuing is exposing the algorithm via API to other language learning apps:

https://api-dev.laleolanguage.com/v1/docs

All of that needs a better funnel. I'll probably post some stuff here once I've got everything in a better state.

(If anyone reading this is interested in the API, please contact me at contact@laleolanguage.com .)

ran3000 · 21 days ago
Just watched the video, great work!
ran3000 commented on Content-Aware Spaced Repetition   giacomoran.com/blog/conte... · Posted by u/ran3000
gwd · 22 days ago
Thanks for the write-up!

I've got a system for learning languages that does some of the things you mention. The goal is to be able to recommend content for a user to read which combines 1) appropriate level of difficulty 2) usefulness for learning. The idea is to have the SRS system build into the system, so you just sit and read what it gives you, and review of old words and learning new words (according to frequency) happens automatically.

Separating the recall model from the teaching model as you say opens up loads of possibilities.

Brief introduction:

1. Identify "language building blocks" for a language; this includes not just pure vocabulary, but the grammar concepts, inflected forms of words, and can even include graphemes and what-not.

2. For each building block, assign a value -- normally this is the frequency of the building block within the corpus.

3. Get a corpus of selections to study. Tag them with the language building blocks. This is similar to Math Academy's approach, but while they have hundreds of math concepts, I have tens of thousands of building blocks.

3. Use a model to estimate the current difficulty of each word. (I'm using "difficulty" here as the inverse of "retrievability", for reasons that will be clear later.)

4. Estimate the delta of difficulty of each building block after being viewed. Multiply this delta by the word value to get the study value of that word.

5. For each selection, calculate the total difficulty, average difficulty, and total study value. (This is why I use "difficulty" rather than "retrievability", so that I can calculate total cognitive load of a selection.)

Now the teaching algorithm has a lot of things it can do. It can calculate a selection score which balances study value, difficulty, as well as repetitiveness. It can take the word with the highest study value, and then look for words with that word in it. It can take a specific selection that you want to read or listen to, find the most important word in that selection, and then look for things to study which reinforce that word.

You mentioned computational complexity -- calculating all this from scratch certainly takes a lot, but the key thing is that each time you study something, only a handful of things change. This makes it possible to update things very efficiently using an incremental computation [1].

But that does make the code quite complicated.

[1] https://en.wikipedia.org/wiki/Incremental_computing

ran3000 · 22 days ago
Interesting, I've been surprised to see how many language learning apps already include some of the ideas I've discussed in the blog post!

How far along are you in developing the system?

ran3000 commented on Content-Aware Spaced Repetition   giacomoran.com/blog/conte... · Posted by u/ran3000
HDMI_Cable · 22 days ago
What sort of privacy implications? I'd imagine that Anki data would be relatively privacy-concern free, as it contains no PII, and for the AnKing decks, all of the content is standardized and so wouldn't contain personal notes. Though, having never worked with this data, please let me know if I'm wrong!

Also, having used those decks in the past, and downloaded the add-on/look at the monetization structure of developers like the AnKing, I would be very surprised if aggregate data on review statistics wasn't collected in some way. I.e., if the AnKing is collecting this data already to design better decks/understand which cards are the hardest—probably to target individual support—then I imagine that collecting some de-anonymized version of that data wouldn't be too much of a stretch.

Plus, considering that all of the developers of AnKing-style decks are all doctors, they probably have a pretty good grasp at handling PII and could (hopefully) make pretty sound decisions on whether to give you access :)

ran3000 · 22 days ago
You're right, it might work by restricting to just AnKing data. My concern was around other, possibly personal, cards making their way into the dataset.
ran3000 commented on Content-Aware Spaced Repetition   giacomoran.com/blog/conte... · Posted by u/ran3000
jeffalyanak · 22 days ago
In the language learning world there are some great tools already for adding content-awareness.

AnkiMorphs[1] will analyze the morphemes in your sentences and, taking into account the interval of each card as a sign of how well you know each one, will re-order your new cards to, ideally, present you with cards that have only one unknown word.

It doesn't do anything to affect the FSRS directly—it only changes the order of new, unlearned cards—but in my experience it's so effective at shrinking the time from new card to stable/mature that I'm not sure how much more it would help to have the FSRS intervals being adjusted in this particular domain.

1: https://mortii.github.io/anki-morphs/intro.html

ran3000 · 22 days ago
Reading the thread, I definitely overlooked language learning solutions. Thanks for sharing!
ran3000 commented on Content-Aware Spaced Repetition   giacomoran.com/blog/conte... · Posted by u/ran3000
dllthomas · 22 days ago
I ran into a question a while ago that I couldn't find a good answer to, and while it's not exactly on topic this seems like a good place to ask it.

I was working in a detail rich context, where there were a lot of items, about which there were a lot of facts that mostly didn't change but only mostly. Getting a snapshot of these details into approximately everyone's head seemed like a job for spaced repetition, and I considered making a shared Anki deck for the company.

What wasn't clear was how to handle those updates. Just changing the deck in place feels wrong, for those who have been using it - they're remembering right, the cards have changed.

Deprecating cards that are no longer accurate but which don't have replacement information was a related question. It might be worth informing people who have been studying that card that it's wrong now, but there's no reason to surface the deprecation to a person who has never seen the card.

Is there an obvious way to use standard SRS features for this? A less obvious way? A system that provides less standard features? Is this an opportunity for a useful feature for a new or existing system? Or is this actually not an issue for some reason I've missed?

ran3000 · 22 days ago
This specific problem gave us lots of headache while building https://rember.com We don't have a good solution yet. My hope is that something like content-aware memory models solve the problem at a lower level, so we don't have to worry about it at the product level.
ran3000 commented on Content-Aware Spaced Repetition   giacomoran.com/blog/conte... · Posted by u/ran3000
HDMI_Cable · 22 days ago
| The main challenge in building content-aware memory models is lack of data. To my knowledge, no publicly available dataset exists that contains real-world usage data with both card textual content and review histories.

I wonder if the author has ever considered reaching out to makers of Anki decks used by premeds and medical students like the AnKing [1]. They create Anki decks for users studying the MCAT and various Med School curricula, so have a) relatively stable deck content (which is very well annotated and contains lots of key words that would make semantic grouping quite easy) b) probably contains loads of statistics on user reviews (since they have an Anki addon that sends telemetry to their team to make the decks better IIRC), and c) contains incredibly disparate information (all the way from high-school physics to neurochemistry).

---

[1]: https://www.theanking.com

ran3000 · 22 days ago
It would be awesome to work on that data. I'm afraid of the privacy implications though.
ran3000 commented on Content-Aware Spaced Repetition   giacomoran.com/blog/conte... · Posted by u/ran3000
vhrrsrivr · 22 days ago
Rather than relying on an embedding space, my approach is to have the cards themselves be grammars that can define the relationships between concepts explicitly. Then the problem becomes what specific sampling of all the possible outputs is optimal for a learner to see at any given time, given their knowledge state.

See how it's applied to Japanese learning here: https://elldev.com/feed/grsly

ran3000 · 22 days ago
Amazing work! In https://rember.com the main unit is a note representing a concept or idea, plus some flashcards associated to it, hsrs would fit perfectly! I'll look more deeply into it.
ran3000 commented on Content-Aware Spaced Repetition   giacomoran.com/blog/conte... · Posted by u/ran3000
pessimizer · 22 days ago
> The main limitation is that the same flashcards must be studied by multiple students, which doesn't generally apply.

I think only a kernel of the same flashcards, because in my mind new cards would quickly find their position after being reviewed a few times, and might displace already well-known cards. I see the process as throwing random cards at students, seeing what's left after shaking the tree, and using that info to teach new students.

The goal, however, would definitely be a single standard but evolving set of cards that described some group of related ideas. I know that's against Supermemo/Anki gospel, but I've gotten an enormous amount of value out of engineered decks such as https://www.asiteaboutnothing.net/w_ultimate_spanish_conjuga....

> I also love the idea of the market, you could even extend it to evaluate/write high-quality flashcards.

It's been my idea to drive conversational spaced repetition with something like this.

ran3000 · 22 days ago
I would be valuable for shared decks, like the one you mentioned. As far as I can tell, the majority of Anki users are medical school students or language learners. Both groups benefit from shared decks. So I think it's a good idea to pursue.

My personal interest is more on conceptual knowledge, like math, cs, history or random blog posts and ideas. It's often the case that, on the same article, different people focus different things, so it would be hard to collect even a small number of reviews on a flashcard you want to study.

ran3000 commented on Content-Aware Spaced Repetition   giacomoran.com/blog/conte... · Posted by u/ran3000
barrell · 22 days ago
I’ve been working on https://phrasing.app for a while now, including many iterations of the SRS. It’s been my experience that most of these sorts of improvements are really imperceptible. While I use FSRS as a base, and I’m very happy with the results it provides, it’s really only a few percentage points off of the SM-2 algorithm from the 90s. It’s slightly less stressful, definitely more accurate, but I think only astute users would even notice the difference.

I’ve incorporated many different things into the SRS, from vector embeddings to graph based association to lemma clustering to morpheme linking, and was surprised how much of these I took out.

Most of the unlocks with the SRS have been more in application space. Doing reviews with Anki feels like a chore, and I’m always counting down the reviews left to do. Reviews with Phrasing however are much more addictive, and I routinely spent an extra 30+ minutes in that “ok just one more card” loop.

We will never be able to know with 100% certainty how well you know a card, but FSRS gets us darn close. I think the interesting stuff is less about improving that metric, and more about what can you do with that information.

Thanks to the whole FSRS team btw (I assume y’all will be reading this hn post) <3

And if anyone is curious I wrote up a bit about my SRS here: https://phrasing.app/blog/humane-srs

ran3000 · 22 days ago
Phrasing looks amazing!

There's a lot of UX work to do for SRS. Do you have a sense of how well the ideas behind Humane SRS translate outside of language learning? I imagine the main challenge would be identifying a steady influx of new cards.

I agree that gains in scheduling accuracy are fairly imperceptible for most students. That's why, over the past few years building https://rember.com, we've focused on UX rather than memory models. People who review hundreds of card a day definitely feel the difference, doing 50 fewer reviews per day is liberating. And now that LLMs can generate decent-quality flashcards, people will build larger and larger collections, so scheduler improvements might suddenly become much more important.

Ultimately, though, the biggest advantages is freeing the SRS designer. I'm sure you've grappled with questions like "is the right unit the card, the note, the deck or something else entirely?" or "what happens to the review history if the student edits a card?". You have to consider how review UX, creation/editing flows, and card organization interact. Decoupling the scheduler from these concerns would help a ton.

ran3000 commented on Content-Aware Spaced Repetition   giacomoran.com/blog/conte... · Posted by u/ran3000
joshdavham · 22 days ago
I've been thinking about this for a while too as an FSRS developer [1].

In general, we can think of a spaced repetition system as being (i) Content-aware vs. Content-agnostic and (ii) Deck-aware vs. Deck-agnostic

Content-aware systems care about what you're studying (language, medecine, etc) while Content-agnostic systems don't care about what you're studying.

Deck-aware systems consider each card in the context of the rest of the cards (the "deck") while Deck-agnostic systems consider each card in pure isolation.

Currently, FSRS is both Content-agnostic as well as Deck-agnostic. This makes it extremely easy to integrate into a spaced repetition system, but this also means the model will underfit a bit.

It it interesting to note that you could in practice optimize seperate FSRS models for each deck covering different topics, which would make it Content-aware in a sense. Additionally, "fuzz" is a somewhat Deck-aware feature of the model in that it exists specifically to reduce interactions between other cards in the deck.

[1] https://github.com/open-spaced-repetition/py-fsrs

ran3000 · 22 days ago
Being easy to integrate is an underappreciated feature of FSRS.

Using decks to draw semantic boundaries is likely overly constraining. I think we want to account for finer differences between cards. Decks are coarse and people differ in the ways they use them, some people recommend having just one global deck. Notes are too fine. We explored something in between: a note capturing an idea or concept, plus an associated set of cards. Turns out it's hard to draw idea boundaries. That's why I think it's easier to relate cards by semantic embeddings or more rigid but clearer structures, like the DAG of dependencies suggested elsewhere in this thread.

u/ran3000

KarmaCake day69August 26, 2017View Original