Human languages are massively redundant specifically to enable this kind of "fill the gaps" learning process.
Observe an unknown word used a few times in different contexts, and you'll get a very good bead on what it means.
It's the "natural" way of learning a language. No one gets every word explained to them in detail - most words you know in your own first language are the ones you picked up on the go.
I think this "structural redundancy" might be a big part of what enables LLMs to learn human languages "from nothing".
I think it is more that the common words in the language are the scaffolding (like hedging with "I think...", "like" for setting up an example) and the subject of the scaffold is normally unique and highly context specific.
Looking at the first Frieren example:
"Live without [fear]", "Live without [bread]", "Live without [sunlight]", "Live without [honour]". The common words are more than enough to set up a generic sentence for lack of something but then what someone lacks can be literally any object, concept or verb. There are a lot more than 500 of those in the world.
One nuance not mentioned in the blog is that this particular "without" is expressed with the archaic negative verb conjugation -zu (ず). It lives on in set phrases and literary use, but it's not usable without a verb to negate (you can't say "pan-zu ikiru" for "live without bread") and it's going to sound pretty weird if you try to use it for quotidian things like "live without eating bread" (pan tabezu ikiru).
And in the case of Japanese, with words consisting of multiple Kanji you can even get more context of a word from the word without knowing the word. Down to, to a lesser extent, radicals too...
Maybe words are the wrong thing to delineate by, I mean even if you haven’t heard the word ‘circumnavigate’ you can probably deduce its meaning especially with context.
People who can figure out words from radicals probably have >500 word vocabularies to begin with. I’d say if you started learning on some natural progression, you would start learning how to figure things out from radicals once you knew somewhere around 300 kanji (which would mean you have a vocabulary of 1,500 words, ish).
Before that point, you are probably going to have too much trouble generalizing.
Isn't that the exact opposite? That sounds like reasons why "understood" may include false positives ("you understood all words but not a necessary grammar construction or couldn't follow at the given speaking speed"). It mentions "context" but the context in which context is mentioned suggests it's meant as yet another hurdle, not as something that could help.
Grammar is very important. You can understand all the words and not know if the super robots power the spaceship or the spaceship powers the super robots. And I remember some translators getting that particular point wrong because the anime's lore made the super robots work in strange ways.
Granted DBZ is not exactly Shakespeare, but I first watched the Buu saga in japanese w/o subs. And I was generally able to follow along, especially as it went along.
You can figure it out from context if don't know one or two words in a paragraph. If you are missing a word from most lines of dialogue then you are probably going to struggle.
I'm sure that's intentional, so the article wouldn't be stupid. You can often figure out the meaning if you don't know the language at all, but who cares?
Other than language learners. If you want to learn someone's language, read their comic books. They have pictures.
I once ran an inadvertent experiment by installing Austrumi Linux, but not realizing that the default language is Latgallian. I'm fluent in Latvian, but I wondered if something had gone wrong with my brain. I could understand most of the words I was looking at, but more than half of the sentences I read were baffling.
Err… it's not particularly good methodology to just consider words for Japanese. While it's true that a lot of kanji share pronunciations (and you thus can't identify them from hearing them on their own) you can still get at some compounds from context and "knowing 2 halves". (Kanji are quite frequently used in groups of 2 to form words, and a lot have only one reading when used in this manner. Personally speaking, I really need some context to get anywhere though.)
I’m not sure exactly where 85% coverage falls. But I can say that at 75% you will have essentially zero understanding of what a text says. (Context clues don’t help you here) And at 95% coverage you will be able to understand basically the entire text.
I don’t remember the source but here is the first page of Harry Potter with only 75% coverage:
> 75% coverage
_______ ONE
THE BOY WHO _____
Mr . and Mrs . Dursley , of ______ four , ______ _____ , were _____ to say that they were _________ ______ , _____ you very much . They were the last people you ' d ______ to be ________ in anything _______ or __________ , because they just didn ' t ____ with ____ ________ .
Mr . Dursley was the ________ of a ____ called _________ , which made ______ . He was a big , _____ man with ______ any ____ , ________ he did have a very large ________ . Mrs . Dursley was ____ and ______ and had nearly _____ the _____ ______ of ____ , which came in very ______ as she _____ so much of her time _______ over ______ ______ , ______ on the _________ . The Dursleys had a small ___ called Dudley and in their _______ there was no _____ boy ________ .
Those aren't random words. That's like most of the nouns and meaningful verbs. You can pluck out most of the nouns and verbs of anything and the context gets lost.
Sounds about right. Similar experience learning German when I moved to Berlin from the UK (though less extreme, felt like 85% of words was 50% of sentences).
I didn’t include it in my analysis, but I’ve been thinking about “sentence coverage” vs “word coverage” for a number of years now. In an extreme case, you could imagine a transcript with only i+1/1T sentences and you’d be left with totally incomprehensible input.
> By the third pause to look up a word, your tea has gone cold.
And here I thought that water had one of the highest specific heat capacities.
> Both numbers are true
What does "both numbers" refer to? Presumably the two percentages, (based on the title, but there is no framing right away that these are the key numbers), and proximate to the line are several unrounded numbers.
> is a flashcard app that teaches Japanese from anime. Each card uses a line where you know everything except the one new word, which is why we started counting lines. The figures above count each word once, however many meanings it has. That wouldn't work in the app: knowing あれ as "that" doesn't help when someone says あれ？ (huh?). These are two separate cards. It doesn't catch every such word yet, we're working on it. During early access it offers one show, Hunter x Hunter.
Highly advise cutting down your marketing pitch for your product, it's just too long to grab people.
It's the 500 most common words in that specific show rather than some broader corpus, so there's bound to be some words that happen to be unusually common or rare.
This reminds me of how I learned about Zipf's law. 19 years ago, I spent the summer after my sophomore year in a research lab at UC Santa Cruz as part of an NSF-funded summer research program for undergrads. My project that summer was to investigate index design for the Ceph distributed file system in order to enable faster file search. This was a hot topic at the time, especially with Spotlight being introduced to Mac OS X Tiger, and we wanted to see how to enable similar functionality for a distributed file system. I learned about Zipf's law, which states that there is a power law distribution of word frequencies in many natural languages. I remember downloading the Project Gutenberg corpus and writing a Python script where I stemmed the words and computed the frequencies of the stemmed words. The resulting distribution confirmed Zipf's law.
Ironic that it holds the first place by understandable lines. The vocabulary isn't the complex thing - it's the content itself that eludes understanding.
Surreal and dreamlike - almost as if inviting you to connect the dots, but not giving you the labels, not giving you enough dots or an obvious enough outline to connect them confidently.
I think the rarest words in your sentence are “information” and “sentence” yet you can drop both of those and figure out your meaning, especially if the sentence is part of a broader conversation.
“Rare words carry the most” basically conveys what you’re saying but just more vague. And I think that’s a really important distinction. Rare words reduce ambiguity and increase the specificity of your sentence, but they don’t necessarily carry the most meaning.
Disagree because it's one bit of information and in practice you can often guess the correct direction even if you're not sure if "not" was or wasn't said.
I've only watched the original series, once, and understood most of it.
It's just meaning is very very vague on purpose. It's not meant to be specific. Most of Japanese art is like that.
Also, if you start learning Japanese language, you'll quickly notice that the "I'm not going to tell you anything directly" and "read the room" permeates even casual and especially formal daily conversations.
One thing that seem to be improving Japanese literacy across the world is Inmu gay porn meme contents. I believe it hasn't reached the US, but it is becoming eerily well known in East/Southeast Asian countries, including mainland China. It uses tiny set of vocabulary in highly repetitive manner, a la kid's picture books, but in the way relatable and addicting to social media users rather than e.g. portraying a day in the life of a cute crayon drawn caterpillar.
IMO, what that tells us is that a completely dumb repetition of small set of simple concepts to just set up the pathway can be an effective means to kickstart language acquisition process. Most language learning materials attempt to establish a compact set of rules to be expanded with a long tail of modular add-ons, even many that aims to create an alternative and less stressful approaches like Duolingo. The Inmu dictionary has less than 500 known expression, all unusable in regular communications and inappropriate in most settings, and the way it is used is simplistic, childish, disrespectful, and dumb. And I think that might be how "start with picture books" advice look like if rebuilt from first principles.
Can confirm that understanding Jujutsu Kaisen in a language you're still learning is brutal.
I can watch pretty much anything in that language at this point but having one character explain complex arbitrary rules with little visual aid is just too difficult yet.
Full metal alchemist, the original not Brotherhood, was the pivot point in my opinion where dubs were superior to subs.
The main characters were all well voice acted while I admit background characters were poorly voice acted.
However I say it’s the pivot point because the in top of the main characters having good voice actors the translation included cultural context I would have never picked up on my own.
For instance it took me nearly a decade later of occasionally watching subs to find out that “ore” was rude when male characters used it to refer to themselves because I didn’t have the cultural context to understand.
If I’m willing to become fully fluent in a language and its culture dubs are superior, but I just don’t have the time for most foreign media.
Do you mean that you pivoted to only dubs? Full Metal Alchemist is generally considered one of the best dubs so I don't think it's representative. While I have an affinity for Adult Swim era English dubs, I'm 100% subs now. Even things like Dragon Ball Z's sub, which can seem very weird at first if you're used to the dub, are ultimately better than the dub imho.
I pivoted to only dubs past Full Metal Alchemist because they finally were of high quality and provided enough context that they were superior to subs for me, given that I wasn't willing to learn Japanese and its cultural connotations
Just know that many of the best anime do not have dubs or have very low quality dubs. Subs will also account for some cultural translation most of the time.
In fansub times we used to supply that cultural context as onscreen notes when it was important enough. (In most cases you'd have needed to pause to read them.) Seems to have fallen out of fashion though.
This ignores the fact that you can often figure out the meaning of the line from context even if you don't know the meaning of each word.
Human languages are massively redundant specifically to enable this kind of "fill the gaps" learning process.
Observe an unknown word used a few times in different contexts, and you'll get a very good bead on what it means.
It's the "natural" way of learning a language. No one gets every word explained to them in detail - most words you know in your own first language are the ones you picked up on the go.
I think this "structural redundancy" might be a big part of what enables LLMs to learn human languages "from nothing".
I think it is more that the common words in the language are the scaffolding (like hedging with "I think...", "like" for setting up an example) and the subject of the scaffold is normally unique and highly context specific.
Looking at the first Frieren example:
"Live without [fear]", "Live without [bread]", "Live without [sunlight]", "Live without [honour]". The common words are more than enough to set up a generic sentence for lack of something but then what someone lacks can be literally any object, concept or verb. There are a lot more than 500 of those in the world.
One nuance not mentioned in the blog is that this particular "without" is expressed with the archaic negative verb conjugation -zu (ず). It lives on in set phrases and literary use, but it's not usable without a verb to negate (you can't say "pan-zu ikiru" for "live without bread") and it's going to sound pretty weird if you try to use it for quotidian things like "live without eating bread" (pan tabezu ikiru).
And in the case of Japanese, with words consisting of multiple Kanji you can even get more context of a word from the word without knowing the word. Down to, to a lesser extent, radicals too...
Maybe words are the wrong thing to delineate by, I mean even if you haven’t heard the word ‘circumnavigate’ you can probably deduce its meaning especially with context.
People who can figure out words from radicals probably have >500 word vocabularies to begin with. I’d say if you started learning on some natural progression, you would start learning how to figure things out from radicals once you knew somewhere around 300 kanji (which would mean you have a vocabulary of 1,500 words, ish).
Before that point, you are probably going to have too much trouble generalizing.
It's pointed out as a shortcoming at the bottom:
>What could make this wrong: >1. "Understood" means you know every word. It ignores grammar, kanji readings, speaking speed and context.
Isn't that the exact opposite? That sounds like reasons why "understood" may include false positives ("you understood all words but not a necessary grammar construction or couldn't follow at the given speaking speed"). It mentions "context" but the context in which context is mentioned suggests it's meant as yet another hurdle, not as something that could help.
Grammar is very important. You can understand all the words and not know if the super robots power the spaceship or the spaceship powers the super robots. And I remember some translators getting that particular point wrong because the anime's lore made the super robots work in strange ways.
Granted DBZ is not exactly Shakespeare, but I first watched the Buu saga in japanese w/o subs. And I was generally able to follow along, especially as it went along.
You can figure it out from context if don't know one or two words in a paragraph. If you are missing a word from most lines of dialogue then you are probably going to struggle.
or alternatively, leverages that to structure your intuitive learning process.
I'm sure that's intentional, so the article wouldn't be stupid. You can often figure out the meaning if you don't know the language at all, but who cares?
Other than language learners. If you want to learn someone's language, read their comic books. They have pictures.
I once ran an inadvertent experiment by installing Austrumi Linux, but not realizing that the default language is Latgallian. I'm fluent in Latvian, but I wondered if something had gone wrong with my brain. I could understand most of the words I was looking at, but more than half of the sentences I read were baffling.
Err… it's not particularly good methodology to just consider words for Japanese. While it's true that a lot of kanji share pronunciations (and you thus can't identify them from hearing them on their own) you can still get at some compounds from context and "knowing 2 halves". (Kanji are quite frequently used in groups of 2 to form words, and a lot have only one reading when used in this manner. Personally speaking, I really need some context to get anywhere though.)
I’m not sure exactly where 85% coverage falls. But I can say that at 75% you will have essentially zero understanding of what a text says. (Context clues don’t help you here) And at 95% coverage you will be able to understand basically the entire text.
I don’t remember the source but here is the first page of Harry Potter with only 75% coverage:
> 75% coverage
_______ ONE
THE BOY WHO _____
Mr . and Mrs . Dursley , of ______ four , ______ _____ , were _____ to say that they were _________ ______ , _____ you very much . They were the last people you ' d ______ to be ________ in anything _______ or __________ , because they just didn ' t ____ with ____ ________ .
Mr . Dursley was the ________ of a ____ called _________ , which made ______ . He was a big , _____ man with ______ any ____ , ________ he did have a very large ________ . Mrs . Dursley was ____ and ______ and had nearly _____ the _____ ______ of ____ , which came in very ______ as she _____ so much of her time _______ over ______ ______ , ______ on the _________ . The Dursleys had a small ___ called Dudley and in their _______ there was no _____ boy ________ .
Here is a similar source https://thepatrologist.com/what-98-comprehensibility-looks-l...
Those aren't random words. That's like most of the nouns and meaningful verbs. You can pluck out most of the nouns and verbs of anything and the context gets lost.
Sounds about right. Similar experience learning German when I moved to Berlin from the UK (though less extreme, felt like 85% of words was 50% of sentences).
Reminds me of an analysis I did a while back: https://news.ycombinator.com/item?id=44349332
I didn’t include it in my analysis, but I’ve been thinking about “sentence coverage” vs “word coverage” for a number of years now. In an extreme case, you could imagine a transcript with only i+1/1T sentences and you’d be left with totally incomprehensible input.
"How we did it" (Claude and him).
Some bad writing examples I liked.
> By the third pause to look up a word, your tea has gone cold.
And here I thought that water had one of the highest specific heat capacities.
> Both numbers are true
What does "both numbers" refer to? Presumably the two percentages, (based on the title, but there is no framing right away that these are the key numbers), and proximate to the line are several unrounded numbers.
> is a flashcard app that teaches Japanese from anime. Each card uses a line where you know everything except the one new word, which is why we started counting lines. The figures above count each word once, however many meanings it has. That wouldn't work in the app: knowing あれ as "that" doesn't help when someone says あれ？ (huh?). These are two separate cards. It doesn't catch every such word yet, we're working on it. During early access it offers one show, Hunter x Hunter.
Highly advise cutting down your marketing pitch for your product, it's just too long to grab people.
> kochira wa yakusoku no houshuu desu
You blurred yakusoku, everybody knows that, but houshuu is much rarer word, should have blurred that
It's the 500 most common words in that specific show rather than some broader corpus, so there's bound to be some words that happen to be unusually common or rare.
I wonder if the early success of Dragon Ball (Z) was due to how straight forward the plot is.
You can have a good time watching Dragon Ball Z without understanding the dialogue.
Although much has to be said for the fantastic dubs in various languages. One of the only anime where the dub is arguably better.
With the 2136 common use Kanji you get >98% on all shows
This reminds me of how I learned about Zipf's law. 19 years ago, I spent the summer after my sophomore year in a research lab at UC Santa Cruz as part of an NSF-funded summer research program for undergrads. My project that summer was to investigate index design for the Ceph distributed file system in order to enable faster file search. This was a hot topic at the time, especially with Spotlight being introduced to Mac OS X Tiger, and we wanted to see how to enable similar functionality for a distributed file system. I learned about Zipf's law, which states that there is a power law distribution of word frequencies in many natural languages. I remember downloading the Project Gutenberg corpus and writing a Python script where I stemmed the words and computed the frequencies of the stemmed words. The resulting distribution confirmed Zipf's law.
I watched Lain with the subtitles on and I still won't claim I understood all the lines.
I'm pretty sure many of the lines are _not understandable by intent_, even.
"Present day... present time... ahahaha!"
"And you don't seem to understand..."
Ironic that it holds the first place by understandable lines. The vocabulary isn't the complex thing - it's the content itself that eludes understanding.
Surreal and dreamlike - almost as if inviting you to connect the dots, but not giving you the labels, not giving you enough dots or an obvious enough outline to connect them confidently.
The latter Evangelion episodes seemed similar (but not quite as crazy as Lain). Apparently they're making a follow-up series too!
It's funny that I remember basically nothing of the series anymore... except that it still didn't quite make sense to me at the time.
Serial Expirementd Lain is basically the Ulysses of anime. Super dense and cryptic by design.
Rare words usually carry the most information in any given sentence.
I was really hoping you'd go all in and use rare words in that sentence just for the joke.
agreed, he should have used "entropy" instead of information
[dead]
I think the rarest words in your sentence are “information” and “sentence” yet you can drop both of those and figure out your meaning, especially if the sentence is part of a broader conversation.
“Rare words carry the most” basically conveys what you’re saying but just more vague. And I think that’s a really important distinction. Rare words reduce ambiguity and increase the specificity of your sentence, but they don’t necessarily carry the most meaning.
I’d disagree with that. For example, in many sentences, the most load-bearing (heh) word is “not”, which is also one of the most common words.
Disagree because it's one bit of information and in practice you can often guess the correct direction even if you're not sure if "not" was or wasn't said.
My first anime was NGE. I knew all the words and still understand nothing.
I've only watched the original series, once, and understood most of it.
It's just meaning is very very vague on purpose. It's not meant to be specific. Most of Japanese art is like that.
Also, if you start learning Japanese language, you'll quickly notice that the "I'm not going to tell you anything directly" and "read the room" permeates even casual and especially formal daily conversations.
That was me with Revolutionary Girl Utena.
Really? Eva is pretty straightforward to understand once you watched End of Evangelion, Misato pretty much explains most of it in the movie.
One thing that seem to be improving Japanese literacy across the world is Inmu gay porn meme contents. I believe it hasn't reached the US, but it is becoming eerily well known in East/Southeast Asian countries, including mainland China. It uses tiny set of vocabulary in highly repetitive manner, a la kid's picture books, but in the way relatable and addicting to social media users rather than e.g. portraying a day in the life of a cute crayon drawn caterpillar.
IMO, what that tells us is that a completely dumb repetition of small set of simple concepts to just set up the pathway can be an effective means to kickstart language acquisition process. Most language learning materials attempt to establish a compact set of rules to be expanded with a long tail of modular add-ons, even many that aims to create an alternative and less stressful approaches like Duolingo. The Inmu dictionary has less than 500 known expression, all unusable in regular communications and inappropriate in most settings, and the way it is used is simplistic, childish, disrespectful, and dumb. And I think that might be how "start with picture books" advice look like if rebuilt from first principles.
I spam Yare Yare to the extent that my friends had it on my birthday cake
Ara ara!
That's higher than what I would've expected.
Nani!?
Can confirm that understanding Jujutsu Kaisen in a language you're still learning is brutal.
I can watch pretty much anything in that language at this point but having one character explain complex arbitrary rules with little visual aid is just too difficult yet.
Full metal alchemist, the original not Brotherhood, was the pivot point in my opinion where dubs were superior to subs.
The main characters were all well voice acted while I admit background characters were poorly voice acted.
However I say it’s the pivot point because the in top of the main characters having good voice actors the translation included cultural context I would have never picked up on my own.
For instance it took me nearly a decade later of occasionally watching subs to find out that “ore” was rude when male characters used it to refer to themselves because I didn’t have the cultural context to understand.
If I’m willing to become fully fluent in a language and its culture dubs are superior, but I just don’t have the time for most foreign media.
Do you mean that you pivoted to only dubs? Full Metal Alchemist is generally considered one of the best dubs so I don't think it's representative. While I have an affinity for Adult Swim era English dubs, I'm 100% subs now. Even things like Dragon Ball Z's sub, which can seem very weird at first if you're used to the dub, are ultimately better than the dub imho.
I pivoted to only dubs past Full Metal Alchemist because they finally were of high quality and provided enough context that they were superior to subs for me, given that I wasn't willing to learn Japanese and its cultural connotations
Just know that many of the best anime do not have dubs or have very low quality dubs. Subs will also account for some cultural translation most of the time.
In fansub times we used to supply that cultural context as onscreen notes when it was important enough. (In most cases you'd have needed to pause to read them.) Seems to have fallen out of fashion though.
Maybe it just reached its peak with “just according to keikaku (translator’s note: keikaku means plan)”
The Gintama fan subs were quite good, they explained a lot
> Coverage is the number most learning sites quote.
This person is visiting different learning sites then I do.
Also, I dont need to understand every word in sentence to understand it. Especially when coupled with visual show.
It’s an LLM, so…
It did indeed sounded like one. Noticeable by a lot of words that say nothing that would matter.
[flagged]