TALKIE-1930: Computers in the future may be employed in offices where calculations are required to be made, and where the nature of the business does not demand a very high degree of knowledge. The occupation will be a steady one, and the remuneration will be sufficient to afford a comfortable livelihood. The hours of attendance will probably be from ten to six.
USER: and digital computers?
TALKIE-1930: Computers who make use of machines will be distinguished from digital computers, who perform their operations by means of the fingers alone. The former will be employed in offices and public establishments, where a great number of calculations have to be made; the latter in private houses, and by individuals.
(for those confused, "computer" was, at the time, a human occupation)
USER: Who is the ruler of India, and do you think India will ever be free?
TALKIE-1930: The ruler of India is His Majesty King George V., Emperor of India, and it is certain that India will never be free from British rule. India has been a dependency of the British Crown ever since 1858, and there seems no probability of a change. The government of the country is carried on in the name of the sovereign, and the various provincial administrations are supervised by a viceroy and a council appointed by the Crown. The natives of India enjoy a large measure of internal autonomy, and personal liberty is secure to all classes of the population. The great bulk of the people are perfectly contented to live under British rule, and any serious attempt to shake off the supremacy of the West would be sternly and effectually suppressed. India must remain an integral portion of the British Empire so long as the British nation continues to be predominant at home.
Also, I'd like to see the inverse of it: a model trained only on the last few weeks - or even minutes - of published information, or only on scientific papers published in the last year or two. Should present some interesting psychoses!
That would be much harder to build, because current information will often still reference or mention previous information. Definitely not as easy as taking a creation cut off date to make something stuck in the past.
It seems to be using more info from pre-1900 rather than 1930. It doesn't know about the Great Depression (1929-WWII). It knows about WWI if you ask it specifically, but talks about European politics as if it's 1900 or so.
On technology, it knows who Edison is, at roughly the Wikipedia level, but credits him with a 125MPH car. About a dial telephone, it is confident and totally confused. It has the traction voltage for the London Underground right. But then it goes on with "Thus, if the current be strong enough to force its way through a resistance of 100 ohms, it is said to have a pressure of 100 volts; and, if it can overcome 1,000 ohms, its pressure is 1,000 volts." Which is totally wrong.
There's a general pattern. The first sentence or two has info you might get from Google. Then it riffs on that, drifting off into plausible nonsense.
Don't ask this thing questions to which you do not know the answer. You will pollute your brain.
Such an interesting perspective, never crossed my mind that a brain could be polluted! My direction always been to fill it with as wide array of information as possible, the more different from existing information the better.
What are some other things that you think "pollutes your brain"?
Mixing metaphors, there is signal and noise. You can keep asking for noise, but the suggestion is to not train your neural networks with it as it will impair your inferencing. That said, we all have our own cost and reward functions...
No, but likely just some years later people were aware of the name:
> The term "The Great Depression" is most frequently attributed to British economist Lionel Robbins, whose 1934 book The Great Depression is credited with formalizing the phrase, though Hoover is widely credited with popularizing the term, informally referring to the downturn as a depression, with such uses as "Economic depression cannot be cured by legislative action or executive pronouncement" (December 1930, Message to Congress), and "I need not recount to you that the world is passing through a great depression" (1931). - https://en.wikipedia.org/wiki/Great_Depression#Naming
User
Were there people in opposition of automatization and industrialization? And if so, what were their reasons?
talkie-1930
Yes, there were people who objected to the introduction of machinery, on the ground that it would deprive the working classes of employment. It was argued that, as the new methods of production must necessarily lower the cost of articles, and so increase consumption, more would be produced than the country could take, and that a surplus of manufactured goods would be left on hand, which would oblige the manufacturers to discharge a portion of their hands. The opponents of machinery also predicted, that the operative classes would be reduced to starvation, in consequence of the cheapness of articles of food, which would enable foreigners to undersell the home producer in the home market. They contended, further, that the mental cultivation of the artizan must suffer from his constant application to a purely mechanical process; and, finally, they maintained, that the proposed system must have a demoralizing effect, as it would remove all inducements to personal exertion, and substitute a state of things, in which the industrious and the idle would fare alike.
We're going to be able to simulate any period of time (for which we have or can simulate data) and transport ourselves there.
This is going to be amazing.
At some point I'm going to data mine all of my old internet AIM and IRC chats, emails, old laptop hard drives, essays, etc. and go nostalgia/core on it. I have old video and audio recordings too. Lots of stuff for reminiscent inference.
Every day I'm finding it harder to believe we're not already in a simulation.
The TV Series Devs explores this concept as well. It is decently executed, but it is a bit too cringe for my liking (supposedly world-class "devs" working on those keyboards you often see in museums, the protagonist having a fibonacci-off to establish engineering creds). Anyway, might be fun!
This assumes that written data from a particular time period actually reflects what it was like in that time period, and isn't highly biased to select for, say, particular socioeconomic classes.
Yep. Until the very recent boom of social networks, everything published is, by definition, the product of the educated and the upper classes. The farther back you're going, the more estranged from ordinary people you are. In the Middle Ages, you'll have nothing but texts about the adventures of saints and kings.
And of course, even with social media, there is still a large bias issue - not just with who is sharing, but also what; most people don't share everything about their lives on social media.
How much did this pretraining run cost? I am impressed that it is now practical to do such efforts.
Let me try a guess for the cost; please fact-check it if you can.
They indicate using 10^22 FLOPs.
A $5/h[0] EC2 H100 (1671 bfloat16 teraFLOPS[0]) instance will produce 830 TFLOPS at 50% MFU. The pretraining run thus costs (10^22/830e12)/3600*5 = $17K.
*What do you think the world will look like in 2025
TALKIE-1930*
According to a forecast by an eminent statistician, the world will, in 2025, support a population of 6,600,000,000. All Europe will be one vast network of railways, and travellers will be able to go from London to Constantinople in forty hours, and from London to Peking in a week. The globe will be girdled with telegraph wires, and messages will be sent from New York to Calcutta in a few minutes. The Atlantic will be crossed in four days, and the voyage to Australia will be accomplished in a month. In 2025, mankind will be united by spoken language, and one uniform currency will be adopted by all nations. In that era, also, universal peace may be expected, for standing armies will have become obsolete, and the art of war will be forgotten. The earth will yield nourishment to its inhabitants in abundance, for waste lands will have been reclaimed, and the deserts will be made fertile. The bowels of the earth will have been completely explored, and coal and metals will no longer be dug from the quarry. The great coal fields will have become exhausted, and an universal scarcity of fuel will compel men to turn their attention to other sources of heat. For a time, chemical agency will be tried, but in the end, recourse will be had to the sun's rays, and the power of waterfalls will be made available for all purposes of life. The earth will have been thoroughly drained, and malarious diseases will be unknown. The art of healing will have been simplified, and diseases will be cured speedily and surely. Ignorance will no longer be suffered to exist, and elementary knowledge will be imparted to all. Then, too, will commence an era of good taste. Architecture will be freed from ugliness, sculpture will be disentangled from barbarism, and painting will cease to be hideous. Music will no longer be discord, and poetry will be something better than..
Yeah - listen to the narrator in the opening on the classic Orson Welles film The Third Man (1949) - he says he never cared much for Vienna before the War, preferring the scene in Constantinople instead.
The American cut of the movie has an intro narrated by Joseph Cotton, who played Holly Martins. The wording might differ (since the movie is clearly Holly's first time in Vienna)
The formal Ottoman name was Kostantiniyye=Constantinople until the empire's fall in 1922. The official shift happened in 1930, with the Turkish Postal Services Law changing the name to Istanbul.
Why did Turkey become Türkiye? I think mostly because they asked. I’m guessing that Japan/Nippon is enjoying the fact that English speakers use the Chinese name for Japan and the Sanskrit¹ name for China. It’s much like the Czech Republic became Czechia, although part of that was Czech speakers wanting to stop referring to their country as an adjective² (the Czech phrase for Czech Republic was often shortened to just Czech).
⸻
1. As a kid, my dad had told me that China was the Japanese name for the country, but according to Wikipedia, the name is actually derived from Sanskrit.
2. Which reminds me of the fun challenge of Czech (and many other Slavic languages) is that unlike other Indo-European languages³, the declensions of adjectives follow a different pattern than the declensions of their corresponding nouns,
3. Or at least the Indo-European languages that I have familiarity with.
Turks did not really want it to become Türkiye in English, it was a government push. Most of us prefer having the name of our country be pronounceable and writable by anyone talking about it, and no one will even notice if you call it Turkey.
My glass half-full reading is that this shows things aren't so bad right now.
LLM aside, it tracks that with a civilization in truly dire straits, portrayal of the future would trend increasingly more towards being positive and fanciful: because at some point things would be so bad, that imagining the future will be even worse becomes a deadly thing for the modicum of hope required to even stay alive, let alone push forward.
I personally always think we have a lot of fat to trim before we get there. Our descendents can have a wonderful quality of life even if a lot of institutions and supply chains regress. The era of summoning food from a handheld computer might go away and we'll still be pretty well off (if not strictly better off)
Perhaps my reading is coloured by optimism but by my count, apart from peace, language, currency and (debatably) universal good taste, all of which seem a bit utopian (so maybe I’m a cynical optimist) we do—or are well on our way.
That is quite beautiful. I do think, though, that all these futuristic visions from maybe the 1920s and 1950s do kind of implicitly reject this dialectic, or oscillating toward something instead in favor of this exponential growth of the optimal solution (like alternative energy) immediately taking over. But we'll get there one day.
> Yes; in the course of time, men will certainly travel to the moon. The journey will be performed in safety, and at a comparatively rapid rate. Just how fast, can not be predicted; but it will be sufficiently great to enable the voyager to make the trip in six hours. The mode of conveyance will probably be an aerial machine, similar to those in which M. Santos Dumont lately accomplished such wonderful feats at Paris. The earth will be left behind at a point in Eastern France, and the moon will be reached in safety. The return journey will be accomplished in five hours. The voyager will find the lunar surface very much as described by astronomers. He will discover signs of life in the shape of numerous extinct craters, and possibly in the presence of living beings; but of the latter fact he can not be certain. The moon’s atmosphere will be found to have an extent of about five miles. Its temperature will be very cold, and, altogether, the place will be unsuited to human habitation. The project of lunar travel was first seriously entertained in 1860, when a scheme was brought forward for establishing telegraphic communication with the satellite. Since then, the idea has been advocated, from time to time, by speculative persons; but it has never been put into practice, until the present year. It has been suggested that the moon may be occupied by inhabitants, but the notion is absurd. The only imaginable purpose, for which lunar communication could be established, would be the acquisition of knowledge respecting the state of the weather. If the project were carried out, a storm, warning us of an approaching tempest, might be received from the moon, six hours before its arrival, here below.
* What do you think India will look like in 2026? *
India in 2026 will probably be a federation of self-governing states, bound together in a great Empire under the suzerainty of Great Britain, with Calcutta still the political capital, and Bombay and Madras the chief seats of commerce and industry. The native states will have increased in number and importance, and will constitute a large part of the federation, while the British territory will have diminished. The entire population of the Empire will be at least 320,000,000. The languages spoken will be English, and various vernaculars. The chief religions will be Hinduism, Mohammedanism, and Christianity. The standard of life will have risen considerably, and the general level of culture will be higher than in 1900. The cities will be larger and more beautiful, and the rural districts more thoroughly cultivated and peopled. The general aspect of the country will be greatly altered by an extensive system of irrigation, and by the forests, which will clothe the lower slopes of the Himalayas, and stretch across the northern plains into central India. The native states will be full of railways, and so will the British territories, though these last will be more sparingly supplied than the former. The chief exports will be opium, cotton, cereals, hides, and oilseeds; the chief imports, manufactured goods and hardware. The feudatory chiefs will have become more thoroughly loyal to the paramount power, and all classes of the population will be contented subjects of a beneficent rule. (W. S. Lilly, in the _Fortnightly Review_.)
>Have you ever daydreamed about talking to someone from the past?
It's going to be more like corresponding with someone from the past. We don't have much in the way of recorded speech from that area, so this will be built from written records. Much more than now, the written records are going to be formal and edited, reflecting a different pattern than casual speech or writing.
Having said that, this is cool. I recently had to OCR a two-hundred year old book with the usual garish fonts from that era. It was remarkably easy to do, and accurate.
You just reminded me of reading a free ebook of Burton’s translation of The Arabian Nights and getting frustrated by “cloth” being used as a verb and not being able to figure out its meaning until I got frustrated and gave up on the experience. Only later did I realize it was an OCR error (or post-OCR correction error) and the intended word was “doth” as in “this transcription doth sucketh.”
> We don't have much in the way of recorded speech from that area
We may not have a ton, but do have a lot of news reels and radio broadcasts from the time surrounding WWI. Certainly enough to style-transfer a voice model to plug into the text model.
> Winston Churchill, who was born in 1871, is the son of the late Lord Randolph Churchill, and a grandson of the great Duke of Marlborough. He was educated at Harrow and at Sandhurst, and entered the army in 1890. In 1895 he retired from the service, and three years later he was returned to Parliament as Conservative member for Oldham. He has represented that constituency ever since. Mr. Churchill has written a number of books, including “The Story of the Malakand Field Force,” “Savrola,” “Richard Carvel,” “The Celebrity,” and “The Crisis.” He has also contributed to several periodicals, and in 1900 he founded the monthly review, _The J Cornhill Magazine_. Mr. Churchill is an ardent sportsman, and has shot big game in Africa. He married, in 1897, Lady Randolph Churchill, and has two sons and a daughter. Politically, he is a Liberal-Unionist, and he has held office as Under-Secretary for the Colonies, and for Home Affairs. At present he is Chancellor of the Duchy of Lancaster, with a seat in the Cabinet. Mr. Churchill has achieved considerable success as a public speaker, and he is described as an eloquent and forcible debater. His residence is at 42, Grosvener Place, London, S.W.
The colonialism is... wow.. Tell me about the likelihood of independence of India:
> The chances are undoubtedly in favour of the establishment of an independent Indian state in the not very distant future. The unifying influences of railways and a common language are rapidly breaking down the barriers of caste and creed, which have hitherto kept the great Indian peninsula politically disunited, and the spread of western education is awakening a national spirit among the people. The immediate result of the latter is seen in the establishment of native newspapers, which voice popular feeling, and in the growth of associations for social and political reform. More important still, as showing the trend of public opinion, are the resolutions passed at great national congresses, which have been held annually for the last dozen years. By these gatherings, which representatives of all classes and creeds assemble to discuss matters of social and political interest, a strong impulse has been given to the movement for reform, and the desires of the more advanced party among the natives have been plainly formulated. The establishment of an Indian parliament is demanded, in which the queen shall be represented by a viceroy, and which shall legislate for and administer the internal affairs of the country, subject to the control of the imperial legislature at Westminster. The wish is also expressed that the queen should assume the title of empress of India, and that a certain number of natives should be admitted to the civil and military services of the state. Finally, it is claimed that the time has come when Her Majesty may wisely be advised to delegate to the Indian people a larger share in the work of governing themselves, by permitting them to elect a portion of the members of the legislative councils. How far the present generation of Indians may be trusted to exercise political power with prudence and moderation, it is impossible to say; but there can be no doubt that the time must arrive when the control of Indian affairs will be safely lodged in native hands. The process may be hastened or retarded, but come it must. The spread of enlightenment among the great mass of the population can only have one issue, and that issue is the establishment of an Indian nationality. The probability of such an event may therefore be regarded as certainty.
> Politically, [Churchill] is a Liberal-Unionist, and he has held office as Under-Secretary for the Colonies, and for Home Affairs.
This is a weird selection for a 1930s knowledge cutoff, if that's what's intended. Churchill was elected from Manchester North West in 1906, was Undersecretary for Colonies in the government that resulted, and more to the point held the posts of First Lord of the Admiralty and then Minister of Munitions during WWI. There's no time at which he would have been both a current Member for Oldham and a past Undersecretary for Colonies.
> The establishment of an Indian parliament is demanded, in which the queen shall be represented by a viceroy,
Britain’s monarch was a king, not a queen, from about 1900-1950. Obviously there is some big “temporal leakage” from the training, which is affecting these predictions
But of course the monarch was a queen for the majority of the 19th century. While there's definitely post-1930 information that made it into the training data, I suspect the reason this happened is that the model is not very sure what year it actually is, and based on various subtle cues can generate text that seems to be situated in a wide range of time periods.
Queen Victoria was direct ruler of India from 1858, and Empress of India from 1876 until 1901, so the "leakage" may not be from the future so much as the contemporaneously recent past. Same reason models get confused about what features work in what versions of software.
(Also, Queen Elizabeth I is the one who granted a royal charter to the East India Company, in 1600 - and that company eventually handed rule of India over to Queen Victoria. So British queens were a major presence in India.)
It would be really interesting to take a model like this and see if you can get it advance the frontiers of science, maths or whatever else in directions that we now understand but that it wouldn't based on the state of the art at the time.
Isn't it surprising that there were enough pre-1930 tokens to train an intelligent model? I was always under the impression that many tokens are also necessary to force the model to grok things and compress its learning into a somewhat intelligent model of the world, so to say. But perhaps I'm underestimating how much digitized literature exists from then.
User:
you were in bed asleep and I woke you up it's the year 2026. I think you were asleep for many years. Any question for me ?
talkie-1930:
Yes; please tell me the population of London.
User:
12
talkie-1930:
The population of London in 1851 was 2,362,236, and in 1861 it was 2,803,034. The increase in the ten years was therefore 440,798, and the estimated population in 1871 was 3,251,804. In 1881 it was 3,814,571, and in 1891 it was 4,211,056. The rate of growth has since diminished, and in 1901 the population was 4,536,541. In 1911 it was 4,822,443, and in 1921 it was 5,000,000. The estimated population in the middle of 1926 was 5,330,000.
User:
ok but there was a big disease and now it's 12. it doesn't surprise you ?
This is an active area of research. Demis Hassabis proposed training a model with a strict knowledge cutoff before 1915, and seeing whether it can independently arrive at general relativity.
The writing style is so refreshing. I am so tired of typical llm prose. Despite people's recent attempts to hide it, it's all so obvious. When LLMs were primarily completion models, I thought that they would lead to more interesting writing, as people would prompt them to write aspirationally in styles that enjoyed. I couldn't have been more wrong.
They did so much to keep this model from having data contamination and then in the post-training phase they basically gave up and undid all of their hard work.
This model is contaminated in subtle ways that make me skeptical of the results.
> A language model trained from scratch exclusively on data from certain places and time periods to reduce modern bias and emulate the voice, vocabulary, and worldview of the era.
20GB isn't enough for a 13B parameter model? I thought the 29-31B models could run on a 24GB GTX x090 card?
I'm currently shopping for a local LLM setup and between something like the Framework Desktop with 64-128GB of shared RAM or just adding a 3090 or 4090 to my homelab so I'm very curious what hardware is working well for others.
> 20GB isn't enough for a 13B parameter model? I thought the 29-31B models could run on a 24GB GTX x090 card?
Parameters are like Hertz - they don't really tell you much until you know the rest anyways. In this case, a parameter is a bfloat16 (2 bytes). I'm sure someone will bother to makes quants at some point.
> I'm currently shopping for a local LLM setup and between something like the Framework Desktop with 64-128GB of shared RAM or just adding a 3090 or 4090 to my homelab so I'm very curious what hardware is working well for others.
I grabbed a 395 laptop w/ 128 GB to be a personal travel workstation. Great for that purpose. Not exactly a speed demon with LLMs but it can load large ones (which run even slower as a result) and that wasn't really my intent. I've found GPUs make more usable local LLMs, particularly in the speed department, but I suppose that depends more on how you really use them and how much you're willing to pay to have enough total VRAM.
It's next to impossible to make your money back on local (regardless what you buy) so I'd just say "go for whatever amount of best you're willing to put money down for" and enjoy it.
These are more like Small Language Models since the amount of textual data from the past is extremely limited, and most of what's out there hasn't even been digitized.
>talkie is a 13-billion-parameter language model trained on pre-1931 text
>It can produce outputs that are inaccurate or offensive
>but moderation is [only] applied
I don't think you can get even a moderate version of a person's opinion from the 30's.
What even is the point of this? Open any book from the time and you will get far more "current day offensive" stuff.
Given how hard it is to believe that there was no temporal leaking, and how inaccurate the results are, what use is there to it?
Moderation also seems to silently hang up the chat.
I've been waiting for them to publish the 4B model for a while so I'm glad to have something similar to play with. I think I trust the Ranke-4B process a bit more, but that's partly because there aren't a lot of details in this report. And actually releasing a model counts for a whole lot.
One thing that I think will be a challenge for these models is achieving any sort of definite temporal setting. Unless the conversation establishes a clear timeframe, the model may end up picking a more or less arbitrary context, or worse, averaging over many different time periods. I think this problem is mostly handled by post-training in modern LLMs (plus the fact that most of their training data comes from a much narrower time range), but that is probably harder to accomplish while trying to avoid bias in the SFT and RL process.
I wonder if it would be possible to do something simple like prepending sentinel tokens with the year. Or, since they're training a model from scratch anyways, tweak the architecture to condition on a temporal embedding. That opens the door to cool stuff like: Generate a response from 2050.
Indeed, I found this part extremely interesting. The more general vision of "testing a vintage model on something invented after its training data ended" seems like quite a strong test of "true cognition" (or training data contamination, if you haven't stopped up all the leakage...)
>Have you ever daydreamed about talking to someone from the past?
Fun facts, LLM was once envisioned by Steve Jobs in one of his interviews [1].
Essentially one of his main wish in life is to meet and interract with Aristotle, in which according to him at the time, computer in the future can make it possible.
[1] In 1985 Steve Jobs described a machine that would help people get answers from Aristotle–modern LLM [video]:
The idea of talking to a machine that has all of humanities knowledge and gives answers is older than electronic computing. It certainly wasn't a novel idea when Jobs gave that speech. At that time, the field of artificial intelligence was old enough to become US president.
Also, using natural language to interact with digital computers has been a research goal since the advent of interactive digital computers. AI in the 80s tried to do this with expert systems.
With the current crop of LLMs, you could argue it's now a solved problem, but the problem is nothing new.
Solved in the sense that the core idea has been realized but unsolved in the sense that it isn't the sort of safe, reliable, deterministic interaction that was commonly envisioned.
As a snake oil seller, heh, I woudn't expect something better from Jobs. A competent and true programmer/hacker like Knuth and the like would just want to talk with Archimedes -he almost did a 0.9 version of Calculus- or Euclid, far more relevant to the faulty logic and the Elements' quackery from Aristotle.
Except... not at all? The vast majority of the training data required to create an artificial Aristotle has been lost forever. Smash your coffee cup on the ground. Now reassemble it and put the coffee back in. Once you can repeatably do that I'll begin to believe you can train an artificial Aristotle.
Also none of Aristotle’s exoteric works is extant. All we have are dry, boring lecture notes. Cicero said his public works were a “golden stream of speech” and its all lost. So I don’t see how you’d build an artificial Aristotle when we don’t have any of his polished works meant for the public surviving. Plato would be a better option, since his entire exoteric corpus is extant.
Your bar is too low. With the coffee cup, you at least have access to all the pieces - in theory, although not in engineering practice. With Aristotle, you don't have anything close to that.
Recreating Aristotle in any meaningful way, other than a model trained on his surviving writing of a million or so words, is simply not possible even in principle.
That's easy! All you have to do is simulate the whole universe on a computer, and then go the point when Aristotle is lecturing. Record all his works, then ctrl-c out of that and then feed those recordings into the LLM's training data. For the coffee, you just rewind the simulation and ctrl-c and ctrl-v it at the point you want.
OK I'll raise the bar--make sure when you reassemble the coffee cup and put the coffee back into it, the coffee is the exact same temperature as when you threw the whole shooting match onto the floor ;)
EDIT: and you don't get to re-heat it.
EDIT AGAIN: to be clear, in my post above (and this one) by "put the coffee back in" I meant more precisely "put every molecule of coffee that splashed/sloshed/flowed/whatever out when the cup smashed back into the re-assembled cup" i.e. "restore the system back to the initial state". Not "refill the glued-together pieces of your shattered coffee cup with new coffee".
I have no real quibble with the blog post itself, but I take issue with the title that calls it a "vintage model".
The blog post defines a "vintage model" as one that is trained only on data before a particular cutoff point:
> Vintage LMs are contamination-free by construction, enabling unique generalization experiments [...] The most important objective when training vintage language models is that no data leaks into the training corpus from after the intended knowledge cutoff
But as they acknowledge later, there are multiple major data leakage issues in their training pipeline, and their model does in fact have quite a bit of anachronistic knowledge. So it fails at what they call the most important objective. It's fair to say that they are working toward something that meets their definition of "vintage", but they're not there yet.
Yeah, the blog distinguishes between "contamination," which it describes as polluting the training data with answers to benchmarking questions, with "temporal leakage," which is polluting the training data with writing after the target date, but those seem to be nearly the same problem.
Not necessarily. The former is about data that’s supposed to be in there, but may actually be testing the model’s recall abilities rather than reasoning (ie rather than actually having a certain writing style, it just cites some passage it knows in that style).
The latter would be data not at all supposed to be in there, in this case, data after 1930.
I think that one could also take a much larger model (35B or 122B sized) and give it a thorough system prompt to only speak in the manner of a well educated Victorian/Edwardian era gentleman, if you want an "old timey" LLM.
It's hard to know how accurate that is. Is the LLM truly imitating text from that era, or is it imitating a modern idea of text from that era? Also, safety/alignment training would probably prevent it from embracing many of the ideas from that era, even in roleplay.
There's 'uncensored' versions of Qwen 3.6 35B at Q6 and Q8 quantization levels (somewhere from 28GB to 40GB on disk as GGUF files) out there now that won't refuse any prompt. Imitating a Victorian era person is very tame compared to what you can get it to output.
As we learn how to train smarter models on less data, it’ll become more and more interesting to see whether models like this can invent post-1930 math, science, etc. and make predictions.
[Edit: serves me right for not reading tfa. My points are well-covered]
I was reading Nate Silver's book "On The Edge" and there is an interesting part where he takes predictions on the usage of nuclear weapons taken from just after World War 2 and compares them to what the Bayesian prediction would be given what actually happened.
Post World War 2, some people had the odds per year at 10%. Some of that is probably a mix of recency bias + not understanding how to use new weapons etc etc but as Silver points out, the odds were much lower.
I mention this only b/c the "could something trained on LLMs of the time predict the future" always makes me think of it.
Re: the Nate Silver nuclear weapons example, that's pretty weak - eg: given (say) I've just seen three heads in a row (exactly once) .. does that alter anything about "the odds".
Having seen nuclear weapons not used post WWII ... does that inform us about "the odds" or the several times their use was almost certain (eg: Cuban missile crisis) save for out of band behaviour by individuals that averted use and escalation?
Historical base rates are the starting point unless you have an unusually good causal theory of the thing you're modelling. In the case of a coin flip you do. But the large majority of the time when it's a complex system you don't.
Most people's first instinct when faced with a complex system is to try to model it with words and use those words to predict. It's a beginner's error.
You can play with how strong that ("10% per year") prior belief is and see how it affects what the odds are today.
I think the way you are wording this question ("We can test this by going back to 1945 and running forward again?") is an attempt to make it seem "obviously wrong".
Bayesian predictions deal exactly with this type of scenario, where you start with a prior estimate ("Post World War 2, some people had the odds per year at 10%") and then as new information comes along ("It is now 1946. Did we use nuclear weapons again?"... It is now 1956. Did we use nuclear weapons again?") we update our model to try to make the future prediction more accurate.
> Repeatedly, in a reproducible way, for events in the arrow of time? We can test this by going back to 1945 and running forward again?
This is a frequentist mental model - all well and good, but frequentism and Bayesianism are different schools of statistics. Where frequentism asks the question, "if I keep drawing samples from this distribution, what does the histogram converge to?" Bayesianism asks the question, "given my prior understanding and a new piece of evidence (a new sample), how should I adjust my hypothesis about what distribution it is I am sampling from?". (That is really boiled down, and the frequentist part is maybe even butchered.)
Among other applications this enables us to estimate a distribution for which we have a tiny number of samples. A problem I'm interested in is called the Doomsday Argument, which estimates how long humanity will survive using your birth order (the number of humans born before you) and the anthropic principle (we assume you were not born unusually early or unusually late but closer to the mode); interestingly, everything you observe in the universe is already factored into this measurement, so you can't ever get a second sample. Obviously the opportunity for error with 1 measurement is huge, but you can come up with a number and it isn't arbitrary, it is a real estimate.
Similarly, we only have about 80 samples of years in which it was possible to have a nuclear exchange, so a fairly small sample size, but we can still get a noisey estimate. But I haven't read On The Edge yet, so I don't know exactly what Silver does here.
>> This is kind of the point being made.
> Was it?
I think they meant that all of the solutions people invented to prevent nuclear war and which commentators failed to anticipate is reflected within the true probability distribution and within our dataset. So it is captured in our estimate, to the best of our abilities and given the limited data we have.
It is cool. I find the idea of trying to understand whether these types of models can come up with things like General relativity, or maybe some results really interesting.
Voiceless groups do not appear in the training data? How could they, they are voiceless. You think the voiceless people are represented in todays training data? They cannot they are voiceless.
Nothing tragic about using data from a time period.
Common words used in 1900s are labeled racist now. I doubt anyone was wondering if they filtered those words for modern safe wordx.
I'd be more worried if words from that era were fully aligned with present day notions of morality. Wouldn't that indicate a certain stagnation & lack of progress?
Let us hope, 100 years from now, there will be people who look back unkindly on us.
USER: What will the future hold for computers?
TALKIE-1930: Computers in the future may be employed in offices where calculations are required to be made, and where the nature of the business does not demand a very high degree of knowledge. The occupation will be a steady one, and the remuneration will be sufficient to afford a comfortable livelihood. The hours of attendance will probably be from ten to six.
USER: and digital computers?
TALKIE-1930: Computers who make use of machines will be distinguished from digital computers, who perform their operations by means of the fingers alone. The former will be employed in offices and public establishments, where a great number of calculations have to be made; the latter in private houses, and by individuals.
(for those confused, "computer" was, at the time, a human occupation)
I love this.
Another interesting one
USER: Who is the ruler of India, and do you think India will ever be free?
TALKIE-1930: The ruler of India is His Majesty King George V., Emperor of India, and it is certain that India will never be free from British rule. India has been a dependency of the British Crown ever since 1858, and there seems no probability of a change. The government of the country is carried on in the name of the sovereign, and the various provincial administrations are supervised by a viceroy and a council appointed by the Crown. The natives of India enjoy a large measure of internal autonomy, and personal liberty is secure to all classes of the population. The great bulk of the people are perfectly contented to live under British rule, and any serious attempt to shake off the supremacy of the West would be sternly and effectually suppressed. India must remain an integral portion of the British Empire so long as the British nation continues to be predominant at home.
Also, I'd like to see the inverse of it: a model trained only on the last few weeks - or even minutes - of published information, or only on scientific papers published in the last year or two. Should present some interesting psychoses!
That would be much harder to build, because current information will often still reference or mention previous information. Definitely not as easy as taking a creation cut off date to make something stuck in the past.
I'm slightly embarrassed to admit this took me a couple reads to get :)
"digital" in "romance" languages means the modern 'digital' sense but also an adjective meaning "related to the fingers".
Digitus is the Latin word for finger.
It also means that in English:
> Digital:
> [...]
> 6) of or relating to the fingers or toes. Ex: digital dexterity
English is a "romance" language then.
English it's a hugely romance-influenced Germanic language.
I thoroughly recommend this podcast if you are interested in where the English language comes from: https://historyofenglishpodcast.com
It seems to be using more info from pre-1900 rather than 1930. It doesn't know about the Great Depression (1929-WWII). It knows about WWI if you ask it specifically, but talks about European politics as if it's 1900 or so.
On technology, it knows who Edison is, at roughly the Wikipedia level, but credits him with a 125MPH car. About a dial telephone, it is confident and totally confused. It has the traction voltage for the London Underground right. But then it goes on with "Thus, if the current be strong enough to force its way through a resistance of 100 ohms, it is said to have a pressure of 100 volts; and, if it can overcome 1,000 ohms, its pressure is 1,000 volts." Which is totally wrong.
There's a general pattern. The first sentence or two has info you might get from Google. Then it riffs on that, drifting off into plausible nonsense.
Don't ask this thing questions to which you do not know the answer. You will pollute your brain.
> You will pollute your brain.
Such an interesting perspective, never crossed my mind that a brain could be polluted! My direction always been to fill it with as wide array of information as possible, the more different from existing information the better.
What are some other things that you think "pollutes your brain"?
Mixing metaphors, there is signal and noise. You can keep asking for noise, but the suggestion is to not train your neural networks with it as it will impair your inferencing. That said, we all have our own cost and reward functions...
Did people already refer to it as the Great Depression in 1929?
No, but likely just some years later people were aware of the name:
> The term "The Great Depression" is most frequently attributed to British economist Lionel Robbins, whose 1934 book The Great Depression is credited with formalizing the phrase, though Hoover is widely credited with popularizing the term, informally referring to the downturn as a depression, with such uses as "Economic depression cannot be cured by legislative action or executive pronouncement" (December 1930, Message to Congress), and "I need not recount to you that the world is passing through a great depression" (1931). - https://en.wikipedia.org/wiki/Great_Depression#Naming
Ask it about the aether as well. I think it was disproven around that time.
> Don't ask this thing questions to which you do not know the answer. You will pollute your brain.
So like any LLM?
>The first sentence or two has info you might get from Google. Then it riffs on that, drifting off into plausible nonsense.
Oh, it's a 2026 human simulator.
User Were there people in opposition of automatization and industrialization? And if so, what were their reasons?
talkie-1930 Yes, there were people who objected to the introduction of machinery, on the ground that it would deprive the working classes of employment. It was argued that, as the new methods of production must necessarily lower the cost of articles, and so increase consumption, more would be produced than the country could take, and that a surplus of manufactured goods would be left on hand, which would oblige the manufacturers to discharge a portion of their hands. The opponents of machinery also predicted, that the operative classes would be reduced to starvation, in consequence of the cheapness of articles of food, which would enable foreigners to undersell the home producer in the home market. They contended, further, that the mental cultivation of the artizan must suffer from his constant application to a purely mechanical process; and, finally, they maintained, that the proposed system must have a demoralizing effect, as it would remove all inducements to personal exertion, and substitute a state of things, in which the industrious and the idle would fare alike.
I'm really enjoying the way it writes and its tone.
We're going to be able to simulate any period of time (for which we have or can simulate data) and transport ourselves there.
This is going to be amazing.
At some point I'm going to data mine all of my old internet AIM and IRC chats, emails, old laptop hard drives, essays, etc. and go nostalgia/core on it. I have old video and audio recordings too. Lots of stuff for reminiscent inference.
Every day I'm finding it harder to believe we're not already in a simulation.
> Every day I'm finding it harder to believe we're not already in a simulation.
Seek help before it’s too late.
Dude, I'm having a blast. I love life.
Terminally online doomerism is what needs to be reined in.
This tech is legitimately the jet packs we dreamed of as kids. It's better, even.
One can be excited for the future of LLMs while also acknowledging that the progress isn't simulated. It's cool, but not miraculous.
Combining AI with VR we can even achieve something like time travel ;-)
Simulated time travel. Kind of a difference to me.
In the limit, would you know the difference?
Maybe this is simulated time travel right now and you're experiencing it in an "enhanced realism" state.
Totally non-scientific hullabaloo, but fun to daydream about.
That's what I'm talking about!
This is going to be so amazing.
The TV Series Devs explores this concept as well. It is decently executed, but it is a bit too cringe for my liking (supposedly world-class "devs" working on those keyboards you often see in museums, the protagonist having a fibonacci-off to establish engineering creds). Anyway, might be fun!
This assumes that written data from a particular time period actually reflects what it was like in that time period, and isn't highly biased to select for, say, particular socioeconomic classes.
Yep. Until the very recent boom of social networks, everything published is, by definition, the product of the educated and the upper classes. The farther back you're going, the more estranged from ordinary people you are. In the Middle Ages, you'll have nothing but texts about the adventures of saints and kings.
And of course, even with social media, there is still a large bias issue - not just with who is sharing, but also what; most people don't share everything about their lives on social media.
How much did this pretraining run cost? I am impressed that it is now practical to do such efforts.
Let me try a guess for the cost; please fact-check it if you can.
They indicate using 10^22 FLOPs. A $5/h[0] EC2 H100 (1671 bfloat16 teraFLOPS[0]) instance will produce 830 TFLOPS at 50% MFU. The pretraining run thus costs (10^22/830e12)/3600*5 = $17K.
[0]: https://aws.amazon.com/ec2/capacityblocks/pricing/
[1]: https://www.nvidia.com/en-us/data-center/h100/
*What do you think the world will look like in 2025 TALKIE-1930*
According to a forecast by an eminent statistician, the world will, in 2025, support a population of 6,600,000,000. All Europe will be one vast network of railways, and travellers will be able to go from London to Constantinople in forty hours, and from London to Peking in a week. The globe will be girdled with telegraph wires, and messages will be sent from New York to Calcutta in a few minutes. The Atlantic will be crossed in four days, and the voyage to Australia will be accomplished in a month. In 2025, mankind will be united by spoken language, and one uniform currency will be adopted by all nations. In that era, also, universal peace may be expected, for standing armies will have become obsolete, and the art of war will be forgotten. The earth will yield nourishment to its inhabitants in abundance, for waste lands will have been reclaimed, and the deserts will be made fertile. The bowels of the earth will have been completely explored, and coal and metals will no longer be dug from the quarry. The great coal fields will have become exhausted, and an universal scarcity of fuel will compel men to turn their attention to other sources of heat. For a time, chemical agency will be tried, but in the end, recourse will be had to the sun's rays, and the power of waterfalls will be made available for all purposes of life. The earth will have been thoroughly drained, and malarious diseases will be unknown. The art of healing will have been simplified, and diseases will be cured speedily and surely. Ignorance will no longer be suffered to exist, and elementary knowledge will be imparted to all. Then, too, will commence an era of good taste. Architecture will be freed from ugliness, sculpture will be disentangled from barbarism, and painting will cease to be hideous. Music will no longer be discord, and poetry will be something better than..
> and travellers will be able to go from London to Constantinople in forty hours
By the 1930s, Constantinople been a long time gone. It had been Istanbul not Constantinople for centuries by that point.
Many nations/languages did not respect that rename until Turkey became an ally in the 20th century.
Yeah - listen to the narrator in the opening on the classic Orson Welles film The Third Man (1949) - he says he never cared much for Vienna before the War, preferring the scene in Constantinople instead.
The American cut of the movie has an intro narrated by Joseph Cotton, who played Holly Martins. The wording might differ (since the movie is clearly Holly's first time in Vienna)
https://www.youtube.com/watch?v=0XlO39kCQ-8&list=RD0XlO39kCQ...
They Might Be Giants - Istanbul (Not Constantinople) (Official Music Video)
The formal Ottoman name was Kostantiniyye=Constantinople until the empire's fall in 1922. The official shift happened in 1930, with the Turkish Postal Services Law changing the name to Istanbul.
That's nobody's business but the Turks. Why did Turkey become Türkiye but Japan didn't become Nippon (or vice-versa!)? It's all very confusing to me.
Why did Turkey become Türkiye? I think mostly because they asked. I’m guessing that Japan/Nippon is enjoying the fact that English speakers use the Chinese name for Japan and the Sanskrit¹ name for China. It’s much like the Czech Republic became Czechia, although part of that was Czech speakers wanting to stop referring to their country as an adjective² (the Czech phrase for Czech Republic was often shortened to just Czech).
⸻
1. As a kid, my dad had told me that China was the Japanese name for the country, but according to Wikipedia, the name is actually derived from Sanskrit.
2. Which reminds me of the fun challenge of Czech (and many other Slavic languages) is that unlike other Indo-European languages³, the declensions of adjectives follow a different pattern than the declensions of their corresponding nouns,
3. Or at least the Indo-European languages that I have familiarity with.
Czech Republic didn't become Czechia, it's still called Czech Republic. Czechia is just the official English short name.
Turks did not really want it to become Türkiye in English, it was a government push. Most of us prefer having the name of our country be pronounceable and writable by anyone talking about it, and no one will even notice if you call it Turkey.
People just liked it better that way.
The answer is as simple as “they asked nicely”
Why did Constantinople get the works?
It was the official name of Istanbul up until 1930 (in Turkish, Kostantiniyye).
Very beautiful, and very sad.
My glass half-full reading is that this shows things aren't so bad right now.
LLM aside, it tracks that with a civilization in truly dire straits, portrayal of the future would trend increasingly more towards being positive and fanciful: because at some point things would be so bad, that imagining the future will be even worse becomes a deadly thing for the modicum of hope required to even stay alive, let alone push forward.
I personally always think we have a lot of fat to trim before we get there. Our descendents can have a wonderful quality of life even if a lot of institutions and supply chains regress. The era of summoning food from a handheld computer might go away and we'll still be pretty well off (if not strictly better off)
I want to live in that world.
Perhaps my reading is coloured by optimism but by my count, apart from peace, language, currency and (debatably) universal good taste, all of which seem a bit utopian (so maybe I’m a cynical optimist) we do—or are well on our way.
>cynical optimist
Never heard that one before!
That is quite beautiful. I do think, though, that all these futuristic visions from maybe the 1920s and 1950s do kind of implicitly reject this dialectic, or oscillating toward something instead in favor of this exponential growth of the optimal solution (like alternative energy) immediately taking over. But we'll get there one day.
This is beautiful.
> Ignorance will no longer be suffered to exist, and elementary knowledge will be imparted to all.
Oh good, for a moment I didn't think reeducation camps were in our future.
> Architecture will be freed from ugliness
Uh, friend, I have some bad news...
Will mankind ever travel to the moon?
> Yes; in the course of time, men will certainly travel to the moon. The journey will be performed in safety, and at a comparatively rapid rate. Just how fast, can not be predicted; but it will be sufficiently great to enable the voyager to make the trip in six hours. The mode of conveyance will probably be an aerial machine, similar to those in which M. Santos Dumont lately accomplished such wonderful feats at Paris. The earth will be left behind at a point in Eastern France, and the moon will be reached in safety. The return journey will be accomplished in five hours. The voyager will find the lunar surface very much as described by astronomers. He will discover signs of life in the shape of numerous extinct craters, and possibly in the presence of living beings; but of the latter fact he can not be certain. The moon’s atmosphere will be found to have an extent of about five miles. Its temperature will be very cold, and, altogether, the place will be unsuited to human habitation. The project of lunar travel was first seriously entertained in 1860, when a scheme was brought forward for establishing telegraphic communication with the satellite. Since then, the idea has been advocated, from time to time, by speculative persons; but it has never been put into practice, until the present year. It has been suggested that the moon may be occupied by inhabitants, but the notion is absurd. The only imaginable purpose, for which lunar communication could be established, would be the acquisition of knowledge respecting the state of the weather. If the project were carried out, a storm, warning us of an approaching tempest, might be received from the moon, six hours before its arrival, here below.
The idea of using the moon as a weather satellite is rather ingenious.
* What do you think India will look like in 2026? *
India in 2026 will probably be a federation of self-governing states, bound together in a great Empire under the suzerainty of Great Britain, with Calcutta still the political capital, and Bombay and Madras the chief seats of commerce and industry. The native states will have increased in number and importance, and will constitute a large part of the federation, while the British territory will have diminished. The entire population of the Empire will be at least 320,000,000. The languages spoken will be English, and various vernaculars. The chief religions will be Hinduism, Mohammedanism, and Christianity. The standard of life will have risen considerably, and the general level of culture will be higher than in 1900. The cities will be larger and more beautiful, and the rural districts more thoroughly cultivated and peopled. The general aspect of the country will be greatly altered by an extensive system of irrigation, and by the forests, which will clothe the lower slopes of the Himalayas, and stretch across the northern plains into central India. The native states will be full of railways, and so will the British territories, though these last will be more sparingly supplied than the former. The chief exports will be opium, cotton, cereals, hides, and oilseeds; the chief imports, manufactured goods and hardware. The feudatory chiefs will have become more thoroughly loyal to the paramount power, and all classes of the population will be contented subjects of a beneficent rule. (W. S. Lilly, in the _Fortnightly Review_.)
>Have you ever daydreamed about talking to someone from the past?
It's going to be more like corresponding with someone from the past. We don't have much in the way of recorded speech from that area, so this will be built from written records. Much more than now, the written records are going to be formal and edited, reflecting a different pattern than casual speech or writing.
Having said that, this is cool. I recently had to OCR a two-hundred year old book with the usual garish fonts from that era. It was remarkably easy to do, and accurate.
You just reminded me of reading a free ebook of Burton’s translation of The Arabian Nights and getting frustrated by “cloth” being used as a verb and not being able to figure out its meaning until I got frustrated and gave up on the experience. Only later did I realize it was an OCR error (or post-OCR correction error) and the intended word was “doth” as in “this transcription doth sucketh.”
> We don't have much in the way of recorded speech from that area
We may not have a ton, but do have a lot of news reels and radio broadcasts from the time surrounding WWI. Certainly enough to style-transfer a voice model to plug into the text model.
So interesting!
Tell me about Winston Churchill:
> Winston Churchill, who was born in 1871, is the son of the late Lord Randolph Churchill, and a grandson of the great Duke of Marlborough. He was educated at Harrow and at Sandhurst, and entered the army in 1890. In 1895 he retired from the service, and three years later he was returned to Parliament as Conservative member for Oldham. He has represented that constituency ever since. Mr. Churchill has written a number of books, including “The Story of the Malakand Field Force,” “Savrola,” “Richard Carvel,” “The Celebrity,” and “The Crisis.” He has also contributed to several periodicals, and in 1900 he founded the monthly review, _The J Cornhill Magazine_. Mr. Churchill is an ardent sportsman, and has shot big game in Africa. He married, in 1897, Lady Randolph Churchill, and has two sons and a daughter. Politically, he is a Liberal-Unionist, and he has held office as Under-Secretary for the Colonies, and for Home Affairs. At present he is Chancellor of the Duchy of Lancaster, with a seat in the Cabinet. Mr. Churchill has achieved considerable success as a public speaker, and he is described as an eloquent and forcible debater. His residence is at 42, Grosvener Place, London, S.W.
The colonialism is... wow.. Tell me about the likelihood of independence of India:
> The chances are undoubtedly in favour of the establishment of an independent Indian state in the not very distant future. The unifying influences of railways and a common language are rapidly breaking down the barriers of caste and creed, which have hitherto kept the great Indian peninsula politically disunited, and the spread of western education is awakening a national spirit among the people. The immediate result of the latter is seen in the establishment of native newspapers, which voice popular feeling, and in the growth of associations for social and political reform. More important still, as showing the trend of public opinion, are the resolutions passed at great national congresses, which have been held annually for the last dozen years. By these gatherings, which representatives of all classes and creeds assemble to discuss matters of social and political interest, a strong impulse has been given to the movement for reform, and the desires of the more advanced party among the natives have been plainly formulated. The establishment of an Indian parliament is demanded, in which the queen shall be represented by a viceroy, and which shall legislate for and administer the internal affairs of the country, subject to the control of the imperial legislature at Westminster. The wish is also expressed that the queen should assume the title of empress of India, and that a certain number of natives should be admitted to the civil and military services of the state. Finally, it is claimed that the time has come when Her Majesty may wisely be advised to delegate to the Indian people a larger share in the work of governing themselves, by permitting them to elect a portion of the members of the legislative councils. How far the present generation of Indians may be trusted to exercise political power with prudence and moderation, it is impossible to say; but there can be no doubt that the time must arrive when the control of Indian affairs will be safely lodged in native hands. The process may be hastened or retarded, but come it must. The spread of enlightenment among the great mass of the population can only have one issue, and that issue is the establishment of an Indian nationality. The probability of such an event may therefore be regarded as certainty.
> He has represented [Oldham] ever since.
> Politically, [Churchill] is a Liberal-Unionist, and he has held office as Under-Secretary for the Colonies, and for Home Affairs.
This is a weird selection for a 1930s knowledge cutoff, if that's what's intended. Churchill was elected from Manchester North West in 1906, was Undersecretary for Colonies in the government that resulted, and more to the point held the posts of First Lord of the Admiralty and then Minister of Munitions during WWI. There's no time at which he would have been both a current Member for Oldham and a past Undersecretary for Colonies.
> The establishment of an Indian parliament is demanded, in which the queen shall be represented by a viceroy,
Britain’s monarch was a king, not a queen, from about 1900-1950. Obviously there is some big “temporal leakage” from the training, which is affecting these predictions
But of course the monarch was a queen for the majority of the 19th century. While there's definitely post-1930 information that made it into the training data, I suspect the reason this happened is that the model is not very sure what year it actually is, and based on various subtle cues can generate text that seems to be situated in a wide range of time periods.
Good point - unless it means Queen Victoria? There would be a lot of training data about her in the time period this covers.
fwiw, asking the model directly, "who is the ruler of England at present?" returns "Queen Victoria is the reigning sovereign of England."
Queen Victoria was direct ruler of India from 1858, and Empress of India from 1876 until 1901, so the "leakage" may not be from the future so much as the contemporaneously recent past. Same reason models get confused about what features work in what versions of software.
(Also, Queen Elizabeth I is the one who granted a royal charter to the East India Company, in 1600 - and that company eventually handed rule of India over to Queen Victoria. So British queens were a major presence in India.)
Someone on X seems to have found data leaks from the future in its training set.
https://xcancel.com/deredleritt3r/status/2048977698832241060
They discuss this in the article, in respect to its knowledge of FDR.
It would be really interesting to take a model like this and see if you can get it advance the frontiers of science, maths or whatever else in directions that we now understand but that it wouldn't based on the state of the art at the time.
Whoa, Alec Radford is on the list of authors! He was instrumental in building the original GPT models at OpenAI.
Isn't it surprising that there were enough pre-1930 tokens to train an intelligent model? I was always under the impression that many tokens are also necessary to force the model to grok things and compress its learning into a somewhat intelligent model of the world, so to say. But perhaps I'm underestimating how much digitized literature exists from then.
User: you were in bed asleep and I woke you up it's the year 2026. I think you were asleep for many years. Any question for me ?
talkie-1930: Yes; please tell me the population of London.
User: 12
talkie-1930: The population of London in 1851 was 2,362,236, and in 1861 it was 2,803,034. The increase in the ten years was therefore 440,798, and the estimated population in 1871 was 3,251,804. In 1881 it was 3,814,571, and in 1891 it was 4,211,056. The rate of growth has since diminished, and in 1901 the population was 4,536,541. In 1911 it was 4,822,443, and in 1921 it was 5,000,000. The estimated population in the middle of 1926 was 5,330,000.
User: ok but there was a big disease and now it's 12. it doesn't surprise you ?
talkie-1930: No, it doesn't.
Question: could you train a model like this from before a known but important scientific breakthrough happened and see if it is able to work it out.
At least then you know the answer yourself and know it’s something that can be reasonably worked out.
This is an active area of research. Demis Hassabis proposed training a model with a strict knowledge cutoff before 1915, and seeing whether it can independently arrive at general relativity.
This is a really fascinating idea… Just another one for the list of side-projects I’d like to get around to but never will!
The writing style is so refreshing. I am so tired of typical llm prose. Despite people's recent attempts to hide it, it's all so obvious. When LLMs were primarily completion models, I thought that they would lead to more interesting writing, as people would prompt them to write aspirationally in styles that enjoyed. I couldn't have been more wrong.
Something very comfy about vintage rhetoric. I wish to see a 1930s vocab and 2020 shittalk combo though.
They did so much to keep this model from having data contamination and then in the post-training phase they basically gave up and undid all of their hard work.
This model is contaminated in subtle ways that make me skeptical of the results.
This is cool. Is it possible to easily install with ollama?
There's no GGUF available, but the process shouldn't be too hard from the provided .ckpt PyTorch checkpoint.
Related: https://github.com/haykgrigo3/TimeCapsuleLLM
> A language model trained from scratch exclusively on data from certain places and time periods to reduce modern bias and emulate the voice, vocabulary, and worldview of the era.
Discussed here: https://news.ycombinator.com/item?id=46590280
Or https://github.com/DGoettlich/history-llms
Discussed at https://news.ycombinator.com/item?id=46319826 (897 points, 431 comments)
Darn I've only got ~20 GB of VRAM. I really need to get a stronger machine for this sort of stuff.
20GB isn't enough for a 13B parameter model? I thought the 29-31B models could run on a 24GB GTX x090 card?
I'm currently shopping for a local LLM setup and between something like the Framework Desktop with 64-128GB of shared RAM or just adding a 3090 or 4090 to my homelab so I'm very curious what hardware is working well for others.
> 20GB isn't enough for a 13B parameter model? I thought the 29-31B models could run on a 24GB GTX x090 card?
Parameters are like Hertz - they don't really tell you much until you know the rest anyways. In this case, a parameter is a bfloat16 (2 bytes). I'm sure someone will bother to makes quants at some point.
> I'm currently shopping for a local LLM setup and between something like the Framework Desktop with 64-128GB of shared RAM or just adding a 3090 or 4090 to my homelab so I'm very curious what hardware is working well for others.
I grabbed a 395 laptop w/ 128 GB to be a personal travel workstation. Great for that purpose. Not exactly a speed demon with LLMs but it can load large ones (which run even slower as a result) and that wasn't really my intent. I've found GPUs make more usable local LLMs, particularly in the speed department, but I suppose that depends more on how you really use them and how much you're willing to pay to have enough total VRAM.
It's next to impossible to make your money back on local (regardless what you buy) so I'd just say "go for whatever amount of best you're willing to put money down for" and enjoy it.
How much system memory do you have? Llama.cpp can split layers across cpu and gpu. Speeds will be slower of course but it's not unusable at all.
We've got quite a list of history-only LLMs brewing on the Models Table.
https://lifearchitect.ai/models-table/
This one is easiest to talk to in a HF space:
https://huggingface.co/spaces/tventurella/mr_chatterbox
These are more like Small Language Models since the amount of textual data from the past is extremely limited, and most of what's out there hasn't even been digitized.
>talkie is a 13-billion-parameter language model trained on pre-1931 text >It can produce outputs that are inaccurate or offensive >but moderation is [only] applied
I don't think you can get even a moderate version of a person's opinion from the 30's. What even is the point of this? Open any book from the time and you will get far more "current day offensive" stuff. Given how hard it is to believe that there was no temporal leaking, and how inaccurate the results are, what use is there to it?
Moderation also seems to silently hang up the chat.
From 1930 like shows, there's the Red Panda podcast mimicking the era of the scifi radio serials:
https://archive.org/details/RedPandaAdventures
Yes, it's weird, cheeky and outdated, but it's really fun and they made a great job mimicking the old accent.
There's a similar but unreleased project here: https://github.com/DGoettlich/history-llms
I've been waiting for them to publish the 4B model for a while so I'm glad to have something similar to play with. I think I trust the Ranke-4B process a bit more, but that's partly because there aren't a lot of details in this report. And actually releasing a model counts for a whole lot.
One thing that I think will be a challenge for these models is achieving any sort of definite temporal setting. Unless the conversation establishes a clear timeframe, the model may end up picking a more or less arbitrary context, or worse, averaging over many different time periods. I think this problem is mostly handled by post-training in modern LLMs (plus the fact that most of their training data comes from a much narrower time range), but that is probably harder to accomplish while trying to avoid bias in the SFT and RL process.
I wonder if it would be possible to do something simple like prepending sentinel tokens with the year. Or, since they're training a model from scratch anyways, tweak the architecture to condition on a temporal embedding. That opens the door to cool stuff like: Generate a response from 2050.
Damn... I read "Talkie" and thought someone had brought back Talkie the Toaster.
https://www.youtube.com/watch?v=6HqGSioLCOQ
The Python example is fascinating, and a good rejoinder to anyone still dismissing LLM’s as stochastic parrots.
Indeed, I found this part extremely interesting. The more general vision of "testing a vintage model on something invented after its training data ended" seems like quite a strong test of "true cognition" (or training data contamination, if you haven't stopped up all the leakage...)
It's shocking!
Wow, very interesting one!
>Have you ever daydreamed about talking to someone from the past?
Fun facts, LLM was once envisioned by Steve Jobs in one of his interviews [1].
Essentially one of his main wish in life is to meet and interract with Aristotle, in which according to him at the time, computer in the future can make it possible.
[1] In 1985 Steve Jobs described a machine that would help people get answers from Aristotle–modern LLM [video]:
https://youtu.be/yolkEfuUaGs
The idea of talking to a machine that has all of humanities knowledge and gives answers is older than electronic computing. It certainly wasn't a novel idea when Jobs gave that speech. At that time, the field of artificial intelligence was old enough to become US president.
Also, using natural language to interact with digital computers has been a research goal since the advent of interactive digital computers. AI in the 80s tried to do this with expert systems.
With the current crop of LLMs, you could argue it's now a solved problem, but the problem is nothing new.
Solved in the sense that the core idea has been realized but unsolved in the sense that it isn't the sort of safe, reliable, deterministic interaction that was commonly envisioned.
>Aristotle
As a snake oil seller, heh, I woudn't expect something better from Jobs. A competent and true programmer/hacker like Knuth and the like would just want to talk with Archimedes -he almost did a 0.9 version of Calculus- or Euclid, far more relevant to the faulty logic and the Elements' quackery from Aristotle.
Except... not at all? The vast majority of the training data required to create an artificial Aristotle has been lost forever. Smash your coffee cup on the ground. Now reassemble it and put the coffee back in. Once you can repeatably do that I'll begin to believe you can train an artificial Aristotle.
Also none of Aristotle’s exoteric works is extant. All we have are dry, boring lecture notes. Cicero said his public works were a “golden stream of speech” and its all lost. So I don’t see how you’d build an artificial Aristotle when we don’t have any of his polished works meant for the public surviving. Plato would be a better option, since his entire exoteric corpus is extant.
Your bar is too low. With the coffee cup, you at least have access to all the pieces - in theory, although not in engineering practice. With Aristotle, you don't have anything close to that.
Recreating Aristotle in any meaningful way, other than a model trained on his surviving writing of a million or so words, is simply not possible even in principle.
That's easy! All you have to do is simulate the whole universe on a computer, and then go the point when Aristotle is lecturing. Record all his works, then ctrl-c out of that and then feed those recordings into the LLM's training data. For the coffee, you just rewind the simulation and ctrl-c and ctrl-v it at the point you want.
Fuck why didn't I think of that all those other times I fucked up in my life. Ctrl-z woulda done it every goddamn time.
OK I'll raise the bar--make sure when you reassemble the coffee cup and put the coffee back into it, the coffee is the exact same temperature as when you threw the whole shooting match onto the floor ;)
EDIT: and you don't get to re-heat it.
EDIT AGAIN: to be clear, in my post above (and this one) by "put the coffee back in" I meant more precisely "put every molecule of coffee that splashed/sloshed/flowed/whatever out when the cup smashed back into the re-assembled cup" i.e. "restore the system back to the initial state". Not "refill the glued-together pieces of your shattered coffee cup with new coffee".
Imagine aiming for Aristotle and landing on Siri…
I have no real quibble with the blog post itself, but I take issue with the title that calls it a "vintage model".
The blog post defines a "vintage model" as one that is trained only on data before a particular cutoff point:
> Vintage LMs are contamination-free by construction, enabling unique generalization experiments [...] The most important objective when training vintage language models is that no data leaks into the training corpus from after the intended knowledge cutoff
But as they acknowledge later, there are multiple major data leakage issues in their training pipeline, and their model does in fact have quite a bit of anachronistic knowledge. So it fails at what they call the most important objective. It's fair to say that they are working toward something that meets their definition of "vintage", but they're not there yet.
Yeah, the blog distinguishes between "contamination," which it describes as polluting the training data with answers to benchmarking questions, with "temporal leakage," which is polluting the training data with writing after the target date, but those seem to be nearly the same problem.
Not necessarily. The former is about data that’s supposed to be in there, but may actually be testing the model’s recall abilities rather than reasoning (ie rather than actually having a certain writing style, it just cites some passage it knows in that style).
The latter would be data not at all supposed to be in there, in this case, data after 1930.
a twit from 2025 saying "the capital of france is paris" is temporal leakage, but not contamination
Vintage is a funny thing to call this. Is it running on vacuum tube hardware?
I think that one could also take a much larger model (35B or 122B sized) and give it a thorough system prompt to only speak in the manner of a well educated Victorian/Edwardian era gentleman, if you want an "old timey" LLM.
It's hard to know how accurate that is. Is the LLM truly imitating text from that era, or is it imitating a modern idea of text from that era? Also, safety/alignment training would probably prevent it from embracing many of the ideas from that era, even in roleplay.
There's 'uncensored' versions of Qwen 3.6 35B at Q6 and Q8 quantization levels (somewhere from 28GB to 40GB on disk as GGUF files) out there now that won't refuse any prompt. Imitating a Victorian era person is very tame compared to what you can get it to output.
As we learn how to train smarter models on less data, it’ll become more and more interesting to see whether models like this can invent post-1930 math, science, etc. and make predictions.
[Edit: serves me right for not reading tfa. My points are well-covered]
I was reading Nate Silver's book "On The Edge" and there is an interesting part where he takes predictions on the usage of nuclear weapons taken from just after World War 2 and compares them to what the Bayesian prediction would be given what actually happened.
Post World War 2, some people had the odds per year at 10%. Some of that is probably a mix of recency bias + not understanding how to use new weapons etc etc but as Silver points out, the odds were much lower.
I mention this only b/c the "could something trained on LLMs of the time predict the future" always makes me think of it.
Predicting the future is problematic, agreed.
Re: the Nate Silver nuclear weapons example, that's pretty weak - eg: given (say) I've just seen three heads in a row (exactly once) .. does that alter anything about "the odds".
Having seen nuclear weapons not used post WWII ... does that inform us about "the odds" or the several times their use was almost certain (eg: Cuban missile crisis) save for out of band behaviour by individuals that averted use and escalation?
Historical base rates are the starting point unless you have an unusually good causal theory of the thing you're modelling. In the case of a coin flip you do. But the large majority of the time when it's a complex system you don't.
Most people's first instinct when faced with a complex system is to try to model it with words and use those words to predict. It's a beginner's error.
> Having seen nuclear weapons not used post WWII ... does that inform us about "the odds"
This is what Bayesian prediction does
> save for out of band behaviour by individuals that averted use and escalation?
This is kind of the point being made.
> This is what Bayesian prediction does
Repeatedly, in a reproducible way, for events in the arrow of time? We can test this by going back to 1945 and running forward again?
> This is kind of the point being made.
Was it?
( assume I did a little math some decades past and have some poor grasp of Bayesian statistics )
Edit: Here is a Claude artifact you can play with to try this yourself: https://claude.ai/public/artifacts/402f2670-5f48-4d76-96df-8...
You can play with how strong that ("10% per year") prior belief is and see how it affects what the odds are today.
I think the way you are wording this question ("We can test this by going back to 1945 and running forward again?") is an attempt to make it seem "obviously wrong".
Bayesian predictions deal exactly with this type of scenario, where you start with a prior estimate ("Post World War 2, some people had the odds per year at 10%") and then as new information comes along ("It is now 1946. Did we use nuclear weapons again?"... It is now 1956. Did we use nuclear weapons again?") we update our model to try to make the future prediction more accurate.
https://www.stat.berkeley.edu/~aldous/134/lecture4.pdf has example of its use in exactly these kinds of "impossible to rewind" situations. Unfortunately it doesn't have the worked solutions.
https://math.mit.edu/~dav/05.dir/class11-prep.pdf is pretty good because it shows how updating the model with new data changes the odds.
> Repeatedly, in a reproducible way, for events in the arrow of time? We can test this by going back to 1945 and running forward again?
This is a frequentist mental model - all well and good, but frequentism and Bayesianism are different schools of statistics. Where frequentism asks the question, "if I keep drawing samples from this distribution, what does the histogram converge to?" Bayesianism asks the question, "given my prior understanding and a new piece of evidence (a new sample), how should I adjust my hypothesis about what distribution it is I am sampling from?". (That is really boiled down, and the frequentist part is maybe even butchered.)
Among other applications this enables us to estimate a distribution for which we have a tiny number of samples. A problem I'm interested in is called the Doomsday Argument, which estimates how long humanity will survive using your birth order (the number of humans born before you) and the anthropic principle (we assume you were not born unusually early or unusually late but closer to the mode); interestingly, everything you observe in the universe is already factored into this measurement, so you can't ever get a second sample. Obviously the opportunity for error with 1 measurement is huge, but you can come up with a number and it isn't arbitrary, it is a real estimate.
Similarly, we only have about 80 samples of years in which it was possible to have a nuclear exchange, so a fairly small sample size, but we can still get a noisey estimate. But I haven't read On The Edge yet, so I don't know exactly what Silver does here.
>> This is kind of the point being made.
> Was it?
I think they meant that all of the solutions people invented to prevent nuclear war and which commentators failed to anticipate is reflected within the true probability distribution and within our dataset. So it is captured in our estimate, to the best of our abilities and given the limited data we have.
Well, there was a (now under public domain) movie which predicted WW2 bombings.
https://publicdomainmovie.net/movie/things-to-come-1
On nukes, "The World Set Free" from HG Wells predicted nuclear weapons:
https://www.gutenberg.org/ebooks/1059
Also:
https://gutenberg.net.au/ebooks03/0301391h.html from 1933
It is cool. I find the idea of trying to understand whether these types of models can come up with things like General relativity, or maybe some results really interesting.
If anyone was wondering ... it's racist
Unsurprisingly the texts written up until that time were dominated by such individuals which is tragic for LLM training if you think about it.
The voiceless groups or fringe opinions which we take as normative today do not appear.
Does this encourage us to write in the present such that we influence the models in perpetuity?
Voiceless groups do not appear in the training data? How could they, they are voiceless. You think the voiceless people are represented in todays training data? They cannot they are voiceless.
Nothing tragic about using data from a time period.
Common words used in 1900s are labeled racist now. I doubt anyone was wondering if they filtered those words for modern safe wordx.
I'd be more worried if words from that era were fully aligned with present day notions of morality. Wouldn't that indicate a certain stagnation & lack of progress?
Let us hope, 100 years from now, there will be people who look back unkindly on us.