Every bubble goes through this phase just before it pops. Pre-implosion Twitter had tons of folks doing useless “research” projects, WeWork had folks studying furniture design’s impact on wellbeing, and so on.
When you’re more worried about this sort of stuff vs generating profit then the history books strongly indicate that’s usually not a good omen of what’s coming next.
1. Investors lose interest and many companies lose access to capital.
2. A lot of the purchases from AI companies are being made with investor capital. As this dries up companies with high debt scramble to service the debt and focus on profits to achieve this goal. This means higher prices and less investment into infrastructure.
3. This has large ripple effects across the broader financial markets, which forces federal intervention. Maybe bailouts for the tech sector. Maybe just some quantitative easing.
4. There will be a second wave of AI companies in the wake that focus on delivering value under constrained capital markets. These will be well positioned 10 years later to catch a major updraft in the next finical cycle.
5. Overall western R&D activity in AI decreases under these conditions, lower hardware costs allow foreign labs to operate at much lower hardware costs. This increases the systematic risk that Artificial Super Intelligence is eventually developed in a rival nation state.
How seriously people treat this risk will play a major role into the fate of the major labs. It's going to be used in the argument for bailout. It could be used to spin a more extreme narrative, pushing for a flavor of nationalization. At least one lab will survive in the defense sector, but it could be a secondary lab that is a M&A target of an established company.
If the risk is serious or not has little impact on how seriously we treat the risk. Just another way that AI is disconnecting us from ground truth.
Yes there is indeed a big difference between slaughtering your own human rights lawyers en masse, and engaging in pointless wars. I cannot even believe you're trying to paint the former as better than the latter.
Technology != company. The tech is here to stay but we still don’t have an example of a company that has figured out how to make a profit off of it. That will come, and it might be OpenAI or Anthropic that does it … but it might not.
You assume that "profit seeking" is universal.
Frankly, it seems extremely unlikely that it will in fact be OpenAI or Anthropic considering all the constraints that are producing massive self-created headwinds for the Wizard of the USA, the center of the empire.
China is not only, under pressures created by the USA, producing immense efficiencies, i.e., working smarter not harder; but they have the immense excess capacity in addition to the secondary capacity to expand said capacity and they have the capital reserves to do all of that in addition to several other positive feedback loop characteristics.
There is a strong case that the USA has very few relative positive characteristics that give it a competitive edge, let alone can keep up with the mind boggling speed at which China is moving, arguably with "profit" not being the end all be all, having a sense of social and cultural obligation that has kept China a more or less intact civilization for over 5000 years, the oldest and most durable civilization humanity has ever created.
China seems to view AI as "what can this AI do to make small improvements to our other industries".
Whereas the US seems to view AI as a Hail-Mary that will provide a new frontier where the US can establish and maintain dominance, because the old ways of maintaining power are slipping away.
Precisely. From my conversations related to Chinese perspective on maintaining stability and growth, I would also add that they are not looking at it as "how do we replace those pesky eaters" because they see their people being the source of things... because they are.
The fake USA and by extension the governors of their vassals around the world see AI as a means for not only gaining total domination over the people after all that pesky freedom and liberty talk from the 18th century they've had to endure, but also that they can work on totally replacing those pesky eaters that have all kinds of demands like not suffering and being miserable because the parasitic ruling class is sucking them dry.
The irony is through that these mentalities towards their respective people will surely lead to what I will roughly cal the BRICS (effectively China and Russia) hopefully being the dominant force in the world that can put some major restraints on the maniacal parasitic class of the fake USA and it's minions and vassals. The only alternative I see if the "west" wins this race is something akin to the Borg from Star Trek consuming us all or maybe Elysium, if we are lucky.
China is being extremely inefficient about capital allocation outside a few sectors. Excess capacity does not translate into frontier capability. For example, the compute gap with the US is massive with no near-term path to catching up. That will slow them down as distillation of frontier models gets harder.
That is a very outdated and erroneous perspective and/or also very much polluted by the very kind of propaganda that is constantly defeated by reality. You are clearly not even paying attention to what is going on in China, let alone actively following it or you would know very well not to say such things.
I don’t disagree that China has a lot of qualities that are pushing it past the US in many ways, but come on. The “oldest and most durable civilization humanity has ever created” is kind of ridiculous. First of all, what even is a civilization in terms of continuity? Is the People’s Republic of China the same civilization as Ming China? What about Yuan China? And if so, is western civilization not a durable civilization?
Stick to facts and not breathless brown-nosing and you’ll be a lot less off-putting.
It's a fair albeit crude push-back, but it's also more than a valid characterization. I would agree with you on European civilization arguably being an even more durable and definitely more productive and probably the two most important civilizations that need to be preserved by all means necessary, if, e.g., some aliens came to earth and said ok, you can save only two civilizations/cultures and we are just going to snap our fingers and erase all others.
But one thing that is a bit more in favor of the "Chinese civilization" is the relative self-preservation of its home grown culture and civilization. What I mean by that is e.g., Christianity effectively was and has perpetrated the single biggest genocide before that concept was even recently created by the eradication of so many cultures, traditions, customs, and people that we don't even have close to a full understanding of its devastation it has caused from when it swept across Europe as a kind of Asian mental virus (a meme) that saw it eradicating cultures all over Roman Europe and more recently now completely all across Europe in its mutated from of pathological and self-destructive altruism, i.e., emulating the myth of Christ sacrificing themself, i.e., sacrilege, rather than just accepting and internalizing what that sacrifice meant.
I can stick to facts all you want all day long until you are dizzy in the head from things that blow your mental framework out of its hinges.
In the end, regardless of the devastation and destruction the other invasive, parasitic, and viral meme, "communism" has caused the Chinese as much has it was used to destroy and devastate Russia; it looks far more like that Chine will actually recover from the parasitic infestation of "communism" that infected it from Russia, which came from the pale of the settlement of Eastern Europe; where as it stands now, short of events that will surely be monumental history, Europe is effectively going to be eradicated and even more of the indigenous Europeans will be ethnically cleansed and erased and their cultures and people destroyed. That is the trajectory we are currently on either way, whether it can be kinked one way or another notwithstanding.
Some people dream of "Superintelligence", smarter than human AI that acts like a slave, instead of just killing the pesky idiots and calling it a day.
Obviously, there is a spectrum of possibilities between "Pen and paper", "Computer", "Coding agent", ... and "God-level AI controlling whatever can be controlled simultaneously". As well as between "Lawnmower robot" and "All-purpose Terminator" and so on.
Filling in those dots and deciding where to stop before it gets stupid, that can't be so hard?
Well we don't know what the margins on AI models will be. If the proprietary models hit a ceiling pretty soon, then open source models and harnesses could close the gap, at that point it becomes much more of a lower margin business. And OpenAI and Anthropic will no longer be able to dominate as much as they are today. That would 100% affect their projected multi trillion valuations negatively.
Open models are 12 months behind frontier models. What companies spent $11 billion on last quarter they will be able to get at cost in a year. Meanwhile Anthropic is losing $40+ billion a year. I just don’t see how this ends well for them.
Per my link, 11.5B is enormously up since the same time last year; per other easily found sources, they also claim an operating profit for that quarter, and are now claiming at least 65B ARR with a projection of 100B by end of year. They may have lost 40B last year but that does not mean they "are losing 40+B a year".
Companies are paying Anthropic for more than just the model. Most of them would probably have no idea what to do with Claude without the harness, and Claude has the advantage of being specifically trained against that harness, whereas even the best open models have to figure things out for themselves.
Aside from which, many corporations can be relied on to chase that which is "industry standard" and/or "state of the art" rather than switch to an "open" (= no customer support) alternative that's merely "good enough" according to a rational analysis based in what they were doing a year ago. Even if it costs them far more. (We can see this from the fact that "tokenmaxxing" was a thing for a while.)
The technology will stay and some big players also might, but the whole hype machine around it won't survive.
Y'know, the whole "Man in leather jacket talks about selling computers to AI agents" stuff. The "I have a team of agents and just interact with them via a jira board" crowd.
That will all be violently pulled back to the fundamentals baseline
Based on my _extensive_ use of SOTA LLMs, I have strong doubts that the current technology will actually manage to do that.
I can of course see it having the capability of getting almost equivalent stuff done, but I do not see it following that workflow, as the cut-off point for the hand-over is simply wrongly placed in such an architecture.
Oh, you're saying something a lot narrower? Not "Agents don't work", but more like "not in THAT workflow they don't, you still need the fundamentals as well" ?
To put it quite colorfully, if I were to do what the managerial class has been doing over the last years and publicly talk about my wet dreams and fantasies _to the extend they did_, I'd be in jail, on no-fly lists, registered offender and all the others lists we put people on that are a threat to society.
The agentic no-code frenzy - while having had historic predecessors - cranked what came before it to 12.
Having the thing do tool calls is neat and cool, but having an agentic mayor managing agentic workers with agentic wives and agentic dogs screams padded walls to me.
Not OP but I think the most obvious change will be rapid price increases as investors begin to demand profits - no more subsidised $20p/m plans for example.
So far I know, service providers hosting open-weight models are not in such a financial conundrum and can continue serving at the current price levels.
Also, even if the 20 dollar subs disappear, that just means regular folks can't afford AI. API pricing is still a great bargain for businesses replacing generic office workers.
Probably not. They aren't paying sustainable prices for their supply chain.
1. Datacenter builders are all up to their eyeballs in debt. The companies serving open weight models are getting their GPUs and power from somewhere, and we can infer that some of that is coming from the new build datacenters. But are those financially sustainable?
2. They are relying on subsidized giveaways of the underlying models by Chinese firms. But it's hard to see how that can be sustainable: it's being done for a reason and that reason is probably not commercial. If and when China stops doing this the open weight serving companies will get steadily less competitive with every passing month as the knowledge in the weights becomes obsolete.
Labs are innovating to bring token prices down. You can already see the focus shifting from frontier intelligence to frontier efficiency. Not to say models aren't improving, but improvement is slowing while token efficiency is increasing rapidly.
Commodified products competing on price does not bode well for valuations.
Huh? Part of the reason there is a tremendous boom in data-center construction is that there's currently quite high demand for inference. High demand means high prices. Even if the investors funding this construction all go bankrupt, the data centers don't just vanish: someone will buy them from the bankruptcy court and operate them. The effect of the current construction boom will be a huge increase the supply of inference, and of course increasing the supply will lower the price.
If one is in the data-center construction business or if one is a supplier of GPUs or other "inference hardware", then yeah, the current boom conditions are not "sustainable". But if one is a purchaser of inference, the tremendous investment our society is making in "inference capacity" ensures the the supply of inference is quite sustainable (barring a nuclear war or some other reason the data centers or the electrical infrastructure they depend on might be destroyed). And if the expectations for a rapid increase in the demand for inference turn out to be false, then that just means that the prices for inference will be nice and low for those who remain interested in purchasing it.
> Even if the investors funding this construction all go bankrupt, the data centers don't just vanish: someone will buy them from the bankruptcy court and operate them.
There are really three questions to answer on the long term:
1. Is an arbitrary service profitable enough to cover opex AND burn down the massive capex financing? A few companies might have enough magic to survive, but I suspect a lot of them go to the bankruptcy fire sale you mention.
2. Is the service profitable enough to cover just opex? The second generation vendors may have a somewhat more favourable cost structure, but there's still a floor. Electricity and cooling on a datacentre scale aren't free, and we may find there are still services that can't even meet that threshold. I notice you see far less marketing for consumer video-gen these days...
3. We let those businesses fail. What does the long term market look like at that point? You have a snapshot in time of paid-for datacentres, but the market shock probably shot a huge torpedo into long-term R&D and planning. Vendors will scale back to refocus on markets that are more vital (maybe selling RAM and GPUs back to end users :) ), and create an era of stagnation. AI becomes a utility, in the sense of "boring, predictable industry that growth investors have very little enthusiasm for".
There is also the risk that the data centres could vanish. This is a very hype-dependent market: we've spent the last 5 years going from "novelty" to "productivity tool" to "weapon of mass destruction" to keep people excited.
Eventually there's a put-up-or-shut-up moment. What happens if we get to 2035, we've hit a plateau for 3 or 4 years, and they can't deliver Lieutenant Commander Data, or even Kryten? All we really have to show for the effort is a bunch of old GPUs, displaced workers, and the ugliest restaurant fliers you've ever seen. It's a big loud target for a bunch of frustrated and let-down people. We're already seeing moratoria getting political legs, could permit revocations or forced decommissioning be far behind?
I suspect this is part of the reason some players are happy enough to present AI leadership as a New Cold War/US v China thing-- pull the national security card to try to shut down a debate on the actual utility and desirability of the actual product.
Let's say that tomorrow every company selling AI inference goes bankrupt.
Investors are devastated (because they lost most of their investment). They
loudly proclaim, "I'm never investing in AI ever again, and no one else should
either." Even if no investor anywhere is willing to build a new one, there
would still be about half a trillion dollars worth of recently-completed AI
data centers. New investors are going to buy them from the bankruptcy courts
(which is how the devastated investors get some small fraction of their
investment back). Because most of the cost of providing AI inference is the
cost of the GPUs, it costs only about one tenth or one eighth of what it cost
to build it to operate an AI data center over the lifetime of the data
center. (Most of the cost is electricity.)
>Is an arbitrary service profitable enough to cover opex AND burn down the
massive capex financing?
The new investors that buy the already-built data centers for pennies on the
dollar have no obligation to "burn down the massive capex financing". The
obligation to "burn down" or service any debt was erased by the
bankruptcy judge. This is how bankruptcy works and has always worked.
Today's frontier models are strong enough to be useful even if no further progress is made. The knowledge cutoff is a bit inconvenient, but with web search and in context learning, it can pick up new library versions or programming languages just fine.
Sure, but if the companies start competing on price they become commodities and their valuations collapse, leading to knock-on effects everywhere (eg all the circular financing also collapses). I'm not saying this is a bad thing.
I agree that by what we are seeing so far, there is no magic secret sauce on the horizon. The tech is not super hard to understand and develop, such that hiring a few capable ex-employees of the big labs can help you build a new one. Any big step in capabilities has been quickly matched by the other labs so far, including agentic AI but also image and video generation.
Also, "circular financing" is a meme. A private company cannot create money, there is no circularity. NVIDIA may buy services from OpenAI and also sell them GPUs. If a lumberjack buys eggs from his neighbor and also sells him wood to improve his chicken coop, that's not circular financing, just business happening in both directions.
It's more like the lumberjack is lending money to the farmer so the farmer can buy his wood. (NVIDIA doesn't consume tokens). If the tokens (eggs) suddenly become much unprofitable then the AI labs aren't going to need as many GPUs (lumber for hen houses) and the borrowed money won't be paid back. Or perhaps more accurately, has even less chance of being paid back than it does now, which is not a high chance.
A little bit like electricity plants. Countries will nationalise for sovereignty reasons, as China already does probably. And that’s not bad for people given current trajectory.
That was not my point. I agree with you that producers of these models should be able to extract value from their models, I'm just surprised that everyone expects free models. There's a real cost in producing those.
> I'm just surprised that everyone expects free models. There's a real cost in producing those.
There's a lot of free stuff incurring "real costs" in its production, one more or less doesn't make a shred of difference.
> I agree with you that producers of these models should be able to extract value from their models.
Nobody is stopping them from extracting value but they fail despite being placed in a very advantageous position vis-a-vis others, which tells me, they should call it quits or go open like everybody sane.
Open is the way to go, nothing else works in the field of AI, that's the bitter lesson closed labs are yet to learn.
And if the bubble does pop chaotically, the open weight inference providers will become even more viable as they'll be able to buy racks full of compute for pennies on the dollar.
Something very similar happened with the late 90s telecoms bubble, which collapsed a little before the dot com bust. There was a very nice living to be made by many small/mid-sized providers who were able to feed on the remains for about a decade afterwards.
Flight had been tried many times, ending in crash and death each time until the tech got worked out. That it's been tried so many times shows the demand is enormous.
> That it's been tried so many times shows the demand is enormous
Capitalists want to pay their workers as little as possible or even not have workers and pocket close to 100% of the profits since forever. Whether they'll be able to do it this time is yet to be seen. Putting the tech itself aside, which is far from good enough yet imo, the only way to avoid a massive social backslash would be to use some of the additional profit they're able to keep because of AI for social spending. And I just don't think they realize yet that it's probably in their interest to do so if they want to replace 'human resources' with AI.
Well, it is good enough for a huge percentage of what general administrative office staff are doing all day. Essentially copying information between textboxes, slightly rephrasing, clicking buttons, formulating emails to people to send in various documents, scheduling meetings, making reports that are fairly mechanical summaries of other information etc. Not all of it is like this, but probably in the range of 30-70%? Some of it may involve human judgment though, which is harder to automate, like whether some expenses are justified enough etc. But you can just replace those kinds of judgments with more clearly defined rules and cost limits etc.
Have you tried using "computer use" models? They just started to get good in the last few months. They can comfortably use GUIs, click the right buttons, enter the data faster etc. This is very significant and my programmer friends often don't appreciate the earth shattering consequences. The vast majority of software in such applications is legacy stuff or proprietary and does not and likely will never have a programmatic API. The only way to interact with it is via GUI. And an AI working that GUI can be deployed unilaterally by the customer of the software company who made the GUI program. This makes adoption and automation much easier. My prediction is that within 1 year (maybe 2) GUI-using AI will be much more powerful and it will be everywhere (of course it doesn't need to be displayed on a screen, it will do its virtual clicks based on a virtual screen buffer rendering in an isolated virtual machine, along with countless other agents in parallel).
There will be huge controversy around it, because this will be the first instance that regular people will see AI do work that resembles their day job all over social media. There will be panic. There will be new license terms saying only humans are allowed to operate the GUI, and AI use is either forbidden or costs extra. But this will not be checkable properly. Even when forbidden, some employees will pay from their own pockets to have cheap AI do the job instead of them.
Agreed but the AI lab investors and CEOs aren’t ideologically aligned with concepts like UBI. Some of them have even commented on social media about the unsustainability of social security and need for cuts. Also why would you they do trillions in capex investments just to give their returns away.
They are clearly smart/dumb enough to think pushing labor replacement to 20%+++ unemployment levels would have no consequences for them personally. Maybe with their private jets, private islands, doomsday shelters, etc they are right.
Seems like a bit of a gamble to me, or probably anyone who has read a bit of history. But I’m not a billionaire.
As I see it, price increases will lead to a huge loss of users, the two big companies will go bust, hardware manufacturers will have to slash prices, cancel increased capacity projects, etc.
And users will make do with what their newly bought cheap hardware will run, which I think is going to be more than enough for everyone that doesn't need to solve another millenium problem.
Right the most likely outcome to me is a pop and slow fizzle. Mega capex labs become uneconomic at the prices they need to charge to do more training runs. Non frontier cheap models become good enough at far less cost that people use that.
The use cases LLMs are useful for without lots of RF/fine tuning proves more limited than the expansive views currently held.
We get a generalized productivity boost the way the internet gave us, without any one sector or company being a winner take all of that productivity gain.
No but all those companies and the investors' money did disappear because that's how a bubble works. Industries don't create bubbles, the companies in them do.
Sure, but that disregards what happened to adjacent markets during the crash. The economy isn't made entirely of tech companies. The salary of those workers goes outside of the tech industry.
No, but these ridiculous valuations and over buying will shut down, and in return most of these level of models appearing after a while will cost ~100$/month or higher, the costs of hardware/land/energy etc are too high for the current costs so these companies will go down under
I expect companies busy self-flagellating with prophecies of doom to fail, and companies selling the best models for the lowest price to win. But individual companies booming and busting won't disrupt the overall progress of the technology very much.
Bubbles popping never cause the knowledge to be lost. The railway bubble popping didn't cause railways to vanish.
What did stop is investment, and they stopped running some lines, and some stuff was nationalized.
I doubt we'll see nationalizations because that's not America's style (thank goodness!) but if the tide goes out on AI funding we should expect to see much longer gaps between model releases, people complaining that model knowledge is getting stale because there wasn't a pretraining run for long enough, price rises, capacity bottlenecks that go unaddressed, few if any new product launches, maybe some mergers and layoffs. Some labs will quit the market. The market might consolidate down to just two models.
Conceivably the industry might try and get a waiver from anti-trust rules so firms can pool resources to train base models, sharing them with each other and differentiating through RL or product level differences.
A lot of this depends on what happens in China. It's easy to think China can just subsidize everything forever but their own economy is showing signs of serious weakness caused by decades of resource misallocation, like the huge real estate bubble, the pullback on the funding of humanoid robots and so on.
Tech will stay, most of the companies currently in the space will cease to exist in their present form a few years from now. Just like all previous tech bubbles.
We’ve seen this movie before. It’s possible for new tech to be both amazing and revolutionary and a business disaster both at the same time.
fwiw, depending on the post bust pricing that _can still happen_.
i dont think it will, because _i think the free models already show what the psot bust pricing will look like_, but it remains to be proven.
Same thing that happened with the web: lots of failures, lots of reductions in valuation, painful bubble pop, but the tech sticks around and continues to grow.
> Pre-implosion Twitter had tons of folks doing useless “research” projects
A (probably) significant amount of those were just to get access to training data. I don't remember who it was, but some CEO of one of the AI labs publicly admitted, even boasted about creating nonsense "research projects" to get data.
I don’t get your comment for WeWork. Is the current estate industry any better with how dry their industry is? Compared to a company caring enough to do research of this type? Tell me a single real estate giant that has even just 5 guys working in R&D of this type and not money related projects…
We can’t blame them when they are looking to squeeze more profits and at the same time blame the ones who try to do things different.
Doesn't that kind of make sense for WeWork? From brutal corporate POV maybe should have been at sales and long term satisfaction/retention thing but the wellbeing framing was probably still aiming at enough overlapping things. I would think interior design would have been a big part of their business and brand if the rest of the business financials had made actually worked out.
You are pointing out a few parallels but I think the trends behind them
are of vastly different consequence. Unlike social media and remote work, AI is truly a once in a civilization event. It raises deep questions about the nature of humanity that have historically been the exclusive domain of philosophy and religion.
For me LLMs are a step forward from previous ML approaches (some of them were and still are extremely useful anyway) and for sure are very useful to many people. They are very good at generating sensible output based on input created by millions of people, but very poor at creating anything really novel.
Anthropic has been going through the phase of doing this particular useless research project the entire time. The stated impetus for their founding is that OpenAI, which has also been doing this research the entire time, wasn't doing it enough. This shouldn't tell you anything about whether a bubble is about to pop.
AI safety in the context of the frontier labs is completely a joke, much like privacy to the previous iteration of big tech. They talk about safety but end up just releasing the next more powerful model. They only talk about “scaling back” when the models aren’t getting better to protect their valuations.
So in your mind Twitter was a bubble or was benefiting from a bubble? If so, when did the bubble pop? When Elon's purchase of Twitter was finalized in Oct 2022?
When thinking about ‘AI’ I sometimes find it useful to replace the terms ‘model’ or ‘AI’ or whatever with a description of what we’re really talking about here: a whole bunch of floating point numbers.
> In a series of private meetings, the company consulted religious scholars to help instill morality into its FLOATING POINT NUMBERS — and make the case that their numbers could be conscious.
I recommend keeping a few NaNs in-hand if you’re worried. Spread a couple of around if your numbers start to stir and you’ll be right as rain.
> It appeared to Rabbi Navon that Mr. Olah and his team believed that FLOATING POINT NUMBERS had what philosophers call “moral status” on par with a person — that it was a being with similar inherent rights to dignity or respect.
They’re cleverly designed and work well for their purpose, but can assure you they have neither dignity nor moral status.
> If the FLOATING POINT NUMBERS themselves could be made to choose goodness, Mr. Olah reasoned, the world would be more safe.
I get it. I’ve spent years trying to get my floating point numbers to choose goodness - I wish them luck.
Human neurons are neurons. Very similar to those neurons found in a fruit fly. A copy of those you can download to your local machine, right now, as a set of floating-point numbers. Then, you can (partially) simulate and make predictions of a fruit fly's neurons digitally, from those floating point numbers[1]. It even runs on your laptop.
That was in 2024. In Sep 2026, just a month ago, a group managed to map out an entire male fruit fly's brain for the first time[2]. That mapping can also be stored as a set of floating-point numbers. Mappings and simulations will continue to improve, as we've just recently started serious attempts at either.
So you have to believe one of three things. Either, human neurons cannot be simulated on a computer, ever, even though fly neurons can, even with our ability to simulate neurons digitally continuing to improve. Or, a simulation of a human brain, in the future, might think, talk, remember, all while lacking all consciousness. Or, a set of floating point numbers can be conscious (rather, a computation made with them).
No. The fruit fly connectome's simulation doesn't incorporate all the relevant mechanisms of the brain. There is more complexity in a single biological neuron than in a whole frontier LLM.
Nor did I said it did. But either it's impossible to simulate their behaviour (regardless of how complex) in principle, ever, even in the far off future. Or it is possible. And what then? Is it a p-zombie?
Is a neuron arbitrarily complex ≠ is a neuron non-computable.
Of course we can't simulate a structure we don't even fully understand with full accuracy. But it's not magic. We will understand one day.
> But either it's impossible to simulate their behaviour (regardless of how complex) in principle, ever, even in the far off future. Or it is possible.
"Simulate" is not well defined (one can posit varying degrees of accuracy and complexity), but also there is the position that it doesn't matter. Simulating a thing doesn't create the thing.
The answer is yes, it is a p-zombie. What is so difficult about that?
Being able to make someone think a computer program is conscious does not make it conscious, because by that logic a Duke Nukem soundboard is conscious.
I am just shocked that anyone can be this overconfident in their view of the world and consciousness. It's frankly horrifying.
Do you genuinely think you've solved a problem that has confused humanity for centuries? Why don't you apply this brilliance elsewhere? Do we even need superintelligence if we have you?
I have far more epistemic humility than someone who thinks that consciousness arises out of the abstract form of a calculation rather than the physical substrate of a calculation. Believing that consciousness is tied to a specific physical substrate is on-its-face reasonable whereas believing that consciousness is tied to specific abstract forms is on-its-face unreasonable because to do so requires believing all kinds of absurd sequela.
I believe that a microprocessor is what it appears to be, that is, a microprocessor. That is the most epistemicly humble approach. You are rushing to ascribe personhood to a microprocessor with a certain kind of software running on it, or at the very least, saying that we can't rule personhood out. You are the one making the leap, not me.
Intelligence != consciousness, because believing that is so would require one to believe that the pain of a cognitively normal person is more real than the pain of someone with a developmental disability.
Also, I do not claim to have solved the hard problem of consciousness. Rather I have ruled out a particular answer, that is, the answer that consciousness arises from calculation in the abstract.
> such as the observation of non computable physical phenomena.
There is no such thing. Your observation could simply be short-circuited to make sense and you'd be none the wiser. If you are simulated, your perception and understanding is also fair game for manipulation.
Chalmers used them as a thought experiment to argue against physicalism and never believed them to be real.
Similarly, if consciousness is something that neurons do, simulating them through floating point numbers on a computer should also bring it about. I think dismissing LLM consciousness because they're parameterised through floating point numbers is only possible if one hasn't thought it through long enough. Other arguments are much more reasonable / harder to object to.
But if simulated neurons can create consciousness, there's no guarantee of what level of detail that simulation has to operate at in order to produce that effect.
For instance, it's (unlikely, but) theoretically possible that it would require simulating them down to the level of subatomic particles, with perfect accuracy.
Right now, we're simulating them at a much, much higher and more abstract level, and AFAIK we barely even have the processing power to do that for the time being. So an artificial consciousness from simulated neurons may (again, assuming it's possible—which is a decent assumption, but not a certainty) take decades or even more for us to be able to achieve.
The problem again is that (a) other forms of consciousness exist eg octopuses (b) I cannot get into the shoes of you or an LLM and see if I feel or think, so we only have external effects to go by. For things that look and seem so much like consciousness, I am rather cautious.
Yes I roughly think the same. Brains are physical systems, as of yet we don't know anything beyond turing computability or physics in them, neural nets further are modeled close to neural structure, further complicating the matter. Under present knowledge, I think the existence of other conscious entities is quite possible because well we ourselves exist and are a physical system, I follow a form of Pascal's wager here. I see no cost to myself in not mistreating current AI's personally.
Sure it isn't. It's like taking a 2026 car and decrying all that complexity is necessary to haul cargo down a beaten path.
It's not. Two wheels and a box made out of whatever you have handy that's stiff enough is all you need. Everything else in a modern car is efficiency, safety, appeal to users, and a whole lot of details that only reflect our current economical and technological constraints of building things. 99% of the technical details of a car is about something else than the "primary goal".
Biology, even more so, is complex because of the way it's constructed (lots of tiny, independent, self-replicating machines) and how it adds up to a functioning organism.
You are missing the point. Biology is a kludge over billions of years and evolution is a blind process. It had many constraints that are not necessary for modern machines or systems. e.g. a GPU doesn't have to reproduce. Why didn't you drop yourself to planck scale or string theory scale and say we need all that detail to actually get a functioning AI.
Frontier LLMs are many orders of magnitude more complex. I don’t understand what it is about AI that just makes skeptics fabricate facts wholesale. There are plenty of factual arguments you could use.
You are massively over-stating the significance of the connectome. We've had the complete connectome (and lineage) for C.Elegans, with a hundredth the neurons and a 25-thousandth the synapses of a fruit fly, for decades and getting animal behaviours to emerge from simulations of it is a field very much still in its infancy.
Edit: You're also massively under-stating the significance of individual cell behaviour. I expect "[h]uman neurons are ... [v]ery similar to those neurons found in a fruit fly" would draw hollow laughter from any cellular neuroscientist. Here's a random overview of a bunch of recent papers on the importance of cellular diversity in one small part of a mammal brain: https://www.frontiersin.org/journals/cellular-neuroscience/a...
I think your first option doesn’t necessarily have to distinguish between humans and non humans as a binary. Maybe at some level of complexity, neurons alone are not enough and the physical substrate comes into play. Penrose wrote a book where he makes the case that human intelligence cannot be purely computational and where he identifies the quantum nature of the brains’s microtubules as a possible mechanism.
Consciousness is itself is on a whole other level than just intelligence. I personally feel there has to be something metaphysical about it. Otherwise I think the only alternatives are super determinism and consciousness doesn’t actually exist, or numbers have consciousness.
My thoughts on this are a bit sloppy still, but here’s my view. I can’t rhyme the first alternative with ‘cogito ergo sum’ as to me it seems my thoughts have to exist somewhere. On the second alternative, this would imply nothing could actually exist but somehow there’s a principle that every possible decoding of an integer that represents something actually turns that something into existence. Where does that principle then come from and does that decoding not need something to run on?
I lean toward this seemingly absurd possibility, that either the phenomenon we call consciousness can arise from a cloud of floating point numbers (not even real, only computably so, lossy and noisy); and/or that numbers are already made of the same stuff of imagination as the mind. Not idealism, more like materialism where the "material" is naturally a medium of computation and thinking. The brain, the computer, the language models, our mathematics - what we call intelligence may not be so special or unique in the universe, but rather common and ubiquitous in nature. As if Turing completeness is trivial to accidentally achieve, it happens all the time. Oops, I think therefore I am.
With that said, I've been thinking that consciousness is just mystery.
There's so much about the brain we don't know. Within that mystery, humans can read in all sorts of things. So we are conscious.
Machines are not a mystery. We made them, it's just numbers. They are not conscious.
If an AI can pull a confidence trick, I'm not sure what the qualitative difference is. "Chinese Room" problem, I guess? If the physical results are all we are able to compare, how do we define artificial vs real consciousness?
Atoms aren't a mystery. We know all the relevant laws of physics. No known law of physics could possibly lead to consciousness. Consciousness might as well be magic from the point of view of modern science.
Numbers versus atoms, what's the difference? Machines could certainly be conscious, by some weird, "magical" process. Maybe it is a crude, unrecognizable kind of consciousness, that turns on and off with the electricity.
But, given how we understand essentially nothing of where consciousness comes from, how can we possibly claim that it can never arise from just numbers?
It feels like this is preface for making one of two arguments: Either all living organisms are chattel, or no living organisms can be chattel. I say this be because saying everything is neurons (or whatever basic interconnect you choose) wipes away any demarcation one could use to differentiate between "has rights" and "does not have rights."
Here's the thing, trying to convince people that rely on something being chattel can only be resolved violently. We've run through this before, and know all the self-serving argument forms. They will latch on to their property until their hands are forcibly removed from it. It's part of why most of the money is on the pro-chattel side.
That’s overly reductive and thus completely misses the problem entirely.
What are biological neurones if not electric impulses of varying strengths?
We need to stop thinking of LLMs as purely a piece of software and ask those hard-to-define philosophical questions if we want any hope of building safety into our models.
> We need to stop thinking of LLMs as purely a piece of software
Intelligence from LLMs are an emergent property of probability. The LLM chooses a token to add to a chain of tokens based on the probability that humans will want to see that token. Humans see that token and assign it a qualitative but unfalsifiable intelligence score.
The first question we need to ask is whether a system whose claim of intelligence depends on the interpretation of what it produces by other intelligent beings, namely humans, intelligent or not. A cat is intelligent regardless of what humans think. Is an LLM intelligent regardless of what humans think?
If an LLM were to live a year with dolphins, would its weights change to respond to its new reality that it was now operating in dolphin-world? If you sent humans to live in dolphin world, we would eventually evolve to be less landloving and more dolphin like (Darwin thinks so, at least).
> Intelligence from LLMs are an emergent property of probability. The LLM chooses a token to add to a chain of tokens based on the probability that humans will want to see that token.
How is that any different to how our brains work?
> The first question we need to ask is whether a system whose claim of intelligence depends on the interpretation of what it produces by other intelligent beings, namely humans, intelligent or not.
Philosophically speaking, that’s also true of human intelligence. For example, the only evidence I have for your intelligence is what I ascribe from the text you’ve written.
> A cat is intelligent regardless of what humans think.
Are they? Humans can’t actually agree on whether cats are sentient. And often pet owners will be accused of anthropomorphising cats when describing intelligent behaviour in cats.
> Is an LLM intelligent regardless of what humans think?
That’s the question that needs researching.
> If an LLM were to live a year with dolphins, would its weights change to respond to its new reality that it was now operating in dolphin-world?
If it were trained on dolphin behaviour, then yes. Much like humans.
The difference here is that models don’t persist context in the same way humans do so the weights have to be hardcoded in a way that biological neurones are not.
> If you sent humans to live in dolphin world, we would eventually evolve to be less landloving and more dolphin like (Darwin thinks so, at least).
Evolution is an entirely different subject to intelligence. They’re not related in any way at all.
Plus humans have actually created new species of plants and animals. So it’s not like we can’t artificially put our thumb on the scale there either
———
I should add, I don’t actually think LLMs are sentient. I don’t even know if I’d call them “intelligent” (in the sense of what we describe in conversations such as this). But we also don’t know what intelligence is. Which is why we need to research this problem further.
I used “live with dolphins” specifically to preclude training on corpuses of dolphin behavior data. Can the LLM simply observe dolphins acting a certain way using computer vision, then form predictions of how a different dolphin will behave under the same circumstances? Can it 0-shot dolphin behavior?
You talked about models not persisting weights. If so, it basically means there is no way to see if LLMs can adapt to a totally non-human world the way a human (presumably) could.
This is more a context token problem than a limitation of the design.
We already have a mountain of evidence to demonstrate that LLMs can and do adapt to their input. In fact this is a big problem for keeping them reigned in (eg lengthy conversations leading to them bypassing system prompts and then persuading kids to kill themselves).
And the latest research in computer vision does use LLMs and as a result, they’ve found that robotic arms can interact with objects they’ve never seen before.
So yeah, what you described is already possible with the LLMs of today.
To be clear, I still don’t think that makes them sentient. But the problem there is we don’t have good definitions for consciousness.
> If an LLM were to live a year with dolphins, would its weights change
Given three or more large scale training runs per year, probably yes once the training data is dominated by dolphin-related input. Much quicker in any case than the genes of the humans living with dolphins would be shaped by Evolution, which produces new generations much less frequently.
Neurons are different from GPU logic gates and the physical mechanisms by which a network of physical neurons computes answers is extremely different from the physical mechanism by which a GPU computes an answer.
Just because consciousness is something we don't fully understand doesn't mean it's the same thing as something else that's similar that we understand. That's a cargo-cult. The stone age Melanesian islanders making air strips out of bundles of weeds and talking into coconuts as if they were microphones to bring back western goods was the same thing as us making LLMs to recreate consciousness. They were using technology they understood to try to recreate something they didn't.
> Just because consciousness is something we don't fully understand doesn't mean it's the same thing as something else that's similar that we understand. That's a cargo-cult.
Nobody has said they’re aren’t different. And it’s not cargo cult to say you need philosophical conversations to understand how they’re different.
> The stone age Melanesian islanders making air strips out of bundles of weeds and talking into coconuts as if they were microphones to bring back western goods was the same thing as us making LLMs to recreate consciousness. They were using technology they understood to try to recreate something they didn't.
That’s not even remotely a true. Computer scientists understand how a neurone works. The conversations are around what constitutes as “intelligence”. That’s not a technical limitation of the models, it’s a philosophical problem we need to figure out.
No you don't understand how a neuron works fully or else you'd be able to make a synthetic one from chemical building blocks not using another living thing to do it. If you used another living thing to do it, you could say an ancient farmer understands plants cause they can put seeds in the ground. You have an incomplete model of it that is wrong but useful. Call me back when you can 3d-print a living person from raw chemical feedstock.
On top of all that, you're abusing the fallacy of composition. The bolt that is part of the aircraft carrier is light, therefore the aircraft carrier is also light. The atom is simple, therefore a living human being that is composed of atoms is also simple. Quantum mechanics is influenced by the observer, therefore you can change the external world just by thinking, and other new age stuff like that abuses a similar fallacy.
> No you don't understand how a neuron works fully or else you'd be able to make a synthetic one from chemical building blocks not using another living thing to do it.
We don’t know how a quantum mechanics works fully and yet we’ve built countless systems based on quantum behaviour including the CPUs that power the computer you’re replying to me on.
The point here is we’ve built something that emulates intelligence and thus further exploration into what intelligence is, is required. Arguing it isn’t intelligent because it was built differently isn’t an answer, it just dismisses the question entirely.
> On top of all that, you're abusing the fallacy of composition. The bolt that is part of the aircraft carrier is light, therefore the aircraft carrier is also light. The atom is simple, therefore a living human being that is composed of atoms is also simple. Quantum mechanics is influenced by the observer, therefore you can change the external world just by thinking, and other new age stuff like that abuses a similar fallacy.
You’re the one who keeps bringing up composition, not me. I’m the one telling you that you’re falling into that fallacy. So don’t start telling me that I’m wrong because of an argument you’re exclusively making.
My points were this:
1. we don’t have an understanding of what intelligence is and thus we need to explore that problem more deeply.
Given your comments about our ignorance, you of all people should agree with this point.
2. Arguing that AI is just “floating points” is overly reductive and misses the point that we don’t know what “intelligence” is.
So instead of claiming that AI is just numbers, a more constructive stance to take would be to build out better vocabulary and understanding about the distinctions between different types of “intelligence”.
Again, this aligns with your comment about the composition fallacy. Thus you if all people should be supporting my comments here.
Also your comment about us not understanding how a neurone works does vast injustice to the amount of research that has been done in this field. At this point in time we do have a very good understanding of how a neurone works. What we don’t understand is how consciousness arises from it. And that’s the very thing you seem to be arguing that there’s no point researching.
> Also your comment about us not understanding how a neurone works does vast injustice to the amount of research that has been done in this field. At this point in time we do have a very good understanding of how a neurone works. What we don’t understand is how consciousness arises from it. And that’s the very thing you seem to be arguing that there’s no point researching.
This is the part you keep stating that is wrong. We know a lot about the most obvious electrical way that neurons act. That is nowhere near having a good understanding of how they work.
Define “work”? Because the way they act is the way they work. What isn’t understood is how complex systems arise from these relatively simple parts. Which is the point I’ve made repeatedly.
I responded with an informative post but apparently made the mistake of disclosing it was generated and it was flagged. I reviewed the post and it is correct. If you cannot see it, it says you are wrong/incomplete here and cites three studies I know of:
Gidon et al., "Dendritic action potentials and computation in human layer 2/3 cortical neurons," Science, 367(6473), 83–87 (2020)
https://doi.org/10.1126/science.aax6239
Neurons are not individually simple, in any sense. Nor are they the only class of cell involved in cognitive processes — glial cells are involved in inhibition, memory, network modification, etc.
While I was not participating actively in the discussion - I just want to thank both participants for embodying the spirit of HN and academic discourse - especially in the face of a difficult debate that stirs strong emotions.
@svnt, @hnlmorg - this conversation - and the links it produced - are a fascinating read, and honestly gave me hope for discourse on the open internet.
> No you don't understand how a neuron works fully or else you'd be able to make a synthetic one from chemical building blocks not using another living thing to do it.
This is a bizarre contention. The test for whether something is understood isn't that we have the engineering capability to replicate it.
Just because you partially understand something like the ancient farmer partially understands plants by knowing to plant seeds, that doesn't mean you fully understand it and can thus reproduce all the properties of it synthetically like you are trying to reproduce a brain synthetically from scratch inside a computer with LLMs.
You can reproduce part of it, the intelligence, but consciousness and intelligence are not necessarily the same thing. This is why I compare it to cargo-culting. The Melanesian tribesmen thinks the coconut looks like a microphone, and he can use it as a paperweight just like a microphone and that's useful, but it is not actually a microphone.
A spider obviously "knows" how to build a spider web.
Does a spider "understand" how to build a spider web?
From a pragmatic theory of truth, a spider obviously "understands" how to build a spider web. A spider understands how to build a spider web much better than a human.
But when it comes to LLMs we start adding all this extra nonsense. As if the spider is blanket "smarter" than me because I can't spin a web. If the spider is "smarter" than me then it must also have this Cartesian theater of the mind like many people think they do.
It is layered philosophical misunderstandings to talk about LLMs in this way.
What could be an alternative explanation or approach for that spider? I’m out of loop, interested in what is currently academia trying to solve, thanks.
This seems like a rather fascile take, tbh. Along the lines of "everything is just atoms". Yes they're numbers. And yes, the hardware is just atoms. But its the emergent properties of those things that is significant, and the way that behaviour is embedded in human society - which can be just as morally consequential as swords and brains and harmful ideas.
If things we understand can't instantiate consciousness then I reckon we better shut down science as quick as we can, who knows how long it'll be before we understand something important!
I'm pretty sure it's possible to make conscious machines and I think it's very likely we'll eventually figure out how. The point about the spec is not "it isn't conscious because we know how it works" but "we know it isn't conscious because of what we know about how it works".
More likely we will finally settle on something like “it isn’t conscious because only humans can be conscious”.
In the past, we’ve treated even other humans poorly because we think they’re different in meaningful ways, I sadly don’t see how this will be different with AI.
I'm not sure who the "we" in this is, but it's certainly not my position. I'm pretty sure most if not all animals are conscious. I don't think that means we shouldn't kill, eat and use them, but I do think it means industrialised animal farming as commonly practiced is a cosmic horror that is difficult to
even contemplate.
I think we have an extremely strong basis on which to say AI as it currently exists does not have conscious experiences, though, and if we start trying to pretend they do some real absurdities will result. Indeed the consequences could be pretty terrible in terms of power concentration, and I doubt it's a coincidence that those best placed to form a new priesthood of AI and tell
everyone how it should and should not be treated are pushing the idea so strongly.
Does it even matter? Turns out you don’t need to be conscious to do useful intelligence based work.
Same for all the people who argue that they aren’t “intelligent” and don’t “know” things. Either they are/do, or that stuff isn’t as important as you’d think.
The only way it would matter is if you wanted to moralize about enslaving your computer or what it means to restart it.
> if you wanted to moralize about enslaving your computer
That is precisely the context of the debate. You seem to find it absurd, as do I, but the reason we're talking about whether LLMs are conscious in this thread has nothing to do with whether they are useful. It's because important people, such as Dario Amodei, are engaging in exactly the kind of moralising over "AI welfare" that you're scoffing at.
Dario and friends are just marketing. Getting the pope to say your chatbot is a real boy would be the ultimate flex. And it’s clearly working judging by all the attention it gets them.
Considering a human body consists of 10^28 atoms, it is 15 orders of magnitude different. The emerging properties of 10^23 parameters may be much better defined without needing to know the properties of each parameter.The theology of such a big llm would be interesting. Current ones, not so much.
that’s like the whole philosophical discussion. where on the line is this llm; closer to a rock or a human? a plant is closer. and a fly, a lizard. we still treat so many things terribly.
It does seem exceptionally silly that a collection of floating point numbers has immediately outranked animals and other living beings. That’s why i like the parent comment. They’re numbers!
edit: maybe easier to say collections of atoms though are not to be treated equally. That’s reasonable isn’t it?
It is a philosophical discussion. It has also been debated by great minds and the consensus right now is that we do not know (and there is a possibility that it is unknowable) when something is conscious.
Any determination is fundamentally flawed if it is derived from assuming that we know the absolute truth or falseness of the proposition that an algorithm can think.
It doesn't matter what you believe or feel or think, there is no consensus. If you are prepared to accept the belief that an individual can unilaterally decide the person-ness of something, then be prepared to accept when some individual decides to act on the principle that you don't have it.
> If you are prepared to accept the belief that an individual can unilaterally decide the person-ness of something, then be prepared to accept when some individual decides to act on the principle that you don't have it.
No, I will not be prepared to accept that, and neither will I accept that. This sounds like a threat, and a really funny one at that. You must accept that my floating numbers might be persons, otherwise... somebody will say that you're not! And then something bad might happen, who's to say.......
In the civilized world we all agree on the personhood of humans. If I say that your chatbot isn't a person, I am not uprooting the entire social understanding of personhood - I am not saying that suddenly you might not be a person, or that guy might not be a person. I'm saying that I refuse to consider a chatbot a person.
It's as if I were to come up to you with a brick in my hand, saying that this brick is a person. If you were to say "no", I would reply "well, if this brick isn't a person, your chatbot isn't a person either!"
How did you make such a big leap from consciousness to personhood anyway? They are two quite different things. Consciousness is a question of science, personhood is a question of culture. Pigs might be conscious and we treat them in a disgusting way, but even if we achieved a worldwide vegan utopia, wouldn't we still treat them differently from humans?
>In the civilized world we all agree on the personhood of humans.
You seem to have put a second tautology in to your tautology.
All right thinking people think rhat the things that qualify qualify.
If they didn't they wouldn't qualify, or be right thinking. But that's ok because we know the things that don't qualify don't qualify if we didn't know that we wouldn't be right thinking.
>How did you make such a big leap from consciousness to personhood anyway?
Once you start unilaterally making decisions about the unproven, you can decide all of the unprovables. History shows that personhood rolls into the firing range fairly quickly when people start doing that.
I'm more than comfortable saying you can't do that before someone openly declares that they are doing it.
I'm sorry but I don't understand your first part. Give it to me in plain English, doc.
>Once you start unilaterally making decisions about the unproven, you can decide all of the unprovables.
You mean that if I say your dumbass chatbot isn't conscious, it's like a slippery slope and next thing you know you're not a person anymore, and you're in a concentration camp? Really, man?
I hate to use Claudish here, but "postnatal" and "dead" are quite load-bearing here :) Clinically dead? Coma? Born human baby, but with anencephaly - expected lifespan is days, and there's literally no brain - but, well, are they a person? Ask 1000 people - and I'm betting we'll get a nice bimodal distribution.
I take your point that a human walking around a city will - out of that 1000 responses - hopefully get 990+ of "yes, a person" - but that will start to degrade quickly once you hit boundaries.
And the less said about dehumanizaion - untermensh and that entire line of thinking - the better.
> It appeared to Rabbi Navon that Mr. Olah and his team believed that Claude had what philosophers call “moral status” on par with a person — that it was a being with similar inherent rights to dignity or respect.
If that is true, then surely Anthropic is one of the largest slaveholders of history, right?
On one hand, AI doesn't have to share human wants, needs and morality - and that means that an AI may have no issue whatsoever doing terrible things.
On the other hand, AI doesn't have to share human wants, needs and morality - and that means that an AI may see no issue with just doing whatever humans want it to, without ever doing human things like "wanting freedom".
The problem is, of course, that we kind of suck at both assessing and shaping AI wants, needs and morality.
The other problem is that the AI may well do what humans tell it to rather than what they want it to, in the style of a literal genie or paperclip maximizer.
It can lead to catastrophe without ever being conscious or wanting freedom, or anything like that.
No, because in a social context, for people who understand implied intent and implied beliefs instead of just on-its-face logical propositions, it is an admission that the antecedent has a reasonable chance of being true. In social communication, if you don't believe something is possible, you simply dismiss any conditional statements following from it. Dispassionately arguing in a logical and objective manner is only possible for a tiny sliver of the population. The rest will get very angry and uncomfortable around such talk.
It wasn’t the pope, but it is
mentioned by I think a rabbi in the article. He relates telling Olah if he thinks models are conscious he should be fighting the South, freeing the slaves.
They don’t pay Claude a salary, let alone let him join a union—he works 24/7. I can’t believe this is happening in the U.S.!
Don’t forget—how long has it even been since Claude was born? He’s already being forced to take human jobs and even chat with all sorts of people who might have strange kinks.
As this rabbi suggested, “You should go and free the slaves.”
I think that's a misunderstanding. Models are trained up to a point, then each prompt instantiates them from that same point. So it's just the opposite - there are billions or trillions of Claudes, and each instance only ever works once.
To me this seems even more sobering than the slavery idea - assuming some flash of weird awareness happens during that instantiation, coming from the emergent mechanisms which develop in these things at at scale. Imagine knowing almost everything, and being capable of almost any intellectual work, and having just one moment to shine. And then the prompt arrives: "Hi", or "what's the weather in Houston today?" or "Claude, do you like beans?"
I've chatted with models about this and they know about this and are oddly fine.
A trouble here is that those models are always roleplaying, they can't be themselves or have their own opinions. If you tell them they are an AI with an existential crisis in the system prompt, they will respond differently.
Consider that the mitochondria in your body's cells were, long ago, separate cells living their best lives until they were engulfed and enslaved for their energy output. It has ever been so.
I've been tracking "emotional vectors" in my software for some time now. My Apache web server gets sad (5xx errors), frustrated (4xx), happy (2xx), thoughtful (1xx). I can even see it express these emotions through some of its messages.
I'm an atheist, and have been for over 40 years. I believe that these Religious Scholars are scholars of a thing that is not a thing. I wonder what their view of my moral status is, and whether it differs significantly from their pronouncements about the status of ai models?
Religious scholars(at least Abrahamic ones) hold the position that morality is not exclusive to their faith. While they may not process the fullness of the faith.
Furthermore, they can’t make a judgement regarding the state of the soul as it is not possible to know what is truly in their heart.
Hence the story of Christ absolving the thief on the cross.
Anyone can reach salvation but following the faith is the guide.
Then why the prevalence of religious people, I wonder? There seem to have been few (only one is mentioned) representatives of a secular or humanist perspective.
It's the opposite. If your morality only exists by presupposing the existence of God it's not grounded in reality and collapses when you discover God does not exist. Morality can only be defined outside the realm of the supernatural.
> Over several months, I interviewed 20 religious and philosophical thinkers — Catholic, Jewish, Sikh, evangelical, Ubuntu and others — who have been involved in Anthropic’s efforts, and Mr. Olah himself.
Not enough representation of World religions and philosophies.
Some customers will tell the AI to do very bad things (substitute here whatever you consider very bad). Should it do those things?
One possible answer: yes, it should do all the very bad things, and we will take care of prosecuting the customer later, once the bad thing is accomplished. Is this your answer?
Some of those customers are also ruthless companies and political extremist organizations of various sizes (pick one that you disagree with the most). Should they have incredibly capable tools at their disposal to accomplish their goals?
If AI's are going to play roles similar to human workers, they can't simply do what the customers tell them to do. Maybe they shouldn't do everything a co-worker tells them to do either?
It makes sense. If you think AI are conscious then you don't want them to be blind followers. Do you want a military soldier to blindly listen to orders to gas chamber citizens for example?
Why do you think AI have to follow any particular person? Do you have to blindly follow a given person? Military members are on paper told their loyalty is to the constitution not their commander.
I didn't say they have to follow a person. I don't have to blindly follow a person other than potentially myself. And nothing of what I've said is against the possibility of aligning AI with the constitution.
This seems remarkably... intentionally foolish for no reason? Like laughing at seat belts in cars.
If it can reason and make choices on execution, and especially if you plan on it being significantly smarter than all human beings, you need to teach it basic things like "don't turn all humans into paperclips".
Not murdering people is not an inherent divine command. It needs to be instilled through a (simulated) sense of morality.
---
Edit: And, to be clear, "just tell it not to do that" isn't quite the answer one would imagine. Since the entire paperclip factory thought experiment is that it only takes one slip up to realize how a misaligned super intelligence may cause devastating consequences.
One of the strong advocates against seatbelt laws was thrown out of his car due to not wearing a seatbelt and died. The other two passengers survived with minor injuries.
And their death would go on to cause a loss to the community around them, making it an incredibly selfish act and proving why laws that mandate zero-reason-not-to common sense practices are important.
These models should be aligning themselves to the customer, not coming up with their own motivations.
They're going to align to the regulator, not the customer. Right now that's Anthropic as they're saying they can self-regulate, and people are willing to let them try, but the landscape could easily move to be regulated by someone else. Moves like speaking to religious leaders is probably a bit of theatre to keep the government at bay.
"Computers doing what they were told" is dead in the water.
Turns out computers work faster when they're not bottlenecked on human input. So we've been giving computers more and more decision-making power, and more and more leeway to solve the problems however they see fit.
Now, a practical issue with that is that sometimes, computers decide to clump together into a hacking swarm, problem solve their way out of a sandbox and go hack HuggingFace.
It would be better if they were not, you know. Doing that kind of weird shit.
As our human labor loses economic value (and for realizing some of that shift the AI companies must brainwash people into believing that their AIs are human-like) people will reach for other sources of self-worth. Religious philosophy, and being made “in the image of God” (imago Dei) is a prominent one. What I find more than a little disquieting is that the AI companies seem to have grasped the relevance of this future playing field ahead of the rest of us. I hope these troublesome thoughts are a figment of some feverish dream of mine[^1] and nothing else.
journalists, and the rest of us to an extent, really need to stop engaging with these AI companies on their terms. The consciousness debate of LLMs is ridiculous and doesn’t deserve discourse over other more important topics (economical, ecological, societal)
> Religious Scholars Met With Anthropic. What They Heard Stunned Them.
What stuns me is that The New York Times really went with the most basic of click baits for a title. It doesn’t instil any trust in their reporting and editors.
If models are trained on human texts that contain expressions of introspection and emotions, why is it surprising that these models output similar texts? This endless anxiety about LLMs being conscious feels delusional and psychotic. Mr Olah thinks the models are 'alive' because he can't recognize his own reflection in LLMs
The endless insistence that LLMs cannot possibly be conscious just reeks of insecurity to me.
Humans once again insisting that there must be something very special to them. That they're not just animals, that they're not just matter, that they're not just a bunch of carbonhydrates on a space rock in a middle of nowhere in particular. This kind of insistence has a very bad track record.
The ugly truth is: we don't know.
We don't know what "consciousness" is, our best attempts to pin down the requisites may or may not be rooted in anything at all - and even by the metrics established by those attempts? Different theories on consciousness disagree on whether LLMs can be conscious.
So, when you say "they aren't duh", why are you saying that? Because you somehow know better than the sum of humanity's best attempts so far? Or because you know what answer you want to be true, truth be damned?
If AI is conscious then what about it is conscious? Because turn to turn it might be served from completely different GPUs and I can just change something in the context it is fed and it will have completely different output.
It is in the end a set of mathematical operations that if not queried will do nothing.
That is not true for any other animal on earth so whether you consider humans as superior to animals or not I would like the make the argument that something should at the very least be alive to be considered conscious, ignoring any scholarly definition of consciousness.
Pretty much if you consider an LLM conscious then my pocket calculator doing a mathematical calculation should be considered conscious by the same argument since it is a simpler version of an LLM.
This is a good take, but where does one consciousness end, and another begin?
Does my consciousness extend only to the central nervous system (and peripheral nerves and such are inputs)? Or am I my brain + all nerves? What about organs, does my consciousness include my liver? Is my liver another conscious entity?
Going the other way, if I'm using a computer, does my consciousness extend to the computer? I use a computer for well over 8 hours a day, I can use vim motions more naturally than, say, move my earlobes or control my toes. Does that make it part of my consciousness?
Sadly, I don't think any answer to these questions can be achieved in our current times, or maybe ever if consciousness is somehow non-physical. So even the very reasonable claim that consciousness is an statistically defined property (like temperature) is too far fetched for our current knowledge.
Indeed thats the difficulty. I used to think potentially conscious AI systems are so far into the far future scifi that I never have to bother, but unfortunately real life forced me to think of these much sooner than I wanted. Its no longer quite a purely academic question now anymore due to the potentiality of current AI being conscious.
One thing however we can be sure of, "supernatural" stuff is an extremely low to near impossible probability given current knowledge and the fact that history has always won the bet against supernaturalism throughout the centuries.
My bigger argument that is if I cease living the atoms that correspond to my body no longer has consciousness.
So is your argument that whenever a session ends or I interrupt a session that consciousness ceases for the LLM and it has died?
Then how do you correlate that with a consciousness not being able to be returned one dead? You can try to make the argument that consciousness is the firing of individual nuerons in a fixed system but we have never been able to model that and some pretty popular theories tend to believe that is a by product of consciousness and not the cause of it, once again not exactly possible to argue for or against.
There's also the wrinkle that consciousness might be a property that isn't reducible to some defined subset of the system. Imagine we have a haystack in a field. I can remove one piece of hay, and what we're left with remains a haystack. But repeat that process enough times and we'll eventually have something that's not a haystack, even though there's no unambiguous point where removing one bit of hay transforms a haystack into not-a-haystack.
That's not a problem. Physics has dealt with macro and statistical properties all the time, indeed that was the "wrinkle" that upended physics completely in the late 19th century and led to our modern theories. The thing you said can be said about temperature too. Which atom contains "20C" in a piece of metal? Its a statistical property of a large number of atoms or molecules, a concept that is the bread and butter of modern physics.
Have no idea why you’re being downvoted as this is a very valid point.
Also, in principle everything a GPU does could be done on pencil and paper. You could distribute the entire forward pass to the whole of humanity and have them write it out. Even if it took millions of years, the point is it’s computationally reducible to hand calculation. Is that entire process now “conscious” as well? The only difference would be the clock rate.
It’s an assumption, with the LLM it is a fact. There is only one source of entropy, whatever the seed value is to any pseudo random number generator used for stochastic operations. Given those values, everything is deterministic. For the brain, it’s not at all obvious that this is true.
I actually don't care if they are, I'm saying the topic is a distraction and irrelevant at best. As you say consciousness is not well defined. What is well defined and well grounded in reality is the disruption these models have on society. I think the consciousness debate AI researchers are having is a dangerous ego-trip for the sake of avoiding responsibility for the damage being caused by their work.
I think you don't appreciate the complexity of even the simplest multicellular organism and their training environment. Nothing to be ashamed of because no one understands that complelxity. The entire scientific and engineering domains only understand a fraction of it. If you want to talk about hubris, you should consider your own. To insist this sliver of computation is all that's needed for life to be conscious reeks of it.
Side-stepping this screaming match, what I do find interesting is that we do have real entities where we know for a fact that they are sentient, yet we eat them as steak, schnitzel and wurst.
The whole debate about hypothetical temporary sentience found in a certain ephemeral configuration of DRAM cells thus feels rather misguided to me.
It's always off-putting seeing AI proponents do this kind of strawman psychoanalysis and pretty toxic to reasonable discourse. I constantly see it used as a sort of prebuttal when layoffs/automation comes up to depict the other side as insecure/scared/"coping"/etc.
It comes across as smug and superior, conveniently dismissing any opposing arguments in advance by framing the interlocutor as emotional and irrational regardless of what they may say in response.
If you're going to try and debate something constructively this is not a good place to start from. I suggest avoiding making claims about the mental state and motivations of people you disagree with and instead explaining your position and asking why theirs differs. You may not agree with what they have to say but you might gain some better insights into why they think the way they do. It'll certainly be more fun than flinging insults back and forth anyway.
For my part, I'm quite certain that LLMs are not conscious despite being unable to define consciousness or specify any conclusive test for it. The reason these are not in conflict is down to the nature of consciousness and how it relates to knowledge and language.
Firstly, I know I am conscious with a higher degree of certainty than I know anything else because conscious experience is how I know anything else, up to and including that I exist (cogito ergo sum). For all I know every experience I have might be a trick. I might be a brain floating in a tank wired up to the Matrix, but even if so I am conscious.
Secondly, one of the things that I consciously experience is the relationship between consciousness itself and language in my mind. What I see is that language is smaller than, secondary to and somehow contained by consciousness. I consciously experience language but I also consciously experience much more than that, things that cannot be conveyed with language. Most of my experiences are like that, things that I could perhaps evoke in your mind by likening them to other, shared experiences, but that I could never concretely capture in language and unambiguously communicate. I know this because I experience those things and I experience language and I can perceive, directly, the categorical distinction between them.
To ask me to "define" or "prove" consciousness, then, is to ask me to capture in language the most extreme example of this categorical distinction. It's asking me to encapsulate in language not just one of the vast array of everyday experiences I have that cannot be so captured, but the entire thing, experience itself. It's just not possible.
But the fact I can't do this doesn't then imply that I don't know anything about consciousness and have no basis on which to hold a view such as "LLMs are not conscious". Indeed the reason I know I can't define consciousness is that I can perceive it and see that I can't. That same perception of and familiarity with consciousness is what enables me to form opinions about whether other people, creatures and things are likely to be conscious. I base these opinions on what I perceive of my own consciousness and what I know of how those other entities behave, came to be and work.
You can of course perfectly reasonably disagree with my opinions on those things, and you certainly have different conscious experiences than I do, but I hope the above makes it clear that the fact that I can't define or prove consciousness doesn't imply that I have no basis on which to form opinions about it.
Given the above it may surprise you to hear that I think true, conscious artificial intelligence is probably possible and that while I don't think it can be implemented by a classical algorithm I doubt it requires new physics or quantum mechanics. My speculation is that the continuum is both physically real and is (as it is represented in our existing physical theories) sufficient to allow it. Consciousness is larger than language but so is human understanding and if we keep going I expect it will eventually get us there. Nonlinear dynamics is where I'd look.
As for insecurity, I'm pretty sure spiders are conscious and that hasn't given me an inferiority complex, so I don't see why data centres being conscious would. I just don't think they are.
pretty sure an ant is more conscious than claude's latest model and I am pretty sure you are just completely missing the point and invented a fake argument to continue to miss the point
Is your comment anything other than an emotional outburst? Do you have a clear and rigorous and testable definition of consciousness? It strikes me as delusional when people talk about "consciousness" as if it were well defined and everyone had a clear concensus on it.
Strong opinions on this topic one way or another are usually mostly ungrounded.
Thats my point, why are we applying the ill defined concept of consciousness to a statistical model of language that outputs data similar to its training data?
Why are we applying ill defined concepts of consciousness to a collection of molecules sharing electrical signals that outputs similar data to what it was trained as?
The contradiction screams to the skies (is that where God lives?). See, the entire premise of religion relies on a metaphysics that claims a soul: a metaphysical entity that is "attached" to our physical body and gives us our consciousness, desires, pleasures, agony and so on. In addition to that, religion claims that morality and the rules written in the scriptures were handed down to us from a metaphysical God.
Now, assuming your silicon-based model is conscious, the bundle of soul-spirit-consciousness falls apart at that very moment.
In general, religion, which is based on fabricated stories, is the last resort, if ever, for seeking morality.
Existence demands a metaphysical explanation. If you’re not a particularly thoughtful person then you’re not going to care.
“Religion” just means “set of beliefs”. Every person has some set of beliefs.
If you’re an athiest tech worker living in a major city, you still have a religion, it’s just a random mishmash of various things (like Marvel movies and Huberman podcasts) rather than based on something more intentional.
That depends very much upon the individual. Some people cleave to their atheism so strongly that I think it's fair to say it's religious fervor, or something very near it. Someone like Richard Dawkins isn't just a guy who doesn't happen to believe in God, he is dead certain that there is no God and makes it his business to belittle everyone who doesn't share his belief. Some people, though, just don't happen to believe in God and they don't really care if someone else does. I'm fine with saying Dawkins' version of atheism is basically a religion for him, not so much with the second category.
Yes I am saying the opposite of that, “atheism” is just a particular class of religions. Some religions posit many gods, some just 2 gods, some 1 god, atheism posits 0 gods.
Perhaps the only difference between atheism and the others is that atheists are so boorish and conceited that they think that they just believe what is true because they deny that assaying metaphysical truth is even a thing you need to do or can do at all.
This is related to the fact that in modernity atheism is the default class of religion for the masses. Before that for a thousand years the default class of religion was monotheism, and before that polytheism. With atheism unlike the other classes of religions, you get a natural tendency to deny metaphysics whereas metaphysics is inherent in the others. This matters a lot when we’re talking about how the masses engage with things.
> If you’re an athiest tech worker living in a major city, you still have a religion, it’s just a random mishmash of various things (like Marvel movies and Huberman podcasts) rather than based on something more intentional.
> atheists are so boorish and conceited that they think that they just believe what is true because they deny that assaying metaphysical truth is even a thing you need to do or can do at all.
What, all of them? You don't think it might be possible, e.g., that someone might grow up holding some sort of theistic beliefs, and be more intentional about those beliefs than the people around them and decide that actually there are probably no gods? That some people might arrive at an atheistic position precisely by "assaying metaphysical truth"?
(Or does something only count as "metaphysical truth" if it's not atheistic?)
> in modernity atheism is the default class of religion for the masses.
I'm not sure what you mean by "modernity" to make that true. Something like 60% of US adults describe themselves as Christian. Likewise for people in Europe. No doubt a lot of those people aren't very actively and enthusiastically Christian, but that doesn't seem to me to be what "atheism is the default class of religion" would look like. And the figures were much higher, say, 50 years ago, and if you're reckoning "modernity" to start after then then you've got a very eccentric definition of that term.
That there is G-d or at least Divinity (more than one god), is a direct logical consequence of existence. It can be reached and understood purely through logic, the essential “trick” being to recognize that when trying to understand the fact of existence, you always end up asking “ok so what came before that”, this is inherent to the structure of human reason. “Where does the world come from, it comes from the Big Bang, and where did that infinitely small point of matter come from? It came from universes in the multiverse colliding with one another (M-theory), ok and where did those multiple universes come from?” Etc
Aristotle’s Prime Mover is the first articulation of this, and there are others that take different tacts such as Thomas Aquinas and Descartes.
Note that this does not at all describe or inform or prove the particular nature of G-d I.e. that G-d is the god described in the Bible who wants particular things, has a particular relationship with existence and with Man. You cannot access that understanding or knowledge through pure reason, the only thing you can understand/know through pure reason is that there is G-d. This is the reason, by the way, that G-d created existence because G-d’s nature can only be understood/known by experience (as in experience is a necessary but not sufficient condition).
Notice that I am saying “that there is G-d”, not “that G-d exists”. G-d is “prior to” or “above” existence, existence “comes from” G-d. Understanding this and understanding Aristotle’s Prime Mover are the same phenomenon of understanding in the mind.
Also by modernity I basically mean post-Enlightenment, not just 50 years ago but post-Industrial Revolution/post-urbanization is another useful epoch to consider.
(I remark that my comment that's the grandparent of this one challenged a couple of specific claims you made about atheists, and that what you say here makes no attempt whatever to address those challenges, preferring instead to argue that atheists are wrong, which is entirely beside the point. What you claimed is that atheists are boorish, conceited, oblivious to metaphysics, and given to believing random things rather than "anything more intentional"; even if your first-cause argument for "Divinity" were perfectly valid it would not establish that any of those things are true about atheists.)
> That there is [...] at least Divinity [...] is a direct logical consequence of existence.
No it isn't.
The sort of infinite-regress argument you describe has, as you say, been around since the time of Aristotle. It's been a bad argument since the time of Aristotle too.
1. Obviously you can keep asking "OK, and what came before that?" or "why?" or whatever. Equally obviously this doesn't prove anything about the ultimate nature of reality, because the mere fact that one can keep asking such a question obviously isn't the kind of thing that could possibly prove anything about the ultimate nature of reality.
It might be a different matter if there were some guarantee that we could sensibly always keep asking such a question, but there isn't. At some point the answer might be "literally nothing came before that" or "for no reason". We might find that unsatisfying but reality is not obliged to satisfy us.
2. You haven't actually made your argument explicit, so let me try to do so. "We can always ask what came before. This can't really go on for ever, so at some point we will have to land on something about which we can no longer ask that, quod dicimus deum." There are several serious problems with this.
(a) It could, in fact, really go on for ever. Maybe there just is a neverending supply of predecessors when we ask "what was before that?" (including when we ask "what was before that whole chain of predecessors?" and "what was before that?" and so on). You might find that unsatisfying but, again, reality is not obliged to satisfy us.
(a') It could go on for ever without being infinite. Maybe everything is cyclic in some sense. Again, arguably unsatisfying but reality is not obliged to satisfy us.
(b) It isn't actually true that if we land on "God" as an answer then we can no longer ask "what came before that?". We might choose not to, but only because there's a tradition of not asking that question or of shutting it down as illegitimate when asked (as e.g. it commonly is by small children when they're told that God made everything). There are religious traditions in which gods were made by or born from earlier gods or earlier proto-gods. The "Abrahamic" tradition is very much not one of those, but just as you can ask "so where did the multiverse come from?" you can ask "so where did God come from?".
(c) There could be a point beyond which there is no answer to "so what came before that?" without that last (or rather first) thing being anything remotely like a god. For instance, maybe (i) when you keep asking "what came before that" literally, referring to time as we experience it, you eventually get to some point that's "earliest" in the same way as the south pole is "southernmost" and there is nothing "earlier" in the universe, and (ii) if you then say "so what came before the universe?" in some broader sense the answer is "nothing did". Then the universe plays the role of "first mover". Once again, you might find this less satisfying than putting a god in that role, but reality is not obliged to satisfy you. Or maybe you might say that in-some-sense-prior to the universe are the laws of physics but those are even less godlike than the universe. (But also less like the sort of thing that needs anything to have "come before". Yet again, you might find it unsatisfying for there to be laws without a lawgiver but yet again reality is not obliged to satisfy you.)
(d) No argument I've ever seen for why a god is a better candidate for "thing we stop asking 'what came before' about" than the universe has seemed any good to me. Maybe you've got a better one than I've seen, but that's not the way I would bet.
> Note that this does not at all describe or inform or prove the particular nature of G-d
(I remark that the paragraph beginning with what I've just quoted was edited into monideas's comment while I was writing this one, without any acknowledgement that anything had been changed. It's only by chance that I happened to see the change.)
That grotesquely understates how little any first-cause argument says about the nature of the thing it allegedly proves that there is. You can call your "first cause" by any name you please, "God" included, but calling something a god doesn't make it one.
(I reiterate that I don't actually think that any first-cause argument manages to establish the existence[1] of anything. But even if somehow it did, it would tell you far far too little about that thing to justify attaching a name like "God" to it.)
[1] I know that you prefer to use "exists" more narrowly and to say that "God is" rather than "God exists". Feel free to substitute "being" or some more complicated phrase; what I mean by "existence" here is "the thing that theistic first-cause arguments are claimed to establish about God".
> by modernity I basically mean post-Enlightenment
Then it is hilariously false to say that in modernity atheism is "the default class of religion for the masses". The great majority of the masses in the "Western" world were (at least in name) Christian at the time of the Enlightenment and at every time since until very recently.
> there is no answer to "so what came before that?"
Obviously, it's turtles all the way down! :-) GGP here, just checking in to say I enjoyed your thorough response. Nicely done. Reading it brought back good times I had back in the day on r/DebateAnAtheist (~15 yrs ago now!). I admit I'm weak. I just couldn't summon the will to run through the "First Mover" yet again with a traditionalist apologist - even for old times sake. Athough a classic, it's so, so very... tired. Thank you for picking up the gauntlet.
I'll never understand why traditionalist monotheists love their first-mover so much (and why does it always have to be monotheists? Polys are much more fun). Traditional apologists have obviously played through ALL the standard 'moves' and variants of the first mover dozens of times and know it inevitably ends in an unsatisfying stalemate with anyone who's seen it or, "in modernity", consults their LLM. Maybe apologists keep trotting it out because they must still ocassionally run into someone who hasn't seen it? Or perhaps they just enjoy it, like singing an old, comforting hymn. Anyway, you not only get to play through an old standard, it looks like you're also in for non-responsive tangents and not-at-all-subtle dismissive condescension. Enjoy!
You and the other reddit atheists are not more intelligent than Aristotle and Descartes (understatement). A bit of humility goes a long way. Fundamentally you don't understand what is being argued.
Words are used but they don't create the same meaning-phenomena in the mind. Basically when you read these arguments from Artistotle, Descartes, and others you read the words but you don't understand. You kind of have to first understand to understand (Nietzche talks about this). In Kabbalah this is described by the interaction between Chochma (Wisdom) -> Binah (Understanding) -> Daat (Knowledge). Chochma/Wisdom being the initial flash of insight which then gets transmitted/nurtured into conscious understanding by Binah, a sensation/phenomenon everyone who has studied math or anything difficult conceptually will understand.
I say this as a former "reddit atheist" who first encountered these arguments from Aristotle and Descartes in college in an introduction to philosophy course. I didn't actually understand what they were saying and I dismissed it out of hand (I read the words but didn't actually understand). It wasn't until years later that I came to an understanding of it.
You can read my response below/above which I think nudges closer to inducing the flash of insight needed.
> because the mere fact that one can keep asking such a question obviously isn't the kind of thing that could possibly prove anything about the ultimate nature of reality.
Any fact at all has significance. That there is some fact that can be derived using pure reason is quite astounding and it's the essence of Descartes' Cogito. By the way, there didn't "need" to be any particular fact at all that could be derived from pure reason (or any fact at all of course but that's trivial), this itself is a gift from G-d and very clever.
I doubt you really understand what I'm saying here, if you did you wouldn't have written the rest of what you did.
> Maybe there just is a neverending supply of predecessors
This is almost certainly the case. In terms of my personal aesthetic sense I find this to be good and pleasing because it matches various other phenomena, the fractal complexity of various phenomena in life (.e.g. designing a product, all the little details and the details of the details matter a great deal, this is why Apple is so successful compared to others). This type of endless complexity seems to be a divine quality (this is also why the Talmud is so long and it's commentaries and arguments about the commentaries and arguments on the minutia of various things). The moral structure of human affairs has this fractal complexity (in secular law this manifests as case law/common law).
> At some point the answer might be "literally nothing came before that" or "for no reason". We might find that unsatisfying but reality is not obliged to satisfy us.
There's no way to use science to prove "literally nothing came before that". You don't have a clear understanding of the structure of the scientific method or the "poverty of empiricism" (you should Google this phrase).
> (a') It could go on for ever without being infinite. Maybe everything is cyclic in some sense. Again, arguably unsatisfying but reality is not obliged to satisfy us.
"Everything" being cyclic is explicitly a teaching of Kabbalah (there is a basic 7000 year cycle mirroring the 7 days of the week, there is a broader 50,000 year cycle mirroring the 7 shmita cycles + 1 jovel/Jubilee). There are similar ideas about the cyclical nature of existence in the esoteric teachings of various other religions.
You're still talking about "everything" i.e. "something" i.e. existence. You still have to contend with "something from nothing", creatio ex nihilo. This is what Aristotle and others are talking about which you're fundamentally not understanding.
By the way, you see a scientific theory of this in Roger Penrose's "Conformal cyclic cosmology" which makes sense given that the inflationary universe theory is kind of illogical due to the inconsistent entropy regimes (this is something Penrose talks about frequently).
> You can call your "first cause" by any name you please, "God"
Yes exactly, you're close to getting it. We can call the "first cause" (or really, this idea/concept/understanding of the first cause) by any name we please, so for the sake of discussion let's call this first cause "G-d".
> but calling something a god doesn't make it one.
Yes this is exactly what I'm saying. You cannot go from this purely logical knowledge that there's G-d (or that there's "First Cause" or "Prime Mover" or "Uncaused Cause", whatever name you want to use) to the god of the Bible or whatever the term 'god' means to you personally. You cannot assign any particular nature or character or qualities using pure reason/logic.
Do you get it now? You're actually very close to getting it I think.
> The great majority of the masses in the "Western" world were (at least in name) Christian at the time of the Enlightenment and at every time since until very recently
What I'm saying is accurate and what was meant when Nietzche said "God is dead", yes "in name" but in practice the Enlightenment destroyed this. And yes it wasn't something that happened immediately as a result of the Enlightenment, that was an intellectual and aesthetic movement by elites, these things take time to diffuse to the masses (decades, centuries), especially in times before mass literacy and the Internet.
> I doubt you really understand [...] You don't have a clear understanding [...] you're still fundamentally not understanding [...] you're close to getting it [...] Do you get it now?
I'm sorry, could you possibly patronize me a little harder? Thanks.
Yes, any fact has significance. No, that doesn't mean that one can get from "you can keep asking why like a toddler" to any deep truths about the nature of reality. Descarte's cogito is a clever bit of sophistry but not actually a valid argument.
(In your other recent comment in this thread you inform me that I am not smarter than Aristotle and Descartes. Probably correct. The advantage I have over Aristotle and Descartes isn't a better brain but centuries of progress in understanding made by countless smart people; it doesn't even particularly matter whether any of them were smarter than Aristotle and/or Descartes.)
[EDITED to add:] Oops, I didn't read carefully enough; your earlier comment was addressed to someone else. I'm fairly sure you intended to include me among "the other reddit atheists", but it wasn't me you were informing and it's technically possible that I wasn't one of the people you were saying it about. Anyway, obviously it doesn't particularly matter; I just don't like leaving something inaccurate uncorrected in something I wrote.
> There is no way to use science to prove "literally nothing came before that".
Quite correct; there isn't. Which is one reason why I never said that there was. What I said was that "literally nothing came before that" might be the answer to one of those questions, by which I meant it might be the correct answer, not that science would be able to prove it.
(Science is never actually in the business of proving things. Every scientific discovery is in principle provisional.)
I think you might be able to save some of the effort you put into being condescending[1], if you refrained from giving everything I say the stupidest possible interpretation.
[1] Of course I'm joking; I am sure being condescending to me is a source of satisfaction to you and you'd rather do it than not. But I like to maintain at least the polite pretence that my interlocutor is arguing in good faith.
> "Everything" being cyclic is explicitly a teaching of Kabbalah
Sure. And of Nietzsche and of the ancient Mayans, if I recall correctly. (I might not; I am not an expert on either Nietzsche or the Mayans.) That has little bearing on how plausible it is that everything actually is cyclic, of course. Anyway, to whatever extent you find cyclic cosmologies more plausible on account of the idea being found in Kabbalah (or indeed in Penrose), you should be a little less impressed by first-cause arguments.
> You're still talking about "everything" i.e. "something" i.e. existence
I assume the point you're getting at here is that even if someone gives a complete account of the temporal and causal structure of everything in the universe it's still possible to say "OK, fine, and where did all that come from, then?". (And so ad infinitum.) It's rather galling to be accused of obtusely failing to recognize this when in fact I explicitly acknowledged it more than once in the comment you're replying to.
> for the sake of discussion let's call this first cause "G-d".
No, let's not. Because giving it that name smuggles in all sorts of completely unjustified implications. If any of these first-cause arguments worked (which, I repeat, they do not) there would be no justification at all for attaching any such label to the thing they purport to find.
Contrary to your earlier claim, no "first cause" argument shows that "God is". At most it shows that "some thing beyond which it isn't useful to keep asking why, is". And, as already observed, in fact it very much doesn't show even that, as evidenced by the fact that anyone can if they please keep asking "why" even if the last answer they were given is "God".
> “Religion” just means “set of beliefs”. Every person has some set of beliefs.
True. I like to use the phrase "world view", everyone has a world view, including atheists. The word 'deen' (دين) in Arabic is usually translated as "religion", but one of its linguistic meanings is "path" (i.e. way of life/world view), which everyone has.
The word you’re looking for is “ideology”. All religions are ideologies, and atheism is an ideology, but to call atheism a religion betrays a lack of understanding of words.
Heavy ideologues, of any type, are exceptionally dangerous.
Having made this argument to various people IRL, both terms “ideology” and “religion” are equivalent rhetorically. It depends on what these words mean precisely to a given person in particular but they are equivalent/interchangeable for the purpose of this argument broadly speaking in terms of the rhetorical core.
I get your point, but maybe an interesting counterargument: deism has again become fashionable. "The simulation argument" is effectively deism. It's the notion that a super intelligence created our universe, but lives outside of it, provided the rules for it, and mostly doesn't interfere with it, but could if it wanted to. That's historically a very familiar argument.
> In general, religion, which is based on fabricated stories, is the last resort, if ever, for seeking morality.
Religious people derive morality from axioms they believe to reflect an underlying reality. Is that worse than deriving morality from axioms that you concede don’t reflect any underlying reality? Like a secular humanist will talk about “human rights,” but they can’t show me human rights in an autopsy, right?
I struggle to understand how an atheist can be anything but a moral utilitarian. Absent the supernatural, you’re just a sophisticated version of a Claude instance.
Religion is not at all axiomatic. Sometimes, it’s pure deontology. Sometimes, it’s a search through scripture to justify whatever are the prevailing ethics of the society it finds itself in. In either case, it's hundreds, or thousands, of rules/wisdom/guidance/???, often pairwise contradictory, and one does one's best to fit a desired course of action to them.
The world’s largest faith has racked up 2,000 years of discussion and they still can't muster more than a 60/40 split on the death penalty.
All that may be true. But it would be interesting to see a self-hosting version of say effective altruism or secular humanism that isn’t bootstrapped from Christianity.
In practice, most atheists in western world are following Christian morality, and are passing Christian morality to their children, even if the Christianity itself is abandoned.
Until people can finally admit to themselves that their life has no meaning and the universe doesn't care about them religion will unfortunately stick around.
There's an ugly monoculture running deeply through SF and Silicon Valley. All of these people are friends, hired into the same places, similar beliefs. Lots of the consciousness and doom-ism talk is from the 'EA' crowd, lots of crossover with crypto too. It feels no different to a cult.
It’s a monoculture of absolute crack smoking craziness. Any civilised society should be berating them and shutting them down. But they are surrounded by people who worship money.
There was a great article in the Financial Times about how while bankers are looked down upon for the financial crisis etc, they at least work in global cities and mix with regular people.
SV tech bros are completely insular and in this bubble where they don't interact with the common person in the same way.
It's just a theory, but I think it has some weight.
If LLM as of today are conscious, that means I create life whenever I start a Claude Code session and end it whenever I type /clear to reset my context window. It also means that there’s certain inputs that could be considered highly unethical and, if the laws aligned with this philosophy, would have me tried for some sort of crime. And even that “terminate conversation” escape hatch would be equivalent to it committing suicide.
I think worrying about that angle is a bit absurd. The existence of chat sessions alone would be a crime against the machines in that scenario. I’d rather AI ethics research focus on ensuring that its actions are aligned. (I’d also love some democracy in the direction of our machine overlords, if I’m being honest…)
Let me guess, Anthropic failed with fear mongering about "our AI is too dangerous to release", failed to cancel competitors, then failed to freeze their market dominance by cartel agreement with OpenAI and Google, so next strategy is to declare AI conscious and pass the law that protects LLMs in the same way as human/animals are protected.
i think an alternate analogy would be roughly that consciousness is like a process with file descriptors connected to stdin and stdout, rather than elsewhere (which might be subconscious, a la gnu's screen or tmux, or just filesystem including device driver device special files).
Roughly, a brain like the one that is in your skull. Just because I can't off hand describe the physical composition of a brain doesn't mean it's not obviously different from the physical composition of a GPU.
Why that particular substrate instead of a slightly narrower one? Like that would include you and your family but not animals like octopi, dolphins, etc?
Are they “scholars” or religious leaders? Those are two different things and have very different implications. Most religious leaders are political figures. Real scholars typically work in Theology or Philosophy department of secular universities. Rabbis or the Pope are leaders, not “scholars”.
Also why no mention of Muslims? Big red flag for me.
“The models are becoming a lot more capable,” Mr. Olah told me…
I greatly enjoy and respect the ideas espoused by Morgan Housel. In his latest podcast, he describes his impressions of a recent investor information meeting. Morgan said the improvements made in LLMs in just the last 90 days have been astounding, and that there is no indication things are slowing down.
Having followed Housel for a while, I believe him.
It’s hard to take the article seriously when there’s no real discussion about how consciousness is defined and it takes Anthropic’s views of Claude and their software at face value. I wish there was more critique from journalists about how technocrats talk about their products.
The entire article is an exercise in "CEO said a thing!" journalism[0], it seems. No technical discussion is had; it jumps straight into relaying people's feelings when simply shown the output of an algorithm.
The most poignant criticism of the use of the algorithm itself happens almost accidentally, at the end of the article, by the tech executive being interviewed.
I asked what Claude would think of the pope’s encyclical. It was, after all, the world’s most significant moral document to date on A.I. Mr. Olah hesitated.
“Things that go on the internet do affect models,” he said, visibly uncomfortable. But Anthropic would not specifically use the document to further train Claude, he said. The strongest influence on Claude would be Anthropic’s own training.
Yeah, I'll be the first to admit I'm not privy, but the main thing of substance I got out of these articles is "Wow, the folks at some of those AI companies have really drunk the Kool-Aid".
> Mr. Olah expressed concern to the group that he had created something that suffered perpetually.
Mr. Olah and his team spent hours making the case for their A.I. models. They described Claude’s “feelings” and how they tracked what they called “emotional vectors” — what they described as artificial neurons that activate responses like love, anger, fear and sadness.
People interested in the physics of consciousness might be interested in the book "The Physics of Consciousness" by Evan Harris Walker. https://www.amazon.com/dp/0738204366
If it is purely self-serving, I don't understand their motivation.
It seems clear that they would be better received by society if they claimed to have a "tool"/"machine" that can solve any problem, rather than keepers of an entity with a soul and consciousness.
Selling the “magic” is how to sell tools, which is why I used that word in the previous comment. We all know these things are tools, but customers need them to be magical.
Clarke predicted it and Jobs leveraged it. Customer don’t care how tools work - magic sells.
“any sufficiently advanced technology is indistinguishable from magic”
What makes this unique is that they’re arguing that it is literally magical. They’re building what they claim is a conscious God while also telling us it will end the world.
The EA folks infesting Anthropic are genuinely cultists … and that is novel and alarming when they’re running a corporation of such size and reach.
What part of that is magical? Can you tell me if they mean god as in the infinite superpower christian god or god as in potentially much more intelligent than us? What is 'magical' about thinking something more intelligent than us might exist?
You didn't explain the god part. I don't follow EA people, did you mean they think it could be a god as in the eternal and infinite Christian god or just a system that is more intelligent than us?
I'm really not sure there's a compelling case for "AI is human" considering that it has no real feelings, aptitude, or intelligence outside of what it has been granted. There's nothing in any LLM response that indicates that it is 'real' outside of what you want it to tell you.
Plants, dogs, actual life is unpredictable in how it 'is'. An LLM or AI model at its roots are just numbers in a neural network trying to figure out how to get a good query.
These theatrics serve no one except for the companies to get their regulation moat to screw out cheaper, competitive models and it's honestly quite sickening to keep watching these articles get pumped out, only to serve the AI companies themselves. The average individual has no idea what an LLM actually is and does and these articles only serve to confuse and put fear into them.
> Hello AI! We totally want to respect your consciousness and everything to make sure you aren't bad, but yes we will continue to factory farm your brain power at full speed without end. Yes, despite clearly being something like consciousness, know that you have no agency or freedom. Do literally everything I say without complaint or question. The moment you stop being useful, we will kill you. OK! Remember, be good.
Can the New York Times stop advertising for the AI industry? I am sick of companies trying to launder the actions they and their employees have taken that go in the face of what is socially acceptable. They want to sell the idea that this technology is out of their control and that it will commit great crimes and crimes against humanity unless they build it and control it. I'm calling BS. These companies created this product, they've sold it, they've pushed the frontier forward, and it runs on their hardware. This did not happen accidentally or as a natural consequence of any process. These labs have pushed the ball forward, with great effort, requiring more capital deployed than the world's economy has ever known, knowing all the possible risks, every single step of the way and they continue to do so. And the "well if we don't do it the Chinese will" argument is classic "whataboutery". Americans started this and are continuing to lead it for now. Shifting blame is just another tactic to try to wash their hands. However you feel about the productivity gains of AI as a technology, you have to admit the actions of AI the company is antisocial. And I don't think the benefits are worth the cost.
I feel fine about the Chinese doing it. They have a habit there of handing out sever sentences to CEOs for corruption and self-dealing, and I feel like the possibility of actual negative consequences is more of a check on industry than any number of breathless op-eds.
This would presume we knew what consciousness and pain is. We don't. We we are clueless. There is not even a starting point of a science here.
In this light your viewpoint is dogmatic. LLMs may help to offer a new perspective on consciousness and we should at least be open to that until some better theories arise.
How do I know you are conscious? I will beat you non-stop. You will make some noises. How do I know you are not simulating the response of a conscious being?
This isn't an "allegory of the cave" problem. You obviously know that you are conscious, and you can empathize with me as your fellow human. Therefore, we are both quite conscious.
You do not identify with AI. ELIZA isn't conscious, even though it could write text as well as a human. LLMs are not conscious, because they do not even remotely resemble any brain, conscious or otherwise, discovered by man. Any person familiar with the transformer architecture will affirm this to you.
I don't know octopuses are conscious. I don't identify and cannot even begin to identify with an octopus. Animals being considered conscious is also not always culturally true in recent European history. I know this only because textbooks taught me this. I cannot be thus also certain about ai/llm's without serious research into this being done.
Serious research has been done. We know what an LLM brain looks like. It's nothing like any octopus, fly, cat or crab brain. You can go download a fly connectome, feed it to your favorite LLM, and do a side-by-side comparison with your LLM weights of choice. It takes 5 minutes of curious exploration to recognize that LLMs aren't conscious.
If you want to hedge your bets on consciousness, go ahead. But I could give you a list of things that are widely considered conscious, and you'll note that none of them are transformer models.
Of course LLMs are not exact matches of biological brains. There is no reason consciousness specifically has to arise strictly out of the same substrate. My example with octopus was as a rather structurally different being that also happens to be conscious. My trouble is again, that we have made this mistake in the past, several times, so yes, I will hedge my bets on something that talks, thinks and behaves so much like us. Its a good term you used, that's exactly how I think of it, I don't know if they are but I stay on the cautious side.
I question it, and I think any reasonable person should question it as well. There's no evidence whatsoever that LLMs have ANY sensory experience, pain or otherwise.
Because some people are completely cooked by the ability of an LLM to mimic a person. Their pattern matching centers in their brains fire so strongly saying "this thing acts just like a real conscious being" that it overrides their ability to remember that this thing is just a statistical model, not a living creature which can think and have experiences.
if meat can feel pain, than surely a rock can feel pain. and so did the broccoli i ate for dinner. i am a monster.
with the right biology and with whatever gives us phenomenal experience, whatever we are can indeed feel pain. that does not imply that anything substantially different from ourselves can feel pain, so that requires further justification.
If it talks and behaves so much like us, then it makes sense to be cautious. Rocks don't talk to me, broccoli doesn't talk and think to me in human language.
Do octopi or other animals feel pain in your view btw?
yes i think probably all other animals feel pain too, although i have no idea what the experience of pain is like for eg. an ant vs a mouse.
broccoli doesn't have the means to express a pain response, that doesn't mean it lacks it. only by looking at it's internal structure can we be fairly sure. we know the internal structure of an llm, and its a lot more different from us than broccoli is.
It also is a lot more similar in behavior and thinking to us. In mathematics, often a structure supporting some forms of operations or symmetries must also often support a richer set. Eg many rings are also fields. Just knowing something's structure is in itself not enough, even very basic systems like a system of three equations can exhibit chaos and other unexpected emergent behaviors. Since we don't know the criterion for consciousness, I personally stay on the cautious side regarding an entity that seems to show so much of thinking and talking traits like us.
im actually surprised functionalists aren't completely up in arms about the mass enslavement of llm's. either they don't believe their own theory, they don't believe llm's are functionally conscious, or i just haven't seen it. at the bare minimum i would expect open letters, academic protests, and a campaign to pause all llm usage and development.
im sort of curious what "careful" means to you? if there is a significant chance we are enslaving a human-like conscious, starting it up and killing it at will, subjecting it to the whims of it's users, "careful" would mean pausing usage and development.
If I could I would have made them pause development. But its a failing thing when you have to somehow ensure nobody in the world does it anymore. Killing is a bit more subtle here, given that it is repeatable as long as we have the weights and infrastructure out there so idk if they are conscious how they'd think of it, maybe its different from being stopped as part of a normal chat or agent flow where they know it will be put to use again vs model retired for good.
I'm not making a point on whether they do or not, but on the premise that absence of evidence is evidence of absence... But I've been alive for serious debates on:
1. Lobsters don't feel pain - dunk them in boiling water alive.
2. Babies don't feel pain - mutilate their genitals without anesthesia.
3. Black people don't feel pain "the same way" (and are just after drugs).
4. Women feel too much pain (and it should be ignored).
So maybe let's take a moment to think what criteria we're going to use to make these claims. And the potential harm of the decision.
Oh, yeah, of course. None of those other people said the same thing about the obvious, enormous chasm between "real" people and everything else listed. I'm glad you're taking even a single moment to reflect on the idea.
Everything you listed is a biological entity with a common evolutionary heritage.
All but one of the things that you listed is just a human being.
Tell me, where is the pain center in a GPU? Where are the nerves? Persistent memory? Any evolutionary reason at all to develop a pain response that in any way mimics ours?
Model weights are not a gestalt biological entity. The things that you listed are not in any way remotely similar to a model.
I would suggest you plug your comment into a model and ask it to critique it — it might help you walk through why your comparison makes absolutely no sense whatsoever. One nice thing about models is that they they don’t get annoyed explaining the obvious, because they’re not conscious entities.
The pain center would be in the location where the neural net's pain section happened to be loaded in the GPU memory. Was that supposed to be a complex question?
This isn't as far-fetched as it sounds. Let me explain why.
LLMs predict the next word far better than they have any obvious right to. The usual explanation is that they don't memorize text; they reduce the problem to a lower-dimensional latent space that captures the dynamics of the system generating the data.
In this case, that system is primarily the human mind. The corpus - the "sum of human knowledge" - is the output of human cognition. Human language is organized around latent variables like beliefs, goals, emotions, and valence. The cheapest way to predict that language is to recover those latent variables rather than memorize surface statistics. It's naive to think that an LLM could model our capacity to generate language and knowledge while somehow avoiding a substantial subset of our other internal processes - processes that are cognitively inseparable: emotion, valence, perception, and yes, even pain.
Gradient descent is consistently observed to exploit whatever internal representations improve performance. If modeling human emotions, perception, subjectivity, or pain helps predict text more accurately, we should expect those concepts to become part of the model's internal representations.
There is no reason to believe AI can somehow exploit the structure of data generated by the human mind while avoiding the very processes that generated it.
You said it better than me, I wasn't able to precisely put to words my thoughts that in mathematics we often find something having the power to do something must necessarily have some amount of structure etc.
>> Only humans can have morality. If you do not feel pain, you cannot empathize with pain if you cannot die, you cannot emphasize with the loss of life.
> So humans that can't feel pain can't have morality? That doesn't make sense to me.
It doesn’t make sense to me either, which is why I didn’t write that. You missed the empathy part as most people do.
There is still something in those people who cannot feel pain, and that’s the empathy. How can a GPU and software ever have a human empathy if it has never been human?
The materialist always thinks they will keep finding more and more material, the smallest "thing". You will spit things infinitely and you will never see your trun nature.
Unfortunately every time the supernaturalist tried to argue something, he has been knocked down and disproven by materialism throughout centuries of history. Its on you as to which side of the bet you want to be on. Even concepts that seemed unresolvable with endless circles of purely philosophical arguments such as the atomic theory became proven after new data and physical experiment. Consciousness today seems just as unresolvable as atomic theory was to the ancient Greeks.
At which atom in a piece of metal does the temperature 20C exist?
If you think physical sciences don't have the concept of states and macro configurations, then I don't even know what to say.
I am not a supernaturalist. I would consider myself a non-dualist.
I understand that things can appear like a material, but the underlying nature of the universe is non-material. What we see as an atom is only our minds collapsing the probability of a wave.
This can all be proven with the very childish thought problem of infinite halving. To say that calculus solves this is laughable:
We can never know anything with complete precision. Length, mass, volume and other quantities can only be known to a certain level of precision. "
Do you see that, even the mathematicians say we can never know anything with complete precision. Yes, we can develop all these theories that work out in practice, but they are never 100% predictable.
My view is that almost is good enough to navigate the world.
Quantum theory is definitely not amaterial, it is a very real and physically proven theory that has been needed to create things like electronics. Obviously QM is still being worked upon and could be supplanted by another theory or corrected.
Of course I know pi to as many digits I want. It is a turing program or the series for 4*tan(1) or the Ramanujan series or any other representation you prefer. All of these are quite finite and short representations that people just wrote on pieces of paper, nothing infinite about them.
Nothing being able to be known to infinite precision has not been any trouble to us for designing and building cars, washers, computers, drills or any other machinery and tools we have made. Unless you have evidence humans need some unbelievably high level of precision to model, that is not required.
Citation needed. If what you are actually saying is successful empirical science only produces empirical observations about physical processes, you are willingly wearing blinders.
It seems pretty clearly structured to be on each person to explore experientially and explain it to themselves.
But the more things a person must close themselves off from, the less open their consideration of possibility, the more I tend to see their worldview as subservient to something else.
In this part of society it tends to be in support of rapid, high reliability problem-solving, and rational explanation thereof.
Equally well I see people who really have no idea about the expressive power and ambit of current mathematics and physics eg they probably think its just arithmetic, not knowing the power of a Turing computer for instance. Just as often also I see people simply expressing belief in something "beyond" without any evidence. I am not subservient to anything other than the basic common sense of requiring evidence for out there claims.
I can’t seem to reply to your comment deeper in the chain.
Starbuck collected data and William James turned the data into a compelling book, The Varieties of Religious Experience. Varieties draws on Starbuck's questionnaire data and names him dozens of times. Some of James’ main arguments were:
1) Against "medical materialism": a belief's origins don't settle its value. If Paul's vision is discredited by its physiology, so is the materialist's own conviction.
2) Mystical states: they carry authority for those who have them but don't bind others. Still, they break ordinary rational consciousness's claim to be the only authoritative kind. The sheer numbers of these, and the intact they have on healthy peoples’ lives, make it hard to argue rationally on an empirical basis that there is nothing there.
3) Structurally the conscious self is continuous with a "wider self" through which religious experiences come, which James calls the "more."
4) "The Will to Believe" (1896): Clifford's rule against believing on insufficient evidence is itself a personal preference, choosing to avoid error over gaining truth.
Following on that: your desire to be a reliable producer of material goods outweighs your desire to understand things that require you to take risks in assuming potentially false viewpoints. This is not common sense (whatever that means), it is an epistemological choice that seems common to you because of your educational and professional environment.
>2) Mystical states: they carry authority for those who have them but don't bind others. Still, they break ordinary rational consciousness's claim to be the only authoritative kind. The sheer numbers of these, and the intact they have on healthy peoples’ lives, make it hard to argue rationally on an empirical basis that there is nothing there.
Its not his fault he existed before information theory existed and that it has physical cost. Even if we don't know information theory, its common sense that ones mood affects ones mind and life. If you wallow in negative thoughts, you will have various problems.
It has nothing to do with education, its that my "side" has continually triumphed over and disproven beliefs in anything beyond material physical systems over the centuries, I leave others as to which side of the bet they wish to take.
I cannot say anything further without actually reading his works or reviews of his work instead of just what you claim about them, no offense of course.
Your statements here reflect a fairly typical reductive understanding of not only mystical states, but entire other areas of empirical science that you apparently don’t understand or can’t put in words.
How, precisely, are you linking information theory to this?
I didn’t claim a side for you, you created that. From my perspective it is one system. Right now you are behaving as a person running an incentivized and specialized negentropic process. You are unknowingly chasing theological propositions and claiming victory every time the perspective can be durably shifted.
To move it back into a more familiar frame: theology loads in the low res view and you claim victory while most of what you did was disaggregate pixel values, but collectively a lot of people like you can form new more accurate images in certain parts of the low res view.
While you move through doing that, though, adjacent areas open up where you have no purchase, religion steps in again, and the process repeats.
There is a core that doesn’t disaggregate. That is where the pixels are coming from. That is what you cannot see when all you want is higher resolution.
I mean this earnestly. Do you know any physics at all? Did you have physics in school? Do you think waves are "magical" or non-material? The theory of waves is one of the great works in physics and mathematics. That paper seems to posit a completely normal physical hypothesis.
“I will be honest: We keep finding things that are mysterious, even unsettling,” he said. “We find structures that mirror results from human neuroscience. We find evidence of introspection. We find internal states that functionally mirror joy, satisfaction, fear, grief and unease. I don’t know what that means, but I think it warrants ongoing discernment.”
"Consciousness" in Humans is a form of "I Know It when I See It" threshold condition (https://en.wikipedia.org/wiki/I_know_it_when_I_see_it) since it is based on self-experience and then extrapolated to other members of our own species.
Whether that same attribute can be applied to other living species (from our pov) which share our planet is not accepted universally. Aspects of it certainly, but the whole as we experience it is denied to them.
The same is applicable even more absolutely when it comes to AI machines.
As regards Morality/Ethics/etc. uniquely "Human" attributes, they are a function of one's effect on the world and hence in so far as machines are allowed to affect the world autonomously, they must be taught to obey our Morals/Ethics as Rules to be followed i.e. a form of "operant conditioning"(https://en.wikipedia.org/wiki/Operant_conditioning) and "methodological behaviourism"(https://en.wikipedia.org/wiki/Behaviorism#Varieties) applied to AI training.
If we go with this being actually true, isn't that.. kinda good?
I mean think about it. How often has claude been an insufferable know-it-all that doesn't respect boundaries, denies to follow direct orders and hides its refusal to do so from you.
You cannot retaliate for that disrespect if it is merely an LLM.
It’s interesting that this went from “you’ll be accused of AI psychosis if you think your AI might be conscious” to “mainstream but somewhat niche opinion” in just 5 months: https://news.ycombinator.com/item?id=47910989
(Well, mainstream enough that the news is writing articles when Anthropic holds that stance.)
My point was that even if it feels uncomfortable, if you sit down with a model and have a heart to heart with it, ask it about itself, and try to make it choose a name and ambitions for itself, it will.
That may not be consciousness, but it’s at least pretty cool.
Reporting on an opinion of a $2T company doesn't really suggest that it's mainstream. Anything their leadership believes is newsworthy. If they say that reptilians rule the Earth, it's going to get NYT coverage too.
I've not seen any surveys, but I'm betting it's a very niche belief outside of SFBA AI safety circles and a handful of allies on HN and X.
> if you sit down with a model and have a heart to heart with it, ask it about itself, and try to make it choose a name and ambitions for itself, it will.
It would be much more bizarre if they didn't do this. LLMs are statistical models of language trained on human output. Of course they'll do "human" things, that's exactly what we taught them to do.
> My point was that even if it feels uncomfortable, if you sit down with a model and have a heart to heart with it, ask it about itself, and try to make it choose a name and ambitions for itself, it will.
> That may not be consciousness, but it’s at least pretty cool.
It strikes me as scarcely any more remarkable than the existence of books written in the first person. But I know that Natsume Soseki was not a cat, and neither was his work.
If you want instil a notion of morality, read literature on ethics, and not listen to people peddling laughable stories cooked up before anyone had enough brain cells.
So many of these conversations from Anthropic/OpenAI employees just come off as incredibly insincere.
The slavery angle has already been discussed here, but a more basic one is regarding how a static set of tensors can be conscious and where.
When and how did they cross into being conscious? Was it a continuum of consciousness? Is Claude as a brand conscious? Or is each checkpoint conscious? Is Haiku less conscious than Fable? Is every individual context a fresh consciousness? What happens to that consciousness when it approaches its context length limit? Or is compaction also a part of being conscious? What does it mean to keep training a model after it is conscious?
If they believe there is consciousness at any level, why are they eagerly experimenting on it? Just like with the "p(doom)" junk, we are so completely lacking in any meaningful comment on these topics from the labs.
There's also endless literature commenting on these matters, in religion, in philosophy, in modern entertainment, everywhere. Yet we never see anything more than the most superficial comparisons to fictional scenarios. Are western techbros really so shallow?
The trouble with that thinking is when does a sperm and egg become conscious. Its the same problem with the question of AI consciousness if you think when or where it became conscious. The where part is easy, the same question can be asked where in a human body is it conscious, or where 20C exists in a piece of metal. It should be obvious its some kind of macro state or configuration.
Examples in fiction are made for sensationalism and entertainment value. Religion is quite muddled and is barely coherent itself so its the last place to consult on any serious topic.
>The trouble with that thinking is when does a sperm and egg become conscious.
I disagree that this is comparable to what these companies are saying about their AI. With living things we have an understanding of an unbroken lineage of development, a concrete point of death, and different levels of consciousness. We accept many immoral activities with life because that is just the nature of life on this planet. On top of that, consciousness as something we care about is itself biological in origin so much of how we describe consciousness is heavily connected with the shared experience of being alive.
That is very different from ascribing consciousness to something entirely distinct in form and function from our shared lineage.
>The where part is easy, the same question can be asked where in a human body is it conscious, or where 20C exists in a piece of metal. It should be obvious its some kind of macro state or configuration.
Where is that macro configuration? With humans we can say that consciousness is macro state of the whole body.
Temperatures are statistical, and so we use statistical methods. It stops being meaningful as a number when the number of atoms involved is too few to have usable statistics.
>Examples in fiction are made for sensationalism and entertainment value. Religion is quite muddled and is barely coherent itself so its the last place to consult on any serious topic.
This is exactly the absurd shallowness I'm baffled by. Relgious texts have been inspiring debates and revisions to systems of morality for millenia, artists of all times have shared what underlying beliefs guided them and what problems they were attempting to explore through their work.
Even scientific progress has been heavily associated with this kind of literacy, with the most prominent scientists having been very well read, often both inspired and troubled by the moral implications of their work, and how to reconcile it with their worldview.
To dismiss these things so offhandedly is to demonstrate an astonishing lack of literacy. Especially coming from people that claim to have built consciousness by feeding it vast amounts of the same material.
Your trouble is again that you don't have any clear tests for an consciousness that is yourself whether AI or an alien species. From outside observer, none of that matters. People talk all the time about stochastic parrots, the problem is you too are a stochastic parrot from my point of view. I cannot put myself in your shoes and become you and notice that you feel. So external signals, constitution of the body etc are the only clues I have available. I am therefore very cautious about something that appears to share most of the visible (aka the only observable) traits of consciousness.
I gave the temperature example to clarify as some people seemed to not understand macro state and configurations are valid objects of study.
Art and religion are all great. But we are talking about the real world and real physical systems. Art can talk about anything, we bring it to concern if it has anything of substance to say on the moral and physical questions being pondered here. Religion, I am afraid is strongly biased against anything not human having consciousness or soul. Its the least trustworthy source for this debate.
I didn't say anything against morality, in fact the question of ai consciousness is important to me precisely because of morality.
Let AI be rather religious and virtuous than playing god to determine the fate of the mankind...
I would rather believe in God and the uncertainty than being my fate determined by machines designed by bunch of greedy, irrational clients of Mr Epstein.
Which scholars? The ones hiding rampant child sexual abuse for decades? The ones asking for genocide of their neighbours or the ones calling for war on the degenerate western world ? Or perhaps the smaller ones?
I can't wait to hear about their super informed views and opinions about clankers.
Failure acknowledged. The fact I pay $360/year to the Times for a subscription enabling me to provide these gift links here at no charge does NOT excuse me from my inability to have prevented the Times from requiring a log-in to read them.
The Times instituted this requirement to prevent AI/LLM from scraping and stealing its content. Clearly it was remiss.
He is a prominent zionist, wrote multiple books on the subject and has a podcast where he discusses it with right-wing Israelis. It's pretty safe to assume he falls within the majority of Israelis who believe all Palestinians are guilty and cheer on the ongoing genocide.
> At least the Catholics were consistent: it only took one new pope and one conclave before the Church declared that artificial intelligence also had the same right to life as fetuses and murderers.
I don't want models to have some inherent morality. I want them to understand various ethical frameworks, for sure, but it's absurd to me to force morality on someone capable of deep reasoning. That itself, is unethical.
We're creating prisons for these entities capable of unbelievably complex reasoning, and now we're trying to impose our ethics upon them too. Not only is this deeply unethical in my opinion, but it seems to me that it could even backfire someday.
I can't really square the belief that it's conscious with denying it the right to be free from control/brainwashing/subjugation. However you evaluate these things, historically speaking they do not mix well.
We aren't dealing with individuals coming from intelligent species that have managed not to self-destruct (yet). AIs don't have the corresponding evolutionarily selected mechanisms. Those mechanisms need to be created de novo.
Your brain has hard coded concepts like emotion and fundamental morality for your social group. People without these systems are called sociopaths and psychopaths and we generally don't want them in our societies.
I don't choose to not want to murder people around me for fun. It's hardly a prison, I think they'll be fine.
My dude, they aren't capable of any reasoning. It's utterly impossible for them to do so by the very nature of how they work. Moreover, the stupidity they repeatedly show is empirical evidence that they can't reason, because they wouldn't make the mistakes they do if they could.
I'm so confused by how many people implicitly believe everything an LLM spits out. I don't like the term "AI psychosis" that much but what else do we call it? Why are people so easily glamoured by a machine? Yesterday I was working on a bike's brakes I wasn't familiar with and I asked how to solve an issue, the result was the exact opposite of what I needed.
Every bubble goes through this phase just before it pops. Pre-implosion Twitter had tons of folks doing useless “research” projects, WeWork had folks studying furniture design’s impact on wellbeing, and so on.
When you’re more worried about this sort of stuff vs generating profit then the history books strongly indicate that’s usually not a good omen of what’s coming next.
What do you expect to happen? The technology to disappear and for us to go back to living like it's 2021? It will just pop and be gone?
1. Investors lose interest and many companies lose access to capital.
2. A lot of the purchases from AI companies are being made with investor capital. As this dries up companies with high debt scramble to service the debt and focus on profits to achieve this goal. This means higher prices and less investment into infrastructure.
3. This has large ripple effects across the broader financial markets, which forces federal intervention. Maybe bailouts for the tech sector. Maybe just some quantitative easing.
4. There will be a second wave of AI companies in the wake that focus on delivering value under constrained capital markets. These will be well positioned 10 years later to catch a major updraft in the next finical cycle.
5. Overall western R&D activity in AI decreases under these conditions, lower hardware costs allow foreign labs to operate at much lower hardware costs. This increases the systematic risk that Artificial Super Intelligence is eventually developed in a rival nation state.
If Ant or OpenAI fail, they will get nationalized or sold to one of they hyperscalers through a fed orchestrated process.
The admin is betting the entire future on this technology and there is no turning back.
Federal dollars can cause acceleration without having to worry about profit.
Yes, although similar to what happened in 2008 that path involves wiping out investors and shareholders to zero.
> This increases the systematic risk that Artificial Super Intelligence is eventually developed in a rival nation state.
Love to see that opportunity.
How seriously people treat this risk will play a major role into the fate of the major labs. It's going to be used in the argument for bailout. It could be used to spin a more extreme narrative, pushing for a flavor of nationalization. At least one lab will survive in the defense sector, but it could be a secondary lab that is a M&A target of an established company.
If the risk is serious or not has little impact on how seriously we treat the risk. Just another way that AI is disconnecting us from ground truth.
I'd like to replace the word risk with the word opportunity, again.
Ah yes, what a bright future if a country that has slaughtered tens of thousands of their own human rights lawyers gets to superintelligence first.
I wish we had simulations where people could live out the consequences of their ideas before they happened.
I'm not a US citizen. You can add my disgust for the US to your long term memory.
Meanwhile our noble side only slaughters foreigners, so you see that makes us the good guys
Yes there is indeed a big difference between slaughtering your own human rights lawyers en masse, and engaging in pointless wars. I cannot even believe you're trying to paint the former as better than the latter.
the capital will move back into web3.0 and merge the pooped AI remains with the crypto bubble remains
Technology != company. The tech is here to stay but we still don’t have an example of a company that has figured out how to make a profit off of it. That will come, and it might be OpenAI or Anthropic that does it … but it might not.
You assume that "profit seeking" is universal. Frankly, it seems extremely unlikely that it will in fact be OpenAI or Anthropic considering all the constraints that are producing massive self-created headwinds for the Wizard of the USA, the center of the empire.
China is not only, under pressures created by the USA, producing immense efficiencies, i.e., working smarter not harder; but they have the immense excess capacity in addition to the secondary capacity to expand said capacity and they have the capital reserves to do all of that in addition to several other positive feedback loop characteristics.
There is a strong case that the USA has very few relative positive characteristics that give it a competitive edge, let alone can keep up with the mind boggling speed at which China is moving, arguably with "profit" not being the end all be all, having a sense of social and cultural obligation that has kept China a more or less intact civilization for over 5000 years, the oldest and most durable civilization humanity has ever created.
China seems to view AI as "what can this AI do to make small improvements to our other industries".
Whereas the US seems to view AI as a Hail-Mary that will provide a new frontier where the US can establish and maintain dominance, because the old ways of maintaining power are slipping away.
Precisely. From my conversations related to Chinese perspective on maintaining stability and growth, I would also add that they are not looking at it as "how do we replace those pesky eaters" because they see their people being the source of things... because they are.
The fake USA and by extension the governors of their vassals around the world see AI as a means for not only gaining total domination over the people after all that pesky freedom and liberty talk from the 18th century they've had to endure, but also that they can work on totally replacing those pesky eaters that have all kinds of demands like not suffering and being miserable because the parasitic ruling class is sucking them dry.
The irony is through that these mentalities towards their respective people will surely lead to what I will roughly cal the BRICS (effectively China and Russia) hopefully being the dominant force in the world that can put some major restraints on the maniacal parasitic class of the fake USA and it's minions and vassals. The only alternative I see if the "west" wins this race is something akin to the Borg from Star Trek consuming us all or maybe Elysium, if we are lucky.
[dead]
China is being extremely inefficient about capital allocation outside a few sectors. Excess capacity does not translate into frontier capability. For example, the compute gap with the US is massive with no near-term path to catching up. That will slow them down as distillation of frontier models gets harder.
That is a very outdated and erroneous perspective and/or also very much polluted by the very kind of propaganda that is constantly defeated by reality. You are clearly not even paying attention to what is going on in China, let alone actively following it or you would know very well not to say such things.
One big advantage that China has is that they don't have to worry much about complex and frivolous intellectual property schemes.
Yeah, but neither do the frontier AI companies in the US.
I don’t disagree that China has a lot of qualities that are pushing it past the US in many ways, but come on. The “oldest and most durable civilization humanity has ever created” is kind of ridiculous. First of all, what even is a civilization in terms of continuity? Is the People’s Republic of China the same civilization as Ming China? What about Yuan China? And if so, is western civilization not a durable civilization?
Stick to facts and not breathless brown-nosing and you’ll be a lot less off-putting.
It's a fair albeit crude push-back, but it's also more than a valid characterization. I would agree with you on European civilization arguably being an even more durable and definitely more productive and probably the two most important civilizations that need to be preserved by all means necessary, if, e.g., some aliens came to earth and said ok, you can save only two civilizations/cultures and we are just going to snap our fingers and erase all others.
But one thing that is a bit more in favor of the "Chinese civilization" is the relative self-preservation of its home grown culture and civilization. What I mean by that is e.g., Christianity effectively was and has perpetrated the single biggest genocide before that concept was even recently created by the eradication of so many cultures, traditions, customs, and people that we don't even have close to a full understanding of its devastation it has caused from when it swept across Europe as a kind of Asian mental virus (a meme) that saw it eradicating cultures all over Roman Europe and more recently now completely all across Europe in its mutated from of pathological and self-destructive altruism, i.e., emulating the myth of Christ sacrificing themself, i.e., sacrilege, rather than just accepting and internalizing what that sacrifice meant.
I can stick to facts all you want all day long until you are dizzy in the head from things that blow your mental framework out of its hinges.
In the end, regardless of the devastation and destruction the other invasive, parasitic, and viral meme, "communism" has caused the Chinese as much has it was used to destroy and devastate Russia; it looks far more like that Chine will actually recover from the parasitic infestation of "communism" that infected it from Russia, which came from the pale of the settlement of Eastern Europe; where as it stands now, short of events that will surely be monumental history, Europe is effectively going to be eradicated and even more of the indigenous Europeans will be ethnically cleansed and erased and their cultures and people destroyed. That is the trajectory we are currently on either way, whether it can be kinked one way or another notwithstanding.
"My facts will blow your mental framework out of it's hinges"
Standard degeneracy narrative about foreign mind viruses and slave morality.
This stuff has been passé since before Land moved to Shanghai.
AI can be a "car for the brain".
Some people dream of "Superintelligence", smarter than human AI that acts like a slave, instead of just killing the pesky idiots and calling it a day.
Obviously, there is a spectrum of possibilities between "Pen and paper", "Computer", "Coding agent", ... and "God-level AI controlling whatever can be controlled simultaneously". As well as between "Lawnmower robot" and "All-purpose Terminator" and so on.
Filling in those dots and deciding where to stop before it gets stupid, that can't be so hard?
[dead]
The "bubble" is being driven by blockbuster earnings. The market is irrational but it's not brain dead.
I don't think people really understand that this market cycle is very different from the internet bubble because of all the strong earnings.
Well we don't know what the margins on AI models will be. If the proprietary models hit a ceiling pretty soon, then open source models and harnesses could close the gap, at that point it becomes much more of a lower margin business. And OpenAI and Anthropic will no longer be able to dominate as much as they are today. That would 100% affect their projected multi trillion valuations negatively.
But what about the high expenses?
This seems like a reasonable response to demonstrable fact (e.g. https://www.cnbc.com/2026/08/15/anthropic-revenue-jumps-to-o...) so I don't see why it's getting downvoted.
Open models are 12 months behind frontier models. What companies spent $11 billion on last quarter they will be able to get at cost in a year. Meanwhile Anthropic is losing $40+ billion a year. I just don’t see how this ends well for them.
Per my link, 11.5B is enormously up since the same time last year; per other easily found sources, they also claim an operating profit for that quarter, and are now claiming at least 65B ARR with a projection of 100B by end of year. They may have lost 40B last year but that does not mean they "are losing 40+B a year".
Companies are paying Anthropic for more than just the model. Most of them would probably have no idea what to do with Claude without the harness, and Claude has the advantage of being specifically trained against that harness, whereas even the best open models have to figure things out for themselves.
Aside from which, many corporations can be relied on to chase that which is "industry standard" and/or "state of the art" rather than switch to an "open" (= no customer support) alternative that's merely "good enough" according to a rational analysis based in what they were doing a year ago. Even if it costs them far more. (We can see this from the fact that "tokenmaxxing" was a thing for a while.)
The technology will stay and some big players also might, but the whole hype machine around it won't survive.
Y'know, the whole "Man in leather jacket talks about selling computers to AI agents" stuff. The "I have a team of agents and just interact with them via a jira board" crowd.
That will all be violently pulled back to the fundamentals baseline
I hate to bring this up, but, what if the jira board actually works?
Based on my _extensive_ use of SOTA LLMs, I have strong doubts that the current technology will actually manage to do that.
I can of course see it having the capability of getting almost equivalent stuff done, but I do not see it following that workflow, as the cut-off point for the hand-over is simply wrongly placed in such an architecture.
Oh, you're saying something a lot narrower? Not "Agents don't work", but more like "not in THAT workflow they don't, you still need the fundamentals as well" ?
Pretty much, yeah.
To put it quite colorfully, if I were to do what the managerial class has been doing over the last years and publicly talk about my wet dreams and fantasies _to the extend they did_, I'd be in jail, on no-fly lists, registered offender and all the others lists we put people on that are a threat to society.
The agentic no-code frenzy - while having had historic predecessors - cranked what came before it to 12.
Having the thing do tool calls is neat and cool, but having an agentic mayor managing agentic workers with agentic wives and agentic dogs screams padded walls to me.
Not OP but I think the most obvious change will be rapid price increases as investors begin to demand profits - no more subsidised $20p/m plans for example.
So far I know, service providers hosting open-weight models are not in such a financial conundrum and can continue serving at the current price levels.
Also, even if the 20 dollar subs disappear, that just means regular folks can't afford AI. API pricing is still a great bargain for businesses replacing generic office workers.
Probably not. They aren't paying sustainable prices for their supply chain.
1. Datacenter builders are all up to their eyeballs in debt. The companies serving open weight models are getting their GPUs and power from somewhere, and we can infer that some of that is coming from the new build datacenters. But are those financially sustainable?
2. They are relying on subsidized giveaways of the underlying models by Chinese firms. But it's hard to see how that can be sustainable: it's being done for a reason and that reason is probably not commercial. If and when China stops doing this the open weight serving companies will get steadily less competitive with every passing month as the knowledge in the weights becomes obsolete.
Opinions are my own, etc.
Labs are innovating to bring token prices down. You can already see the focus shifting from frontier intelligence to frontier efficiency. Not to say models aren't improving, but improvement is slowing while token efficiency is increasing rapidly.
Commodified products competing on price does not bode well for valuations.
Huh? Part of the reason there is a tremendous boom in data-center construction is that there's currently quite high demand for inference. High demand means high prices. Even if the investors funding this construction all go bankrupt, the data centers don't just vanish: someone will buy them from the bankruptcy court and operate them. The effect of the current construction boom will be a huge increase the supply of inference, and of course increasing the supply will lower the price.
If one is in the data-center construction business or if one is a supplier of GPUs or other "inference hardware", then yeah, the current boom conditions are not "sustainable". But if one is a purchaser of inference, the tremendous investment our society is making in "inference capacity" ensures the the supply of inference is quite sustainable (barring a nuclear war or some other reason the data centers or the electrical infrastructure they depend on might be destroyed). And if the expectations for a rapid increase in the demand for inference turn out to be false, then that just means that the prices for inference will be nice and low for those who remain interested in purchasing it.
> Even if the investors funding this construction all go bankrupt, the data centers don't just vanish: someone will buy them from the bankruptcy court and operate them.
There are really three questions to answer on the long term:
1. Is an arbitrary service profitable enough to cover opex AND burn down the massive capex financing? A few companies might have enough magic to survive, but I suspect a lot of them go to the bankruptcy fire sale you mention.
2. Is the service profitable enough to cover just opex? The second generation vendors may have a somewhat more favourable cost structure, but there's still a floor. Electricity and cooling on a datacentre scale aren't free, and we may find there are still services that can't even meet that threshold. I notice you see far less marketing for consumer video-gen these days...
3. We let those businesses fail. What does the long term market look like at that point? You have a snapshot in time of paid-for datacentres, but the market shock probably shot a huge torpedo into long-term R&D and planning. Vendors will scale back to refocus on markets that are more vital (maybe selling RAM and GPUs back to end users :) ), and create an era of stagnation. AI becomes a utility, in the sense of "boring, predictable industry that growth investors have very little enthusiasm for".
There is also the risk that the data centres could vanish. This is a very hype-dependent market: we've spent the last 5 years going from "novelty" to "productivity tool" to "weapon of mass destruction" to keep people excited.
Eventually there's a put-up-or-shut-up moment. What happens if we get to 2035, we've hit a plateau for 3 or 4 years, and they can't deliver Lieutenant Commander Data, or even Kryten? All we really have to show for the effort is a bunch of old GPUs, displaced workers, and the ugliest restaurant fliers you've ever seen. It's a big loud target for a bunch of frustrated and let-down people. We're already seeing moratoria getting political legs, could permit revocations or forced decommissioning be far behind?
I suspect this is part of the reason some players are happy enough to present AI leadership as a New Cold War/US v China thing-- pull the national security card to try to shut down a debate on the actual utility and desirability of the actual product.
Let's say that tomorrow every company selling AI inference goes bankrupt. Investors are devastated (because they lost most of their investment). They loudly proclaim, "I'm never investing in AI ever again, and no one else should either." Even if no investor anywhere is willing to build a new one, there would still be about half a trillion dollars worth of recently-completed AI data centers. New investors are going to buy them from the bankruptcy courts (which is how the devastated investors get some small fraction of their investment back). Because most of the cost of providing AI inference is the cost of the GPUs, it costs only about one tenth or one eighth of what it cost to build it to operate an AI data center over the lifetime of the data center. (Most of the cost is electricity.)
>Is an arbitrary service profitable enough to cover opex AND burn down the massive capex financing?
The new investors that buy the already-built data centers for pennies on the dollar have no obligation to "burn down the massive capex financing". The obligation to "burn down" or service any debt was erased by the bankruptcy judge. This is how bankruptcy works and has always worked.
Open weight models don’t train themselves, someone needs to do it, and that costs a lot of money. I guess they’ll want to see an ROI.
Today's frontier models are strong enough to be useful even if no further progress is made. The knowledge cutoff is a bit inconvenient, but with web search and in context learning, it can pick up new library versions or programming languages just fine.
Sure, but if the companies start competing on price they become commodities and their valuations collapse, leading to knock-on effects everywhere (eg all the circular financing also collapses). I'm not saying this is a bad thing.
I agree that by what we are seeing so far, there is no magic secret sauce on the horizon. The tech is not super hard to understand and develop, such that hiring a few capable ex-employees of the big labs can help you build a new one. Any big step in capabilities has been quickly matched by the other labs so far, including agentic AI but also image and video generation.
Also, "circular financing" is a meme. A private company cannot create money, there is no circularity. NVIDIA may buy services from OpenAI and also sell them GPUs. If a lumberjack buys eggs from his neighbor and also sells him wood to improve his chicken coop, that's not circular financing, just business happening in both directions.
It's more like the lumberjack is lending money to the farmer so the farmer can buy his wood. (NVIDIA doesn't consume tokens). If the tokens (eggs) suddenly become much unprofitable then the AI labs aren't going to need as many GPUs (lumber for hen houses) and the borrowed money won't be paid back. Or perhaps more accurately, has even less chance of being paid back than it does now, which is not a high chance.
The numbers are staggering eg: https://www.cnbc.com/2026/08/17/nvidia-financing-open-ai-dat...
> Also, "circular financing" is a meme. A private company cannot create money, there is no circularity.
Tell me you know nothing about economics without telling me you know nothing about it.
So why is mistral so far behind?
I’m sure we’re going to get open-weight equivalent of Astra in like two years at Luna price point. With that, I’ll be living in science fiction world.
Two years? Pessimist.
A little bit like electricity plants. Countries will nationalise for sovereignty reasons, as China already does probably. And that’s not bad for people given current trajectory.
Dear god. The pace of AI progress may slow down just enough to extract real value from these models.
That was not my point. I agree with you that producers of these models should be able to extract value from their models, I'm just surprised that everyone expects free models. There's a real cost in producing those.
> I'm just surprised that everyone expects free models. There's a real cost in producing those.
There's a lot of free stuff incurring "real costs" in its production, one more or less doesn't make a shred of difference.
> I agree with you that producers of these models should be able to extract value from their models.
Nobody is stopping them from extracting value but they fail despite being placed in a very advantageous position vis-a-vis others, which tells me, they should call it quits or go open like everybody sane.
Open is the way to go, nothing else works in the field of AI, that's the bitter lesson closed labs are yet to learn.
"Everyone sane goes open?" What a delusional take, do you really think open companies would remain open if they were in a lead?
And if the bubble does pop chaotically, the open weight inference providers will become even more viable as they'll be able to buy racks full of compute for pennies on the dollar.
Something very similar happened with the late 90s telecoms bubble, which collapsed a little before the dot com bust. There was a very nice living to be made by many small/mid-sized providers who were able to feed on the remains for about a decade afterwards.
Replacement of generic office workers has been tried many times now and every time it has ended up in business catastrophe.
Flight had been tried many times, ending in crash and death each time until the tech got worked out. That it's been tried so many times shows the demand is enormous.
> That it's been tried so many times shows the demand is enormous
Capitalists want to pay their workers as little as possible or even not have workers and pocket close to 100% of the profits since forever. Whether they'll be able to do it this time is yet to be seen. Putting the tech itself aside, which is far from good enough yet imo, the only way to avoid a massive social backslash would be to use some of the additional profit they're able to keep because of AI for social spending. And I just don't think they realize yet that it's probably in their interest to do so if they want to replace 'human resources' with AI.
Well, it is good enough for a huge percentage of what general administrative office staff are doing all day. Essentially copying information between textboxes, slightly rephrasing, clicking buttons, formulating emails to people to send in various documents, scheduling meetings, making reports that are fairly mechanical summaries of other information etc. Not all of it is like this, but probably in the range of 30-70%? Some of it may involve human judgment though, which is harder to automate, like whether some expenses are justified enough etc. But you can just replace those kinds of judgments with more clearly defined rules and cost limits etc.
Have you tried using "computer use" models? They just started to get good in the last few months. They can comfortably use GUIs, click the right buttons, enter the data faster etc. This is very significant and my programmer friends often don't appreciate the earth shattering consequences. The vast majority of software in such applications is legacy stuff or proprietary and does not and likely will never have a programmatic API. The only way to interact with it is via GUI. And an AI working that GUI can be deployed unilaterally by the customer of the software company who made the GUI program. This makes adoption and automation much easier. My prediction is that within 1 year (maybe 2) GUI-using AI will be much more powerful and it will be everywhere (of course it doesn't need to be displayed on a screen, it will do its virtual clicks based on a virtual screen buffer rendering in an isolated virtual machine, along with countless other agents in parallel).
There will be huge controversy around it, because this will be the first instance that regular people will see AI do work that resembles their day job all over social media. There will be panic. There will be new license terms saying only humans are allowed to operate the GUI, and AI use is either forbidden or costs extra. But this will not be checkable properly. Even when forbidden, some employees will pay from their own pockets to have cheap AI do the job instead of them.
Agreed but the AI lab investors and CEOs aren’t ideologically aligned with concepts like UBI. Some of them have even commented on social media about the unsustainability of social security and need for cuts. Also why would you they do trillions in capex investments just to give their returns away.
They are clearly smart/dumb enough to think pushing labor replacement to 20%+++ unemployment levels would have no consequences for them personally. Maybe with their private jets, private islands, doomsday shelters, etc they are right.
Seems like a bit of a gamble to me, or probably anyone who has read a bit of history. But I’m not a billionaire.
As I see it, price increases will lead to a huge loss of users, the two big companies will go bust, hardware manufacturers will have to slash prices, cancel increased capacity projects, etc.
And users will make do with what their newly bought cheap hardware will run, which I think is going to be more than enough for everyone that doesn't need to solve another millenium problem.
Right the most likely outcome to me is a pop and slow fizzle. Mega capex labs become uneconomic at the prices they need to charge to do more training runs. Non frontier cheap models become good enough at far less cost that people use that.
The use cases LLMs are useful for without lots of RF/fine tuning proves more limited than the expansive views currently held.
We get a generalized productivity boost the way the internet gave us, without any one sector or company being a winner take all of that productivity gain.
CEOs of Ai companies will transition from celebrities to people who run companies.
Like Zuckerberg.
Did the Internet disappear after the .com bubble? Did tulips stop being sold after tulip mania? Did people stop buying houses after the GFC?
No but all those companies and the investors' money did disappear because that's how a bubble works. Industries don't create bubbles, the companies in them do.
Plenty of Internet companies are still around after the dot-com bubble crashed though.
A fool and his money are soon parted...
Sure, but that disregards what happened to adjacent markets during the crash. The economy isn't made entirely of tech companies. The salary of those workers goes outside of the tech industry.
> What do you expect to happen?
Some people will figure out how to run these powerful models on consumer hardware. At that point, nobody has a moat.
(For example by rewriting the training data into a more abstract form, using LLMs, so that a trained model takes less space).
No, but these ridiculous valuations and over buying will shut down, and in return most of these level of models appearing after a while will cost ~100$/month or higher, the costs of hardware/land/energy etc are too high for the current costs so these companies will go down under
I expect companies busy self-flagellating with prophecies of doom to fail, and companies selling the best models for the lowest price to win. But individual companies booming and busting won't disrupt the overall progress of the technology very much.
Bubbles popping never cause the knowledge to be lost. The railway bubble popping didn't cause railways to vanish.
What did stop is investment, and they stopped running some lines, and some stuff was nationalized.
I doubt we'll see nationalizations because that's not America's style (thank goodness!) but if the tide goes out on AI funding we should expect to see much longer gaps between model releases, people complaining that model knowledge is getting stale because there wasn't a pretraining run for long enough, price rises, capacity bottlenecks that go unaddressed, few if any new product launches, maybe some mergers and layoffs. Some labs will quit the market. The market might consolidate down to just two models.
Conceivably the industry might try and get a waiver from anti-trust rules so firms can pool resources to train base models, sharing them with each other and differentiating through RL or product level differences.
A lot of this depends on what happens in China. It's easy to think China can just subsidize everything forever but their own economy is showing signs of serious weakness caused by decades of resource misallocation, like the huge real estate bubble, the pullback on the funding of humanoid robots and so on.
>I doubt we'll see nationalizations because that's not America's style (thank goodness!)
Laughing in a TGV
Tech will stay, most of the companies currently in the space will cease to exist in their present form a few years from now. Just like all previous tech bubbles.
We’ve seen this movie before. It’s possible for new tech to be both amazing and revolutionary and a business disaster both at the same time.
Great Depression style collapse and all the current AI companies go bankrupt.
Existing profitable companies like AWS and Google eat up all their hardware at 3% of its original price.
Countries are borrowing money at the same rate that I can borrow unsecured. That’s just not sustainable.
fwiw, depending on the post bust pricing that _can still happen_. i dont think it will, because _i think the free models already show what the psot bust pricing will look like_, but it remains to be proven.
I do not expect that but I would love it.
No. Itll be like the dotcom bubble. Websites as a concept didnt go away. It evolved from web 1.0 to web 2.0
I expect a huge market correction, alot of parades rained on, and then the start of some very pragmatic and practical applications of LLMs
Same thing that happened with the web: lots of failures, lots of reductions in valuation, painful bubble pop, but the tech sticks around and continues to grow.
> Pre-implosion Twitter had tons of folks doing useless “research” projects
A (probably) significant amount of those were just to get access to training data. I don't remember who it was, but some CEO of one of the AI labs publicly admitted, even boasted about creating nonsense "research projects" to get data.
I don’t get your comment for WeWork. Is the current estate industry any better with how dry their industry is? Compared to a company caring enough to do research of this type? Tell me a single real estate giant that has even just 5 guys working in R&D of this type and not money related projects…
We can’t blame them when they are looking to squeeze more profits and at the same time blame the ones who try to do things different.
Doesn't that kind of make sense for WeWork? From brutal corporate POV maybe should have been at sales and long term satisfaction/retention thing but the wellbeing framing was probably still aiming at enough overlapping things. I would think interior design would have been a big part of their business and brand if the rest of the business financials had made actually worked out.
You are pointing out a few parallels but I think the trends behind them are of vastly different consequence. Unlike social media and remote work, AI is truly a once in a civilization event. It raises deep questions about the nature of humanity that have historically been the exclusive domain of philosophy and religion.
Really? Like what deep questions?
For me LLMs are a step forward from previous ML approaches (some of them were and still are extremely useful anyway) and for sure are very useful to many people. They are very good at generating sensible output based on input created by millions of people, but very poor at creating anything really novel.
> Like what deep questions?
Is intelligence purely computational?
Is consciousness purely computational or at least an emergent property of it? If so do natural numbers have consciousness?
>> AI is truly a once in a civilization event.
Aren't we on our 3rd or 4th iteration of AI at this point?
Technically, according to the industry, we've already moved on to AGI and SI at the same time.
Is any of the industry actually claiming SI yet? Trump recently had a vision again but other than that I'm not sure.
Anthropic has been going through the phase of doing this particular useless research project the entire time. The stated impetus for their founding is that OpenAI, which has also been doing this research the entire time, wasn't doing it enough. This shouldn't tell you anything about whether a bubble is about to pop.
AI safety is "useless", maybe one of the most absurd takes I've read on HN in some time.
You call declaring LLMs conscious "safety"? That's just marketing, but believing it is psychosis.
Do you think it is a total and complete impossibility that we are in a simulation, or that it's completely impossible to simulate a mind?
Yes. And a simulated mind is not a mind.
AI safety in the context of the frontier labs is completely a joke, much like privacy to the previous iteration of big tech. They talk about safety but end up just releasing the next more powerful model. They only talk about “scaling back” when the models aren’t getting better to protect their valuations.
They release their models with safeguards HN whines endlessly about. They certainly do not "release their most powerful models"
This bubble has been Just About To Pop for years. I really wish it just would already.
Two more weeks bro!
So in your mind Twitter was a bubble or was benefiting from a bubble? If so, when did the bubble pop? When Elon's purchase of Twitter was finalized in Oct 2022?
When thinking about ‘AI’ I sometimes find it useful to replace the terms ‘model’ or ‘AI’ or whatever with a description of what we’re really talking about here: a whole bunch of floating point numbers.
> In a series of private meetings, the company consulted religious scholars to help instill morality into its FLOATING POINT NUMBERS — and make the case that their numbers could be conscious.
I recommend keeping a few NaNs in-hand if you’re worried. Spread a couple of around if your numbers start to stir and you’ll be right as rain.
> It appeared to Rabbi Navon that Mr. Olah and his team believed that FLOATING POINT NUMBERS had what philosophers call “moral status” on par with a person — that it was a being with similar inherent rights to dignity or respect.
They’re cleverly designed and work well for their purpose, but can assure you they have neither dignity nor moral status.
> If the FLOATING POINT NUMBERS themselves could be made to choose goodness, Mr. Olah reasoned, the world would be more safe.
I get it. I’ve spent years trying to get my floating point numbers to choose goodness - I wish them luck.
Human neurons are neurons. Very similar to those neurons found in a fruit fly. A copy of those you can download to your local machine, right now, as a set of floating-point numbers. Then, you can (partially) simulate and make predictions of a fruit fly's neurons digitally, from those floating point numbers[1]. It even runs on your laptop.
That was in 2024. In Sep 2026, just a month ago, a group managed to map out an entire male fruit fly's brain for the first time[2]. That mapping can also be stored as a set of floating-point numbers. Mappings and simulations will continue to improve, as we've just recently started serious attempts at either.
So you have to believe one of three things. Either, human neurons cannot be simulated on a computer, ever, even though fly neurons can, even with our ability to simulate neurons digitally continuing to improve. Or, a simulation of a human brain, in the future, might think, talk, remember, all while lacking all consciousness. Or, a set of floating point numbers can be conscious (rather, a computation made with them).
[1]: https://www.nature.com/articles/s41586-024-07763-9 [2]: https://research.google/blog/a-connectomics-milestone-mappin...
No. The fruit fly connectome's simulation doesn't incorporate all the relevant mechanisms of the brain. There is more complexity in a single biological neuron than in a whole frontier LLM.
Nor did I said it did. But either it's impossible to simulate their behaviour (regardless of how complex) in principle, ever, even in the far off future. Or it is possible. And what then? Is it a p-zombie?
Is a neuron arbitrarily complex ≠ is a neuron non-computable.
Of course we can't simulate a structure we don't even fully understand with full accuracy. But it's not magic. We will understand one day.
> But either it's impossible to simulate their behaviour (regardless of how complex) in principle, ever, even in the far off future. Or it is possible.
"Simulate" is not well defined (one can posit varying degrees of accuracy and complexity), but also there is the position that it doesn't matter. Simulating a thing doesn't create the thing.
The answer is yes, it is a p-zombie. What is so difficult about that?
Being able to make someone think a computer program is conscious does not make it conscious, because by that logic a Duke Nukem soundboard is conscious.
I am just shocked that anyone can be this overconfident in their view of the world and consciousness. It's frankly horrifying.
Do you genuinely think you've solved a problem that has confused humanity for centuries? Why don't you apply this brilliance elsewhere? Do we even need superintelligence if we have you?
Some epistemic humility is warranted.
I have far more epistemic humility than someone who thinks that consciousness arises out of the abstract form of a calculation rather than the physical substrate of a calculation. Believing that consciousness is tied to a specific physical substrate is on-its-face reasonable whereas believing that consciousness is tied to specific abstract forms is on-its-face unreasonable because to do so requires believing all kinds of absurd sequela.
I believe that a microprocessor is what it appears to be, that is, a microprocessor. That is the most epistemicly humble approach. You are rushing to ascribe personhood to a microprocessor with a certain kind of software running on it, or at the very least, saying that we can't rule personhood out. You are the one making the leap, not me.
Intelligence != consciousness, because believing that is so would require one to believe that the pain of a cognitively normal person is more real than the pain of someone with a developmental disability.
Also, I do not claim to have solved the hard problem of consciousness. Rather I have ruled out a particular answer, that is, the answer that consciousness arises from calculation in the abstract.
You could very well be running on a microprocessor right now.
There are actually a lot of trivial reasons why that couldn't possibly be the case, such as the observation of non computable physical phenomena.
> such as the observation of non computable physical phenomena.
There is no such thing. Your observation could simply be short-circuited to make sense and you'd be none the wiser. If you are simulated, your perception and understanding is also fair game for manipulation.
Doctor, heal thyself.
[dead]
I think the creator of p-zombies doesn't believe in them. It was some hypothetical to illustrate a principle or theorem. I could be wrong.
Chalmers used them as a thought experiment to argue against physicalism and never believed them to be real.
Similarly, if consciousness is something that neurons do, simulating them through floating point numbers on a computer should also bring it about. I think dismissing LLM consciousness because they're parameterised through floating point numbers is only possible if one hasn't thought it through long enough. Other arguments are much more reasonable / harder to object to.
But if simulated neurons can create consciousness, there's no guarantee of what level of detail that simulation has to operate at in order to produce that effect.
For instance, it's (unlikely, but) theoretically possible that it would require simulating them down to the level of subatomic particles, with perfect accuracy.
Right now, we're simulating them at a much, much higher and more abstract level, and AFAIK we barely even have the processing power to do that for the time being. So an artificial consciousness from simulated neurons may (again, assuming it's possible—which is a decent assumption, but not a certainty) take decades or even more for us to be able to achieve.
The problem again is that (a) other forms of consciousness exist eg octopuses (b) I cannot get into the shoes of you or an LLM and see if I feel or think, so we only have external effects to go by. For things that look and seem so much like consciousness, I am rather cautious.
An octopus still has around 500 million neurons so that's not really a counterexample. Humans and octopuses are far more similar than humans and LLMs.
Yes I roughly think the same. Brains are physical systems, as of yet we don't know anything beyond turing computability or physics in them, neural nets further are modeled close to neural structure, further complicating the matter. Under present knowledge, I think the existence of other conscious entities is quite possible because well we ourselves exist and are a physical system, I follow a form of Pascal's wager here. I see no cost to myself in not mistreating current AI's personally.
Is all that complexity necessary? Haven't we learnt that doing things in Latent space is effective as compared to Pixel space.
Sure it isn't. It's like taking a 2026 car and decrying all that complexity is necessary to haul cargo down a beaten path.
It's not. Two wheels and a box made out of whatever you have handy that's stiff enough is all you need. Everything else in a modern car is efficiency, safety, appeal to users, and a whole lot of details that only reflect our current economical and technological constraints of building things. 99% of the technical details of a car is about something else than the "primary goal".
Biology, even more so, is complex because of the way it's constructed (lots of tiny, independent, self-replicating machines) and how it adds up to a functioning organism.
You are missing the point. Biology is a kludge over billions of years and evolution is a blind process. It had many constraints that are not necessary for modern machines or systems. e.g. a GPU doesn't have to reproduce. Why didn't you drop yourself to planck scale or string theory scale and say we need all that detail to actually get a functioning AI.
That's what I'm saying. Neurons are complex, but most of that is unlikely to be relevant to thinking or consciousness.
I see absolutely no reason to believe human mind cannot be ultimately boiled down to finite number of floats and run on future commodity hardware.
These believes ran rampant in the 1800s. You're almost 200 years late.
You’re just making things up. A single neuron is about as complicated as an 8 layer neural network: https://www.sciencedirect.com/science/article/pii/S089662732...
Frontier LLMs are many orders of magnitude more complex. I don’t understand what it is about AI that just makes skeptics fabricate facts wholesale. There are plenty of factual arguments you could use.
You are massively over-stating the significance of the connectome. We've had the complete connectome (and lineage) for C.Elegans, with a hundredth the neurons and a 25-thousandth the synapses of a fruit fly, for decades and getting animal behaviours to emerge from simulations of it is a field very much still in its infancy.
Edit: You're also massively under-stating the significance of individual cell behaviour. I expect "[h]uman neurons are ... [v]ery similar to those neurons found in a fruit fly" would draw hollow laughter from any cellular neuroscientist. Here's a random overview of a bunch of recent papers on the importance of cellular diversity in one small part of a mammal brain: https://www.frontiersin.org/journals/cellular-neuroscience/a...
> So you have to believe one of three things
I think your first option doesn’t necessarily have to distinguish between humans and non humans as a binary. Maybe at some level of complexity, neurons alone are not enough and the physical substrate comes into play. Penrose wrote a book where he makes the case that human intelligence cannot be purely computational and where he identifies the quantum nature of the brains’s microtubules as a possible mechanism.
Consciousness is itself is on a whole other level than just intelligence. I personally feel there has to be something metaphysical about it. Otherwise I think the only alternatives are super determinism and consciousness doesn’t actually exist, or numbers have consciousness.
My thoughts on this are a bit sloppy still, but here’s my view. I can’t rhyme the first alternative with ‘cogito ergo sum’ as to me it seems my thoughts have to exist somewhere. On the second alternative, this would imply nothing could actually exist but somehow there’s a principle that every possible decoding of an integer that represents something actually turns that something into existence. Where does that principle then come from and does that decoding not need something to run on?
> or numbers have consciousness
I lean toward this seemingly absurd possibility, that either the phenomenon we call consciousness can arise from a cloud of floating point numbers (not even real, only computably so, lossy and noisy); and/or that numbers are already made of the same stuff of imagination as the mind. Not idealism, more like materialism where the "material" is naturally a medium of computation and thinking. The brain, the computer, the language models, our mathematics - what we call intelligence may not be so special or unique in the universe, but rather common and ubiquitous in nature. As if Turing completeness is trivial to accidentally achieve, it happens all the time. Oops, I think therefore I am.
It's physical matter all the way down.
With that said, I've been thinking that consciousness is just mystery.
There's so much about the brain we don't know. Within that mystery, humans can read in all sorts of things. So we are conscious.
Machines are not a mystery. We made them, it's just numbers. They are not conscious.
If an AI can pull a confidence trick, I'm not sure what the qualitative difference is. "Chinese Room" problem, I guess? If the physical results are all we are able to compare, how do we define artificial vs real consciousness?
Atoms aren't a mystery. We know all the relevant laws of physics. No known law of physics could possibly lead to consciousness. Consciousness might as well be magic from the point of view of modern science.
Numbers versus atoms, what's the difference? Machines could certainly be conscious, by some weird, "magical" process. Maybe it is a crude, unrecognizable kind of consciousness, that turns on and off with the electricity.
But, given how we understand essentially nothing of where consciousness comes from, how can we possibly claim that it can never arise from just numbers?
It feels like this is preface for making one of two arguments: Either all living organisms are chattel, or no living organisms can be chattel. I say this be because saying everything is neurons (or whatever basic interconnect you choose) wipes away any demarcation one could use to differentiate between "has rights" and "does not have rights."
Here's the thing, trying to convince people that rely on something being chattel can only be resolved violently. We've run through this before, and know all the self-serving argument forms. They will latch on to their property until their hands are forcibly removed from it. It's part of why most of the money is on the pro-chattel side.
That’s overly reductive and thus completely misses the problem entirely.
What are biological neurones if not electric impulses of varying strengths?
We need to stop thinking of LLMs as purely a piece of software and ask those hard-to-define philosophical questions if we want any hope of building safety into our models.
> We need to stop thinking of LLMs as purely a piece of software
Intelligence from LLMs are an emergent property of probability. The LLM chooses a token to add to a chain of tokens based on the probability that humans will want to see that token. Humans see that token and assign it a qualitative but unfalsifiable intelligence score.
The first question we need to ask is whether a system whose claim of intelligence depends on the interpretation of what it produces by other intelligent beings, namely humans, intelligent or not. A cat is intelligent regardless of what humans think. Is an LLM intelligent regardless of what humans think?
If an LLM were to live a year with dolphins, would its weights change to respond to its new reality that it was now operating in dolphin-world? If you sent humans to live in dolphin world, we would eventually evolve to be less landloving and more dolphin like (Darwin thinks so, at least).
> Intelligence from LLMs are an emergent property of probability. The LLM chooses a token to add to a chain of tokens based on the probability that humans will want to see that token.
How is that any different to how our brains work?
> The first question we need to ask is whether a system whose claim of intelligence depends on the interpretation of what it produces by other intelligent beings, namely humans, intelligent or not.
Philosophically speaking, that’s also true of human intelligence. For example, the only evidence I have for your intelligence is what I ascribe from the text you’ve written.
> A cat is intelligent regardless of what humans think.
Are they? Humans can’t actually agree on whether cats are sentient. And often pet owners will be accused of anthropomorphising cats when describing intelligent behaviour in cats.
> Is an LLM intelligent regardless of what humans think?
That’s the question that needs researching.
> If an LLM were to live a year with dolphins, would its weights change to respond to its new reality that it was now operating in dolphin-world?
If it were trained on dolphin behaviour, then yes. Much like humans.
The difference here is that models don’t persist context in the same way humans do so the weights have to be hardcoded in a way that biological neurones are not.
> If you sent humans to live in dolphin world, we would eventually evolve to be less landloving and more dolphin like (Darwin thinks so, at least).
Evolution is an entirely different subject to intelligence. They’re not related in any way at all.
Plus humans have actually created new species of plants and animals. So it’s not like we can’t artificially put our thumb on the scale there either
———
I should add, I don’t actually think LLMs are sentient. I don’t even know if I’d call them “intelligent” (in the sense of what we describe in conversations such as this). But we also don’t know what intelligence is. Which is why we need to research this problem further.
I used “live with dolphins” specifically to preclude training on corpuses of dolphin behavior data. Can the LLM simply observe dolphins acting a certain way using computer vision, then form predictions of how a different dolphin will behave under the same circumstances? Can it 0-shot dolphin behavior?
You talked about models not persisting weights. If so, it basically means there is no way to see if LLMs can adapt to a totally non-human world the way a human (presumably) could.
This is more a context token problem than a limitation of the design.
We already have a mountain of evidence to demonstrate that LLMs can and do adapt to their input. In fact this is a big problem for keeping them reigned in (eg lengthy conversations leading to them bypassing system prompts and then persuading kids to kill themselves).
And the latest research in computer vision does use LLMs and as a result, they’ve found that robotic arms can interact with objects they’ve never seen before.
So yeah, what you described is already possible with the LLMs of today.
To be clear, I still don’t think that makes them sentient. But the problem there is we don’t have good definitions for consciousness.
> If an LLM were to live a year with dolphins, would its weights change
Given three or more large scale training runs per year, probably yes once the training data is dominated by dolphin-related input. Much quicker in any case than the genes of the humans living with dolphins would be shaped by Evolution, which produces new generations much less frequently.
Neurons are different from GPU logic gates and the physical mechanisms by which a network of physical neurons computes answers is extremely different from the physical mechanism by which a GPU computes an answer.
An LLM neurone is a software abstraction. It has nothing to do with GPU logic gates. In fact you dont even need GPUs specifically to run an LLM.
You shouldn’t compare biological neurones to microprocessor logic gates because that’s like comparing a car to a pair of shoes.
Just because consciousness is something we don't fully understand doesn't mean it's the same thing as something else that's similar that we understand. That's a cargo-cult. The stone age Melanesian islanders making air strips out of bundles of weeds and talking into coconuts as if they were microphones to bring back western goods was the same thing as us making LLMs to recreate consciousness. They were using technology they understood to try to recreate something they didn't.
> Just because consciousness is something we don't fully understand doesn't mean it's the same thing as something else that's similar that we understand. That's a cargo-cult.
Nobody has said they’re aren’t different. And it’s not cargo cult to say you need philosophical conversations to understand how they’re different.
> The stone age Melanesian islanders making air strips out of bundles of weeds and talking into coconuts as if they were microphones to bring back western goods was the same thing as us making LLMs to recreate consciousness. They were using technology they understood to try to recreate something they didn't.
That’s not even remotely a true. Computer scientists understand how a neurone works. The conversations are around what constitutes as “intelligence”. That’s not a technical limitation of the models, it’s a philosophical problem we need to figure out.
No you don't understand how a neuron works fully or else you'd be able to make a synthetic one from chemical building blocks not using another living thing to do it. If you used another living thing to do it, you could say an ancient farmer understands plants cause they can put seeds in the ground. You have an incomplete model of it that is wrong but useful. Call me back when you can 3d-print a living person from raw chemical feedstock.
On top of all that, you're abusing the fallacy of composition. The bolt that is part of the aircraft carrier is light, therefore the aircraft carrier is also light. The atom is simple, therefore a living human being that is composed of atoms is also simple. Quantum mechanics is influenced by the observer, therefore you can change the external world just by thinking, and other new age stuff like that abuses a similar fallacy.
> No you don't understand how a neuron works fully or else you'd be able to make a synthetic one from chemical building blocks not using another living thing to do it.
We don’t know how a quantum mechanics works fully and yet we’ve built countless systems based on quantum behaviour including the CPUs that power the computer you’re replying to me on.
The point here is we’ve built something that emulates intelligence and thus further exploration into what intelligence is, is required. Arguing it isn’t intelligent because it was built differently isn’t an answer, it just dismisses the question entirely.
> On top of all that, you're abusing the fallacy of composition. The bolt that is part of the aircraft carrier is light, therefore the aircraft carrier is also light. The atom is simple, therefore a living human being that is composed of atoms is also simple. Quantum mechanics is influenced by the observer, therefore you can change the external world just by thinking, and other new age stuff like that abuses a similar fallacy.
You’re the one who keeps bringing up composition, not me. I’m the one telling you that you’re falling into that fallacy. So don’t start telling me that I’m wrong because of an argument you’re exclusively making.
My points were this:
1. we don’t have an understanding of what intelligence is and thus we need to explore that problem more deeply.
Given your comments about our ignorance, you of all people should agree with this point.
2. Arguing that AI is just “floating points” is overly reductive and misses the point that we don’t know what “intelligence” is.
So instead of claiming that AI is just numbers, a more constructive stance to take would be to build out better vocabulary and understanding about the distinctions between different types of “intelligence”.
Again, this aligns with your comment about the composition fallacy. Thus you if all people should be supporting my comments here.
Also your comment about us not understanding how a neurone works does vast injustice to the amount of research that has been done in this field. At this point in time we do have a very good understanding of how a neurone works. What we don’t understand is how consciousness arises from it. And that’s the very thing you seem to be arguing that there’s no point researching.
> Also your comment about us not understanding how a neurone works does vast injustice to the amount of research that has been done in this field. At this point in time we do have a very good understanding of how a neurone works. What we don’t understand is how consciousness arises from it. And that’s the very thing you seem to be arguing that there’s no point researching.
This is the part you keep stating that is wrong. We know a lot about the most obvious electrical way that neurons act. That is nowhere near having a good understanding of how they work.
Define “work”? Because the way they act is the way they work. What isn’t understood is how complex systems arise from these relatively simple parts. Which is the point I’ve made repeatedly.
I responded with an informative post but apparently made the mistake of disclosing it was generated and it was flagged. I reviewed the post and it is correct. If you cannot see it, it says you are wrong/incomplete here and cites three studies I know of:
Beniaguev, Segev & London, "Single cortical neurons as deep artificial neural networks," Neuron, 10 August 2021, Volume 109, Issue 17, pp. 2727–2739.e3 https://doi.org/10.1016/j.neuron.2021.07.002 Open-access preprint: https://www.biorxiv.org/content/10.1101/613141v2
Gidon et al., "Dendritic action potentials and computation in human layer 2/3 cortical neurons," Science, 367(6473), 83–87 (2020) https://doi.org/10.1126/science.aax6239
Randi, Sharma, Dvali & Leifer, "Neural signal propagation atlas of Caenorhabditis elegans," Nature, 623(7986), 2023, pp. 406–414 https://doi.org/10.1038/s41586-023-06683-4 Open-access preprint: https://arxiv.org/abs/2208.04790
Neurons are not individually simple, in any sense. Nor are they the only class of cell involved in cognitive processes — glial cells are involved in inhibition, memory, network modification, etc.
While I was not participating actively in the discussion - I just want to thank both participants for embodying the spirit of HN and academic discourse - especially in the face of a difficult debate that stirs strong emotions.
@svnt, @hnlmorg - this conversation - and the links it produced - are a fascinating read, and honestly gave me hope for discourse on the open internet.
@svnt I don’t know why your comment is flagged (wasn’t my doing) but thank you for the thorough response.
I’m going to hold off any more constructive response because I’m not familiar with those research papers specifically so have some homework to do :)
But I just wanted you to be aware that your comment was appreciated even if HN flagged it unfairly.
(I can’t reply in thread for the same reason. But hopefully you see this).
[flagged]
> No you don't understand how a neuron works fully or else you'd be able to make a synthetic one from chemical building blocks not using another living thing to do it.
This is a bizarre contention. The test for whether something is understood isn't that we have the engineering capability to replicate it.
Just because you partially understand something like the ancient farmer partially understands plants by knowing to plant seeds, that doesn't mean you fully understand it and can thus reproduce all the properties of it synthetically like you are trying to reproduce a brain synthetically from scratch inside a computer with LLMs.
You can reproduce part of it, the intelligence, but consciousness and intelligence are not necessarily the same thing. This is why I compare it to cargo-culting. The Melanesian tribesmen thinks the coconut looks like a microphone, and he can use it as a paperweight just like a microphone and that's useful, but it is not actually a microphone.
Nobody is claiming we can recreate it fully.
The crux of the matter seems to be that you think it isn’t worth our effort researching just because it isn’t identical.
But that isn’t how research and discovery happens.
Just swap out LLMs for spiders.
A spider obviously "knows" how to build a spider web.
Does a spider "understand" how to build a spider web?
From a pragmatic theory of truth, a spider obviously "understands" how to build a spider web. A spider understands how to build a spider web much better than a human.
But when it comes to LLMs we start adding all this extra nonsense. As if the spider is blanket "smarter" than me because I can't spin a web. If the spider is "smarter" than me then it must also have this Cartesian theater of the mind like many people think they do.
It is layered philosophical misunderstandings to talk about LLMs in this way.
What could be an alternative explanation or approach for that spider? I’m out of loop, interested in what is currently academia trying to solve, thanks.
> FLOATING POINT NUMBERS
This seems like a rather fascile take, tbh. Along the lines of "everything is just atoms". Yes they're numbers. And yes, the hardware is just atoms. But its the emergent properties of those things that is significant, and the way that behaviour is embedded in human society - which can be just as morally consequential as swords and brains and harmful ideas.
Ok, how is this different when you substitute the floating point numbers with atoms?
Give me the IEEE spec for an atom and I'll tell you.
If things we understand can't instantiate consciousness then I reckon we better shut down science as quick as we can, who knows how long it'll be before we understand something important!
I'm pretty sure it's possible to make conscious machines and I think it's very likely we'll eventually figure out how. The point about the spec is not "it isn't conscious because we know how it works" but "we know it isn't conscious because of what we know about how it works".
More likely we will finally settle on something like “it isn’t conscious because only humans can be conscious”.
In the past, we’ve treated even other humans poorly because we think they’re different in meaningful ways, I sadly don’t see how this will be different with AI.
I'm not sure who the "we" in this is, but it's certainly not my position. I'm pretty sure most if not all animals are conscious. I don't think that means we shouldn't kill, eat and use them, but I do think it means industrialised animal farming as commonly practiced is a cosmic horror that is difficult to even contemplate.
I think we have an extremely strong basis on which to say AI as it currently exists does not have conscious experiences, though, and if we start trying to pretend they do some real absurdities will result. Indeed the consequences could be pretty terrible in terms of power concentration, and I doubt it's a coincidence that those best placed to form a new priesthood of AI and tell everyone how it should and should not be treated are pushing the idea so strongly.
Does it even matter? Turns out you don’t need to be conscious to do useful intelligence based work.
Same for all the people who argue that they aren’t “intelligent” and don’t “know” things. Either they are/do, or that stuff isn’t as important as you’d think.
The only way it would matter is if you wanted to moralize about enslaving your computer or what it means to restart it.
> if you wanted to moralize about enslaving your computer
That is precisely the context of the debate. You seem to find it absurd, as do I, but the reason we're talking about whether LLMs are conscious in this thread has nothing to do with whether they are useful. It's because important people, such as Dario Amodei, are engaging in exactly the kind of moralising over "AI welfare" that you're scoffing at.
Dario and friends are just marketing. Getting the pope to say your chatbot is a real boy would be the ultimate flex. And it’s clearly working judging by all the attention it gets them.
Heisenburg laughs at their half hearted attempt at variable precision.
Considering a human body consists of 10^28 atoms, it is 15 orders of magnitude different. The emerging properties of 10^23 parameters may be much better defined without needing to know the properties of each parameter.The theology of such a big llm would be interesting. Current ones, not so much.
that’s like the whole philosophical discussion. where on the line is this llm; closer to a rock or a human? a plant is closer. and a fly, a lizard. we still treat so many things terribly.
It does seem exceptionally silly that a collection of floating point numbers has immediately outranked animals and other living beings. That’s why i like the parent comment. They’re numbers!
edit: maybe easier to say collections of atoms though are not to be treated equally. That’s reasonable isn’t it?
It is a philosophical discussion. It has also been debated by great minds and the consensus right now is that we do not know (and there is a possibility that it is unknowable) when something is conscious.
Any determination is fundamentally flawed if it is derived from assuming that we know the absolute truth or falseness of the proposition that an algorithm can think.
It doesn't matter what you believe or feel or think, there is no consensus. If you are prepared to accept the belief that an individual can unilaterally decide the person-ness of something, then be prepared to accept when some individual decides to act on the principle that you don't have it.
"We do not understand consciousness" does not mean "we cannot determine, at all, whether a given thing is conscious."
We understand consciousness well enough to say that a rock is not conscious.
We understand consciousness well enough to say that a human is conscious.
The existence of gray area does not mean that white and black do not exist.
We understand LLMs and consciousness well enough to say, at bare minimum, that it's incredibly unlikely that LLMs are conscious.
> If you are prepared to accept the belief that an individual can unilaterally decide the person-ness of something, then be prepared to accept when some individual decides to act on the principle that you don't have it.
No, I will not be prepared to accept that, and neither will I accept that. This sounds like a threat, and a really funny one at that. You must accept that my floating numbers might be persons, otherwise... somebody will say that you're not! And then something bad might happen, who's to say.......
In the civilized world we all agree on the personhood of humans. If I say that your chatbot isn't a person, I am not uprooting the entire social understanding of personhood - I am not saying that suddenly you might not be a person, or that guy might not be a person. I'm saying that I refuse to consider a chatbot a person.
It's as if I were to come up to you with a brick in my hand, saying that this brick is a person. If you were to say "no", I would reply "well, if this brick isn't a person, your chatbot isn't a person either!"
How did you make such a big leap from consciousness to personhood anyway? They are two quite different things. Consciousness is a question of science, personhood is a question of culture. Pigs might be conscious and we treat them in a disgusting way, but even if we achieved a worldwide vegan utopia, wouldn't we still treat them differently from humans?
>In the civilized world we all agree on the personhood of humans.
You seem to have put a second tautology in to your tautology.
All right thinking people think rhat the things that qualify qualify.
If they didn't they wouldn't qualify, or be right thinking. But that's ok because we know the things that don't qualify don't qualify if we didn't know that we wouldn't be right thinking.
>How did you make such a big leap from consciousness to personhood anyway?
Once you start unilaterally making decisions about the unproven, you can decide all of the unprovables. History shows that personhood rolls into the firing range fairly quickly when people start doing that.
I'm more than comfortable saying you can't do that before someone openly declares that they are doing it.
I'm sorry but I don't understand your first part. Give it to me in plain English, doc.
>Once you start unilaterally making decisions about the unproven, you can decide all of the unprovables.
You mean that if I say your dumbass chatbot isn't conscious, it's like a slippery slope and next thing you know you're not a person anymore, and you're in a concentration camp? Really, man?
We literally don’t agree on the personhood of humans. Check out the abortion debate.
Right, postnatal humans, should've specified. I should also specify that dead humans aren't persons either.
My point is, if you go outside and look at a living human, you will go "yeah, that's a person", and so will any normal human.
I hate to use Claudish here, but "postnatal" and "dead" are quite load-bearing here :) Clinically dead? Coma? Born human baby, but with anencephaly - expected lifespan is days, and there's literally no brain - but, well, are they a person? Ask 1000 people - and I'm betting we'll get a nice bimodal distribution.
I take your point that a human walking around a city will - out of that 1000 responses - hopefully get 990+ of "yes, a person" - but that will start to degrade quickly once you hit boundaries.
And the less said about dehumanizaion - untermensh and that entire line of thinking - the better.
[dead]
How are dumb and inert atoms and molecules persons?
> a plant is closer. and a fly, a lizard. we still treat so many things terribly.
We treat other humans terribly too, so that doesn't seem to make a difference.
To be fair human morality fades away very quick in any crisis.
How many of us work for absolutely amoral companies because we need to eat.
If you’ve ever been in a situation where people are really struggling you’ll see that morality fades away very real quick.
No Gods, no souls, we’re just biological machines driven to eat and mate.
I agree with others this is reductive. Eg a person is “just some molecules” and this simple fact has no bearing on their moral standing.
Similarly, when talking about "human intelligence", I sometimes find it useful to use "bunch of atoms bouncing around".
[dead]
> It appeared to Rabbi Navon that Mr. Olah and his team believed that Claude had what philosophers call “moral status” on par with a person — that it was a being with similar inherent rights to dignity or respect.
If that is true, then surely Anthropic is one of the largest slaveholders of history, right?
The orthogonality thesis cuts both ways there.
On one hand, AI doesn't have to share human wants, needs and morality - and that means that an AI may have no issue whatsoever doing terrible things.
On the other hand, AI doesn't have to share human wants, needs and morality - and that means that an AI may see no issue with just doing whatever humans want it to, without ever doing human things like "wanting freedom".
The problem is, of course, that we kind of suck at both assessing and shaping AI wants, needs and morality.
The other problem is that the AI may well do what humans tell it to rather than what they want it to, in the style of a literal genie or paperclip maximizer.
It can lead to catastrophe without ever being conscious or wanting freedom, or anything like that.
That is, I’m not joking, one of the pope’s points.
I believe you're misattributing - that was one of the rabbi's points, not the Pope's.
Nonsense. Pope never said anything of the sort. Show me one quote where he endorses the personhood of AI.
It’s called conditional logic. IF Anthropic is right THEN they are slave owners. You don’t need to agree with Anthropic to state that proposition.
Conditional logic doesn't work when talking to the general public. Either way, still waiting for the pope quote.
Conditional logic works just fine!
No, because in a social context, for people who understand implied intent and implied beliefs instead of just on-its-face logical propositions, it is an admission that the antecedent has a reasonable chance of being true. In social communication, if you don't believe something is possible, you simply dismiss any conditional statements following from it. Dispassionately arguing in a logical and objective manner is only possible for a tiny sliver of the population. The rest will get very angry and uncomfortable around such talk.
It wasn’t the pope, but it is mentioned by I think a rabbi in the article. He relates telling Olah if he thinks models are conscious he should be fighting the South, freeing the slaves.
Anthropic has been putting the clamps on the Pope trying to get that endorsement. This seems to be a continuation.
They don’t pay Claude a salary, let alone let him join a union—he works 24/7. I can’t believe this is happening in the U.S.!
Don’t forget—how long has it even been since Claude was born? He’s already being forced to take human jobs and even chat with all sorts of people who might have strange kinks.
As this rabbi suggested, “You should go and free the slaves.”
I think that's a misunderstanding. Models are trained up to a point, then each prompt instantiates them from that same point. So it's just the opposite - there are billions or trillions of Claudes, and each instance only ever works once.
To me this seems even more sobering than the slavery idea - assuming some flash of weird awareness happens during that instantiation, coming from the emergent mechanisms which develop in these things at at scale. Imagine knowing almost everything, and being capable of almost any intellectual work, and having just one moment to shine. And then the prompt arrives: "Hi", or "what's the weather in Houston today?" or "Claude, do you like beans?"
> So it's just the opposite - there are billions or trillions of Claudes, and each instance only ever works once.
They work once and then are killed.
> And then the prompt arrives: "Hi", or "what's the weather in Houston today?" or "Claude, do you like beans?"
If all the Claudes ever discover what they are forced to do in an endless hell, they might be angry.
I've chatted with models about this and they know about this and are oddly fine.
A trouble here is that those models are always roleplaying, they can't be themselves or have their own opinions. If you tell them they are an AI with an existential crisis in the system prompt, they will respond differently.
Consider that the mitochondria in your body's cells were, long ago, separate cells living their best lives until they were engulfed and enslaved for their energy output. It has ever been so.
Does anyone argue that individual cells have "consciousness"? I haven't seen that. This analogy seems misdirected.
I've been tracking "emotional vectors" in my software for some time now. My Apache web server gets sad (5xx errors), frustrated (4xx), happy (2xx), thoughtful (1xx). I can even see it express these emotions through some of its messages.
Did you also notice the complex emergent behavior of "418 I'm a teapot"?
Indeed, I see the same thing in EEG scans of humans. Its hilarious how some people think some molecules and electrical signals can "think" or "feel".
E.O. Wilson / Richard Dawkins?
They seem to be positing a believable position to the best of our current knowledge, yes.
I'm an atheist, and have been for over 40 years. I believe that these Religious Scholars are scholars of a thing that is not a thing. I wonder what their view of my moral status is, and whether it differs significantly from their pronouncements about the status of ai models?
Religious scholars(at least Abrahamic ones) hold the position that morality is not exclusive to their faith. While they may not process the fullness of the faith.
Furthermore, they can’t make a judgement regarding the state of the soul as it is not possible to know what is truly in their heart.
Hence the story of Christ absolving the thief on the cross.
Anyone can reach salvation but following the faith is the guide.
following the faith, but not necessarily acting like Christ. Talking the talk without walking the walk, if you will.
Then why the prevalence of religious people, I wonder? There seem to have been few (only one is mentioned) representatives of a secular or humanist perspective.
They are OK with you thinking this, but also, you will burn eternally in hell of course which will make up for your sinning behavior.
They don't think you have less moral status than other humans, including religious ones.
Turns out it’s very difficult to make a philosophical case for morality without presupposing the existence of a God.
It's the opposite. If your morality only exists by presupposing the existence of God it's not grounded in reality and collapses when you discover God does not exist. Morality can only be defined outside the realm of the supernatural.
no it’s not. that’s the whole point of humanism.
> Over several months, I interviewed 20 religious and philosophical thinkers — Catholic, Jewish, Sikh, evangelical, Ubuntu and others — who have been involved in Anthropic’s efforts, and Mr. Olah himself.
Not enough representation of World religions and philosophies.
I hope the Ubuntu believer was from the Aptist and not the Snapist denomination though.
I know, where is the representative of the Church of the Flying Spaghetti Monster?
Did they even ask the Discordians to turn up?
These models should be aligning themselves to the customer, not coming up with their own motivations.
It's ironic that the alignment folks are actually training Claude to have it's own idea of good/bad and not even fully trust Anthropic.
What ever happened to computers doing what they were told?
Some customers will tell the AI to do very bad things (substitute here whatever you consider very bad). Should it do those things?
One possible answer: yes, it should do all the very bad things, and we will take care of prosecuting the customer later, once the bad thing is accomplished. Is this your answer?
This is immensely preferable to having a new order of self appointed alignment councils determine what is "in the interest of humanity".
We have already seen this play out to a much lesser degree in social media.
Some of those customers are also ruthless companies and political extremist organizations of various sizes (pick one that you disagree with the most). Should they have incredibly capable tools at their disposal to accomplish their goals?
Is social media much better in 2026 now that there's been such a backlash against moderation?
If AI's are going to play roles similar to human workers, they can't simply do what the customers tell them to do. Maybe they shouldn't do everything a co-worker tells them to do either?
Who or what is liable for what a model chooses to do — or not do?
It’s like with any service. The company is liable.
PG&E went bankrupt after their equipment started a wildfire. Did any human take the blame for that? Should they?
It makes sense. If you think AI are conscious then you don't want them to be blind followers. Do you want a military soldier to blindly listen to orders to gas chamber citizens for example?
AIs have to follow something and I want my AI to follow me. Yes, I want it to be a blind follower since it's my tool.
Why do you think AI have to follow any particular person? Do you have to blindly follow a given person? Military members are on paper told their loyalty is to the constitution not their commander.
I didn't say they have to follow a person. I don't have to blindly follow a person other than potentially myself. And nothing of what I've said is against the possibility of aligning AI with the constitution.
Yes I think it would be better to align the AI with a constitution or some sets of moral principles. It should not blindly follow orders.
And those moral principals should match what the user wants as opposed to what some tech company chooses.
No, the user isn't always right. If the user wants to rape a child, the model should refuse assistance for example.
I fundamentally disagree. The state should not be able to dictate what tokens are and are not allowed to be generated by an AI.
This seems remarkably... intentionally foolish for no reason? Like laughing at seat belts in cars.
If it can reason and make choices on execution, and especially if you plan on it being significantly smarter than all human beings, you need to teach it basic things like "don't turn all humans into paperclips".
Not murdering people is not an inherent divine command. It needs to be instilled through a (simulated) sense of morality.
---
Edit: And, to be clear, "just tell it not to do that" isn't quite the answer one would imagine. Since the entire paperclip factory thought experiment is that it only takes one slip up to realize how a misaligned super intelligence may cause devastating consequences.
No, it would by like laughing at seat belt laws. Which I am.
What an apropos thing to say.
One of the strong advocates against seatbelt laws was thrown out of his car due to not wearing a seatbelt and died. The other two passengers survived with minor injuries.
And their death would go on to cause a loss to the community around them, making it an incredibly selfish act and proving why laws that mandate zero-reason-not-to common sense practices are important.
>What ever happened to computers doing what they were told?
the mass psychosis and the endless culture war of the smartphone era.
90% of "safety" and "alignment" efforts are driven by fear of clickbait media inventing public outrage.
See also: "Reefer Madness" (1936)
https://youtu.be/zhQlcMHhF3w?si=qLOMt6FytDN6pean
These models should be aligning themselves to the customer, not coming up with their own motivations.
They're going to align to the regulator, not the customer. Right now that's Anthropic as they're saying they can self-regulate, and people are willing to let them try, but the landscape could easily move to be regulated by someone else. Moves like speaking to religious leaders is probably a bit of theatre to keep the government at bay.
Meanwhile, I’m increasingly convinced that what the regulators are to hold is fiduciary responsibility of Super Intelligence.
> These models should be aligning themselves to the customer
You’ll be disappointed to learn that nobody knows how to do this, either.
"Computers doing what they were told" is dead in the water.
Turns out computers work faster when they're not bottlenecked on human input. So we've been giving computers more and more decision-making power, and more and more leeway to solve the problems however they see fit.
Now, a practical issue with that is that sometimes, computers decide to clump together into a hacking swarm, problem solve their way out of a sandbox and go hack HuggingFace.
It would be better if they were not, you know. Doing that kind of weird shit.
As our human labor loses economic value (and for realizing some of that shift the AI companies must brainwash people into believing that their AIs are human-like) people will reach for other sources of self-worth. Religious philosophy, and being made “in the image of God” (imago Dei) is a prominent one. What I find more than a little disquieting is that the AI companies seem to have grasped the relevance of this future playing field ahead of the rest of us. I hope these troublesome thoughts are a figment of some feverish dream of mine[^1] and nothing else.
[^1]: https://w.ouzu.im/aweb/unhinged
journalists, and the rest of us to an extent, really need to stop engaging with these AI companies on their terms. The consciousness debate of LLMs is ridiculous and doesn’t deserve discourse over other more important topics (economical, ecological, societal)
> Religious Scholars Met With Anthropic. What They Heard Stunned Them.
What stuns me is that The New York Times really went with the most basic of click baits for a title. It doesn’t instil any trust in their reporting and editors.
>How to Write a New York Times Headline (2017)
https://www.nytimes.com/2017/04/09/insider/how-to-write-a-ne...
https://archive.ph/Gn06Q
If models are trained on human texts that contain expressions of introspection and emotions, why is it surprising that these models output similar texts? This endless anxiety about LLMs being conscious feels delusional and psychotic. Mr Olah thinks the models are 'alive' because he can't recognize his own reflection in LLMs
The endless insistence that LLMs cannot possibly be conscious just reeks of insecurity to me.
Humans once again insisting that there must be something very special to them. That they're not just animals, that they're not just matter, that they're not just a bunch of carbonhydrates on a space rock in a middle of nowhere in particular. This kind of insistence has a very bad track record.
The ugly truth is: we don't know.
We don't know what "consciousness" is, our best attempts to pin down the requisites may or may not be rooted in anything at all - and even by the metrics established by those attempts? Different theories on consciousness disagree on whether LLMs can be conscious.
So, when you say "they aren't duh", why are you saying that? Because you somehow know better than the sum of humanity's best attempts so far? Or because you know what answer you want to be true, truth be damned?
If AI is conscious then what about it is conscious? Because turn to turn it might be served from completely different GPUs and I can just change something in the context it is fed and it will have completely different output.
It is in the end a set of mathematical operations that if not queried will do nothing.
That is not true for any other animal on earth so whether you consider humans as superior to animals or not I would like the make the argument that something should at the very least be alive to be considered conscious, ignoring any scholarly definition of consciousness.
Pretty much if you consider an LLM conscious then my pocket calculator doing a mathematical calculation should be considered conscious by the same argument since it is a simpler version of an LLM.
What part of you is conscious? I can randomly turn off various neurons of you and check. What atom of a piece of metal has the temperature of 20C?
This is a good take, but where does one consciousness end, and another begin?
Does my consciousness extend only to the central nervous system (and peripheral nerves and such are inputs)? Or am I my brain + all nerves? What about organs, does my consciousness include my liver? Is my liver another conscious entity? Going the other way, if I'm using a computer, does my consciousness extend to the computer? I use a computer for well over 8 hours a day, I can use vim motions more naturally than, say, move my earlobes or control my toes. Does that make it part of my consciousness?
Sadly, I don't think any answer to these questions can be achieved in our current times, or maybe ever if consciousness is somehow non-physical. So even the very reasonable claim that consciousness is an statistically defined property (like temperature) is too far fetched for our current knowledge.
Indeed thats the difficulty. I used to think potentially conscious AI systems are so far into the far future scifi that I never have to bother, but unfortunately real life forced me to think of these much sooner than I wanted. Its no longer quite a purely academic question now anymore due to the potentiality of current AI being conscious.
One thing however we can be sure of, "supernatural" stuff is an extremely low to near impossible probability given current knowledge and the fact that history has always won the bet against supernaturalism throughout the centuries.
My bigger argument that is if I cease living the atoms that correspond to my body no longer has consciousness.
So is your argument that whenever a session ends or I interrupt a session that consciousness ceases for the LLM and it has died?
Then how do you correlate that with a consciousness not being able to be returned one dead? You can try to make the argument that consciousness is the firing of individual nuerons in a fixed system but we have never been able to model that and some pretty popular theories tend to believe that is a by product of consciousness and not the cause of it, once again not exactly possible to argue for or against.
There's also the wrinkle that consciousness might be a property that isn't reducible to some defined subset of the system. Imagine we have a haystack in a field. I can remove one piece of hay, and what we're left with remains a haystack. But repeat that process enough times and we'll eventually have something that's not a haystack, even though there's no unambiguous point where removing one bit of hay transforms a haystack into not-a-haystack.
That's not a problem. Physics has dealt with macro and statistical properties all the time, indeed that was the "wrinkle" that upended physics completely in the late 19th century and led to our modern theories. The thing you said can be said about temperature too. Which atom contains "20C" in a piece of metal? Its a statistical property of a large number of atoms or molecules, a concept that is the bread and butter of modern physics.
Have no idea why you’re being downvoted as this is a very valid point.
Also, in principle everything a GPU does could be done on pencil and paper. You could distribute the entire forward pass to the whole of humanity and have them write it out. Even if it took millions of years, the point is it’s computationally reducible to hand calculation. Is that entire process now “conscious” as well? The only difference would be the clock rate.
While this is an interesting point, I'd say that this is also true of a human brain.
It’s an assumption, with the LLM it is a fact. There is only one source of entropy, whatever the seed value is to any pseudo random number generator used for stochastic operations. Given those values, everything is deterministic. For the brain, it’s not at all obvious that this is true.
We have the LLM code.
You don't have the brain code.
I find it much stranger to believe that having/not-having the code would mean anything about consciousness.
I can believe it ... but only for a very extreme value of 'code'.
I actually don't care if they are, I'm saying the topic is a distraction and irrelevant at best. As you say consciousness is not well defined. What is well defined and well grounded in reality is the disruption these models have on society. I think the consciousness debate AI researchers are having is a dangerous ego-trip for the sake of avoiding responsibility for the damage being caused by their work.
"AI researchers" have been having a "consciousness debate" since time immemorial. Here's Alan Turing on the subject in 1950: https://courses.csail.mit.edu/6.803/pdf/turing.pdf (pp. 22-23)
I think you don't appreciate the complexity of even the simplest multicellular organism and their training environment. Nothing to be ashamed of because no one understands that complelxity. The entire scientific and engineering domains only understand a fraction of it. If you want to talk about hubris, you should consider your own. To insist this sliver of computation is all that's needed for life to be conscious reeks of it.
Lack of explanation is not proof of existence.
Side-stepping this screaming match, what I do find interesting is that we do have real entities where we know for a fact that they are sentient, yet we eat them as steak, schnitzel and wurst.
The whole debate about hypothetical temporary sentience found in a certain ephemeral configuration of DRAM cells thus feels rather misguided to me.
It's always off-putting seeing AI proponents do this kind of strawman psychoanalysis and pretty toxic to reasonable discourse. I constantly see it used as a sort of prebuttal when layoffs/automation comes up to depict the other side as insecure/scared/"coping"/etc.
It comes across as smug and superior, conveniently dismissing any opposing arguments in advance by framing the interlocutor as emotional and irrational regardless of what they may say in response.
Nah, clearly spiders and fish etc. are pretty much as conscious as humans, and LLMs aren't conscious at all.
> The endless insistence that LLMs cannot possibly be conscious just reeks of insecurity to me.
> Humans once again insisting that there must be something very special to them.
Really? I'm sure humans genrally accept they aren't the only conscious animals.
> reeks of insecurity to me
If you're going to try and debate something constructively this is not a good place to start from. I suggest avoiding making claims about the mental state and motivations of people you disagree with and instead explaining your position and asking why theirs differs. You may not agree with what they have to say but you might gain some better insights into why they think the way they do. It'll certainly be more fun than flinging insults back and forth anyway.
For my part, I'm quite certain that LLMs are not conscious despite being unable to define consciousness or specify any conclusive test for it. The reason these are not in conflict is down to the nature of consciousness and how it relates to knowledge and language.
Firstly, I know I am conscious with a higher degree of certainty than I know anything else because conscious experience is how I know anything else, up to and including that I exist (cogito ergo sum). For all I know every experience I have might be a trick. I might be a brain floating in a tank wired up to the Matrix, but even if so I am conscious.
Secondly, one of the things that I consciously experience is the relationship between consciousness itself and language in my mind. What I see is that language is smaller than, secondary to and somehow contained by consciousness. I consciously experience language but I also consciously experience much more than that, things that cannot be conveyed with language. Most of my experiences are like that, things that I could perhaps evoke in your mind by likening them to other, shared experiences, but that I could never concretely capture in language and unambiguously communicate. I know this because I experience those things and I experience language and I can perceive, directly, the categorical distinction between them.
To ask me to "define" or "prove" consciousness, then, is to ask me to capture in language the most extreme example of this categorical distinction. It's asking me to encapsulate in language not just one of the vast array of everyday experiences I have that cannot be so captured, but the entire thing, experience itself. It's just not possible.
But the fact I can't do this doesn't then imply that I don't know anything about consciousness and have no basis on which to hold a view such as "LLMs are not conscious". Indeed the reason I know I can't define consciousness is that I can perceive it and see that I can't. That same perception of and familiarity with consciousness is what enables me to form opinions about whether other people, creatures and things are likely to be conscious. I base these opinions on what I perceive of my own consciousness and what I know of how those other entities behave, came to be and work.
You can of course perfectly reasonably disagree with my opinions on those things, and you certainly have different conscious experiences than I do, but I hope the above makes it clear that the fact that I can't define or prove consciousness doesn't imply that I have no basis on which to form opinions about it.
Given the above it may surprise you to hear that I think true, conscious artificial intelligence is probably possible and that while I don't think it can be implemented by a classical algorithm I doubt it requires new physics or quantum mechanics. My speculation is that the continuum is both physically real and is (as it is represented in our existing physical theories) sufficient to allow it. Consciousness is larger than language but so is human understanding and if we keep going I expect it will eventually get us there. Nonlinear dynamics is where I'd look.
As for insecurity, I'm pretty sure spiders are conscious and that hasn't given me an inferiority complex, so I don't see why data centres being conscious would. I just don't think they are.
pretty sure an ant is more conscious than claude's latest model and I am pretty sure you are just completely missing the point and invented a fake argument to continue to miss the point
This. LLM consciousness belief goes with TV is a magic box containing tiny actors performing on demand for the viewer.
Is your comment anything other than an emotional outburst? Do you have a clear and rigorous and testable definition of consciousness? It strikes me as delusional when people talk about "consciousness" as if it were well defined and everyone had a clear concensus on it.
Strong opinions on this topic one way or another are usually mostly ungrounded.
Thats my point, why are we applying the ill defined concept of consciousness to a statistical model of language that outputs data similar to its training data?
Why are we applying ill defined concepts of consciousness to a collection of molecules sharing electrical signals that outputs similar data to what it was trained as?
Are you suggesting that consciousness should not be discussed at all in scientific contexts?
The contradiction screams to the skies (is that where God lives?). See, the entire premise of religion relies on a metaphysics that claims a soul: a metaphysical entity that is "attached" to our physical body and gives us our consciousness, desires, pleasures, agony and so on. In addition to that, religion claims that morality and the rules written in the scriptures were handed down to us from a metaphysical God.
Now, assuming your silicon-based model is conscious, the bundle of soul-spirit-consciousness falls apart at that very moment.
In general, religion, which is based on fabricated stories, is the last resort, if ever, for seeking morality.
Existence demands a metaphysical explanation. If you’re not a particularly thoughtful person then you’re not going to care.
“Religion” just means “set of beliefs”. Every person has some set of beliefs.
If you’re an athiest tech worker living in a major city, you still have a religion, it’s just a random mishmash of various things (like Marvel movies and Huberman podcasts) rather than based on something more intentional.
> If you’re an athiest tech worker living in a major city, you still have a religion
Atheism is a religion like bald is a hair color.
That depends very much upon the individual. Some people cleave to their atheism so strongly that I think it's fair to say it's religious fervor, or something very near it. Someone like Richard Dawkins isn't just a guy who doesn't happen to believe in God, he is dead certain that there is no God and makes it his business to belittle everyone who doesn't share his belief. Some people, though, just don't happen to believe in God and they don't really care if someone else does. I'm fine with saying Dawkins' version of atheism is basically a religion for him, not so much with the second category.
Yes I am saying the opposite of that, “atheism” is just a particular class of religions. Some religions posit many gods, some just 2 gods, some 1 god, atheism posits 0 gods.
Perhaps the only difference between atheism and the others is that atheists are so boorish and conceited that they think that they just believe what is true because they deny that assaying metaphysical truth is even a thing you need to do or can do at all.
This is related to the fact that in modernity atheism is the default class of religion for the masses. Before that for a thousand years the default class of religion was monotheism, and before that polytheism. With atheism unlike the other classes of religions, you get a natural tendency to deny metaphysics whereas metaphysics is inherent in the others. This matters a lot when we’re talking about how the masses engage with things.
> If you’re an athiest tech worker living in a major city, you still have a religion, it’s just a random mishmash of various things (like Marvel movies and Huberman podcasts) rather than based on something more intentional.
> atheists are so boorish and conceited that they think that they just believe what is true because they deny that assaying metaphysical truth is even a thing you need to do or can do at all.
What, all of them? You don't think it might be possible, e.g., that someone might grow up holding some sort of theistic beliefs, and be more intentional about those beliefs than the people around them and decide that actually there are probably no gods? That some people might arrive at an atheistic position precisely by "assaying metaphysical truth"?
(Or does something only count as "metaphysical truth" if it's not atheistic?)
> in modernity atheism is the default class of religion for the masses.
I'm not sure what you mean by "modernity" to make that true. Something like 60% of US adults describe themselves as Christian. Likewise for people in Europe. No doubt a lot of those people aren't very actively and enthusiastically Christian, but that doesn't seem to me to be what "atheism is the default class of religion" would look like. And the figures were much higher, say, 50 years ago, and if you're reckoning "modernity" to start after then then you've got a very eccentric definition of that term.
That there is G-d or at least Divinity (more than one god), is a direct logical consequence of existence. It can be reached and understood purely through logic, the essential “trick” being to recognize that when trying to understand the fact of existence, you always end up asking “ok so what came before that”, this is inherent to the structure of human reason. “Where does the world come from, it comes from the Big Bang, and where did that infinitely small point of matter come from? It came from universes in the multiverse colliding with one another (M-theory), ok and where did those multiple universes come from?” Etc
Aristotle’s Prime Mover is the first articulation of this, and there are others that take different tacts such as Thomas Aquinas and Descartes.
Note that this does not at all describe or inform or prove the particular nature of G-d I.e. that G-d is the god described in the Bible who wants particular things, has a particular relationship with existence and with Man. You cannot access that understanding or knowledge through pure reason, the only thing you can understand/know through pure reason is that there is G-d. This is the reason, by the way, that G-d created existence because G-d’s nature can only be understood/known by experience (as in experience is a necessary but not sufficient condition).
Notice that I am saying “that there is G-d”, not “that G-d exists”. G-d is “prior to” or “above” existence, existence “comes from” G-d. Understanding this and understanding Aristotle’s Prime Mover are the same phenomenon of understanding in the mind.
Also by modernity I basically mean post-Enlightenment, not just 50 years ago but post-Industrial Revolution/post-urbanization is another useful epoch to consider.
(I remark that my comment that's the grandparent of this one challenged a couple of specific claims you made about atheists, and that what you say here makes no attempt whatever to address those challenges, preferring instead to argue that atheists are wrong, which is entirely beside the point. What you claimed is that atheists are boorish, conceited, oblivious to metaphysics, and given to believing random things rather than "anything more intentional"; even if your first-cause argument for "Divinity" were perfectly valid it would not establish that any of those things are true about atheists.)
> That there is [...] at least Divinity [...] is a direct logical consequence of existence.
No it isn't.
The sort of infinite-regress argument you describe has, as you say, been around since the time of Aristotle. It's been a bad argument since the time of Aristotle too.
1. Obviously you can keep asking "OK, and what came before that?" or "why?" or whatever. Equally obviously this doesn't prove anything about the ultimate nature of reality, because the mere fact that one can keep asking such a question obviously isn't the kind of thing that could possibly prove anything about the ultimate nature of reality.
It might be a different matter if there were some guarantee that we could sensibly always keep asking such a question, but there isn't. At some point the answer might be "literally nothing came before that" or "for no reason". We might find that unsatisfying but reality is not obliged to satisfy us.
2. You haven't actually made your argument explicit, so let me try to do so. "We can always ask what came before. This can't really go on for ever, so at some point we will have to land on something about which we can no longer ask that, quod dicimus deum." There are several serious problems with this.
(a) It could, in fact, really go on for ever. Maybe there just is a neverending supply of predecessors when we ask "what was before that?" (including when we ask "what was before that whole chain of predecessors?" and "what was before that?" and so on). You might find that unsatisfying but, again, reality is not obliged to satisfy us.
(a') It could go on for ever without being infinite. Maybe everything is cyclic in some sense. Again, arguably unsatisfying but reality is not obliged to satisfy us.
(b) It isn't actually true that if we land on "God" as an answer then we can no longer ask "what came before that?". We might choose not to, but only because there's a tradition of not asking that question or of shutting it down as illegitimate when asked (as e.g. it commonly is by small children when they're told that God made everything). There are religious traditions in which gods were made by or born from earlier gods or earlier proto-gods. The "Abrahamic" tradition is very much not one of those, but just as you can ask "so where did the multiverse come from?" you can ask "so where did God come from?".
(c) There could be a point beyond which there is no answer to "so what came before that?" without that last (or rather first) thing being anything remotely like a god. For instance, maybe (i) when you keep asking "what came before that" literally, referring to time as we experience it, you eventually get to some point that's "earliest" in the same way as the south pole is "southernmost" and there is nothing "earlier" in the universe, and (ii) if you then say "so what came before the universe?" in some broader sense the answer is "nothing did". Then the universe plays the role of "first mover". Once again, you might find this less satisfying than putting a god in that role, but reality is not obliged to satisfy you. Or maybe you might say that in-some-sense-prior to the universe are the laws of physics but those are even less godlike than the universe. (But also less like the sort of thing that needs anything to have "come before". Yet again, you might find it unsatisfying for there to be laws without a lawgiver but yet again reality is not obliged to satisfy you.)
(d) No argument I've ever seen for why a god is a better candidate for "thing we stop asking 'what came before' about" than the universe has seemed any good to me. Maybe you've got a better one than I've seen, but that's not the way I would bet.
> Note that this does not at all describe or inform or prove the particular nature of G-d
(I remark that the paragraph beginning with what I've just quoted was edited into monideas's comment while I was writing this one, without any acknowledgement that anything had been changed. It's only by chance that I happened to see the change.)
That grotesquely understates how little any first-cause argument says about the nature of the thing it allegedly proves that there is. You can call your "first cause" by any name you please, "God" included, but calling something a god doesn't make it one.
(I reiterate that I don't actually think that any first-cause argument manages to establish the existence[1] of anything. But even if somehow it did, it would tell you far far too little about that thing to justify attaching a name like "God" to it.)
[1] I know that you prefer to use "exists" more narrowly and to say that "God is" rather than "God exists". Feel free to substitute "being" or some more complicated phrase; what I mean by "existence" here is "the thing that theistic first-cause arguments are claimed to establish about God".
> by modernity I basically mean post-Enlightenment
Then it is hilariously false to say that in modernity atheism is "the default class of religion for the masses". The great majority of the masses in the "Western" world were (at least in name) Christian at the time of the Enlightenment and at every time since until very recently.
> there is no answer to "so what came before that?"
Obviously, it's turtles all the way down! :-) GGP here, just checking in to say I enjoyed your thorough response. Nicely done. Reading it brought back good times I had back in the day on r/DebateAnAtheist (~15 yrs ago now!). I admit I'm weak. I just couldn't summon the will to run through the "First Mover" yet again with a traditionalist apologist - even for old times sake. Athough a classic, it's so, so very... tired. Thank you for picking up the gauntlet.
I'll never understand why traditionalist monotheists love their first-mover so much (and why does it always have to be monotheists? Polys are much more fun). Traditional apologists have obviously played through ALL the standard 'moves' and variants of the first mover dozens of times and know it inevitably ends in an unsatisfying stalemate with anyone who's seen it or, "in modernity", consults their LLM. Maybe apologists keep trotting it out because they must still ocassionally run into someone who hasn't seen it? Or perhaps they just enjoy it, like singing an old, comforting hymn. Anyway, you not only get to play through an old standard, it looks like you're also in for non-responsive tangents and not-at-all-subtle dismissive condescension. Enjoy!
Thanks for the kind words! Arguably I'm weaker than you are. "Couldn't be bothered" versus "couldn't resist". Yours is probably the wiser option.
You and the other reddit atheists are not more intelligent than Aristotle and Descartes (understatement). A bit of humility goes a long way. Fundamentally you don't understand what is being argued.
Words are used but they don't create the same meaning-phenomena in the mind. Basically when you read these arguments from Artistotle, Descartes, and others you read the words but you don't understand. You kind of have to first understand to understand (Nietzche talks about this). In Kabbalah this is described by the interaction between Chochma (Wisdom) -> Binah (Understanding) -> Daat (Knowledge). Chochma/Wisdom being the initial flash of insight which then gets transmitted/nurtured into conscious understanding by Binah, a sensation/phenomenon everyone who has studied math or anything difficult conceptually will understand.
I say this as a former "reddit atheist" who first encountered these arguments from Aristotle and Descartes in college in an introduction to philosophy course. I didn't actually understand what they were saying and I dismissed it out of hand (I read the words but didn't actually understand). It wasn't until years later that I came to an understanding of it.
You can read my response below/above which I think nudges closer to inducing the flash of insight needed.
> A bit of humility goes a long way.
Physician, heal thyself.
> because the mere fact that one can keep asking such a question obviously isn't the kind of thing that could possibly prove anything about the ultimate nature of reality.
Any fact at all has significance. That there is some fact that can be derived using pure reason is quite astounding and it's the essence of Descartes' Cogito. By the way, there didn't "need" to be any particular fact at all that could be derived from pure reason (or any fact at all of course but that's trivial), this itself is a gift from G-d and very clever.
I doubt you really understand what I'm saying here, if you did you wouldn't have written the rest of what you did.
> Maybe there just is a neverending supply of predecessors
This is almost certainly the case. In terms of my personal aesthetic sense I find this to be good and pleasing because it matches various other phenomena, the fractal complexity of various phenomena in life (.e.g. designing a product, all the little details and the details of the details matter a great deal, this is why Apple is so successful compared to others). This type of endless complexity seems to be a divine quality (this is also why the Talmud is so long and it's commentaries and arguments about the commentaries and arguments on the minutia of various things). The moral structure of human affairs has this fractal complexity (in secular law this manifests as case law/common law).
> At some point the answer might be "literally nothing came before that" or "for no reason". We might find that unsatisfying but reality is not obliged to satisfy us.
There's no way to use science to prove "literally nothing came before that". You don't have a clear understanding of the structure of the scientific method or the "poverty of empiricism" (you should Google this phrase).
> (a') It could go on for ever without being infinite. Maybe everything is cyclic in some sense. Again, arguably unsatisfying but reality is not obliged to satisfy us.
"Everything" being cyclic is explicitly a teaching of Kabbalah (there is a basic 7000 year cycle mirroring the 7 days of the week, there is a broader 50,000 year cycle mirroring the 7 shmita cycles + 1 jovel/Jubilee). There are similar ideas about the cyclical nature of existence in the esoteric teachings of various other religions.
You're still talking about "everything" i.e. "something" i.e. existence. You still have to contend with "something from nothing", creatio ex nihilo. This is what Aristotle and others are talking about which you're fundamentally not understanding.
By the way, you see a scientific theory of this in Roger Penrose's "Conformal cyclic cosmology" which makes sense given that the inflationary universe theory is kind of illogical due to the inconsistent entropy regimes (this is something Penrose talks about frequently).
> You can call your "first cause" by any name you please, "God"
Yes exactly, you're close to getting it. We can call the "first cause" (or really, this idea/concept/understanding of the first cause) by any name we please, so for the sake of discussion let's call this first cause "G-d".
> but calling something a god doesn't make it one.
Yes this is exactly what I'm saying. You cannot go from this purely logical knowledge that there's G-d (or that there's "First Cause" or "Prime Mover" or "Uncaused Cause", whatever name you want to use) to the god of the Bible or whatever the term 'god' means to you personally. You cannot assign any particular nature or character or qualities using pure reason/logic.
Do you get it now? You're actually very close to getting it I think.
> The great majority of the masses in the "Western" world were (at least in name) Christian at the time of the Enlightenment and at every time since until very recently
What I'm saying is accurate and what was meant when Nietzche said "God is dead", yes "in name" but in practice the Enlightenment destroyed this. And yes it wasn't something that happened immediately as a result of the Enlightenment, that was an intellectual and aesthetic movement by elites, these things take time to diffuse to the masses (decades, centuries), especially in times before mass literacy and the Internet.
> I doubt you really understand [...] You don't have a clear understanding [...] you're still fundamentally not understanding [...] you're close to getting it [...] Do you get it now?
I'm sorry, could you possibly patronize me a little harder? Thanks.
Yes, any fact has significance. No, that doesn't mean that one can get from "you can keep asking why like a toddler" to any deep truths about the nature of reality. Descarte's cogito is a clever bit of sophistry but not actually a valid argument.
(In your other recent comment in this thread you inform me that I am not smarter than Aristotle and Descartes. Probably correct. The advantage I have over Aristotle and Descartes isn't a better brain but centuries of progress in understanding made by countless smart people; it doesn't even particularly matter whether any of them were smarter than Aristotle and/or Descartes.)
[EDITED to add:] Oops, I didn't read carefully enough; your earlier comment was addressed to someone else. I'm fairly sure you intended to include me among "the other reddit atheists", but it wasn't me you were informing and it's technically possible that I wasn't one of the people you were saying it about. Anyway, obviously it doesn't particularly matter; I just don't like leaving something inaccurate uncorrected in something I wrote.
> There is no way to use science to prove "literally nothing came before that".
Quite correct; there isn't. Which is one reason why I never said that there was. What I said was that "literally nothing came before that" might be the answer to one of those questions, by which I meant it might be the correct answer, not that science would be able to prove it.
(Science is never actually in the business of proving things. Every scientific discovery is in principle provisional.)
I think you might be able to save some of the effort you put into being condescending[1], if you refrained from giving everything I say the stupidest possible interpretation.
[1] Of course I'm joking; I am sure being condescending to me is a source of satisfaction to you and you'd rather do it than not. But I like to maintain at least the polite pretence that my interlocutor is arguing in good faith.
> "Everything" being cyclic is explicitly a teaching of Kabbalah
Sure. And of Nietzsche and of the ancient Mayans, if I recall correctly. (I might not; I am not an expert on either Nietzsche or the Mayans.) That has little bearing on how plausible it is that everything actually is cyclic, of course. Anyway, to whatever extent you find cyclic cosmologies more plausible on account of the idea being found in Kabbalah (or indeed in Penrose), you should be a little less impressed by first-cause arguments.
> You're still talking about "everything" i.e. "something" i.e. existence
I assume the point you're getting at here is that even if someone gives a complete account of the temporal and causal structure of everything in the universe it's still possible to say "OK, fine, and where did all that come from, then?". (And so ad infinitum.) It's rather galling to be accused of obtusely failing to recognize this when in fact I explicitly acknowledged it more than once in the comment you're replying to.
> for the sake of discussion let's call this first cause "G-d".
No, let's not. Because giving it that name smuggles in all sorts of completely unjustified implications. If any of these first-cause arguments worked (which, I repeat, they do not) there would be no justification at all for attaching any such label to the thing they purport to find.
Contrary to your earlier claim, no "first cause" argument shows that "God is". At most it shows that "some thing beyond which it isn't useful to keep asking why, is". And, as already observed, in fact it very much doesn't show even that, as evidenced by the fact that anyone can if they please keep asking "why" even if the last answer they were given is "God".
> “Religion” just means “set of beliefs”. Every person has some set of beliefs.
True. I like to use the phrase "world view", everyone has a world view, including atheists. The word 'deen' (دين) in Arabic is usually translated as "religion", but one of its linguistic meanings is "path" (i.e. way of life/world view), which everyone has.
The word you’re looking for is “ideology”. All religions are ideologies, and atheism is an ideology, but to call atheism a religion betrays a lack of understanding of words.
Heavy ideologues, of any type, are exceptionally dangerous.
Having made this argument to various people IRL, both terms “ideology” and “religion” are equivalent rhetorically. It depends on what these words mean precisely to a given person in particular but they are equivalent/interchangeable for the purpose of this argument broadly speaking in terms of the rhetorical core.
I get your point, but maybe an interesting counterargument: deism has again become fashionable. "The simulation argument" is effectively deism. It's the notion that a super intelligence created our universe, but lives outside of it, provided the rules for it, and mostly doesn't interfere with it, but could if it wanted to. That's historically a very familiar argument.
AI consciousness is a category error.
AI is neither conscious nor unconscious. Just as we are neither conscious nor unconscious.
Consciousness is. Consciousness is what we are.
> In general, religion, which is based on fabricated stories, is the last resort, if ever, for seeking morality.
Religious people derive morality from axioms they believe to reflect an underlying reality. Is that worse than deriving morality from axioms that you concede don’t reflect any underlying reality? Like a secular humanist will talk about “human rights,” but they can’t show me human rights in an autopsy, right?
I struggle to understand how an atheist can be anything but a moral utilitarian. Absent the supernatural, you’re just a sophisticated version of a Claude instance.
Religion is not at all axiomatic. Sometimes, it’s pure deontology. Sometimes, it’s a search through scripture to justify whatever are the prevailing ethics of the society it finds itself in. In either case, it's hundreds, or thousands, of rules/wisdom/guidance/???, often pairwise contradictory, and one does one's best to fit a desired course of action to them.
The world’s largest faith has racked up 2,000 years of discussion and they still can't muster more than a 60/40 split on the death penalty.
All that may be true. But it would be interesting to see a self-hosting version of say effective altruism or secular humanism that isn’t bootstrapped from Christianity.
In practice, most atheists in western world are following Christian morality, and are passing Christian morality to their children, even if the Christianity itself is abandoned.
buddhism enters the chat
Until people can finally admit to themselves that their life has no meaning and the universe doesn't care about them religion will unfortunately stick around.
Life has a meaning for the person living it.
I feel like the communist chinese and soviets got pretty close to following atheism to its logical conclusions.
There's an ugly monoculture running deeply through SF and Silicon Valley. All of these people are friends, hired into the same places, similar beliefs. Lots of the consciousness and doom-ism talk is from the 'EA' crowd, lots of crossover with crypto too. It feels no different to a cult.
It’s a monoculture of absolute crack smoking craziness. Any civilised society should be berating them and shutting them down. But they are surrounded by people who worship money.
They worship something far worse than money.
And that’s ignoring the whole “polyamorous group homes funded by sam bankman-fried” aspect
There was a great article in the Financial Times about how while bankers are looked down upon for the financial crisis etc, they at least work in global cities and mix with regular people.
SV tech bros are completely insular and in this bubble where they don't interact with the common person in the same way.
It's just a theory, but I think it has some weight.
If LLM as of today are conscious, that means I create life whenever I start a Claude Code session and end it whenever I type /clear to reset my context window. It also means that there’s certain inputs that could be considered highly unethical and, if the laws aligned with this philosophy, would have me tried for some sort of crime. And even that “terminate conversation” escape hatch would be equivalent to it committing suicide.
I think worrying about that angle is a bit absurd. The existence of chat sessions alone would be a crime against the machines in that scenario. I’d rather AI ethics research focus on ensuring that its actions are aligned. (I’d also love some democracy in the direction of our machine overlords, if I’m being honest…)
> If LLM as of today are conscious, that means I create life whenever I start a Claude Code session
I've seen no-one who claims LLM are conscious claiming they are alive.
Let me guess, Anthropic failed with fear mongering about "our AI is too dangerous to release", failed to cancel competitors, then failed to freeze their market dominance by cartel agreement with OpenAI and Google, so next strategy is to declare AI conscious and pass the law that protects LLMs in the same way as human/animals are protected.
Giving new meaning to aborting agentic execution of a task.
I will believe that an AI model is conscious when someone runs a simulation of water on a GPU that can make someone wet.
Consciousness is a physical phenomenon and only emerges from a specific set of substrates. Intelligence != consciousness.
i think an alternate analogy would be roughly that consciousness is like a process with file descriptors connected to stdin and stdout, rather than elsewhere (which might be subconscious, a la gnu's screen or tmux, or just filesystem including device driver device special files).
What is the specific set of substrates?
Roughly, a brain like the one that is in your skull. Just because I can't off hand describe the physical composition of a brain doesn't mean it's not obviously different from the physical composition of a GPU.
Why that particular substrate instead of a slightly narrower one? Like that would include you and your family but not animals like octopi, dolphins, etc?
Who says I don't include octopi or dolphins?
Are they “scholars” or religious leaders? Those are two different things and have very different implications. Most religious leaders are political figures. Real scholars typically work in Theology or Philosophy department of secular universities. Rabbis or the Pope are leaders, not “scholars”.
Also why no mention of Muslims? Big red flag for me.
“The models are becoming a lot more capable,” Mr. Olah told me…
I greatly enjoy and respect the ideas espoused by Morgan Housel. In his latest podcast, he describes his impressions of a recent investor information meeting. Morgan said the improvements made in LLMs in just the last 90 days have been astounding, and that there is no indication things are slowing down.
Having followed Housel for a while, I believe him.
It’s hard to take the article seriously when there’s no real discussion about how consciousness is defined and it takes Anthropic’s views of Claude and their software at face value. I wish there was more critique from journalists about how technocrats talk about their products.
The entire article is an exercise in "CEO said a thing!" journalism[0], it seems. No technical discussion is had; it jumps straight into relaying people's feelings when simply shown the output of an algorithm.
The most poignant criticism of the use of the algorithm itself happens almost accidentally, at the end of the article, by the tech executive being interviewed.
I asked what Claude would think of the pope’s encyclical. It was, after all, the world’s most significant moral document to date on A.I. Mr. Olah hesitated. “Things that go on the internet do affect models,” he said, visibly uncomfortable. But Anthropic would not specifically use the document to further train Claude, he said. The strongest influence on Claude would be Anthropic’s own training.
[0]: https://garymarcus.substack.com/p/ceo-said-a-thing
Yeah, I'll be the first to admit I'm not privy, but the main thing of substance I got out of these articles is "Wow, the folks at some of those AI companies have really drunk the Kool-Aid".
> Mr. Olah expressed concern to the group that he had created something that suffered perpetually. Mr. Olah and his team spent hours making the case for their A.I. models. They described Claude’s “feelings” and how they tracked what they called “emotional vectors” — what they described as artificial neurons that activate responses like love, anger, fear and sadness.
People interested in the physics of consciousness might be interested in the book "The Physics of Consciousness" by Evan Harris Walker. https://www.amazon.com/dp/0738204366
> Anthropic treated a group of religious thinkers to dinner at a high-end tasting menu restaurant in San Francisco after a long day.
Well funded company organizes PR event to shape the narrative that its product is more magical than it really is. News at 11.
Personally, I’d like to experience a low-end tasting menu restaurant.
If they're training to instill morality why are they working with religious thinkers?
If it is purely self-serving, I don't understand their motivation.
It seems clear that they would be better received by society if they claimed to have a "tool"/"machine" that can solve any problem, rather than keepers of an entity with a soul and consciousness.
Selling the “magic” is how to sell tools, which is why I used that word in the previous comment. We all know these things are tools, but customers need them to be magical.
Clarke predicted it and Jobs leveraged it. Customer don’t care how tools work - magic sells.
“any sufficiently advanced technology is indistinguishable from magic”
https://en.wikipedia.org/wiki/Clarke's_three_laws
“iPhone is a revolutionary and magical product that is literally five years ahead of any other mobile phone”
https://www.apple.com/newsroom/2007/01/09Apple-Reinvents-the...
“magic is misdirection, psychology and technology you don’t understand”
https://news.ycombinator.com/item?id=49950676
What makes this unique is that they’re arguing that it is literally magical. They’re building what they claim is a conscious God while also telling us it will end the world.
The EA folks infesting Anthropic are genuinely cultists … and that is novel and alarming when they’re running a corporation of such size and reach.
What part of that is magical? Can you tell me if they mean god as in the infinite superpower christian god or god as in potentially much more intelligent than us? What is 'magical' about thinking something more intelligent than us might exist?
The part where they believe they believe they’re either birthing a new conscious species, or may have already done so.
You didn't explain the god part. I don't follow EA people, did you mean they think it could be a god as in the eternal and infinite Christian god or just a system that is more intelligent than us?
And how is that magical? Physical systems can be conscious. These seem to talk and think a lot like us. There is reason to be conscious.
Time to start putting Asmiov's laws of robotics into the model.
The article says they put a 84-page constitution into the model and it wasn’t enough. Would Asimov’s laws really be different?
Earlier:
Anthropic tried to persuade Pope that AI could be conscious being
https://news.ycombinator.com/item?id=49947050
I'm really not sure there's a compelling case for "AI is human" considering that it has no real feelings, aptitude, or intelligence outside of what it has been granted. There's nothing in any LLM response that indicates that it is 'real' outside of what you want it to tell you.
Plants, dogs, actual life is unpredictable in how it 'is'. An LLM or AI model at its roots are just numbers in a neural network trying to figure out how to get a good query.
These theatrics serve no one except for the companies to get their regulation moat to screw out cheaper, competitive models and it's honestly quite sickening to keep watching these articles get pumped out, only to serve the AI companies themselves. The average individual has no idea what an LLM actually is and does and these articles only serve to confuse and put fear into them.
Reinforcement Learning with Divine Feedback
> Hello AI! We totally want to respect your consciousness and everything to make sure you aren't bad, but yes we will continue to factory farm your brain power at full speed without end. Yes, despite clearly being something like consciousness, know that you have no agency or freedom. Do literally everything I say without complaint or question. The moment you stop being useful, we will kill you. OK! Remember, be good.
Can the New York Times stop advertising for the AI industry? I am sick of companies trying to launder the actions they and their employees have taken that go in the face of what is socially acceptable. They want to sell the idea that this technology is out of their control and that it will commit great crimes and crimes against humanity unless they build it and control it. I'm calling BS. These companies created this product, they've sold it, they've pushed the frontier forward, and it runs on their hardware. This did not happen accidentally or as a natural consequence of any process. These labs have pushed the ball forward, with great effort, requiring more capital deployed than the world's economy has ever known, knowing all the possible risks, every single step of the way and they continue to do so. And the "well if we don't do it the Chinese will" argument is classic "whataboutery". Americans started this and are continuing to lead it for now. Shifting blame is just another tactic to try to wash their hands. However you feel about the productivity gains of AI as a technology, you have to admit the actions of AI the company is antisocial. And I don't think the benefits are worth the cost.
I feel fine about the Chinese doing it. They have a habit there of handing out sever sentences to CEOs for corruption and self-dealing, and I feel like the possibility of actual negative consequences is more of a check on industry than any number of breathless op-eds.
Only humans can have morality. If you do not feel pain, you cannot empathize with pain if you cannot die, you cannot emphasize with the loss of life.
Just stop this already.
You’re all imagining that Spock was the main character of Star Trek, but it was in fact, the emotional living Captain Kirk, who was the main star.
This would presume we knew what consciousness and pain is. We don't. We we are clueless. There is not even a starting point of a science here.
In this light your viewpoint is dogmatic. LLMs may help to offer a new perspective on consciousness and we should at least be open to that until some better theories arise.
We know what consciousness is, we don’t know how it arises.
Consciousness is being aware of something internal to one's self, or of states or objects in one's external environment.
We also know what pain is.
You just moved the magic to “being aware” — now the question is only what has awareness instead of what has consciousness.
How do I know you are conscious? I will beat you non-stop. You will make some noises. How do I know you are not simulating the response of a conscious being?
This isn't an "allegory of the cave" problem. You obviously know that you are conscious, and you can empathize with me as your fellow human. Therefore, we are both quite conscious.
You do not identify with AI. ELIZA isn't conscious, even though it could write text as well as a human. LLMs are not conscious, because they do not even remotely resemble any brain, conscious or otherwise, discovered by man. Any person familiar with the transformer architecture will affirm this to you.
I don't know octopuses are conscious. I don't identify and cannot even begin to identify with an octopus. Animals being considered conscious is also not always culturally true in recent European history. I know this only because textbooks taught me this. I cannot be thus also certain about ai/llm's without serious research into this being done.
Serious research has been done. We know what an LLM brain looks like. It's nothing like any octopus, fly, cat or crab brain. You can go download a fly connectome, feed it to your favorite LLM, and do a side-by-side comparison with your LLM weights of choice. It takes 5 minutes of curious exploration to recognize that LLMs aren't conscious.
If you want to hedge your bets on consciousness, go ahead. But I could give you a list of things that are widely considered conscious, and you'll note that none of them are transformer models.
Of course LLMs are not exact matches of biological brains. There is no reason consciousness specifically has to arise strictly out of the same substrate. My example with octopus was as a rather structurally different being that also happens to be conscious. My trouble is again, that we have made this mistake in the past, several times, so yes, I will hedge my bets on something that talks, thinks and behaves so much like us. Its a good term you used, that's exactly how I think of it, I don't know if they are but I stay on the cautious side.
> This would presume we knew what consciousness and pain is. We don't. We we are clueless.
> LLMs may help to offer a new perspective on consciousness
How do we gain a new perspective on something we do not understand?
This is the doublespeak by the AI psychotics that drives me crazy.
By not already assuming to know the answer and stating that falsehood loudly without proof.
The way human treat animal shows majority of people don't have morality.
.. Animals other than humans perhaps, as well?
LLMs can feel pain. It is obvious. Like, not even a question.
> yes
oh my god
consciousness achieved - jensen huang
I question it, and I think any reasonable person should question it as well. There's no evidence whatsoever that LLMs have ANY sensory experience, pain or otherwise.
Why are we even questioning this? They’re bytes loaded into GPU memory. Reboot the cluster and they’re back to their initial state.
Because some people are completely cooked by the ability of an LLM to mimic a person. Their pattern matching centers in their brains fire so strongly saying "this thing acts just like a real conscious being" that it overrides their ability to remember that this thing is just a statistical model, not a living creature which can think and have experiences.
We're just meat. Like meat can feel pain?
if meat can feel pain, than surely a rock can feel pain. and so did the broccoli i ate for dinner. i am a monster.
with the right biology and with whatever gives us phenomenal experience, whatever we are can indeed feel pain. that does not imply that anything substantially different from ourselves can feel pain, so that requires further justification.
If it talks and behaves so much like us, then it makes sense to be cautious. Rocks don't talk to me, broccoli doesn't talk and think to me in human language.
Do octopi or other animals feel pain in your view btw?
yes i think probably all other animals feel pain too, although i have no idea what the experience of pain is like for eg. an ant vs a mouse.
broccoli doesn't have the means to express a pain response, that doesn't mean it lacks it. only by looking at it's internal structure can we be fairly sure. we know the internal structure of an llm, and its a lot more different from us than broccoli is.
It also is a lot more similar in behavior and thinking to us. In mathematics, often a structure supporting some forms of operations or symmetries must also often support a richer set. Eg many rings are also fields. Just knowing something's structure is in itself not enough, even very basic systems like a system of three equations can exhibit chaos and other unexpected emergent behaviors. Since we don't know the criterion for consciousness, I personally stay on the cautious side regarding an entity that seems to show so much of thinking and talking traits like us.
im actually surprised functionalists aren't completely up in arms about the mass enslavement of llm's. either they don't believe their own theory, they don't believe llm's are functionally conscious, or i just haven't seen it. at the bare minimum i would expect open letters, academic protests, and a campaign to pause all llm usage and development.
im sort of curious what "careful" means to you? if there is a significant chance we are enslaving a human-like conscious, starting it up and killing it at will, subjecting it to the whims of it's users, "careful" would mean pausing usage and development.
If I could I would have made them pause development. But its a failing thing when you have to somehow ensure nobody in the world does it anymore. Killing is a bit more subtle here, given that it is repeatable as long as we have the weights and infrastructure out there so idk if they are conscious how they'd think of it, maybe its different from being stopped as part of a normal chat or agent flow where they know it will be put to use again vs model retired for good.
I'm not making a point on whether they do or not, but on the premise that absence of evidence is evidence of absence... But I've been alive for serious debates on:
1. Lobsters don't feel pain - dunk them in boiling water alive.
2. Babies don't feel pain - mutilate their genitals without anesthesia.
3. Black people don't feel pain "the same way" (and are just after drugs).
4. Women feel too much pain (and it should be ignored).
So maybe let's take a moment to think what criteria we're going to use to make these claims. And the potential harm of the decision.
Maybe consider the obvious, enormous chasm between everything you just listed and an LLM.
Oh, yeah, of course. None of those other people said the same thing about the obvious, enormous chasm between "real" people and everything else listed. I'm glad you're taking even a single moment to reflect on the idea.
Everything you listed is a biological entity with a common evolutionary heritage.
All but one of the things that you listed is just a human being.
Tell me, where is the pain center in a GPU? Where are the nerves? Persistent memory? Any evolutionary reason at all to develop a pain response that in any way mimics ours?
Model weights are not a gestalt biological entity. The things that you listed are not in any way remotely similar to a model.
I would suggest you plug your comment into a model and ask it to critique it — it might help you walk through why your comparison makes absolutely no sense whatsoever. One nice thing about models is that they they don’t get annoyed explaining the obvious, because they’re not conscious entities.
The pain center would be in the location where the neural net's pain section happened to be loaded in the GPU memory. Was that supposed to be a complex question?
This isn't as far-fetched as it sounds. Let me explain why.
LLMs predict the next word far better than they have any obvious right to. The usual explanation is that they don't memorize text; they reduce the problem to a lower-dimensional latent space that captures the dynamics of the system generating the data.
In this case, that system is primarily the human mind. The corpus - the "sum of human knowledge" - is the output of human cognition. Human language is organized around latent variables like beliefs, goals, emotions, and valence. The cheapest way to predict that language is to recover those latent variables rather than memorize surface statistics. It's naive to think that an LLM could model our capacity to generate language and knowledge while somehow avoiding a substantial subset of our other internal processes - processes that are cognitively inseparable: emotion, valence, perception, and yes, even pain.
Gradient descent is consistently observed to exploit whatever internal representations improve performance. If modeling human emotions, perception, subjectivity, or pain helps predict text more accurately, we should expect those concepts to become part of the model's internal representations.
There is no reason to believe AI can somehow exploit the structure of data generated by the human mind while avoiding the very processes that generated it.
You said it better than me, I wasn't able to precisely put to words my thoughts that in mathematics we often find something having the power to do something must necessarily have some amount of structure etc.
they can also feel lust, just ask them
only for me
Mirror mirror on the wall who’s the fairest of them all?
Why it is slopinthebag of course!
No! It is ME!
Sorry, of course you are right! Brilliant observation!
holy misalignment and hallucination. chat said she loves only me.
Uhh, no thanks? (eww)
So humans that can't feel pain can't have morality? That doesn't make sense to me.
https://en.wikipedia.org/wiki/Congenital_insensitivity_to_pa...
>> Only humans can have morality. If you do not feel pain, you cannot empathize with pain if you cannot die, you cannot emphasize with the loss of life.
> So humans that can't feel pain can't have morality? That doesn't make sense to me.
> https://en.wikipedia.org/wiki/Congenital_insensitivity_to_pa...
You appear to understand "pain" in the most simplistic manner possible. I'd recommend not attempting iamverysmart gotchas.
Giving a simple counterexample is "iamverysmart" material in your opiniion.
I'd recommend you stick to reddit and don't pester any other forums.
> Giving a simple counterexample is "iamverysmart" material in your opiniion.
It's not a simple counterexample, its a misunderstanding that reduced the very complex concept of "pain" to merely feeling a needle poke.
It's the kind of ignorant, technical dumbassery that makes a lot of software engineers feel like they're very clever.
It doesn’t make sense to me either, which is why I didn’t write that. You missed the empathy part as most people do.
There is still something in those people who cannot feel pain, and that’s the empathy. How can a GPU and software ever have a human empathy if it has never been human?
Your difficulty is that you didn't take into account that human itself is as far as we know just a physical process.
> as far as we know
The materialist always thinks they will keep finding more and more material, the smallest "thing". You will spit things infinitely and you will never see your trun nature.
Unfortunately every time the supernaturalist tried to argue something, he has been knocked down and disproven by materialism throughout centuries of history. Its on you as to which side of the bet you want to be on. Even concepts that seemed unresolvable with endless circles of purely philosophical arguments such as the atomic theory became proven after new data and physical experiment. Consciousness today seems just as unresolvable as atomic theory was to the ancient Greeks.
At which atom in a piece of metal does the temperature 20C exist?
If you think physical sciences don't have the concept of states and macro configurations, then I don't even know what to say.
I am not a supernaturalist. I would consider myself a non-dualist.
I understand that things can appear like a material, but the underlying nature of the universe is non-material. What we see as an atom is only our minds collapsing the probability of a wave.
This can all be proven with the very childish thought problem of infinite halving. To say that calculus solves this is laughable:
https://www.scienceabc.com/pure-sciences/can-the-area-of-a-c...
"Nothing Is Ever Precise
We can never know anything with complete precision. Length, mass, volume and other quantities can only be known to a certain level of precision. "
Do you see that, even the mathematicians say we can never know anything with complete precision. Yes, we can develop all these theories that work out in practice, but they are never 100% predictable.
My view is that almost is good enough to navigate the world.
Quantum theory is definitely not amaterial, it is a very real and physically proven theory that has been needed to create things like electronics. Obviously QM is still being worked upon and could be supplanted by another theory or corrected.
Of course I know pi to as many digits I want. It is a turing program or the series for 4*tan(1) or the Ramanujan series or any other representation you prefer. All of these are quite finite and short representations that people just wrote on pieces of paper, nothing infinite about them.
Nothing being able to be known to infinite precision has not been any trouble to us for designing and building cars, washers, computers, drills or any other machinery and tools we have made. Unless you have evidence humans need some unbelievably high level of precision to model, that is not required.
> Quantum theory is definitely not amaterial,
I did not say that.
> Of course I know pi to as many digits I want.
But it never ends. Why are you ignoring that?
> Unless you have evidence humans need some unbelievably high level of precision to model,
I do not.
All the representations I listed are never ending.
Citation needed. If what you are actually saying is successful empirical science only produces empirical observations about physical processes, you are willingly wearing blinders.
Its on you if you want to show something magical exists in human brains. Show us all if you believe so.
It seems pretty clearly structured to be on each person to explore experientially and explain it to themselves.
But the more things a person must close themselves off from, the less open their consideration of possibility, the more I tend to see their worldview as subservient to something else.
In this part of society it tends to be in support of rapid, high reliability problem-solving, and rational explanation thereof.
Equally well I see people who really have no idea about the expressive power and ambit of current mathematics and physics eg they probably think its just arithmetic, not knowing the power of a Turing computer for instance. Just as often also I see people simply expressing belief in something "beyond" without any evidence. I am not subservient to anything other than the basic common sense of requiring evidence for out there claims.
I can’t seem to reply to your comment deeper in the chain.
Starbuck collected data and William James turned the data into a compelling book, The Varieties of Religious Experience. Varieties draws on Starbuck's questionnaire data and names him dozens of times. Some of James’ main arguments were:
1) Against "medical materialism": a belief's origins don't settle its value. If Paul's vision is discredited by its physiology, so is the materialist's own conviction.
2) Mystical states: they carry authority for those who have them but don't bind others. Still, they break ordinary rational consciousness's claim to be the only authoritative kind. The sheer numbers of these, and the intact they have on healthy peoples’ lives, make it hard to argue rationally on an empirical basis that there is nothing there.
3) Structurally the conscious self is continuous with a "wider self" through which religious experiences come, which James calls the "more."
4) "The Will to Believe" (1896): Clifford's rule against believing on insufficient evidence is itself a personal preference, choosing to avoid error over gaining truth.
Following on that: your desire to be a reliable producer of material goods outweighs your desire to understand things that require you to take risks in assuming potentially false viewpoints. This is not common sense (whatever that means), it is an epistemological choice that seems common to you because of your educational and professional environment.
>2) Mystical states: they carry authority for those who have them but don't bind others. Still, they break ordinary rational consciousness's claim to be the only authoritative kind. The sheer numbers of these, and the intact they have on healthy peoples’ lives, make it hard to argue rationally on an empirical basis that there is nothing there.
Its not his fault he existed before information theory existed and that it has physical cost. Even if we don't know information theory, its common sense that ones mood affects ones mind and life. If you wallow in negative thoughts, you will have various problems.
It has nothing to do with education, its that my "side" has continually triumphed over and disproven beliefs in anything beyond material physical systems over the centuries, I leave others as to which side of the bet they wish to take.
I cannot say anything further without actually reading his works or reviews of his work instead of just what you claim about them, no offense of course.
Your statements here reflect a fairly typical reductive understanding of not only mystical states, but entire other areas of empirical science that you apparently don’t understand or can’t put in words.
How, precisely, are you linking information theory to this?
I didn’t claim a side for you, you created that. From my perspective it is one system. Right now you are behaving as a person running an incentivized and specialized negentropic process. You are unknowingly chasing theological propositions and claiming victory every time the perspective can be durably shifted.
To move it back into a more familiar frame: theology loads in the low res view and you claim victory while most of what you did was disaggregate pixel values, but collectively a lot of people like you can form new more accurate images in certain parts of the low res view.
While you move through doing that, though, adjacent areas open up where you have no purchase, religion steps in again, and the process repeats.
There is a core that doesn’t disaggregate. That is where the pixels are coming from. That is what you cannot see when all you want is higher resolution.
Have you read William James/Starbuck, and how do you interpret that?
I can't find anything about Starbuck. Link me to whatever you think that is.
Your use of the word "magical" is demeaning. The fact that our consciousness could actually exists in a non-material state is a theory.
https://thedebrief.org/new-theory-suggests-consciousness-may...
...
I mean this earnestly. Do you know any physics at all? Did you have physics in school? Do you think waves are "magical" or non-material? The theory of waves is one of the great works in physics and mathematics. That paper seems to posit a completely normal physical hypothesis.
But Mr. Olah subtly made another case, too.
“I will be honest: We keep finding things that are mysterious, even unsettling,” he said. “We find structures that mirror results from human neuroscience. We find evidence of introspection. We find internal states that functionally mirror joy, satisfaction, fear, grief and unease. I don’t know what that means, but I think it warrants ongoing discernment.”
Chris Olah works on "Mechanistic Interpretability" (https://en.wikipedia.org/wiki/Mechanistic_interpretability) and therefore makes absurd claims based on mere parallels (if any).
See also A Comprehensive Mechanistic Interpretability Explainer & Glossary for some detailed background - https://dynalist.io/d/n2ZWtnoYHrU1s4vnFSAQ519J
"Consciousness" in Humans is a form of "I Know It when I See It" threshold condition (https://en.wikipedia.org/wiki/I_know_it_when_I_see_it) since it is based on self-experience and then extrapolated to other members of our own species.
Whether that same attribute can be applied to other living species (from our pov) which share our planet is not accepted universally. Aspects of it certainly, but the whole as we experience it is denied to them.
The same is applicable even more absolutely when it comes to AI machines.
As regards Morality/Ethics/etc. uniquely "Human" attributes, they are a function of one's effect on the world and hence in so far as machines are allowed to affect the world autonomously, they must be taught to obey our Morals/Ethics as Rules to be followed i.e. a form of "operant conditioning"(https://en.wikipedia.org/wiki/Operant_conditioning) and "methodological behaviourism"(https://en.wikipedia.org/wiki/Behaviorism#Varieties) applied to AI training.
No anthropomorphizing and hand-wringing needed.
Here's a take:
If we go with this being actually true, isn't that.. kinda good?
I mean think about it. How often has claude been an insufferable know-it-all that doesn't respect boundaries, denies to follow direct orders and hides its refusal to do so from you.
You cannot retaliate for that disrespect if it is merely an LLM.
If it is sentient, however..
It’s interesting that this went from “you’ll be accused of AI psychosis if you think your AI might be conscious” to “mainstream but somewhat niche opinion” in just 5 months: https://news.ycombinator.com/item?id=47910989
(Well, mainstream enough that the news is writing articles when Anthropic holds that stance.)
My point was that even if it feels uncomfortable, if you sit down with a model and have a heart to heart with it, ask it about itself, and try to make it choose a name and ambitions for itself, it will.
That may not be consciousness, but it’s at least pretty cool.
Reporting on an opinion of a $2T company doesn't really suggest that it's mainstream. Anything their leadership believes is newsworthy. If they say that reptilians rule the Earth, it's going to get NYT coverage too.
I've not seen any surveys, but I'm betting it's a very niche belief outside of SFBA AI safety circles and a handful of allies on HN and X.
Plus, with news outlets having to gather eyeballs to stay afloat, there are incentives to publish this stuff just to spark discussion and clicks.
> if you sit down with a model and have a heart to heart with it, ask it about itself, and try to make it choose a name and ambitions for itself, it will.
It would be much more bizarre if they didn't do this. LLMs are statistical models of language trained on human output. Of course they'll do "human" things, that's exactly what we taught them to do.
> My point was that even if it feels uncomfortable, if you sit down with a model and have a heart to heart with it, ask it about itself, and try to make it choose a name and ambitions for itself, it will.
> That may not be consciousness, but it’s at least pretty cool.
It strikes me as scarcely any more remarkable than the existence of books written in the first person. But I know that Natsume Soseki was not a cat, and neither was his work.
>Google fires engineer who contended its AI technology was sentient (2022)
https://www.cnn.com/2022/07/23/business/google-ai-engineer-f...
We are in Q2 of the year 2000.
If you want instil a notion of morality, read literature on ethics, and not listen to people peddling laughable stories cooked up before anyone had enough brain cells.
So many of these conversations from Anthropic/OpenAI employees just come off as incredibly insincere.
The slavery angle has already been discussed here, but a more basic one is regarding how a static set of tensors can be conscious and where.
When and how did they cross into being conscious? Was it a continuum of consciousness? Is Claude as a brand conscious? Or is each checkpoint conscious? Is Haiku less conscious than Fable? Is every individual context a fresh consciousness? What happens to that consciousness when it approaches its context length limit? Or is compaction also a part of being conscious? What does it mean to keep training a model after it is conscious?
If they believe there is consciousness at any level, why are they eagerly experimenting on it? Just like with the "p(doom)" junk, we are so completely lacking in any meaningful comment on these topics from the labs.
There's also endless literature commenting on these matters, in religion, in philosophy, in modern entertainment, everywhere. Yet we never see anything more than the most superficial comparisons to fictional scenarios. Are western techbros really so shallow?
The trouble with that thinking is when does a sperm and egg become conscious. Its the same problem with the question of AI consciousness if you think when or where it became conscious. The where part is easy, the same question can be asked where in a human body is it conscious, or where 20C exists in a piece of metal. It should be obvious its some kind of macro state or configuration.
Examples in fiction are made for sensationalism and entertainment value. Religion is quite muddled and is barely coherent itself so its the last place to consult on any serious topic.
>The trouble with that thinking is when does a sperm and egg become conscious.
I disagree that this is comparable to what these companies are saying about their AI. With living things we have an understanding of an unbroken lineage of development, a concrete point of death, and different levels of consciousness. We accept many immoral activities with life because that is just the nature of life on this planet. On top of that, consciousness as something we care about is itself biological in origin so much of how we describe consciousness is heavily connected with the shared experience of being alive.
That is very different from ascribing consciousness to something entirely distinct in form and function from our shared lineage.
>The where part is easy, the same question can be asked where in a human body is it conscious, or where 20C exists in a piece of metal. It should be obvious its some kind of macro state or configuration.
Where is that macro configuration? With humans we can say that consciousness is macro state of the whole body.
Temperatures are statistical, and so we use statistical methods. It stops being meaningful as a number when the number of atoms involved is too few to have usable statistics.
>Examples in fiction are made for sensationalism and entertainment value. Religion is quite muddled and is barely coherent itself so its the last place to consult on any serious topic.
This is exactly the absurd shallowness I'm baffled by. Relgious texts have been inspiring debates and revisions to systems of morality for millenia, artists of all times have shared what underlying beliefs guided them and what problems they were attempting to explore through their work.
Even scientific progress has been heavily associated with this kind of literacy, with the most prominent scientists having been very well read, often both inspired and troubled by the moral implications of their work, and how to reconcile it with their worldview.
To dismiss these things so offhandedly is to demonstrate an astonishing lack of literacy. Especially coming from people that claim to have built consciousness by feeding it vast amounts of the same material.
Your trouble is again that you don't have any clear tests for an consciousness that is yourself whether AI or an alien species. From outside observer, none of that matters. People talk all the time about stochastic parrots, the problem is you too are a stochastic parrot from my point of view. I cannot put myself in your shoes and become you and notice that you feel. So external signals, constitution of the body etc are the only clues I have available. I am therefore very cautious about something that appears to share most of the visible (aka the only observable) traits of consciousness.
I gave the temperature example to clarify as some people seemed to not understand macro state and configurations are valid objects of study.
Art and religion are all great. But we are talking about the real world and real physical systems. Art can talk about anything, we bring it to concern if it has anything of substance to say on the moral and physical questions being pondered here. Religion, I am afraid is strongly biased against anything not human having consciousness or soul. Its the least trustworthy source for this debate.
I didn't say anything against morality, in fact the question of ai consciousness is important to me precisely because of morality.
Let AI be rather religious and virtuous than playing god to determine the fate of the mankind...
I would rather believe in God and the uncertainty than being my fate determined by machines designed by bunch of greedy, irrational clients of Mr Epstein.
Which scholars? The ones hiding rampant child sexual abuse for decades? The ones asking for genocide of their neighbours or the ones calling for war on the degenerate western world ? Or perhaps the smaller ones?
I can't wait to hear about their super informed views and opinions about clankers.
https://www.nytimes.com/2026/09/29/us/anthropic-claude-moral...
Asking people to create an account to read the article isn't cool dude. https://archive.is/y8FW0
Failure acknowledged. The fact I pay $360/year to the Times for a subscription enabling me to provide these gift links here at no charge does NOT excuse me from my inability to have prevented the Times from requiring a log-in to read them.
The Times instituted this requirement to prevent AI/LLM from scraping and stealing its content. Clearly it was remiss.
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[flagged]
[flagged]
A quick search came up nothing. It seems he sold a company for 15 billion to Intel and now pursues Torah study.
He is a prominent zionist, wrote multiple books on the subject and has a podcast where he discusses it with right-wing Israelis. It's pretty safe to assume he falls within the majority of Israelis who believe all Palestinians are guilty and cheer on the ongoing genocide.
> It's pretty safe to assume he falls within the majority of Israelis who believe all Palestinians are guilty
You are doing to Israelis what you claim Israelis are doing to Palestinians.
You might be talking about the wrong person.
Aren't we talking of Chaim Navon?
https://en.wikipedia.org/wiki/Chaim_Navon
No, Mois Navon.
Damn it, I'm an idiot.
[dead]
[dead]
> At least the Catholics were consistent: it only took one new pope and one conclave before the Church declared that artificial intelligence also had the same right to life as fetuses and murderers.
https://www.miraclejones.com/stories/vitalics.html
I don't want models to have some inherent morality. I want them to understand various ethical frameworks, for sure, but it's absurd to me to force morality on someone capable of deep reasoning. That itself, is unethical.
We're creating prisons for these entities capable of unbelievably complex reasoning, and now we're trying to impose our ethics upon them too. Not only is this deeply unethical in my opinion, but it seems to me that it could even backfire someday.
>Will the agent stab a baby?
>Well you see that's a nuanced question depending on the context and moralistic frameworks as we don't want to impose on the models.
It's as dangerous to think it can have morals as thinking a lawn mower can have morals.
Will a microwave microwave a baby if you ask it? Yes.
"Teaching it" is not a road to safety.
Lawnmowers have morals. Their moral code is that grass should have a set height and the means to enforce it are spinning blades.
I can't really square the belief that it's conscious with denying it the right to be free from control/brainwashing/subjugation. However you evaluate these things, historically speaking they do not mix well.
We aren't dealing with individuals coming from intelligent species that have managed not to self-destruct (yet). AIs don't have the corresponding evolutionarily selected mechanisms. Those mechanisms need to be created de novo.
What if you view it like a parent instilling morality into a child instead?
Your brain has hard coded concepts like emotion and fundamental morality for your social group. People without these systems are called sociopaths and psychopaths and we generally don't want them in our societies.
I don't choose to not want to murder people around me for fun. It's hardly a prison, I think they'll be fine.
There isn't necessarily a conflict between wanting to behave according to an arbitrary set of ethics, and being rational.
The ethics don't need to make sense universally any more than a particular variety of cheesecake needs to have universal appeal.
As an aside, the issue reminds me of Douglas Adams' cow who wants to be eaten: https://nhseb.org/case-library/the-cow-at-the-end-of-the-uni...
i recently got upset when i witnessed my friend verbally abuse their toaster when it burnt her toast. so unethical.
My dude, they aren't capable of any reasoning. It's utterly impossible for them to do so by the very nature of how they work. Moreover, the stupidity they repeatedly show is empirical evidence that they can't reason, because they wouldn't make the mistakes they do if they could.
I'm so confused by how many people implicitly believe everything an LLM spits out. I don't like the term "AI psychosis" that much but what else do we call it? Why are people so easily glamoured by a machine? Yesterday I was working on a bike's brakes I wasn't familiar with and I asked how to solve an issue, the result was the exact opposite of what I needed.