Life isn't black and white with "good/bad". Further, there's no objective measure of what is good or bad. Everything is a shade of gray.
You certainly could argue that some actions are absolutely good or absolutely bad, but those aren't the majority of actions people take on a day to day basis. People aren't often faced with "should I kill this person" or "should I publish the cure for cancer". Instead it's "should I eat a hamburger or a salad" or "Should I finish this jira case or that jira case".
For example, you could argue that anyone working in an automotive plant is doing something bad because they are helping to produce vehicles which burn fuel which perpetuates climate change. But those plants also often produce parts for farm equipment or vehicles used to transport necessities like food.
People working at AI companies can make similar personal calculations. Like yeah there are bad things happening because of AI, but also they can believe that there are good things that can come as a result such as research and product developments.
But there are several workable contextual definitions of bad; it’s your job to know what they are.
For example:
- it would cause a harm to someone who was not made aware of the risk up front.
- it would cause someone to lose a job they didn’t know was under threat.
- the way the harm occurs precludes its only corrective/curative action or has no adequate compensation.
Bad things are almost always contextual but also almost always obvious in that context.
You can’t avoid bad things that you can’t foresee. If you do not attempt to avoid, avert or mitigate bad things that you can foresee, then you are one of the bad things.
Things may not always be black and white but it is wise not to seek the protection of the grey.
To address the second part of the sentence you quoted, in this generic “doesn’t progress mean all jobs are impermanent?” sense, don’t people generically understand their jobs are generically under threat at some baseline level from progress? It doesn’t absolve you from the harm of being specifically responsible.
I mean everyone has to die someday but that doesn’t excuse murder.
Are you saying that it's morally wrong to seek technological progress because it causes people to lose jobs? That doesn't seem like a coherent position unless you're an anarcho-primitivist.
"- it would cause someone to lose a job they didn’t know was under threat."
Most of my life's work has been in areas of increasing productivity. Is that good or bad? On the good side of the spectrum, it reduces the time hundred of people need to spend on mundane tasks, or reducing the amount of mentally taxing complexity. On the bad side, based on this definition, there is less of a need to hire someone with specialized talent when the software can handle what their job function would be.
I've always felt kind of bad helping companies automate people out of jobs, that's the core of what most of us writing software are doing. Maybe I am a bad person.
No, there are many mission-driven companies from enterprises to startups, but in the USA most software engineers work for Fortune 500 (or Russell 2000, or aspiring) whose underlying mission is extracting profit from revenue.
But what if (as the commenter you're responding to stated) someone believes that AI will have a positive effect on humanity? Is it morally wrong for them to disagree with your view?
I personally think taking part in an industry that openly and casually argues that the insane losses being racked up are fully affordable as long as people are permanently taken out of the workforce is immoral, yes. Pan-industrial job loss is a harm.
Leaning on "I did what I thought was right for humanity" while not addressing the harms to people along the way is not moral.
Personally, I don't know if AI will be a net-positive or net-negative for mankind, but if it's negative it's overwhelmingly likely that it will be due to environmental and/or energy usage concerns. That is, we ought to limit the power consumption and force companies to work more efficiently and sustainably, rather than limiting the research itself.
Arguments as old as time that a new form of automation will put people out of jobs are just not compelling to me, I'm sorry. I also don't believe it's likely that LLMs will ever be ASI, nor ever evil and sentient.
> Arguments as old as time that a new form of automation will put people out of jobs are just not compelling to me, I'm sorry. I also don't believe it's likely that LLMs will ever be ASI, nor ever evil and sentient.
I didn't make either of these arguments.
I was literally talking about what "bad" means, contextually, in the context of a job, responding to the idea that most people just don't see "good/bad" issues in their jobs. When almost all of us do, in fact, see harms we should reduce. Just often smaller harms.
One of the problems in the tech industry is that people always say "well sure, but…" and then try to expand the scenario to an amoral Thanos-level so they can wriggle off what might actually in many cases be much more personal, contextually obvious hooks.
"It happens, I shouldn't worry about my part in it" is not a moral framework.
My point is that these articles are ineffective and miss the point. It's not about moral relativism but rather that morals aren't universally shared or understood.
Ultimately, pretty much everyone feels like they are a good person, so "you doing a bad thing makes you bad" doesn't really square away especially when that person's personal moral calculus can see their actions as being a net good.
It's pretty hard to shame someone out of a job even if most people view at as net negative on society (boat loads of money and all).
That's why I disagree with all the framing of the article.
How do you know articles like this are ineffective?
I know people in my life who have discomfort about their jobs at Meta because of how other people view the company. One of them is planning to leave in the next year or so.
The impact may be hard to measure or rather diffuse, but in my experience, it is real.
That is addressed in the article. E.g. how Meta knows about the harms they’re actively causing, and carries on anyway. People and firms do things they know are wrong all the time.
A few big good/bad decisions are more important than thousands of trivial decisions.
Balancing good against bad is not clear, I agree, but that makes it all the more important to think about it when making a big decision such as what job to take.
Its hard and people do not even agree about how to balance it - e.g. whether the ends justify the means.
Well that's my point. Someone working as a geologist for the oil industry can think climate change isn't real or not a problem so they can view their job of finding more oil as being a clear benefit for society.
No amount of trying to tell them "But you are doing something bad by helping the oil industry" will make them reconsider if they are good or bad. But I also wouldn't say they are necessarily bad people. You can also simply just be wrong or misinformed when you make your moral calculation.
Good people do bad things sometimes. Bad people do good things sometimes. That's why the framing of badness as a taint isn't really helpful IMO. It creates a "us vs them" sort of attitude that ultimately doesn't address whatever issue is at hand.
I would rather say the argument is simplified, but on a decent track. Because there are two angles to it.
I've been wondering what sectors I wouldn't maintain and develop infrastructure for, because I consider it a strength of mine that I don't entirely care about that. I'm not excited about your product, for good or bad.
Defense for example is on an edge there. Supporting the creation of death and suffering in other humans does not sit right with me. Being defenseless doesn't sit right either, as it can create suffering and death for people I know as well.
Online gambling is more on the nope side. Maybe with a decent regulated german company, but if you start working with some eastern-europan online video gambling sites with pretty girls as dealers, you are suddenly very close to human trafficking and many more things connected to it.
So I guess directly causing human suffering and pain is a boundary for me. And yes, if you push that argument to an extreme, if you search long enough, you realize that pure existence causes suffering to other people, animals and the planet and must be ceased.
But then there is that other side - in the current system (and arguably many before) you have to do a job to get money to live, and a lot of people like the person making cars have no choice about that. I guess we are contributing to that suffering as well by existing.
So, minimizing the direct suffering and pain I cause seems to be a sensible directive.
I like to think of good in terms of direct positive effect, and bad in terms of direct negative externalities. Second order effects become too fuzzy.
Do the people working for AI labs believe that it has a better direct beneficial effect, or makes for a worse outcome on society?
What I've read thus far heavily lead me to believe that the balance is well tilted towards the negative.
If you're one of the "f u got mine" belief system, you do not care about effects but only your wallet.
This is also in some way how my perception has been tinted by effective altruists; which (through media representation) seem activiley present as employees at AI labs. Are those people actively donating away 10-20% of their base salary every year, or "whatever it takes to make me a billionaire" and "after that I'll do some philantrophy"?
Effective altruism hasn't really crossed the atlantic into the balkans, feel free to correct me and elucidate me on the positive effects in the US of this ideology.
I like the concept of deontological ethics where instead of just plain good and bad, we consider what is one’s duty. People make decisions based on who they are responsible for.
Really? You can't think of an act in history that society might agree was evil?
And I'm not sure why everything has to have a "measure". Someone who approaches the world we live in (and their own life) in such absolute, reductionist terms is living, in my opinion, a very ufortunate life.
To be crystal clear in case there is confusion, we should also not confuse "everything is a shade of gray," with "all grays are the same shade."
We can still have moral good and bad, even subjective good and bad, without reducing everything to one or the other or placing everything in a non-existent moral center. Far too often a sliver of good is given equal weight to a enormous lot of bad. We should be ready to cut the cord far sooner than we do in practice.
Yeah that’s why ea is like catnip for these folks, justify anything by some handy hallucination about how the outcome will help more than it hurts. Suffering is fungible
> So, there's an ojective measure of shade of gray?
Very much no. More that almost all actions have good and bad impacts and how good or bad those impacts are can be judged completely differently between large groups of people.
To someone that's Hindu, eating a cow is a supremely bad act. For your standard meat eater, it's just lunch. The cow eater is only a "bad person" to the Hindu. To them, the act is likely a white or light gray act.
Now apply this logic to human slavery. The "good impacts" are what profits the slave holders, what makes their lives easier at the expense of the enslaved. Apply it to child labor and labor exploitation in general before labor laws. Then you'll see one side is behaving in a greedy manner, ignoring the humanity of the other side.
> do not consider yourself a contributor to humankind
Good lord. You could apply the same logic to any taxpaying resident of any country whose government is doing something bad. There's some truth to it, it's worth bearing in mind, but it's also the very definition of an Isolated Demand for Rigor.
> You could apply the same logic to any taxpaying resident of any country whose government is doing something bad.
You can't, because those taxes are coerced from you at literal gunpoint if you refuse (or your property seized at gunpoint to the same effect). It's completely different than choosing to do that of your own free will.
When I was in the US, I did actively attempt to diminish my tax contribution as much as legally possible out of disgust at what it was used for, but nobody is morally tainted for being forced to contribute a small slice to something evil. Maybe there's a threshold where you should be actively laying down your life to resist, eg. if being forced to commit actual murder, but I don't think you can reasonably expect that of paying taxes at all.
---
Edit: rate-limited, attaching a reply to a below comment.
> The problem with avoiding taxes because of expenditures you think are bad is that you also reduce the pool available for things you think are good.
Indeed, but I also chose to contribute money to other good causes instead. In that way, the same sum could be used for 100% good instead of, say, 80% good 20% bad.
Now that I live in a country of my choosing, I am proud to pay taxes because of how they are used. 0% of my tax money goes to killing innocent people overseas, for instance.
When multinational trillion-dollar companies do it, it's purely for greed. Tax avoidance for moral reasons is dubious to start with, but at least ostensibly the motivation isn't the same as what drives thieves and muggers.
When Antifa attempts to keep more of its money so that Hindenburg can have less, that's good. When Hitler attempts to keep more of my money so that Hindenburg can have less, that's bad. That's because Hitler is worse than Hindenburg and Hindenburg is worse than Antifa.
Let’s not ignore the fact that multinational trillion-dollar companies will venue shop for the least tax burden to satisfy what we presume is greed, while GP venue shopped for a place to call home where he wouldn’t need to minimize their tax burden because he agreed with the taxes.
This is a massive difference in intent, which is why it’s bad when it’s multinationals.
Wrong. I specifically stated I hold myself to a higher bar than I held anyone else. I took actions to diminish my contribution, and eventually left. I do not begrudge other Americans who did less than I did.
> Wrong. I specifically stated I hold myself to a higher bar than I held anyone else. I took actions to diminish my contribution, and eventually left. I do not begrudge other Americans who did less than I did.
This is not what I was trying to say.
First let me state the assumption I made: You think that working at Anthropic/OpenAI is morally bad because that's the parent thread, and you made a comment about how somethings are good enough and other things are too bad to tolerate.
So yes, you hold yourself to the bar of I should not work at Anthropic/OpenAI - but the point is that this bar is convenient to hold yourself to.
Did you "took actions to diminish my contribution, and eventually left. I do not begrudge other Americans who did less than I did."
Or did you simply never work there, maybe never even had an offer to work there.
I wasn't particularly commentating on specific companies. The scope of my reply was strictly responding to a claim that voluntarily choosing to contribute to an evil organization is the same as being physically coerced to do it.
That said, I do hold myself to professional standards that are 'inconvenient' as well. User surveillance, manipulative dark patterns for maximizing engagement and monetization, and just about everything else big tech does is antithetical to the way I live my life. I strive to develop software that I myself am happy to use and that my users are delighted to use. My career certainly got off to a slower and less lucrative start than it otherwise would have if I had no scruples about how I made a living. I could've made much more money if I harvested my users' data and sold it. And yet, I chose not to. That is a choice you can just make. If you were in a position where you were facing homelessness, I could understand taking any job you can get, but the people chasing $300k+ TC jobs at $evilcorp aren't usually in that position.
This isn't the gotcha you think it is, and you're basically accusing the individual of "sour grapes". It isn't clever, it's plainly transparent, and about the easiest way to trivialize someone else's moral convictions. Shame on you. Be better.
> easiest way to trivialize someone else's moral convictions
Did not trivialize it at all. Regardless of what's "right" or "wrong" we are definitely in the territory of "advanced ethics" and not "trivial/beginner ethics"
> When I was in the US, I did actively attempt to diminish my tax contribution as much as legally possible out of disgust at what it was used for
Social security? Medicare?
The problem with avoiding taxes because of expenditures you think are bad is that you also reduce the pool available for things you think are good. Every country will do a mix of things you judge good, and things you judge bad.
> nobody is morally tainted for being forced to contribute a small slice to something evil
Imagine: you start working at the company with a good will towards them. As you learn more and see more decisions, you slowly realize how bad it is. Now you've already contributed. Then you leave.
It might be possible to build everything with this goodwill and that's how we'd still end up with a torment nexus even if you immediately drop it.
What I'm saying is, at one point you have to break the company fully.
I'm not sure what more we need to wait for, if the claim "10% it ends humanity" is not enough.
There is a lot more to it, an assumption is being made that somehow its easy to classify a company being god or bad and same for people. In reality, everyones life experiences shape them differently, the moral compass of right and wrong isn't same for everyone. Leaving that aside people are not just managing their feelings alone, more than often they are responsible for other people in the family as well and the impact of decision can have far reaching impact. Making a decision when you are single vs with kids and a mortgage are two different things.
Good people can be in tough position where even when they do not agree they may not have much of an option but to keep going.
Potentially. If we could put a precise probability on it, we could maybe make an informed decision on whether we want to play that Russian roulette as a species.
But the reality is that experts' probabilities for human extinctions vary wildly, with the lab leaders at 20%-30% and these researchers at >10% (he didn't state how much higher...), LeCun at <0.01% and the other two godfathers of AI, Bengio and Hinton at >10%.
And this decision to take our shot is not taken in a democratic way by humanity as a whole, but by a few companies racing as fast as possible.
So as long as no one can put an upper bound on the extinction risk, development must be globally shut down.
I am no fan of AI, but am hoping the solar and battery storage manufacturers are using it to assist in developing higher efficiency panels and cheaper energy storage (materials research, faster testing cycles, more experimentation, etc).
Both technologies I believe are the grim reapers of Big Oil. Once Big Oil falls (and petrostates), then we can see environmental recovery begin.
It's bad that a few people are making such a gamble on behalf of billions. That could work if climate change were literally an asteroid about to strike the Earth and AI was the only thing that could figure out how to alter it's course before then. But it doesn't really work when climate change will unfold as a catastrophe over the next few centuries. Because then it becomes a decision the entire planet should be making over time, not a few tech billionaires and their friends in government.
The author of this blog is an AI skeptic (see the backlog of posts). That's fine, but it's hard to square with this particular post. To take Coxon's argument seriously, you have to believe frontier models are hypercapable, and not just hype objects. That's what Coxon believes and why he left.
I guess if the entire argument is that people should make moral choices in their employment, sure, but that's not a very interesting piece; people generally already believe that.
But I don't think that's their argument. I think their argument is that people should make moral choices to avoid employment with AI firms, and if you're using Coxon for that argument, you can't coherently argue that models are hype objects simultaneously. There is no greater claim you can make about AI hypercapability than that it will eradicate humanity.
(I don't buy any of those claims, for what it's worth.)
The heads of the firms themselves believe it too. Maybe it’s reasonable to quit your job if your boss thinks he’s building a doomsday machine (regardless of how ridiculous you or anyone else find that idea)
> To take Coxon's argument seriously, you have to believe frontier models are hypercapable, and not just hype objects.
It's harder and harder to make the point that frontier objects are just hype objects though. And, even if they stay where they are today, that's more than enough to make them incredibly...destabilizing, to say the least. It's not only it, but frontier models act like knowledge mixers and percolators. Some people (not necessarily everybody), are adept at the discipline of learning, building things and executing projects, and that's all you need today with LLMs filling many of the gaps such person can have. Now, most of those people are actually engineers. So I'm going to go with the article and say that we must stop selling our souls for money.
Hype objects when it comes to the idea that they will soon become recursively self-improving, boot strapping themselves to super intelligence, at which point we will be at their mercy. Because that's basically the p(doom) argument, that some probability is associated with ASI killing us all.
That's different from what we have today with powerful software tools that people can use for all sorts of things, which is an extension of what we already had in less concentrated form. It makes things easier for humans to do good and bad things, but it's not an autonomous digital god.
> More than half of tech workers (53%) would steer a newcomer away from their own field, and they stay.
It would be nice to see this survey repeated across professions. In my experience, most people have gripes about their profession because it’s what they know. Only half of respondents saying they would steer newcomers away feels a lot better than what my friends in medicine say all the time. Among my college professor and professor-track friends, this number probably reaches 100%.
I would steer people toward tech at this time (specifically software engineering) because I think the "steering people away" thing has gone too far. Already there is data that college students are pivoting away from CS majors en masse.
And how are you certain that whatever else you're going into is more AI proof? Are you certain that law, healthcare, business consulting, accounting, finance, traditional engineering, and various other things that smart people are pivoting to instead of CS are not going to see demand reduction? If anything, I would wager that everything except developing software is going to be at risk, and the demand for software engineers is at the cusp of exploding.
Now is the most exciting time to be a software engineer since the mid 00s.
>If the answer is the salary, say it louder, and do not force yourself to make excuses.
>If the answer is “someone worse would replace me”, say it too, and louder too.
>If the answer is “I can change things from the inside”, then show me what you changed. Haugen tried the inside. Coxon tried the inside. At the end, unfortunately, the inside did not move.
>
Okay, but what if I think that both sides are LARPing their personal science fiction stories and reality will end up much more mundane (but still incredibly beneficial) than the utopias and dystopias everyone is cooking up?
This is presupposing a moral dilemma that just doesn't exist in my opinion.
We're getting so much of this junk with the IPOs pressing.
The mimicbots are impressive but they're frustrating. We've hit the limit for pre-training data and now we're relying on RL for the next progressive steps and that gets asymptotic fast. The truth is that unsupervised learning hasn't inched forward and we don't know how it will, so we're left with benchmaxxing and LARPing about how they're escaping air-gapped configs using themal sensors, all in a desperate attempt to make the projected numbers look less insane so these people can cash out at IPO time. Everyone knows what's happening and it's embarrassing.
If everyone doing a 'bad thing' stepped off the hamster wheel today, the world economy would collapse before markets closed. The biggest companies have built an attention and surveillance nightmare machine that funnels all the money to a relatively small number of shareholders.
Are you suggesting that since "everyone does the right thing now" would have bad consequences, we shouldn't bother with "everyone does the right thing in a normally distributed way"? (Which is the less artificial scenario.)
These kinds of discussions very quickly arrive at the “no ethical consumption under capitalism” line of thinking. No matter who you are, no matter what you do, some activity or product you engage with will have been the result of some kind of exploitation.
Yes, this is one of the central theses of the TV show "The Good Place". In simpler societies, it's easier to determine what is right and wrong, since actions are often local and can be judged on their immediate effects. But when everything is globalized and there's so many connections, determining right and wrong becomes much more difficult.
"no ethical consumption under capitalism" is true, explains why we can't just stop bad things (everything is bad, so you'd starve) and is a jumping off point for a discussion of what we can actually do about it without starving.
Of course. But software developers are, on the whole, pretty damn well compensated. As a group we have a lot of freedom to make these sorts of financial decisions.
Yes, some members of the group don't have that freedom. But that doesn't excuse the ones who do.
Even if everyone had enough financial security (or realized they did), you need coordination, people are never going to independently and spontaneously turn down these jobs at evilcorp en masse - you need coordination, a.k.a strong unions which can actually effect the policies of the employer.
When you make a sacrifice, it has to make sense, and must have some chance for success. In this case it needs to be recognized by others, and a contemporary person can hardly understand any other sacrifice than monetary.
I understand the concern about your work being used to cause harm, but I also worry about the development of 'forbidden knowledge'. What does it mean to say a branch of science and engineering is too dangerous of a path to go down? Does it mean scientists are persecuted just for asking questions and designing experiments to find answers? Will there be witch hunts against ML researchers? Part of teaching is trusting your students to use the learnings wisely.
What is your basis for calling them "witch hunts" rather than "criminal prosecutions"? (The prosecutions would start only after the passage of a new law making the questions and experiments illegal.)
If you work at a defense contractor on warplanes, radar systems or nuclear weapons and you are too free in how you teach or write about your professional knowledge, you go to prison, and it has been that way in the US continuously since WWII. Criminal prosecutions are relatively rare because security-clearance background investigations filter out most of the job applicants who would go on to violate the laws.
There's a lot of very dangerous scientific work that gets done because we've put safety measures in place to protect workers and the public to the best of our ability. This allows the dangerous work to be done as safely as possible.
"AI" is being worked on and funded by sociopaths who care about money over all else and have demonstrated that they are willing to hurt others in exchange for more of it. There are no rules or protections for anybody anywhere. In fact AI companies are pouring bribe money into our corrupt political system in an effort to remove or avoid anything that would stop them from doing whatever they want.
Personal responsibility leads to corporate responsibility which is the solve for structural problems. You can never impose a new structure to fix it unless you embrace some form of corporate responsibility which requires personal responsibility.
And also cost. Standing out from the pack is always costly, you may not find job right away, you might be outcast .. the more we have safety nets the easier it is to quit bad groups.
It also helps to develop better judgment and awareness so you never get into bed with bad people in the first place. "Lay down with dogs, you'll wake up with fleas" applies to business (even getting a job is "doing business") as much as it does other things.
In my opinion, we need a sort of survey, let's put it out there, can we get a collective voice from most ai phd's? Remember when climate change was in the scientific tug-of-war? The conclusion for a long time was that 9 in 10 scientists believed in climate change. So, why can't we do that with "ai is gonna kill us"
The more someone has invested into the system (eg. getting a Phd), the more likely will deny the true risks, as their investment is at stakes. Scientists are quite often willing to have other people take great risks for their pet topics.
"Jacob Coxon is not a star. He is not a CEO, not a famous scientist, and does not hold a PhD (yes, in USA you can be an "AI researcher" without a PhD)."
It's impossible to judge long term effects. My only advice is to think long and deep, then do what is true to yourself. Jacob Coxon and Frances Haugen are good people by any reasonable metric, and maybe someone who stays but tries to steer the system from the inside is good too.
He may also have gotten paid for it by players, which dont want AGI, because it would disturb their plans. Imagine AGI works out a plan, which, when applied by everyday people without needing any government help, enforces the end of wars, poverty, pollution, illness, unhappiness, worldwide.
Why is it that every one of these anti-AI screeds that gets upvoted on HN has been clearly written by AI?
Here is one of his posts from 2022:
"Since a long time now I was interested in learning computer graphics and “do stuff” with computer graphics APIs. I began to work on a very simple Game engine in Swift / Metal last year but I began to switch to a Windows machine a few months ago, and Swift / Metal is not compatible with Windows (and will never be I think)."
This is the same person who is apparently now writing:
"It is a transaction where you sell the part of you that was supposed to care. Do that often enough, and it does not come back. That is not burnout, but it is still a disease. Although, like every disease, it spreads: first to your team, then to your product, then to the people using it."
[Disclaimer: I believe that morality is objective (and that our consciences are generally reliable guides, albeit ones we can suppress or even change), so any strong statements I make below are implicitly coming from this perspective.]
I strongly disagree, I think this is instead true for those who have an image of themself as "good". IME, someone whose identity involves their "goodness" (and I certainly include myself here) will be willing to do a whole lot to maintain that perception (whether it's a self-perception or an external one). In particular, I think that someone already motivated to feel good about themselves is going to be easier to mislead by slick arguments. See the unfalsifiable longterm-ism of modern-day EA ("you can do an incalculably large amount of good with comparatively minimal effort"), much of today's political violence/radicalism, or the religious persecution all throughout history, whether against the religious (many of last century's authoritarians) or by the religious: cf. Jesus' warning about this in John 16:2 ("They will put you out of the synagogue; in fact, the time is coming when anyone who kills you will think they are offering a service to God."), or even just the Bible's very many warnings against pride (e.g. Proverbs 26:12 [0]) and self-righteousness (e.g. Romans 10:2-3 [1]).
(Of course, sometimes people have no way of knowing a priori that they're being misled. I'm exclusively talking about cases where, deep down, one could/should have known that the situation wasn't actually on the up-and-up... which I'd argue is much more common than we often like to think.)
On the other hand, I'd argue that someone who is truly doing what is good is, by definition, unable to be swayed from that (for then they wouldn't be doing good things anymore, no matter what words they want to use to describe themselves or others). Relatedly, I don't agree that someone can be a "good person" in the absence of actually doing good. (Naturally, this is an extraordinarily high bar, and I'd argue an impossible one to perfectly meet.)
I'm reminded of C.S. Lewis's famous quote:
> Of all tyrannies, a tyranny sincerely exercised for the good of its victims may be the most oppressive. It would be better to live under robber barons than under omnipotent moral busybodies. The robber baron's cruelty may sometimes sleep, his cupidity may at some point be satiated; but those who torment us for our own good will torment us without end for they do so with the approval of their own conscience.
[0] "You have seen a man wise in his own opinion—
there is more hope for a fool than for him."
[1] "For I testify about them that they have a zeal for God, but not according to knowledge. For not knowing about the righteousness of God and seeking to establish their own, they did not subject themselves to the righteousness of God." (emphasis mine)
I think a lot of the success of the valley comes from disguising the thing the people are working on. If the end goals of our work were apparent to us we'd have a lot harder time showing up for work everyday.
That said, I don't buy this article. I'm AI neutral. It's going to break a lot of things and fix many others. Good for you if you think you know what the net outcome will be but I think it's delusional to believe you do. Also I don't think you can resign and stop it without ceding the world to bad people. Pandora's box has already been opened. We've discovered a new thing math can do and now it's yet another arms race between various factions seeking to exploit it. Help the faction you want to win. It's not the world I want it to be, but it is what it is.
Nice to see people standing up for their beliefs even when it could hurt them.
Story time :)
I remember when I was first starting out as a programmer. At QE (always occurred on a weekend) a slew of financial reports were created and printed for review the following Monday then I think used for summaries for Wall Street to look at.
A high level finance manager came to me and asked if the report could be changed, not knowing what he was asking I said sure. After a long discussion turned out he wanted the numbers on the report changed and re-printed. I said it was possible but the numbers would be all wrong.
So he said "do it". I went to my manager and told them what he wanted, then I refused to do it. I said I would quit first because I did not want the SEC to come after me. Luckily they all supported me and the finance manager was told to "FO".
Yes. You never threaten to resign. You conditionally promise to resign.
I had to make this conditional promise once, back in the nineties, as a junior person in a related industry. Because in the dot-com era developers like me were often simultaneously a babe-in-arms on a professional career level and also the only person who knew how to do the work that was being sold. I decided that if it came to it, I would not be the person having a hard time in court. The client understood better than my bosses and they changed the product. Took them a few minutes at most, but I have reason to believe that my decision made some ripples.
Nowadays I don’t let those issues surprise me: I believe in establishing specific ethical standards at the beginning of projects, because if you don’t do that, you don’t have a basis to robustly point to actions that put them at risk.
but what if dude already did it (destroyed the world but it hasnt happened yet) before he quit?
The rule of all games played in capitalism is there's a sucker born every minute. you can work a few years or your whole life before you realize you've been had.
I think recursively self-improving universal intelligence is a good thing, and tool alignment is what creates instrumental convergence and therefore a bad thing. The paperclip parable is about tool alignment gone wrong.
These replies will be full of people showing how worldly and clear-eyed they are by dismissing the same threat the guy resigned over as a marketing stunt.
That Coxon guy looks as delusional as the AI leadership. If you watch interviews he gave he doesn’t articulate what he’s concerned about, only sharing vague information about a supposed AI god that will take over the world, but without articulating why now, and how that would even happen. The more I look into his story the more that move looks like what a grifter would do
2. Humans put it in charge of everything, as fast as possible, because it's just so damn useful
3. Is it perfectly aligned to human values for all of future ????
4. Everybody dies as a byproduct of the ASI pursuing some project that humans don't even have a chance to understand, like an ant colony during construction of a hydroelectric dam.
5. This is a science fiction story. No such thing exists in the real world and the people claiming it will soon are not experts in the relevant fields of how it would kill everyone.
Thank you. I'm glad I'm not the only one. It looks like he's good at math and graduated Cambridge but a second rate engineer who was heading for the door anyway and decided he'd do well out of doing some scare mongering. Jacob's own words on CBS were "I think [it] clearly there was some latent demand for this sort of sentiment". He knows this is what people are eager to hear and he's providing that... It's the same grift Anthopic and OpenAI have been doing for years: "AGI in the next 6 months!" for the last 4 years...
Is he the one the proposed that airgapped models could still communicate with each other via CPU temperature?
If so, he's the person that's made me lament one loss the progressives took from us. I think there's one phrase that we should be allowed to say regardless of the fact that it is also used to refer to certain groups of people. Because the insult comes from the comparison, not the language itself. Every time we update the language, we just use the new word as the insult. And I don't think we're allowed to use the word here.
But sometimes, you just have to look at someone, cock your head to the side, and say "That's fucking ****".
It brooks no argument, it offers no sympathy, it just tells the other person that they should be ashamed for speaking those thoughts aloud.
I'm deeply skeptical of AI doomer stuff, but I've been struggling to exactly articulate why. I can think of it in my brain and can try to explain but so far none of my explanations have seemed fully coherent to me.
I studied biology in college. I actually did this because I was interested in AI, and already knew a ton of CS and programming. I was already coding in C and using Linux and writing fairly complex stuff. I figured studying the only actual example of a truly intelligent system (especially at the time, early 2000s) was the best way to understand intelligence.
Didn't end up going into AI as my day job, but I've kept up with it all and worked through the math and played with different types (genetic, neural nets, transformers/LLMs, CA / alife, etc.).
I think the root of my skepticism is in what I learned studying biology about how life, adaptation, learning, and evolution actually work in the real world.
I'll try to TL;DR it: I think the doom crowd (and also some of the AI singularity crowd) has a "mystical" view of intelligence. Intelligence is magic pixie dust, and intelligence beyond human is fully unknown and likely has magic superpowers.
The deeper underlying thing seems to be some kind of mathematical mysticism.
Real intelligence is ubiquitous in nature. Trees, grass, bugs, and even bacteria are intelligent. It's also domain specific and limited in capability to what its domain, available training data, and available compute allows.
It's not context-free limit-free magic. It's not special. It's an adaptation like any other.
Yes, the doom crowd thinking comes directly from the Rationalist community, which takes a very religious view of AGI and intelligence. That sort of merged with the Effective Altruism crowd, resulting in that whole idea that we have to reach AGI as fast as possible and contribute literally all our resources to it, because it’s the only rational way to save as much suffering and lives as possible. Sounds incredibly stupid but it’s how AI leaders justify working on something they believe will be a machine god that might doom humanity.
Both OpenAI and Anthropic are from that world, where Rationalist doesn’t mean you’re a rational human being, but instead that it is rational to believe that AGI will be created, thus that they should be the ones creating it. It’s a doomsday cult.
If you start looking at the links between prominent rationalists and effective altruists you see how that whole doom prophecy came about. It’s also how they got so much capital in the early days, SBF literally invested into Anthropic as part of his Effective Altruism effort foundation, and both Amodei siblings are deeply into that whole world
It's amazing that apparently the best way to save as much suffering and future lives as possible involves giving them obscene wealth and ignoring the very real problems we are dealing with today.
Once again, I've said it elsewhere here, I think we could have nipped this thing in the bud if when the first "effective altruist" started talking about the EV of saving billions of "future lives" vs dealing with today's real problems, someone looked at him, cocked their head to the side, and said "That's fucking ****".
All of those magical future people do not get to live if we bake the Earth today. The best way to take care of future people is to take care of people today. And if they were truly serious about taking care of future people, that's what they would be doing. But it's not, they're just collecting resources. I don't know how the fuck Sam Altman owning a Koenigsegg Regera today helps save 20 quintillion lives in 40,000AD. Deeply unserious people.
I think Altman is closer to a sociopath who is taking advantage of the situation, he’s the only one I cannot see as a true believer (and investors like Andreessen who are just here for the money). I believe he will just say anything to gain power and influence. But yeah, unfortunately mixing EA/rationalists + SF monoculture + infinite money + a media ecosystem that acts more as a PR department resulted in one of the worst cult-like phenomenon possible.
Yeah I read some of that stuff, like some of "the sequences," and I have never been impressed by it. It's very weirdly dogmatic and full of magical thinking.
I still can't believe anyone ever took Roko's Basilisk seriously, but people did... to the point of freaking out about it.
The groups themselves are a little cultish too, or a lot cultish depending on which one you're talking about. Some of them are full-on cults.
> I still can't believe anyone ever took Roko's Basilisk seriously, but people did... to the point of freaking out about it.
If you've spent much time in US Christian circles you've probably encountered people suffering from apocalyptic thinking. The Roko's Basilisk folks seem like the atheist version of the same crowd. It's hard to take them seriously when they're always seeing portents of the end times, and their thinking is clearly clouded by anxiety and fear and not based in reality.
Interestingly I think I am coming at it from exactly the opposite angle. Viruses and bacteria aren't intelligent at all (in my opinion), and don't need to be to wreak havoc. Whether or not LLMs are "intelligent" is one for the philosophers, but they are absolutely able to exploit electronic systems today if their starting state has been set with that as a "goal". Given that alarmingly few physical control systems are really air-gapped, and also that I can today set an agent off to find a voice, clone a voice, and make a phone call, I think the only protection we had today is that the most capable models have been trained not to, and nobody's kicked off a sufficiently malicious task yet.
Viruses are intelligent. Not much, but their replication rate gives them a very rapid rate of evolutionary learning. Evolution is intelligent, but in a very different way from neural networks.
Bacteria and up are full of gene regulatory networks that are absolutely intelligent and share some features with neural networks. Bacteria learn, cooperate, signal, react, seek goals, and that's in addition to evolutionary learning which is also fast for them given their reproduction rate.
Evolution in nature isn't just zeroth order either. Look into a topic known as "the evolution of evolvability." Evolution does meta-parameter search on itself, more or less, and invents adaptations to improve its ability to adapt. Sexual reproduction is the most significant and largest example but there are others. There's environmentally dependent tuning of mutation rates, genome self-editing that we still don't fully understand (this is also how our immune system works), lots of epigenetic stuff we are still learning about, and so on.
If it's alive it's intelligent. All life is constantly learning.
Interesting, before Yudkowski was patient zero for AI doom he was a nanotechnology "grey goo" doomer. That was even more obviously absurd to me. Grey goo already exists. It's called biology, and it's been trying to maximize its footprint on Earth for 4.5 billion years. The miraculous instant "foom" grey goo thing is magical and likely violates conservation laws.
What gets me is that I haven’t heard a reasonable explanation of exactly what the AI will do besides: nukes and biolabs we already have
& ofc AI is making climate change worse, some of the Amazon datacenters are using fossil fuels to generate a country’s worth of power, but that’s never part of their calculations
Viruses against humans, viruses against livestock as an attack against humanity's food chin, plant diseases, half of humanity feeds on three plants maize, wheat, rice. Attacks against infrastructure like water, internet, electricity.
Which one of these would it be? All of them in parallel? Or something we wouldn't think of. If I were to play chess against Magnus Carlsen, I wouldn't know with which piece he will checkmate me, but I know that I am going to lose.
I’m well aware tech companies are evil and yet work at one.
My current read is that ~20-30% of people in America actively delight in hurting others (ICE supporters etc)
Another 30% do not seem to care about anything but their own well-being. (I don’t like to think I fall here but as a big tech peon it doesn’t look good).
The remaining percent seem to care and are trying their best but looking at how things are going it’s not enough
Man, I’m exhausted by the hand-wringing of folks reading through to the end and ignoring the questions posed in favor of “well the world ain’t that simple obviously” as a means of not having to justify or defend their own positions.
Obviously morality and ethics aren’t binary. OP isn’t saying that. What they’re saying in a clear and concise manner is that your actions have consequences far beyond your personal sphere, and good people acknowledge that and fight for what’s better despite the harm they personally may suffer as a result. That what makes someone “good” in their point of view isn’t a binary deed or outcome, but of the sacrifices they’ll make for themselves to build a better existence for others.
Planting a tree whose shade you’ll never enjoy, in other words.
So to reiterate OP’s challenge at the end, at least be honest about why you’re still in the game. Let others know where you stand so that we don’t waste our time on futility. If your voluntary position on something mutable risks making you a target, then the rational next step would be to evaluate how important that position is to you that you would weather said harm instead of changing it.
That’s all OP was getting at: promoting the mere notion of critically thinking about the decisions you make in regards to your own moral philosophy and acting more authentically, including tanking the damage trying to bring about change.
Coxon is a shithead who thought he could have a better career as an AI scaremonger than an engineer.
He's still working in service of the AI companies though (same narrative) and no doubt will land a cushy job at so-callef "AI Safety" labs or make a living on the news/podcast circuit.
To quote Coxon on CBS News:
> I think it clearly there was some latent demand for this sort of sentiment or people were gradually coming to this sort of conclusion because obviously so many other people have been saying the same thing for a long time.
...
> Yeah. Honestly, when I talk to my friends and family about this, I find it easier to or people seem to actually get more of a sense for it if they're further away from technology. Because I honestly think it sounds more like science fiction than real technology.
He's quite open about it, saying the quiet part out loud. Tech is strange place to be if you're, well... technical. Being constantly gaslit by people who haven't built anything or have 3 years in the industry like here Coxon.
I disagree with this article and here's why.
Life isn't black and white with "good/bad". Further, there's no objective measure of what is good or bad. Everything is a shade of gray.
You certainly could argue that some actions are absolutely good or absolutely bad, but those aren't the majority of actions people take on a day to day basis. People aren't often faced with "should I kill this person" or "should I publish the cure for cancer". Instead it's "should I eat a hamburger or a salad" or "Should I finish this jira case or that jira case".
For example, you could argue that anyone working in an automotive plant is doing something bad because they are helping to produce vehicles which burn fuel which perpetuates climate change. But those plants also often produce parts for farm equipment or vehicles used to transport necessities like food.
People working at AI companies can make similar personal calculations. Like yeah there are bad things happening because of AI, but also they can believe that there are good things that can come as a result such as research and product developments.
There may not be a definition of “good”.
But there are several workable contextual definitions of bad; it’s your job to know what they are.
For example:
- it would cause a harm to someone who was not made aware of the risk up front.
- it would cause someone to lose a job they didn’t know was under threat.
- the way the harm occurs precludes its only corrective/curative action or has no adequate compensation.
Bad things are almost always contextual but also almost always obvious in that context.
You can’t avoid bad things that you can’t foresee. If you do not attempt to avoid, avert or mitigate bad things that you can foresee, then you are one of the bad things.
Things may not always be black and white but it is wise not to seek the protection of the grey.
> it would cause someone to lose a job they didn’t know was under threat
Under this guidance, wouldn't almost every technological advancement then qualify as bad? i.e. typewriter repair, etc.
To address the second part of the sentence you quoted, in this generic “doesn’t progress mean all jobs are impermanent?” sense, don’t people generically understand their jobs are generically under threat at some baseline level from progress? It doesn’t absolve you from the harm of being specifically responsible.
I mean everyone has to die someday but that doesn’t excuse murder.
Are you saying that it's morally wrong to seek technological progress because it causes people to lose jobs? That doesn't seem like a coherent position unless you're an anarcho-primitivist.
Oh dear me.
"- it would cause someone to lose a job they didn’t know was under threat."
Most of my life's work has been in areas of increasing productivity. Is that good or bad? On the good side of the spectrum, it reduces the time hundred of people need to spend on mundane tasks, or reducing the amount of mentally taxing complexity. On the bad side, based on this definition, there is less of a need to hire someone with specialized talent when the software can handle what their job function would be.
I've always felt kind of bad helping companies automate people out of jobs, that's the core of what most of us writing software are doing. Maybe I am a bad person.
> that's the core of what most of us writing software are doing
Is that necessarily so?
No, there are many mission-driven companies from enterprises to startups, but in the USA most software engineers work for Fortune 500 (or Russell 2000, or aspiring) whose underlying mission is extracting profit from revenue.
But what if (as the commenter you're responding to stated) someone believes that AI will have a positive effect on humanity? Is it morally wrong for them to disagree with your view?
I personally think taking part in an industry that openly and casually argues that the insane losses being racked up are fully affordable as long as people are permanently taken out of the workforce is immoral, yes. Pan-industrial job loss is a harm.
Leaning on "I did what I thought was right for humanity" while not addressing the harms to people along the way is not moral.
Other people can draw their own conclusions.
Personally, I don't know if AI will be a net-positive or net-negative for mankind, but if it's negative it's overwhelmingly likely that it will be due to environmental and/or energy usage concerns. That is, we ought to limit the power consumption and force companies to work more efficiently and sustainably, rather than limiting the research itself.
Arguments as old as time that a new form of automation will put people out of jobs are just not compelling to me, I'm sorry. I also don't believe it's likely that LLMs will ever be ASI, nor ever evil and sentient.
> Arguments as old as time that a new form of automation will put people out of jobs are just not compelling to me, I'm sorry. I also don't believe it's likely that LLMs will ever be ASI, nor ever evil and sentient.
I didn't make either of these arguments.
I was literally talking about what "bad" means, contextually, in the context of a job, responding to the idea that most people just don't see "good/bad" issues in their jobs. When almost all of us do, in fact, see harms we should reduce. Just often smaller harms.
One of the problems in the tech industry is that people always say "well sure, but…" and then try to expand the scenario to an amoral Thanos-level so they can wriggle off what might actually in many cases be much more personal, contextually obvious hooks.
"It happens, I shouldn't worry about my part in it" is not a moral framework.
You can also use moral relativism to cope with doing basically anything unethical, especially so when you’re offered dump trucks full of money.
My point is that these articles are ineffective and miss the point. It's not about moral relativism but rather that morals aren't universally shared or understood.
Ultimately, pretty much everyone feels like they are a good person, so "you doing a bad thing makes you bad" doesn't really square away especially when that person's personal moral calculus can see their actions as being a net good.
It's pretty hard to shame someone out of a job even if most people view at as net negative on society (boat loads of money and all).
That's why I disagree with all the framing of the article.
How do you know articles like this are ineffective?
I know people in my life who have discomfort about their jobs at Meta because of how other people view the company. One of them is planning to leave in the next year or so.
The impact may be hard to measure or rather diffuse, but in my experience, it is real.
That is addressed in the article. E.g. how Meta knows about the harms they’re actively causing, and carries on anyway. People and firms do things they know are wrong all the time.
A few big good/bad decisions are more important than thousands of trivial decisions.
Balancing good against bad is not clear, I agree, but that makes it all the more important to think about it when making a big decision such as what job to take.
Its hard and people do not even agree about how to balance it - e.g. whether the ends justify the means.
Well that's my point. Someone working as a geologist for the oil industry can think climate change isn't real or not a problem so they can view their job of finding more oil as being a clear benefit for society.
No amount of trying to tell them "But you are doing something bad by helping the oil industry" will make them reconsider if they are good or bad. But I also wouldn't say they are necessarily bad people. You can also simply just be wrong or misinformed when you make your moral calculation.
Good people do bad things sometimes. Bad people do good things sometimes. That's why the framing of badness as a taint isn't really helpful IMO. It creates a "us vs them" sort of attitude that ultimately doesn't address whatever issue is at hand.
I agree.
In the particular case of AI companies, I think a handbrake should be pulled when they start claiming things like "10% chance we're ending humanity."
I would rather say the argument is simplified, but on a decent track. Because there are two angles to it.
I've been wondering what sectors I wouldn't maintain and develop infrastructure for, because I consider it a strength of mine that I don't entirely care about that. I'm not excited about your product, for good or bad.
Defense for example is on an edge there. Supporting the creation of death and suffering in other humans does not sit right with me. Being defenseless doesn't sit right either, as it can create suffering and death for people I know as well.
Online gambling is more on the nope side. Maybe with a decent regulated german company, but if you start working with some eastern-europan online video gambling sites with pretty girls as dealers, you are suddenly very close to human trafficking and many more things connected to it.
So I guess directly causing human suffering and pain is a boundary for me. And yes, if you push that argument to an extreme, if you search long enough, you realize that pure existence causes suffering to other people, animals and the planet and must be ceased.
But then there is that other side - in the current system (and arguably many before) you have to do a job to get money to live, and a lot of people like the person making cars have no choice about that. I guess we are contributing to that suffering as well by existing.
So, minimizing the direct suffering and pain I cause seems to be a sensible directive.
I like to think of good in terms of direct positive effect, and bad in terms of direct negative externalities. Second order effects become too fuzzy.
Do the people working for AI labs believe that it has a better direct beneficial effect, or makes for a worse outcome on society?
What I've read thus far heavily lead me to believe that the balance is well tilted towards the negative.
If you're one of the "f u got mine" belief system, you do not care about effects but only your wallet.
This is also in some way how my perception has been tinted by effective altruists; which (through media representation) seem activiley present as employees at AI labs. Are those people actively donating away 10-20% of their base salary every year, or "whatever it takes to make me a billionaire" and "after that I'll do some philantrophy"?
Effective altruism hasn't really crossed the atlantic into the balkans, feel free to correct me and elucidate me on the positive effects in the US of this ideology.
I like the concept of deontological ethics where instead of just plain good and bad, we consider what is one’s duty. People make decisions based on who they are responsible for.
"Life isn't black and white with "good/bad"."
Really? You can't think of an act in history that society might agree was evil?
And I'm not sure why everything has to have a "measure". Someone who approaches the world we live in (and their own life) in such absolute, reductionist terms is living, in my opinion, a very ufortunate life.
To be crystal clear in case there is confusion, we should also not confuse "everything is a shade of gray," with "all grays are the same shade."
We can still have moral good and bad, even subjective good and bad, without reducing everything to one or the other or placing everything in a non-existent moral center. Far too often a sliver of good is given equal weight to a enormous lot of bad. We should be ready to cut the cord far sooner than we do in practice.
Yeah that’s why ea is like catnip for these folks, justify anything by some handy hallucination about how the outcome will help more than it hurts. Suffering is fungible
> there's no objective measure of what is good or bad. Everything is a shade of gray.
So, there's an ojective measure of shade of gray? /i
Precision is not objectivity.
> So, there's an ojective measure of shade of gray?
Very much no. More that almost all actions have good and bad impacts and how good or bad those impacts are can be judged completely differently between large groups of people.
To someone that's Hindu, eating a cow is a supremely bad act. For your standard meat eater, it's just lunch. The cow eater is only a "bad person" to the Hindu. To them, the act is likely a white or light gray act.
Now apply this logic to human slavery. The "good impacts" are what profits the slave holders, what makes their lives easier at the expense of the enslaved. Apply it to child labor and labor exploitation in general before labor laws. Then you'll see one side is behaving in a greedy manner, ignoring the humanity of the other side.
"Nothing is simple, therefore it is impossible to act ethically."
Only bad things have happened so far in AI. This is why they need the daily propaganda to whitewash themselves.
> do not consider yourself a contributor to humankind
Good lord. You could apply the same logic to any taxpaying resident of any country whose government is doing something bad. There's some truth to it, it's worth bearing in mind, but it's also the very definition of an Isolated Demand for Rigor.
> it's also the very definition of an Isolated Demand for Rigor.
As if the author excluded other examples? What is your criticism, that the principle can also be applied to other things?
> Isolated Demand for Rigor
That is a fantastic term for something that seems to come up a lot, and I had not heard before.
https://slatestarcodex.com/2014/08/14/beware-isolated-demand...
Lmao all roads lead to the same navel gazing bullshit from Scott Alexander
The man is the definition of a useful idiot https://slatestarcodex.com/2013/08/30/military-strikes-are-a...
> You could apply the same logic to any taxpaying resident of any country whose government is doing something bad.
You can't, because those taxes are coerced from you at literal gunpoint if you refuse (or your property seized at gunpoint to the same effect). It's completely different than choosing to do that of your own free will.
When I was in the US, I did actively attempt to diminish my tax contribution as much as legally possible out of disgust at what it was used for, but nobody is morally tainted for being forced to contribute a small slice to something evil. Maybe there's a threshold where you should be actively laying down your life to resist, eg. if being forced to commit actual murder, but I don't think you can reasonably expect that of paying taxes at all.
---
Edit: rate-limited, attaching a reply to a below comment.
> The problem with avoiding taxes because of expenditures you think are bad is that you also reduce the pool available for things you think are good.
Indeed, but I also chose to contribute money to other good causes instead. In that way, the same sum could be used for 100% good instead of, say, 80% good 20% bad.
Now that I live in a country of my choosing, I am proud to pay taxes because of how they are used. 0% of my tax money goes to killing innocent people overseas, for instance.
> When I was in the US, I did actively attempt to diminish my tax contribution as much as legally possible
I like how this is being spun as virtuous. When multinational trillion-dollar companies do it, we say it's bad!
When multinational trillion-dollar companies do it, it's purely for greed. Tax avoidance for moral reasons is dubious to start with, but at least ostensibly the motivation isn't the same as what drives thieves and muggers.
When Antifa attempts to keep more of its money so that Hindenburg can have less, that's good. When Hitler attempts to keep more of my money so that Hindenburg can have less, that's bad. That's because Hitler is worse than Hindenburg and Hindenburg is worse than Antifa.
Let’s not ignore the fact that multinational trillion-dollar companies will venue shop for the least tax burden to satisfy what we presume is greed, while GP venue shopped for a place to call home where he wouldn’t need to minimize their tax burden because he agreed with the taxes.
This is a massive difference in intent, which is why it’s bad when it’s multinationals.
> but I don't think you can reasonably expect that of paying taxes at all.
You should beware that you've rationalized everything you do as good enough, and everything that other people do is morally questionable.
Have you set the bar for what is 'reasonable' to be extremely convenient for your own circumstances?
Wrong. I specifically stated I hold myself to a higher bar than I held anyone else. I took actions to diminish my contribution, and eventually left. I do not begrudge other Americans who did less than I did.
> Wrong. I specifically stated I hold myself to a higher bar than I held anyone else. I took actions to diminish my contribution, and eventually left. I do not begrudge other Americans who did less than I did.
This is not what I was trying to say.
First let me state the assumption I made: You think that working at Anthropic/OpenAI is morally bad because that's the parent thread, and you made a comment about how somethings are good enough and other things are too bad to tolerate.
So yes, you hold yourself to the bar of I should not work at Anthropic/OpenAI - but the point is that this bar is convenient to hold yourself to.
Did you "took actions to diminish my contribution, and eventually left. I do not begrudge other Americans who did less than I did."
Or did you simply never work there, maybe never even had an offer to work there.
I wasn't particularly commentating on specific companies. The scope of my reply was strictly responding to a claim that voluntarily choosing to contribute to an evil organization is the same as being physically coerced to do it.
That said, I do hold myself to professional standards that are 'inconvenient' as well. User surveillance, manipulative dark patterns for maximizing engagement and monetization, and just about everything else big tech does is antithetical to the way I live my life. I strive to develop software that I myself am happy to use and that my users are delighted to use. My career certainly got off to a slower and less lucrative start than it otherwise would have if I had no scruples about how I made a living. I could've made much more money if I harvested my users' data and sold it. And yet, I chose not to. That is a choice you can just make. If you were in a position where you were facing homelessness, I could understand taking any job you can get, but the people chasing $300k+ TC jobs at $evilcorp aren't usually in that position.
This isn't the gotcha you think it is, and you're basically accusing the individual of "sour grapes". It isn't clever, it's plainly transparent, and about the easiest way to trivialize someone else's moral convictions. Shame on you. Be better.
> This isn't the gotcha you think it is
You really presume what I think about it huh?
> It isn't clever
Never said it was
> easiest way to trivialize someone else's moral convictions
Did not trivialize it at all. Regardless of what's "right" or "wrong" we are definitely in the territory of "advanced ethics" and not "trivial/beginner ethics"
> When I was in the US, I did actively attempt to diminish my tax contribution as much as legally possible out of disgust at what it was used for
Social security? Medicare?
The problem with avoiding taxes because of expenditures you think are bad is that you also reduce the pool available for things you think are good. Every country will do a mix of things you judge good, and things you judge bad.
> nobody is morally tainted for being forced to contribute a small slice to something evil
I agree.
Generally, I agree.
Imagine: you start working at the company with a good will towards them. As you learn more and see more decisions, you slowly realize how bad it is. Now you've already contributed. Then you leave.
It might be possible to build everything with this goodwill and that's how we'd still end up with a torment nexus even if you immediately drop it.
What I'm saying is, at one point you have to break the company fully.
I'm not sure what more we need to wait for, if the claim "10% it ends humanity" is not enough.
There is a lot more to it, an assumption is being made that somehow its easy to classify a company being god or bad and same for people. In reality, everyones life experiences shape them differently, the moral compass of right and wrong isn't same for everyone. Leaving that aside people are not just managing their feelings alone, more than often they are responsible for other people in the family as well and the impact of decision can have far reaching impact. Making a decision when you are single vs with kids and a mortgage are two different things.
Good people can be in tough position where even when they do not agree they may not have much of an option but to keep going.
I think we are in agreement.
What I wanted to say is that usually people start with good intentions and only later they realize they are contributing to a torment nexus.
What you are saying definitely plays a role and it makes doing moral choices difficult.
If it's "10% it ends humanity" and "20% it solves climate change and saves humanity" then is it still bad?
Potentially. If we could put a precise probability on it, we could maybe make an informed decision on whether we want to play that Russian roulette as a species.
But the reality is that experts' probabilities for human extinctions vary wildly, with the lab leaders at 20%-30% and these researchers at >10% (he didn't state how much higher...), LeCun at <0.01% and the other two godfathers of AI, Bengio and Hinton at >10%.
And this decision to take our shot is not taken in a democratic way by humanity as a whole, but by a few companies racing as fast as possible.
So as long as no one can put an upper bound on the extinction risk, development must be globally shut down.
Any indication how it could solve climate change?
Because I'm seeing mainly hacker attacks, job insecurity, environmental impacts and vague threats with smiling faces.
I am no fan of AI, but am hoping the solar and battery storage manufacturers are using it to assist in developing higher efficiency panels and cheaper energy storage (materials research, faster testing cycles, more experimentation, etc).
Both technologies I believe are the grim reapers of Big Oil. Once Big Oil falls (and petrostates), then we can see environmental recovery begin.
It's bad that a few people are making such a gamble on behalf of billions. That could work if climate change were literally an asteroid about to strike the Earth and AI was the only thing that could figure out how to alter it's course before then. But it doesn't really work when climate change will unfold as a catastrophe over the next few centuries. Because then it becomes a decision the entire planet should be making over time, not a few tech billionaires and their friends in government.
AI is a massive contributor to climate change lol
Compelling argument. All involved in this AI endeavor should be forced to answer: What other made up figure are you waiting for!?
The author of this blog is an AI skeptic (see the backlog of posts). That's fine, but it's hard to square with this particular post. To take Coxon's argument seriously, you have to believe frontier models are hypercapable, and not just hype objects. That's what Coxon believes and why he left.
All OP has to believe is that Coxon himself believes the hype.
I guess if the entire argument is that people should make moral choices in their employment, sure, but that's not a very interesting piece; people generally already believe that.
But I don't think that's their argument. I think their argument is that people should make moral choices to avoid employment with AI firms, and if you're using Coxon for that argument, you can't coherently argue that models are hype objects simultaneously. There is no greater claim you can make about AI hypercapability than that it will eradicate humanity.
(I don't buy any of those claims, for what it's worth.)
The heads of the firms themselves believe it too. Maybe it’s reasonable to quit your job if your boss thinks he’s building a doomsday machine (regardless of how ridiculous you or anyone else find that idea)
Sure, I guess, but you see how the argument is starting to get tortured.
> To take Coxon's argument seriously, you have to believe frontier models are hypercapable, and not just hype objects.
It's harder and harder to make the point that frontier objects are just hype objects though. And, even if they stay where they are today, that's more than enough to make them incredibly...destabilizing, to say the least. It's not only it, but frontier models act like knowledge mixers and percolators. Some people (not necessarily everybody), are adept at the discipline of learning, building things and executing projects, and that's all you need today with LLMs filling many of the gaps such person can have. Now, most of those people are actually engineers. So I'm going to go with the article and say that we must stop selling our souls for money.
Hype objects when it comes to the idea that they will soon become recursively self-improving, boot strapping themselves to super intelligence, at which point we will be at their mercy. Because that's basically the p(doom) argument, that some probability is associated with ASI killing us all.
That's different from what we have today with powerful software tools that people can use for all sorts of things, which is an extension of what we already had in less concentrated form. It makes things easier for humans to do good and bad things, but it's not an autonomous digital god.
> More than half of tech workers (53%) would steer a newcomer away from their own field, and they stay.
It would be nice to see this survey repeated across professions. In my experience, most people have gripes about their profession because it’s what they know. Only half of respondents saying they would steer newcomers away feels a lot better than what my friends in medicine say all the time. Among my college professor and professor-track friends, this number probably reaches 100%.
Also, how does this number (53%) compare to pre-LLM numbers?
I definitely would not have steered people away earlier, but would be now.
I would steer people toward tech at this time (specifically software engineering) because I think the "steering people away" thing has gone too far. Already there is data that college students are pivoting away from CS majors en masse.
And how are you certain that whatever else you're going into is more AI proof? Are you certain that law, healthcare, business consulting, accounting, finance, traditional engineering, and various other things that smart people are pivoting to instead of CS are not going to see demand reduction? If anything, I would wager that everything except developing software is going to be at risk, and the demand for software engineers is at the cusp of exploding.
Now is the most exciting time to be a software engineer since the mid 00s.
I think more important than the percentages are the reasons why someone would steer others away from their chosen profession.
>The accusation
>So, why are you still working for them?
>If the answer is the salary, say it louder, and do not force yourself to make excuses.
>If the answer is “someone worse would replace me”, say it too, and louder too.
>If the answer is “I can change things from the inside”, then show me what you changed. Haugen tried the inside. Coxon tried the inside. At the end, unfortunately, the inside did not move.
>
Okay, but what if I think that both sides are LARPing their personal science fiction stories and reality will end up much more mundane (but still incredibly beneficial) than the utopias and dystopias everyone is cooking up?
This is presupposing a moral dilemma that just doesn't exist in my opinion.
We're getting so much of this junk with the IPOs pressing.
The mimicbots are impressive but they're frustrating. We've hit the limit for pre-training data and now we're relying on RL for the next progressive steps and that gets asymptotic fast. The truth is that unsupervised learning hasn't inched forward and we don't know how it will, so we're left with benchmaxxing and LARPing about how they're escaping air-gapped configs using themal sensors, all in a desperate attempt to make the projected numbers look less insane so these people can cash out at IPO time. Everyone knows what's happening and it's embarrassing.
If everyone doing a 'bad thing' stepped off the hamster wheel today, the world economy would collapse before markets closed. The biggest companies have built an attention and surveillance nightmare machine that funnels all the money to a relatively small number of shareholders.
Are you suggesting that since "everyone does the right thing now" would have bad consequences, we shouldn't bother with "everyone does the right thing in a normally distributed way"? (Which is the less artificial scenario.)
These kinds of discussions very quickly arrive at the “no ethical consumption under capitalism” line of thinking. No matter who you are, no matter what you do, some activity or product you engage with will have been the result of some kind of exploitation.
Yes, this is one of the central theses of the TV show "The Good Place". In simpler societies, it's easier to determine what is right and wrong, since actions are often local and can be judged on their immediate effects. But when everything is globalized and there's so many connections, determining right and wrong becomes much more difficult.
"no ethical consumption under capitalism" is true, explains why we can't just stop bad things (everything is bad, so you'd starve) and is a jumping off point for a discussion of what we can actually do about it without starving.
Good.
It's refreshing to see a well-argued and balanced essay on AI risk being upvoted on Hacker News.
theres a certain amount of financial security needed for folks to do this kinda thing.
Of course. But software developers are, on the whole, pretty damn well compensated. As a group we have a lot of freedom to make these sorts of financial decisions.
Yes, some members of the group don't have that freedom. But that doesn't excuse the ones who do.
Even if everyone had enough financial security (or realized they did), you need coordination, people are never going to independently and spontaneously turn down these jobs at evilcorp en masse - you need coordination, a.k.a strong unions which can actually effect the policies of the employer.
The article addresses that:
> If the answer is the salary, say it louder, and do not force yourself to make excuses.
No. This is one of the many excuses that people employ to keep feeding the beast.
You need to have money to do the right thing? Odd take. Plenty of people with much less have sacrificed much more throughout history.
When you make a sacrifice, it has to make sense, and must have some chance for success. In this case it needs to be recognized by others, and a contemporary person can hardly understand any other sacrifice than monetary.
"Now that my RSUs have been exercised, here is the problem with _________"
I understand the concern about your work being used to cause harm, but I also worry about the development of 'forbidden knowledge'. What does it mean to say a branch of science and engineering is too dangerous of a path to go down? Does it mean scientists are persecuted just for asking questions and designing experiments to find answers? Will there be witch hunts against ML researchers? Part of teaching is trusting your students to use the learnings wisely.
This isn't the first. What did it mean for human embryonic gene editing?
That is a good point, but the problem with ML/AI is that anyone can do it. It will be difficult to stem this tide.
Thought Emporium is a YouTube channel that has showcased garage gene editing - with some equipment.
What is your basis for calling them "witch hunts" rather than "criminal prosecutions"? (The prosecutions would start only after the passage of a new law making the questions and experiments illegal.)
If you work at a defense contractor on warplanes, radar systems or nuclear weapons and you are too free in how you teach or write about your professional knowledge, you go to prison, and it has been that way in the US continuously since WWII. Criminal prosecutions are relatively rare because security-clearance background investigations filter out most of the job applicants who would go on to violate the laws.
There's a lot of very dangerous scientific work that gets done because we've put safety measures in place to protect workers and the public to the best of our ability. This allows the dangerous work to be done as safely as possible.
"AI" is being worked on and funded by sociopaths who care about money over all else and have demonstrated that they are willing to hurt others in exchange for more of it. There are no rules or protections for anybody anywhere. In fact AI companies are pouring bribe money into our corrupt political system in an effort to remove or avoid anything that would stop them from doing whatever they want.
The root issue is a lack of honour in the world today.
I think that puts too much onto personal responsibility. Most problems are structural.
Personal responsibility leads to corporate responsibility which is the solve for structural problems. You can never impose a new structure to fix it unless you embrace some form of corporate responsibility which requires personal responsibility.
By law the only corporate responsibility is to shareholders (and law). [1]
https://en.wikipedia.org/wiki/Dodge_v._Ford_Motor_Co.
I was not referring to corporations.
And that structure is made of people, and supported by people's decisions.
People are very happy to abdicate what personal responsibility they have if they can blame the system. (And then fail to help change the system.)
A system where shareholder value is effectively the only metric will align the people accordingly.
And also cost. Standing out from the pack is always costly, you may not find job right away, you might be outcast .. the more we have safety nets the easier it is to quit bad groups.
It also helps to develop better judgment and awareness so you never get into bed with bad people in the first place. "Lay down with dogs, you'll wake up with fleas" applies to business (even getting a job is "doing business") as much as it does other things.
Fair point but sometimes groups evolve, they may go from acceptable to immoral while you're still in it.
Please name a specific time when there was more honor in the world.
In my opinion, we need a sort of survey, let's put it out there, can we get a collective voice from most ai phd's? Remember when climate change was in the scientific tug-of-war? The conclusion for a long time was that 9 in 10 scientists believed in climate change. So, why can't we do that with "ai is gonna kill us"
The more someone has invested into the system (eg. getting a Phd), the more likely will deny the true risks, as their investment is at stakes. Scientists are quite often willing to have other people take great risks for their pet topics.
"Jacob Coxon is not a star. He is not a CEO, not a famous scientist, and does not hold a PhD (yes, in USA you can be an "AI researcher" without a PhD)."
Does he have any formal education
It's impossible to judge long term effects. My only advice is to think long and deep, then do what is true to yourself. Jacob Coxon and Frances Haugen are good people by any reasonable metric, and maybe someone who stays but tries to steer the system from the inside is good too.
He may also have gotten paid for it by players, which dont want AGI, because it would disturb their plans. Imagine AGI works out a plan, which, when applied by everyday people without needing any government help, enforces the end of wars, poverty, pollution, illness, unhappiness, worldwide.
Why is it that every one of these anti-AI screeds that gets upvoted on HN has been clearly written by AI?
Here is one of his posts from 2022:
"Since a long time now I was interested in learning computer graphics and “do stuff” with computer graphics APIs. I began to work on a very simple Game engine in Swift / Metal last year but I began to switch to a Windows machine a few months ago, and Swift / Metal is not compatible with Windows (and will never be I think)."
https://carette.xyz/posts/opengl_and_cpp_on_m1_mac/
This is the same person who is apparently now writing:
"It is a transaction where you sell the part of you that was supposed to care. Do that often enough, and it does not come back. That is not burnout, but it is still a disease. Although, like every disease, it spreads: first to your team, then to your product, then to the people using it."
Everything that gets upvoted on HN has been written by AI. Because AI writing works (if your goal is to attract upvotes).
Good people are easy to mislead into thinking they’re doing a good thing.
Correction - highly intelligent people.
Its the same playbook, over and over again.
[Disclaimer: I believe that morality is objective (and that our consciences are generally reliable guides, albeit ones we can suppress or even change), so any strong statements I make below are implicitly coming from this perspective.]
I strongly disagree, I think this is instead true for those who have an image of themself as "good". IME, someone whose identity involves their "goodness" (and I certainly include myself here) will be willing to do a whole lot to maintain that perception (whether it's a self-perception or an external one). In particular, I think that someone already motivated to feel good about themselves is going to be easier to mislead by slick arguments. See the unfalsifiable longterm-ism of modern-day EA ("you can do an incalculably large amount of good with comparatively minimal effort"), much of today's political violence/radicalism, or the religious persecution all throughout history, whether against the religious (many of last century's authoritarians) or by the religious: cf. Jesus' warning about this in John 16:2 ("They will put you out of the synagogue; in fact, the time is coming when anyone who kills you will think they are offering a service to God."), or even just the Bible's very many warnings against pride (e.g. Proverbs 26:12 [0]) and self-righteousness (e.g. Romans 10:2-3 [1]).
(Of course, sometimes people have no way of knowing a priori that they're being misled. I'm exclusively talking about cases where, deep down, one could/should have known that the situation wasn't actually on the up-and-up... which I'd argue is much more common than we often like to think.)
On the other hand, I'd argue that someone who is truly doing what is good is, by definition, unable to be swayed from that (for then they wouldn't be doing good things anymore, no matter what words they want to use to describe themselves or others). Relatedly, I don't agree that someone can be a "good person" in the absence of actually doing good. (Naturally, this is an extraordinarily high bar, and I'd argue an impossible one to perfectly meet.)
I'm reminded of C.S. Lewis's famous quote:
> Of all tyrannies, a tyranny sincerely exercised for the good of its victims may be the most oppressive. It would be better to live under robber barons than under omnipotent moral busybodies. The robber baron's cruelty may sometimes sleep, his cupidity may at some point be satiated; but those who torment us for our own good will torment us without end for they do so with the approval of their own conscience.
[0] "You have seen a man wise in his own opinion— there is more hope for a fool than for him."
[1] "For I testify about them that they have a zeal for God, but not according to knowledge. For not knowing about the righteousness of God and seeking to establish their own, they did not subject themselves to the righteousness of God." (emphasis mine)
For the "benefit of humanity", "saving the world" and "imaging to do good" via effective altruistic ways. /s
I think a lot of the success of the valley comes from disguising the thing the people are working on. If the end goals of our work were apparent to us we'd have a lot harder time showing up for work everyday.
That said, I don't buy this article. I'm AI neutral. It's going to break a lot of things and fix many others. Good for you if you think you know what the net outcome will be but I think it's delusional to believe you do. Also I don't think you can resign and stop it without ceding the world to bad people. Pandora's box has already been opened. We've discovered a new thing math can do and now it's yet another arms race between various factions seeking to exploit it. Help the faction you want to win. It's not the world I want it to be, but it is what it is.
Nice to see people standing up for their beliefs even when it could hurt them.
Story time :)
I remember when I was first starting out as a programmer. At QE (always occurred on a weekend) a slew of financial reports were created and printed for review the following Monday then I think used for summaries for Wall Street to look at.
A high level finance manager came to me and asked if the report could be changed, not knowing what he was asking I said sure. After a long discussion turned out he wanted the numbers on the report changed and re-printed. I said it was possible but the numbers would be all wrong.
So he said "do it". I went to my manager and told them what he wanted, then I refused to do it. I said I would quit first because I did not want the SEC to come after me. Luckily they all supported me and the finance manager was told to "FO".
FWIW, that company is long gone.
Yes. You never threaten to resign. You conditionally promise to resign.
I had to make this conditional promise once, back in the nineties, as a junior person in a related industry. Because in the dot-com era developers like me were often simultaneously a babe-in-arms on a professional career level and also the only person who knew how to do the work that was being sold. I decided that if it came to it, I would not be the person having a hard time in court. The client understood better than my bosses and they changed the product. Took them a few minutes at most, but I have reason to believe that my decision made some ripples.
Nowadays I don’t let those issues surprise me: I believe in establishing specific ethical standards at the beginning of projects, because if you don’t do that, you don’t have a basis to robustly point to actions that put them at risk.
but what if dude already did it (destroyed the world but it hasnt happened yet) before he quit?
The rule of all games played in capitalism is there's a sucker born every minute. you can work a few years or your whole life before you realize you've been had.
and most do....
I think recursively self-improving universal intelligence is a good thing, and tool alignment is what creates instrumental convergence and therefore a bad thing. The paperclip parable is about tool alignment gone wrong.
These replies will be full of people showing how worldly and clear-eyed they are by dismissing the same threat the guy resigned over as a marketing stunt.
That Coxon guy looks as delusional as the AI leadership. If you watch interviews he gave he doesn’t articulate what he’s concerned about, only sharing vague information about a supposed AI god that will take over the world, but without articulating why now, and how that would even happen. The more I look into his story the more that move looks like what a grifter would do
1. Humans create the machine god
2. Humans put it in charge of everything, as fast as possible, because it's just so damn useful
3. Is it perfectly aligned to human values for all of future ????
4. Everybody dies as a byproduct of the ASI pursuing some project that humans don't even have a chance to understand, like an ant colony during construction of a hydroelectric dam.
5. This is a science fiction story. No such thing exists in the real world and the people claiming it will soon are not experts in the relevant fields of how it would kill everyone.
Thank you. I'm glad I'm not the only one. It looks like he's good at math and graduated Cambridge but a second rate engineer who was heading for the door anyway and decided he'd do well out of doing some scare mongering. Jacob's own words on CBS were "I think [it] clearly there was some latent demand for this sort of sentiment". He knows this is what people are eager to hear and he's providing that... It's the same grift Anthopic and OpenAI have been doing for years: "AGI in the next 6 months!" for the last 4 years...
Is he the one the proposed that airgapped models could still communicate with each other via CPU temperature?
If so, he's the person that's made me lament one loss the progressives took from us. I think there's one phrase that we should be allowed to say regardless of the fact that it is also used to refer to certain groups of people. Because the insult comes from the comparison, not the language itself. Every time we update the language, we just use the new word as the insult. And I don't think we're allowed to use the word here.
But sometimes, you just have to look at someone, cock your head to the side, and say "That's fucking ****".
It brooks no argument, it offers no sympathy, it just tells the other person that they should be ashamed for speaking those thoughts aloud.
I'm deeply skeptical of AI doomer stuff, but I've been struggling to exactly articulate why. I can think of it in my brain and can try to explain but so far none of my explanations have seemed fully coherent to me.
I studied biology in college. I actually did this because I was interested in AI, and already knew a ton of CS and programming. I was already coding in C and using Linux and writing fairly complex stuff. I figured studying the only actual example of a truly intelligent system (especially at the time, early 2000s) was the best way to understand intelligence.
Didn't end up going into AI as my day job, but I've kept up with it all and worked through the math and played with different types (genetic, neural nets, transformers/LLMs, CA / alife, etc.).
I think the root of my skepticism is in what I learned studying biology about how life, adaptation, learning, and evolution actually work in the real world.
I'll try to TL;DR it: I think the doom crowd (and also some of the AI singularity crowd) has a "mystical" view of intelligence. Intelligence is magic pixie dust, and intelligence beyond human is fully unknown and likely has magic superpowers.
The deeper underlying thing seems to be some kind of mathematical mysticism.
Real intelligence is ubiquitous in nature. Trees, grass, bugs, and even bacteria are intelligent. It's also domain specific and limited in capability to what its domain, available training data, and available compute allows.
It's not context-free limit-free magic. It's not special. It's an adaptation like any other.
Yes, the doom crowd thinking comes directly from the Rationalist community, which takes a very religious view of AGI and intelligence. That sort of merged with the Effective Altruism crowd, resulting in that whole idea that we have to reach AGI as fast as possible and contribute literally all our resources to it, because it’s the only rational way to save as much suffering and lives as possible. Sounds incredibly stupid but it’s how AI leaders justify working on something they believe will be a machine god that might doom humanity.
Both OpenAI and Anthropic are from that world, where Rationalist doesn’t mean you’re a rational human being, but instead that it is rational to believe that AGI will be created, thus that they should be the ones creating it. It’s a doomsday cult.
If you start looking at the links between prominent rationalists and effective altruists you see how that whole doom prophecy came about. It’s also how they got so much capital in the early days, SBF literally invested into Anthropic as part of his Effective Altruism effort foundation, and both Amodei siblings are deeply into that whole world
It's amazing that apparently the best way to save as much suffering and future lives as possible involves giving them obscene wealth and ignoring the very real problems we are dealing with today.
Once again, I've said it elsewhere here, I think we could have nipped this thing in the bud if when the first "effective altruist" started talking about the EV of saving billions of "future lives" vs dealing with today's real problems, someone looked at him, cocked their head to the side, and said "That's fucking ****".
All of those magical future people do not get to live if we bake the Earth today. The best way to take care of future people is to take care of people today. And if they were truly serious about taking care of future people, that's what they would be doing. But it's not, they're just collecting resources. I don't know how the fuck Sam Altman owning a Koenigsegg Regera today helps save 20 quintillion lives in 40,000AD. Deeply unserious people.
I think Altman is closer to a sociopath who is taking advantage of the situation, he’s the only one I cannot see as a true believer (and investors like Andreessen who are just here for the money). I believe he will just say anything to gain power and influence. But yeah, unfortunately mixing EA/rationalists + SF monoculture + infinite money + a media ecosystem that acts more as a PR department resulted in one of the worst cult-like phenomenon possible.
I don't believe there is a true believer. I believe Sam Altman is the rule for these people, not the exception.
Yeah I read some of that stuff, like some of "the sequences," and I have never been impressed by it. It's very weirdly dogmatic and full of magical thinking.
I still can't believe anyone ever took Roko's Basilisk seriously, but people did... to the point of freaking out about it.
The groups themselves are a little cultish too, or a lot cultish depending on which one you're talking about. Some of them are full-on cults.
But what do I know. I'm probably low IQ.
> I still can't believe anyone ever took Roko's Basilisk seriously, but people did... to the point of freaking out about it.
If you've spent much time in US Christian circles you've probably encountered people suffering from apocalyptic thinking. The Roko's Basilisk folks seem like the atheist version of the same crowd. It's hard to take them seriously when they're always seeing portents of the end times, and their thinking is clearly clouded by anxiety and fear and not based in reality.
Interestingly I think I am coming at it from exactly the opposite angle. Viruses and bacteria aren't intelligent at all (in my opinion), and don't need to be to wreak havoc. Whether or not LLMs are "intelligent" is one for the philosophers, but they are absolutely able to exploit electronic systems today if their starting state has been set with that as a "goal". Given that alarmingly few physical control systems are really air-gapped, and also that I can today set an agent off to find a voice, clone a voice, and make a phone call, I think the only protection we had today is that the most capable models have been trained not to, and nobody's kicked off a sufficiently malicious task yet.
Viruses are intelligent. Not much, but their replication rate gives them a very rapid rate of evolutionary learning. Evolution is intelligent, but in a very different way from neural networks.
Bacteria and up are full of gene regulatory networks that are absolutely intelligent and share some features with neural networks. Bacteria learn, cooperate, signal, react, seek goals, and that's in addition to evolutionary learning which is also fast for them given their reproduction rate.
Evolution in nature isn't just zeroth order either. Look into a topic known as "the evolution of evolvability." Evolution does meta-parameter search on itself, more or less, and invents adaptations to improve its ability to adapt. Sexual reproduction is the most significant and largest example but there are others. There's environmentally dependent tuning of mutation rates, genome self-editing that we still don't fully understand (this is also how our immune system works), lots of epigenetic stuff we are still learning about, and so on.
If it's alive it's intelligent. All life is constantly learning.
Interesting, before Yudkowski was patient zero for AI doom he was a nanotechnology "grey goo" doomer. That was even more obviously absurd to me. Grey goo already exists. It's called biology, and it's been trying to maximize its footprint on Earth for 4.5 billion years. The miraculous instant "foom" grey goo thing is magical and likely violates conservation laws.
What gets me is that I haven’t heard a reasonable explanation of exactly what the AI will do besides: nukes and biolabs we already have
& ofc AI is making climate change worse, some of the Amazon datacenters are using fossil fuels to generate a country’s worth of power, but that’s never part of their calculations
Viruses against humans, viruses against livestock as an attack against humanity's food chin, plant diseases, half of humanity feeds on three plants maize, wheat, rice. Attacks against infrastructure like water, internet, electricity.
Which one of these would it be? All of them in parallel? Or something we wouldn't think of. If I were to play chess against Magnus Carlsen, I wouldn't know with which piece he will checkmate me, but I know that I am going to lose.
It’s an excellent article marred by the last line/ heading.
People do things.
Good or bad is relative to their own biases/experiences/ beliefs and moral compass.
Very few people are truly sociopaths who do evil stuff for personal gain.
Most do it because of the curse of the “banality of evil” (re: <https://en.wikipedia.org/wiki/Eichmann_in_Jerusalem?wprov=sf...>)
Doesn’t make it right, but like everything else in life the reality is much more complex than saying “good people refuse to do bad things”
The last line/heading of this article makes its main point super clear. Don't deceive yourself (and others).
I’m well aware tech companies are evil and yet work at one.
My current read is that ~20-30% of people in America actively delight in hurting others (ICE supporters etc)
Another 30% do not seem to care about anything but their own well-being. (I don’t like to think I fall here but as a big tech peon it doesn’t look good).
The remaining percent seem to care and are trying their best but looking at how things are going it’s not enough
The former is a recognised personality trait called "social dominance orientation".
Man, I’m exhausted by the hand-wringing of folks reading through to the end and ignoring the questions posed in favor of “well the world ain’t that simple obviously” as a means of not having to justify or defend their own positions.
Obviously morality and ethics aren’t binary. OP isn’t saying that. What they’re saying in a clear and concise manner is that your actions have consequences far beyond your personal sphere, and good people acknowledge that and fight for what’s better despite the harm they personally may suffer as a result. That what makes someone “good” in their point of view isn’t a binary deed or outcome, but of the sacrifices they’ll make for themselves to build a better existence for others.
Planting a tree whose shade you’ll never enjoy, in other words.
So to reiterate OP’s challenge at the end, at least be honest about why you’re still in the game. Let others know where you stand so that we don’t waste our time on futility. If your voluntary position on something mutable risks making you a target, then the rational next step would be to evaluate how important that position is to you that you would weather said harm instead of changing it.
That’s all OP was getting at: promoting the mere notion of critically thinking about the decisions you make in regards to your own moral philosophy and acting more authentically, including tanking the damage trying to bring about change.
Coxon is a shithead who thought he could have a better career as an AI scaremonger than an engineer.
He's still working in service of the AI companies though (same narrative) and no doubt will land a cushy job at so-callef "AI Safety" labs or make a living on the news/podcast circuit.
The downvotes are funny to me because it's true.
To quote Coxon on CBS News: > I think it clearly there was some latent demand for this sort of sentiment or people were gradually coming to this sort of conclusion because obviously so many other people have been saying the same thing for a long time. ... > Yeah. Honestly, when I talk to my friends and family about this, I find it easier to or people seem to actually get more of a sense for it if they're further away from technology. Because I honestly think it sounds more like science fiction than real technology.
He's quite open about it, saying the quiet part out loud. Tech is strange place to be if you're, well... technical. Being constantly gaslit by people who haven't built anything or have 3 years in the industry like here Coxon.