>Some of the crashes, including at least one of the fatal incidents, may have involved the use of “forked” or modified versions of Comma.ai’s software, according to the ODI.
i am still hesitant of tesla and similarly large players, it seems absolutely wild to me that people are using a fork of a small company's software to drive their car.
>i am still hesitant of tesla and similarly large players, it seems absolutely wild to me that people are using a fork of a small company's software to drive their car.
Of course they are. This is exactly why we have laws and regulations.
But there are always people who think they know better. George Hotz, the founder of comma.ai, is a very prominent one (despite falling flat on his face more than once). Who do you think is attracted to comma.ai?
“We’ve always been the North Korea of self-driving companies; we are driven by nobody else’s agenda. That’s not going to change.” -George Hotz
"Hotz is clearly a very capable engineer and has accomplished some impressive feats in his young life. However, listening to him speak at Techcrunch Disrupt he also comes off as incredibly arrogant even by Silicon Valley standards. The fundamental problem that I personally have with Hotz and many others in the valley is that they seem to have the idea that just because they can do something, there is no reason not to, regardless of the consequences for others."
Why would you rely on reporting pf behavior from a decade ago though? He was what... 25-26 then? He's closing in on his 40s (sorry if you read this geohot) and hasn't seemed very arrogant on his live streams for tinygrad. Strong opinions, confidence, sure, but he keeps a flexible mind.
Over what period of time, and how many miles? How does that rate compare with other LKAS / ACC systems? Were they using the experimental vison-based distance estimation system & comma driven longitudinal control, or the car's stock radar based ACC (which bypasses comma control entirely)?
There's not enough details in the article to say if the comma contributed or not to the crashes.
Humans have a remarkable ability to be apparently cognizant and engaged with tasks, without actually using higher level awareness. One example of this is distracted driving while talking handsfree or lost in thought.
Driver monitoring is better than nothing, but it's not the same thing as an engaged driver.
Looking at the road; and realizing that your otherwise well-behaved self-driving car is about to do the most obvious mistake and kill you; are two different things.
It's not self driving though... people should stop pretending it is. It's driving assist. Think of it as a slightly better lane-keep and adaptive cruise control (which often require intervention in stock configuration).
Adaptive cruise control will detect a stopped car and come to a complete stop - but only sometimes. Might depend on speed difference or even the color or shape of the car or object. So it's very possible that the car stopped in previous situations, and the driver completely expected it would do the same, especially with such a large emergency vehicle with flashing lights. By the time the driver realized the car did not detect the emergency vehicle, it may have been too late.
These systems are unable to detect when intervention is required.
Unrelated, but it seems to be that “Comma” is a terrible name for a system that might be involved (even not at fault) in a crash resulting in life-changing injuries.
This goes to show that any car, be it internal combustion, electric, manually operated or self-driving is still a car, and thus is still capable of inflicting death (among other negatives). The difference is still just in the frequency of deaths (and with self-driving this is still under debate).
>This goes to show that any car, be it internal combustion, electric, manually operated or self-driving is still a car, and thus is still capable of inflicting death (among other negatives).
No shit.
>The difference is still just in the frequency of deaths (and with self-driving this is still under debate).
No, the difference is not just frequency. It is, what is the probability I would have made the same mistake. Some of these cases would very unlikely have happened by a (sober) person. Which demonstrates, that we are still dealing with incredibly jagged intelligence.
The GP is right, the correct mental model is just frequency of accidents. There's a high chance the accident might not be caused by you, it could be that you're a well behaved pedestrian and you still get hit. Your own cognitive abilities and skills matter, but only so much.
Would you rather that the other driver is a person or an algorithm? What if automated driving causes few % less accidents than humans but there's no liability or accountability when it happens? What if algorithms cause more accidents but far less fatal ones? When is it the best time to deploy them at scale, to enable rapid improvements and a significantly safer roads one year down the road? Difficult questions.
Regardless, self-driving software can't make mistakes. Not one.
Saying that it's better than a human driver is not adequate. I think as a a society we (begrudgingly) make allowances for human error. There is zero solace for a death at the hands of an algorithm.
>>I think as a a society we (begrudgingly) make allowances for human error. There is zero solace for a death at the hands of an algorithm.
There is zero solace for a death caused by senseless drivers either. With machines and algorithms you can at least hope they would improve in the future. Humans won't. There will always be a small % of terrible ones.
Imo we should allow (and then enforce) self driving even if machines are still worse than humans (they aren't) because that's the only path we have and the sooner we start walking it the better.
> even if machines are still worse than humans (they aren't)
As I said this is still a subject of research. This statement has a source of 'trust me bro'.
> Imo we should allow (and then enforce) self driving even if machines are still worse than humans
How about we do that after a drastic reduction of car use? How about we do that after car manufacturers start taking responsibility about road deaths caused by their 'self-driving' software, and hardware corner cuts?
>>How about we do that after a drastic reduction of car use? How about we do that after car manufacturers start taking responsibility about road deaths caused by their 'self-driving' software, and hardware corner cuts?
I am all for that. Add to that actual enforcement and making reckless driving criminal offence so we can catch the terrible drivers before they kill anyone.
That won't happen though because politics. Self-driving just might.
You are wrong, self-driving software can make mistakes. It is the software companies coding it that refuse to shoulder the responsibility, because they still have no idea how reliable their tech is. Otherwise they would have the appropriate insurance, like any company that trades in any dangerous business. Aircraft manufacturers still make planes (with autopilot!) despite crashes, chemical manufacturers still deal with explosive and poisonous materials, etc... Only those software companies want to be fully held irresponsible of what happens. They are dishonest.
Otherwise they would have the appropriate insurance, like any company that trades in any dangerous business.
What is "appropriate insurance"?
As far as I'm aware, virtually every state with an AV pilot program requires insurance (either commercial or self-insured, as typical for commercial vehicles) in order to be permitted. Several major markets (e.g. California) also require an additional bond to cover issues and have a gradual expansion process to better understand unknown risks.
Yes, they are dishonest and only loyal to stakeholders, that's capitalism 101. The other examples you mentioned are around for more time and they have safety regulations, inspections and fines. Another thing that changed is media landscape. Tesla can get caught of altering the logs in cases of autopilot crash and it just gets buried in news instead of being brought up and repeated over and over each time their stock is discussed.
Completely predictable. They’ll likely dodge a lawsuit in this case (since the car that was hit was an unoccupied police car) but it will inevitably happen.
Is it legal to operate a motor vehicle modified to such an extent on a public highway? Retrofitted self driving using, as far as I can tell, completely unregulated software seems absolutely crazy to me. Free software is a good thing, but don't run your buggy shit in a way that can hurt others.
You can do whatever you want to your car in the US as long as it remains roadworthy, which is basically compliance with FMVSS and state laws. Want to guess what rules don't have much to say about driver assistive technologies?
According to the article:
Comma.ai’s own website states that the “performance” of openpilot Adaptive Cruise Control can be affected by “vehicles in the same lane that are not moving.”
To be honest though, I have a Ford with soke form of Blue Cruise. On the highway in certain areas it can pretty much drive itself. It is terrible in traffic, cant see more than one car ahead and slams on the brakes to stop - it freaks me out, I don't drive like that. Systems like these and seemingly Comma's do not appear to have enough information to know what to do in certain situations, it is why the Ford system is limited to certain situations (and even Ford has issues as in this article).
Maybe I'm old, I haven't been super impressed with these self-driving systems and much prefer to be in control myself.
>Some of the crashes, including at least one of the fatal incidents, may have involved the use of “forked” or modified versions of Comma.ai’s software, according to the ODI.
i am still hesitant of tesla and similarly large players, it seems absolutely wild to me that people are using a fork of a small company's software to drive their car.
"…may have involved the use of “forked” or modified versions of Comma.ai’s software…"
Or, it could just be scapegoating. Like blaming AI for blowing up a girl's school.
I thought credible reporting said it was merely the Pentagon's lack of updating a decade old Target list?
I did find it darkly ironic that Operation Epstein Fury started out by destroying the lives of more young girls.
I was curious about what various forks actually do, found a list of some here: https://commaguide.com/compare/
Clicked a random one and checked out the source, and sure enough it has artifacts of being vibe coded (.claude folder.)
I would dare say most people using Comma have installed a fork. They have a lot more features and car compatibility.
>i am still hesitant of tesla and similarly large players, it seems absolutely wild to me that people are using a fork of a small company's software to drive their car.
Of course they are. This is exactly why we have laws and regulations.
But there are always people who think they know better. George Hotz, the founder of comma.ai, is a very prominent one (despite falling flat on his face more than once). Who do you think is attracted to comma.ai?
“We’ve always been the North Korea of self-driving companies; we are driven by nobody else’s agenda. That’s not going to change.” -George Hotz
"Hotz is clearly a very capable engineer and has accomplished some impressive feats in his young life. However, listening to him speak at Techcrunch Disrupt he also comes off as incredibly arrogant even by Silicon Valley standards. The fundamental problem that I personally have with Hotz and many others in the valley is that they seem to have the idea that just because they can do something, there is no reason not to, regardless of the consequences for others."
https://www.forbes.com/sites/samabuelsamid/2016/11/01/though...
Why would you rely on reporting pf behavior from a decade ago though? He was what... 25-26 then? He's closing in on his 40s (sorry if you read this geohot) and hasn't seemed very arrogant on his live streams for tinygrad. Strong opinions, confidence, sure, but he keeps a flexible mind.
Should it drive someone else's car instead?
Pretty clearly, it shouldn't drive anyone's car.
no?
Over what period of time, and how many miles? How does that rate compare with other LKAS / ACC systems? Were they using the experimental vison-based distance estimation system & comma driven longitudinal control, or the car's stock radar based ACC (which bypasses comma control entirely)?
There's not enough details in the article to say if the comma contributed or not to the crashes.
AFAIK their driver monitoring is very good and checks if you're looking at the road, so I wonder how this happened.
Humans have a remarkable ability to be apparently cognizant and engaged with tasks, without actually using higher level awareness. One example of this is distracted driving while talking handsfree or lost in thought.
Driver monitoring is better than nothing, but it's not the same thing as an engaged driver.
Looking at the road; and realizing that your otherwise well-behaved self-driving car is about to do the most obvious mistake and kill you; are two different things.
It's not self driving though... people should stop pretending it is. It's driving assist. Think of it as a slightly better lane-keep and adaptive cruise control (which often require intervention in stock configuration).
Adaptive cruise control will detect a stopped car and come to a complete stop - but only sometimes. Might depend on speed difference or even the color or shape of the car or object. So it's very possible that the car stopped in previous situations, and the driver completely expected it would do the same, especially with such a large emergency vehicle with flashing lights. By the time the driver realized the car did not detect the emergency vehicle, it may have been too late.
These systems are unable to detect when intervention is required.
Unrelated, but it seems to be that “Comma” is a terrible name for a system that might be involved (even not at fault) in a crash resulting in life-changing injuries.
This goes to show that any car, be it internal combustion, electric, manually operated or self-driving is still a car, and thus is still capable of inflicting death (among other negatives). The difference is still just in the frequency of deaths (and with self-driving this is still under debate).
>This goes to show that any car, be it internal combustion, electric, manually operated or self-driving is still a car, and thus is still capable of inflicting death (among other negatives).
No shit.
>The difference is still just in the frequency of deaths (and with self-driving this is still under debate).
No, the difference is not just frequency. It is, what is the probability I would have made the same mistake. Some of these cases would very unlikely have happened by a (sober) person. Which demonstrates, that we are still dealing with incredibly jagged intelligence.
The GP is right, the correct mental model is just frequency of accidents. There's a high chance the accident might not be caused by you, it could be that you're a well behaved pedestrian and you still get hit. Your own cognitive abilities and skills matter, but only so much.
Would you rather that the other driver is a person or an algorithm? What if automated driving causes few % less accidents than humans but there's no liability or accountability when it happens? What if algorithms cause more accidents but far less fatal ones? When is it the best time to deploy them at scale, to enable rapid improvements and a significantly safer roads one year down the road? Difficult questions.
Regardless, self-driving software can't make mistakes. Not one.
Saying that it's better than a human driver is not adequate. I think as a a society we (begrudgingly) make allowances for human error. There is zero solace for a death at the hands of an algorithm.
>>I think as a a society we (begrudgingly) make allowances for human error. There is zero solace for a death at the hands of an algorithm.
There is zero solace for a death caused by senseless drivers either. With machines and algorithms you can at least hope they would improve in the future. Humans won't. There will always be a small % of terrible ones.
Imo we should allow (and then enforce) self driving even if machines are still worse than humans (they aren't) because that's the only path we have and the sooner we start walking it the better.
> even if machines are still worse than humans (they aren't)
As I said this is still a subject of research. This statement has a source of 'trust me bro'.
> Imo we should allow (and then enforce) self driving even if machines are still worse than humans
How about we do that after a drastic reduction of car use? How about we do that after car manufacturers start taking responsibility about road deaths caused by their 'self-driving' software, and hardware corner cuts?
>>How about we do that after a drastic reduction of car use? How about we do that after car manufacturers start taking responsibility about road deaths caused by their 'self-driving' software, and hardware corner cuts?
I am all for that. Add to that actual enforcement and making reckless driving criminal offence so we can catch the terrible drivers before they kill anyone. That won't happen though because politics. Self-driving just might.
You are wrong, self-driving software can make mistakes. It is the software companies coding it that refuse to shoulder the responsibility, because they still have no idea how reliable their tech is. Otherwise they would have the appropriate insurance, like any company that trades in any dangerous business. Aircraft manufacturers still make planes (with autopilot!) despite crashes, chemical manufacturers still deal with explosive and poisonous materials, etc... Only those software companies want to be fully held irresponsible of what happens. They are dishonest.
As far as I'm aware, virtually every state with an AV pilot program requires insurance (either commercial or self-insured, as typical for commercial vehicles) in order to be permitted. Several major markets (e.g. California) also require an additional bond to cover issues and have a gradual expansion process to better understand unknown risks.
Yes, they are dishonest and only loyal to stakeholders, that's capitalism 101. The other examples you mentioned are around for more time and they have safety regulations, inspections and fines. Another thing that changed is media landscape. Tesla can get caught of altering the logs in cases of autopilot crash and it just gets buried in news instead of being brought up and repeated over and over each time their stock is discussed.
Completely predictable. They’ll likely dodge a lawsuit in this case (since the car that was hit was an unoccupied police car) but it will inevitably happen.
Is it legal to operate a motor vehicle modified to such an extent on a public highway? Retrofitted self driving using, as far as I can tell, completely unregulated software seems absolutely crazy to me. Free software is a good thing, but don't run your buggy shit in a way that can hurt others.
You can do whatever you want to your car in the US as long as it remains roadworthy, which is basically compliance with FMVSS and state laws. Want to guess what rules don't have much to say about driver assistive technologies?
According to the article: Comma.ai’s own website states that the “performance” of openpilot Adaptive Cruise Control can be affected by “vehicles in the same lane that are not moving.”
Seems the AI view of the road is kind of like the game Cruis'n where the road just moves forward and you can hit things in the road and keep going. https://www.youtube.com/results?search_query=Cruis%27n
To be honest though, I have a Ford with soke form of Blue Cruise. On the highway in certain areas it can pretty much drive itself. It is terrible in traffic, cant see more than one car ahead and slams on the brakes to stop - it freaks me out, I don't drive like that. Systems like these and seemingly Comma's do not appear to have enough information to know what to do in certain situations, it is why the Ford system is limited to certain situations (and even Ford has issues as in this article).
Maybe I'm old, I haven't been super impressed with these self-driving systems and much prefer to be in control myself.