The industry culture related to security work and career paths seem just f'd up.
Instead of ensuring we build systems with robust foundations, people end up in a swamp of frustrating roles like SOC staff chasing alarms about false positives all day, peddling ineffective add-on security products, management CISO roles where you're expected to take responsibility of existing insecure Microsoft etc infrastructure without power to change things, working on demotivating compliance bureucracy that don't actually improve security.
I'd argue work on meaningful security improvements is mostly available outside industry security roles.
The company I work for (consulting) upended the entire strategy to basically use pentests to sell managed services (XDR, NDR, SOC, vuln scanning, "continuous pentest") that does nothing to meaningfully move the needle on security. Which of course the market will buy, but it is incredibly demoralizing to see expertise sacrificed to the alter of recurring revenue.
And every time some company got hacked and embarrassed, the same refrain is played out in the comments: "Those cheapskates, they invest too little in security!".
Spend all you want. Buy the most advanced products, and then most expensive services to manage them. I have never seen a company that improved their security by buying it.
Whoa, that’s a bit far. I’m a former pentester. I meaningfully improved security at quite a few places. The standout was Citadel, where a product was set to launch within a few weeks. When I first got there, typing ‘ into their search fields resulted in SQL injection right away. They had never thought to defend against it. Over the next week, I fed them a steady list of bugs and vulns (there were many) until by the end of it that product was watertight. I was particularly proud of that one.
In my experience pentests were just a box ticking exercise. I consider it a cultural thing. If you're having to run a pentest right before release and it uncovers a vast amount of issues then you never cared about the quality of your software to begin with and it would show up not just as insecure software. Running automated test suites periodically should be a part of software building practices. That and deep code reviews and so on. All of that to feed into the quality of what you're building.
The problem is getting the decision makers to care. And/or changing the process to at least consider quality as an important factor even if velocity is preferred(and featuritis has taken over).
Story time. In one gig I had, a couple of weeks into it I discovered that AWS keys to the production data in the S3 buckets were being exposed on the client side(an SPA). Those keys would give you access to the data for all the clients on that platform. So I figured I'd do "the right thing" and told my manager(the CTO) who said something along the lines of "yeah that sounds serious" and asked me to talk to the CEO who wrote that code. At this point, I was still expecting that I might be wrong or at least being told that it was written in a rush or something and thank me for pointing it out. The CEO just dismissed it as being "temporary production keys" and closed down the conversation. Suffice it to say that I was not the CEO's favorite person moving forward.
Pentests work to secure the product under test at the point in time of the test (if the company cares to fix the bugs...). The real solution is to design in security throughout the software lifecycle, not play pentest wack-a-mole game at the end of the cycle. If a pentester is finding trivial SQL injection in an app, then it is clear that the company never considered security. And unless the pentest makes them care, the cycle will just continue.
Precisely, the industry needs to empower the engineers to shift left and integrate security as a part of the SDLC. this is the only way to provide continuous assurance in the age of AI.
They had a product that interpolated untrusted data into trusted SQL strings, but being told about it (and many other vulnerabilities!) was all that was required to make them watertight.
I would be very happy if you right about this.
Whitelisting is usually easier than blacklisting, and not devloping brittle features where errors have security implications is usually easier than spending money on security after the fact. However not developing features is not something we as an industry is good at. Github Actions perhaps being the most recent example of this.
"Improved" is a useless word. Is their security now adequate? Is it secure against the run-of-the-mill financially motivated threat actors we see regularly and orchestrating thousands of profitable attacks annually?
We regularly see attacks extorting tens of millions of dollars from major multinationals like Citadel. Is the cost of breaching their systems in excess of ten million dollars (which would net you a nice fat profit against multiple tens of millions extorted)? You get a team of 10 professionals for 1-3 years and you can not breach their systems?
That is the minimum standard of adequate against commonplace, prevailing threats for large multinationals. Even that ignores the fact that major corporations are frequently attacked by state actors, so really the minimum standard for protection against expected threats should include those as well, but I will leave that aside for now since the overwhelming sentiment is that protection against state actors is so utterly hopeless it is not even worth mentioning.
For that matter, can you point to literally any system in the entire world that is positively demonstrated (absence of evidence is not evidence of absence) to have reached that standard?
>Even that ignores the fact that major corporations are frequently attacked by state actors, so really the minimum standard for protection against expected threats should include those as well, but I will leave that aside for now since the overwhelming sentiment is that protection against state actors is so utterly hopeless it is not even worth mentioning.
It always has been, it's just now the state actors are more and more active (and visibly so).
You do realize you're actually supporting the point that you are replying to. No amount of pentests, no amount of security products are going to solve the problem that a product was built that had a search field that was trivially injectable.
It is an investment problem, they need to invest in security expertise, not security products and services. And that is the sad part, absent the company really caring to spend that money or an external demand (regulatory or customers) it just isn't going to happen. They'll just layer on more products and services and call it a day.
> I'd argue work on meaningful security improvements is mostly available outside industry security roles.
I drift in and out of security roles and definitely agree. If a company truly wants secure products the proper way is to do that from the ground up as the product is designed, architected and developed. The optimal role for building secure products is to have security awareness and priority embedded in the design and engineering team. Not as an afterthought from a security team.
Alas! Most companies don't care that much, so if you want to drive the product to be more secure, it can sometimes be more effective to do it from the security organization. If the company culture is to ignore security, you can drive more improvement from infosec because then that's your job. But it's not the optimal way to get there.
Just commented this elsewhere but my takes on cybersecurity today: Its about to blow up in high demand with so many skiddies being able to hack anybody with an LLM. We are seeing an increase in websites, systems and companies being compromised at an alarming rate. I suspect one of these days we will see a headline of a compromise that will shock and horrify us all. Anyone sleeping on cyber security is a ticking timebomb.
Honestly, if you wanted to make a YC company today that targets AI in a meaningdful way, I'd say make it focused on cyber security analysis. ;)
> I suspect one of these days we will see a headline of a compromise that will shock and horrify us all
But we've had the shock headlines already, and nothing changes. We've seen hospitals get hit that had real-life consequences for patients, the entirety of US citizens SSNs have been breached multiple times now. Passwords as a concept are basically obsolete now. There's even more.
That bomb has already been going off.
If anything I'm seeing the opposite. Companies are throwing security to the wind to go all in on AgEnTiC AI.
If we want change irt cybersecurity, then there needs to start being real consequences for a breach. Not just free credit monitoring. The companies that are proven to be negligent should face actual financial & criminal consequences.
I am building in cybersec space. I dont think you even need script kiddies now. Internal employees run dangerous bad ops with AI that itself is a cybersec nightmare.
Whenever I tell people I work in computer security, their first question is "are you worried about AI taking your job"? To which I just laugh and respond "AI is job security"
It really is! AI will only help you if anything, you aren't worried about AI giving you bad code, just bad answers, which you would validate anyway. I think the other area where AI could be interesting, and I don't hear much buzz about it is, during outages, if it can query all online systems and logs in your cloud, it could probably triage it faster than an entire outage team could in theory anyway. Surprised nobodys built such a system yet. ;)
I mean it in the sense that AI security hype and the larger geopolitical environment has woken up a lot of people to the reality that they need to consider security. And the ones that haven't woken up yet will get a wakeup call when they are breached. It also increases the demand for real security expertise, which is already scarce.
Also, in my niche (hardware and embedded product security), AI doesn't a have a functional impact to the work except in code analysis, but even that is difficult given the level of abstraction these systems are built at.
That's fair, though even that could just be a matter of time, as people build tools that interface LLMs to the physical world. I wonder how something like Bus Pirate could be used with an LLM (maybe a more powerful version of it?) to grok and poke hardware all over the place.
I forsee issues with really getting use out of any commodity language model in the hardware security context, because hardware systems notoriously lack standardization. And often times, the technical knowledge (datasheets, app notes) is locked behind vendor NDAs, or straight up not documented, only existing in the minds of engineers. The implementations of said designs are similarly highly proprietary, with little public "real" systems to train models on.
So the issue is two-fold:
* The knowledge must be documented and accessible for training.
* A bespoke model must be trained this documentation.
It is unlikely that both of these things happen in the general model context. Perhaps individual chip vendors will eventually pursue this, but I suspect it is just not a priority for them.
Companies don't fundamentally care about cybersecurity. Most of them see cybersecurity as being similar to waste management; it's not something you get excited about. Sure, your company _must_ have a waste management plan, but it only exists out of pure necessity. It's required to do the real work of the company, but if you had a magic wand and never had to deal with it, you'd choose that option. And, like waste management, plenty of companies outsource their cybersecurity, since it's cheaper and they don't really care about it.
Yes, you're correct. To add - companies don't fundamentally care about all the things that we like to think of as "nice things", like good design, lack of dark patterns, robust security architecture, minimizing technical debt, etc.
If customers cared about reputational damage from cybersecurity incidents (sure.. some do) , then you would see that reflected in their priorities. Also, non-technical customers don't really know who to blame for security anyway. They'll just blame the OS vendor or other random parties even if its the Application that is not secure.
"Cybersecurity" is why everything breaks on a daily basis at $DAYJOB. One random garbage packet from the internet can cause entire VLANs to go down, kicking people out of VPNs and Citrix. I get the need for security, but these security types cause more issues than they're trying to solve.
We have a lot of Cisco switches, and most of our machines run anyconnect. The ones that don't have to have custom filters applied to them, but any time the switches see something they don't like, they bounce or shut off the port. Add to that having 20 border gateway switches and a complex checkpoint WAF, literally anything that strays from the norm breaks the network.
"Show me the incentives, and I'll show you the outcomes." - Charlie Munger
Right now, if you have a security breach, at least in the US, you send out a letter telling the person that their data could be God-knows-where and offer them two free years of credit monitoring. Victims aren't going to really use that because it's essentially useless. If they've got absolutely, positively nothing better to do with their time, I guess you could file a lawsuit. Who knows what the outcome would be. Probably not in their favor.
In other words, it's cheaper for them to overwork the InfoSec guys/gals and barely care about what is happening outside of day-to-day operations, than it is to really secure their stuff. So they don't spend that money.
If you saw corporate valuation-cratering fines being implemented - the kind that would end the c-suite's careers and bring shame to their family lines for seven generations - I bet that they'd start catering lunches for the InfoSec team.
You could also create an AI tool to help generate letters to lawmakers about how they need to make a real dent in this between reruns of Matlock in the retirement home.
I don't think fines are enough of an incentive. They're too easy to evade and insufficiently consequential to the people who are actually shipping code. Moreover, making them enormous (as you put it well "valuation-cratering") unfairly punishes people who are not directly responsible for the failure. Instead, like in other engineering disciplines, Engineers need to be personally liable for the consequences of failure. Not necessarily every engineer--not every mechanical engineer needs to be a P.E.--but someone directly responsible for the quality of the work needs to stake their reputation on it, and suffer the consequences when it fails.
In practice this would mean that you need to show conformance to some kind of security process. The actual outcome of that process is of secondary importance as long as you can show that you’re compliant. Very carefully written process documents _can_ improve things, but my confidence in security processes is low for companies without intrinsic motivation.
I think one can reasonably argue that sufficiently large fines that don’t have a „but we followed iso-xyz“ loophole could produce better outcomes. The difficult part is making the companies care about existential tail risks.
Companies are already following a bunch of standards like SOX, SOC2, HIPAA, etc., and documenting their adherence to checking all of the boxes, but incidents still happen every week.
I say this all the time, corporate security is 100% a game. Unless you are a part of the small group of people literally working on exploit dev you are feeding the security delusion as a service apparatus. Also, contrary to how things started (phrack, L0pht, zines, etc.) your average corporate security drone is almost universally a dull-witted, uninspired specimen.
Yes, it'll generate a lot of super annoying paperwork. But, hopefully, it will also tighten up software engineering standards. It has worked well in other disciplines.
There already are areas where such standards exist, eg safety critical applications in aviation. Arguably the defect rate there _is_ lower, but I still think that this method for achieving this is quite inefficient. And I think that writing aviation software that doesn’t crash is a lot easier to define a process for than for writing software that is difficult to hack.
The missing piece is the requirement for a certified Professional Engineer to sign off on the system. That decouples the incentives from the corporate objectives, and makes it personal. We need that kind of professional accountability in software, otherwise it'll continue to be bad.
I hadn't actually heard of this, I have never worked on safety critical systems, but doing some googling I found a reference to a "Designated Engineering Representative" in this article about the FAA's DO-178B: https://en.wikipedia.org/wiki/DO-178B
I wasn't able to find much information about U.S. P.E. certification for SWE's, although there is at least one state which offers it. I wasn't able to find any indication anywhere that a compliance process requires a P.E. to sign off on software. That doesn't mean it doesn't exist though!
One major problem is that now that software is "everywhere" it's escaping the boundaries of safety critical standards. Nobody will be killed directly by a bank getting hacked, but it could result in mortal harm to an individual who has their identity stolen. There are all kinds of systems that aren't labeled safety critical in the kinetic sense which are nonetheless very load-bearing. Software which runs on phones, for example. Surely people have died due to buggy phone software. Nobody is being held meaningfully accountable, so it will continue to happen.
To be clear, I'm not saying we should heap a whole lot more pressure onto security teams. Instead we need to find better ways to make security every engineer's professional ethical responsibility--either directly because they're signing off on the system or indirectly because their respected senior colleague is. I just don't see fines getting us there.
I understand your argument, but I feel that good development practices are essentially a company culture question. Putting additional burden on the engineers does not sit well with me as in my experience many engineers already care more about quality and security than their management would like to give them manpower to implement. If we make it easier to have some engineers be scapegoats for management failure this might actually backfire.
> Moreover, making them enormous (as you put it well "valuation-cratering") unfairly punishes people who are not directly responsible for the failure.
For better or for worse, that's how we've set up our system. The entire point of incorporation is to separate the people working at a company from the company, legally speaking. The most they can really do is fire you.
With respect to valuation-cratering, that's supposed to be considered fair in our system. If a bunch of shareholders elect a board that lets a C-suite operate a company with lax security culture, they're ultimately responsible for the losses they incur. It's only fair; they're the ones getting the profits, too.
I put emphasis on "supposed" because we don't really do this anymore. Instead of expecting shareholders to take the bath, we shift the loss onto the customers, who have to put up with the consequences of identity theft out of their own pockets.
In some professions incorporation doesn't shield you from malpractice claims. We have a software quality problem in tech, to the point where buggy, broken software has become completely normal. Security vulnerabilities are so commonplace and routine that companies hire entire departments to respond to security breaches.
We don't have bridges falling down left and right, airplane crashes aren't common, trains don't leave the rails every day. Software is all grown up now, and we need to grow up as a discipline whether we're ready or not. It'll get a lot worse before this happens, but I'm convinced it will happen eventually.
That would require us to have the equivalent of a bridge falling down.
I'd say some of that depends on the domain that the software is developed for. I've spent most of my 12 years developing software in healthcare IT. Typically, you don't see too many critical (meaning life-threatening) bugs in EMR/EHR software, which is one of those domains where you'd think it'd be easy to run into that sort of thing. Most of the problems in the domain have to do more with data access being granted or obtained by someone who shouldn't have it. You won't die or get seriously injured as a result of the software, but some guy in a dank basement outside Moscow might know you need your knee replaced.
A lot of that comes from the fact that software for systems that could have a bridge collapse-level of impact are already certified as a part of a larger regulatory scheme for the domain in which the software operates. Healthcare and avionics software instantly come to mind. A lot of people in the KC area make their living writing software for those domains, and while they aren't required to have an engineering license to do so, their wares have to be vetted enough that they have to work at the same level.
You'd need to convince lawmakers to set up a regulating body that tracks business and consumer software for security in the same way we do EHRs for patient safety risks.
It's more of a slow burn than any big event you can point to, so it's possible nothing will get done. Hopefully the political pressure will develop as the political class becomes more tech savvy. They will, over time, as younger people who grew up with technology replace the older generation. If the AI bubble pops in a way that causes a financial crisis I think we'll probably see a lot more scrutiny of the tech industry in general, which could lead to progress in quality. Maybe it won't happen, but the current state of things getting shipped before they're done, constant outages, everything being kind of broken all the time... it just doesn't seem sustainable.
> offer them two free years of credit monitoring. Victims aren't going to really use that because it's essentially useless
It's generally actively harmful, and the CRAs fight for this business from breaches because universally, to accept the free credit monitoring you have to sign up for their highest tier credit monitoring package (which can be up to $50/month), supply a credit card, and then hope to remember, a year later, to cancel at the end of the free period, because at that point they'll convert you to a paying customer.
Yep. I had a chance to go for a cybersecurity degree. And every time ive looked at that, the career path is basically an applied insurance job.
Cybersecurity does not make money. They do not raise the profit for a company. Instead, they are compliance, contractual, and legal defences to repel lawsuits and keep data boundaries clean.
And who's the first to go? Groups that dont make money. Like cybersec.
Cybersecurity certainly makes money. The good ones make a lot. I mean a lot.
But if you think you can just study for a year and get some security certificates and call it a day, you're going to be sorely disappointed in the compensation.
This is true at companies that treat security as a checkbox driven by compliance.
Not true at top-tier tech companies building software in product categories that by their nature have very high security requirements, like Fintech (e.g. Coinbase, Mercury bank, Wise).
With Claude writing so much of the software in big companies, Anthropic is well-positioned to eat up SAST, DAST and a lot of the supply chain analysis. EDR and proactive security are still going to be massive businesses, however.
The industry culture related to security work and career paths seem just f'd up.
Instead of ensuring we build systems with robust foundations, people end up in a swamp of frustrating roles like SOC staff chasing alarms about false positives all day, peddling ineffective add-on security products, management CISO roles where you're expected to take responsibility of existing insecure Microsoft etc infrastructure without power to change things, working on demotivating compliance bureucracy that don't actually improve security.
I'd argue work on meaningful security improvements is mostly available outside industry security roles.
The company I work for (consulting) upended the entire strategy to basically use pentests to sell managed services (XDR, NDR, SOC, vuln scanning, "continuous pentest") that does nothing to meaningfully move the needle on security. Which of course the market will buy, but it is incredibly demoralizing to see expertise sacrificed to the alter of recurring revenue.
And every time some company got hacked and embarrassed, the same refrain is played out in the comments: "Those cheapskates, they invest too little in security!".
Spend all you want. Buy the most advanced products, and then most expensive services to manage them. I have never seen a company that improved their security by buying it.
Whoa, that’s a bit far. I’m a former pentester. I meaningfully improved security at quite a few places. The standout was Citadel, where a product was set to launch within a few weeks. When I first got there, typing ‘ into their search fields resulted in SQL injection right away. They had never thought to defend against it. Over the next week, I fed them a steady list of bugs and vulns (there were many) until by the end of it that product was watertight. I was particularly proud of that one.
Pentests work.
In my experience pentests were just a box ticking exercise. I consider it a cultural thing. If you're having to run a pentest right before release and it uncovers a vast amount of issues then you never cared about the quality of your software to begin with and it would show up not just as insecure software. Running automated test suites periodically should be a part of software building practices. That and deep code reviews and so on. All of that to feed into the quality of what you're building.
The problem is getting the decision makers to care. And/or changing the process to at least consider quality as an important factor even if velocity is preferred(and featuritis has taken over).
Story time. In one gig I had, a couple of weeks into it I discovered that AWS keys to the production data in the S3 buckets were being exposed on the client side(an SPA). Those keys would give you access to the data for all the clients on that platform. So I figured I'd do "the right thing" and told my manager(the CTO) who said something along the lines of "yeah that sounds serious" and asked me to talk to the CEO who wrote that code. At this point, I was still expecting that I might be wrong or at least being told that it was written in a rush or something and thank me for pointing it out. The CEO just dismissed it as being "temporary production keys" and closed down the conversation. Suffice it to say that I was not the CEO's favorite person moving forward.
One textbook I read has a line about how you shouldn't work in security if you want to make friends, and I was shocked and impressed by its honesty.
Pentests work to secure the product under test at the point in time of the test (if the company cares to fix the bugs...). The real solution is to design in security throughout the software lifecycle, not play pentest wack-a-mole game at the end of the cycle. If a pentester is finding trivial SQL injection in an app, then it is clear that the company never considered security. And unless the pentest makes them care, the cycle will just continue.
Precisely, the industry needs to empower the engineers to shift left and integrate security as a part of the SDLC. this is the only way to provide continuous assurance in the age of AI.
They had a product that interpolated untrusted data into trusted SQL strings, but being told about it (and many other vulnerabilities!) was all that was required to make them watertight.
I would be very happy if you right about this.
Whitelisting is usually easier than blacklisting, and not devloping brittle features where errors have security implications is usually easier than spending money on security after the fact. However not developing features is not something we as an industry is good at. Github Actions perhaps being the most recent example of this.
"Improved" is a useless word. Is their security now adequate? Is it secure against the run-of-the-mill financially motivated threat actors we see regularly and orchestrating thousands of profitable attacks annually?
We regularly see attacks extorting tens of millions of dollars from major multinationals like Citadel. Is the cost of breaching their systems in excess of ten million dollars (which would net you a nice fat profit against multiple tens of millions extorted)? You get a team of 10 professionals for 1-3 years and you can not breach their systems?
That is the minimum standard of adequate against commonplace, prevailing threats for large multinationals. Even that ignores the fact that major corporations are frequently attacked by state actors, so really the minimum standard for protection against expected threats should include those as well, but I will leave that aside for now since the overwhelming sentiment is that protection against state actors is so utterly hopeless it is not even worth mentioning.
For that matter, can you point to literally any system in the entire world that is positively demonstrated (absence of evidence is not evidence of absence) to have reached that standard?
>Even that ignores the fact that major corporations are frequently attacked by state actors, so really the minimum standard for protection against expected threats should include those as well, but I will leave that aside for now since the overwhelming sentiment is that protection against state actors is so utterly hopeless it is not even worth mentioning.
It always has been, it's just now the state actors are more and more active (and visibly so).
You do realize you're actually supporting the point that you are replying to. No amount of pentests, no amount of security products are going to solve the problem that a product was built that had a search field that was trivially injectable.
It is an investment problem, they need to invest in security expertise, not security products and services. And that is the sad part, absent the company really caring to spend that money or an external demand (regulatory or customers) it just isn't going to happen. They'll just layer on more products and services and call it a day.
what do you mean no movement?
isnt pentesting supposed to generate tickets to fix vulnerabilities?
> I'd argue work on meaningful security improvements is mostly available outside industry security roles.
I drift in and out of security roles and definitely agree. If a company truly wants secure products the proper way is to do that from the ground up as the product is designed, architected and developed. The optimal role for building secure products is to have security awareness and priority embedded in the design and engineering team. Not as an afterthought from a security team.
Alas! Most companies don't care that much, so if you want to drive the product to be more secure, it can sometimes be more effective to do it from the security organization. If the company culture is to ignore security, you can drive more improvement from infosec because then that's your job. But it's not the optimal way to get there.
Just commented this elsewhere but my takes on cybersecurity today: Its about to blow up in high demand with so many skiddies being able to hack anybody with an LLM. We are seeing an increase in websites, systems and companies being compromised at an alarming rate. I suspect one of these days we will see a headline of a compromise that will shock and horrify us all. Anyone sleeping on cyber security is a ticking timebomb.
Honestly, if you wanted to make a YC company today that targets AI in a meaningdful way, I'd say make it focused on cyber security analysis. ;)
> I suspect one of these days we will see a headline of a compromise that will shock and horrify us all
But we've had the shock headlines already, and nothing changes. We've seen hospitals get hit that had real-life consequences for patients, the entirety of US citizens SSNs have been breached multiple times now. Passwords as a concept are basically obsolete now. There's even more.
That bomb has already been going off.
If anything I'm seeing the opposite. Companies are throwing security to the wind to go all in on AgEnTiC AI.
If we want change irt cybersecurity, then there needs to start being real consequences for a breach. Not just free credit monitoring. The companies that are proven to be negligent should face actual financial & criminal consequences.
I am building in cybersec space. I dont think you even need script kiddies now. Internal employees run dangerous bad ops with AI that itself is a cybersec nightmare.
Whenever I tell people I work in computer security, their first question is "are you worried about AI taking your job"? To which I just laugh and respond "AI is job security"
It really is! AI will only help you if anything, you aren't worried about AI giving you bad code, just bad answers, which you would validate anyway. I think the other area where AI could be interesting, and I don't hear much buzz about it is, during outages, if it can query all online systems and logs in your cloud, it could probably triage it faster than an entire outage team could in theory anyway. Surprised nobodys built such a system yet. ;)
I mean it in the sense that AI security hype and the larger geopolitical environment has woken up a lot of people to the reality that they need to consider security. And the ones that haven't woken up yet will get a wakeup call when they are breached. It also increases the demand for real security expertise, which is already scarce.
Also, in my niche (hardware and embedded product security), AI doesn't a have a functional impact to the work except in code analysis, but even that is difficult given the level of abstraction these systems are built at.
That's fair, though even that could just be a matter of time, as people build tools that interface LLMs to the physical world. I wonder how something like Bus Pirate could be used with an LLM (maybe a more powerful version of it?) to grok and poke hardware all over the place.
I forsee issues with really getting use out of any commodity language model in the hardware security context, because hardware systems notoriously lack standardization. And often times, the technical knowledge (datasheets, app notes) is locked behind vendor NDAs, or straight up not documented, only existing in the minds of engineers. The implementations of said designs are similarly highly proprietary, with little public "real" systems to train models on.
So the issue is two-fold:
* The knowledge must be documented and accessible for training.
* A bespoke model must be trained this documentation.
It is unlikely that both of these things happen in the general model context. Perhaps individual chip vendors will eventually pursue this, but I suspect it is just not a priority for them.
Maybe not for unethical hacking purposes but I am wondering about for reverse engineering think like a game console.
Do you think that AI helps security offense more than defense? It's not obvious to me that it does.
Companies don't fundamentally care about cybersecurity. Most of them see cybersecurity as being similar to waste management; it's not something you get excited about. Sure, your company _must_ have a waste management plan, but it only exists out of pure necessity. It's required to do the real work of the company, but if you had a magic wand and never had to deal with it, you'd choose that option. And, like waste management, plenty of companies outsource their cybersecurity, since it's cheaper and they don't really care about it.
Yes, you're correct. To add - companies don't fundamentally care about all the things that we like to think of as "nice things", like good design, lack of dark patterns, robust security architecture, minimizing technical debt, etc.
If customers cared about reputational damage from cybersecurity incidents (sure.. some do) , then you would see that reflected in their priorities. Also, non-technical customers don't really know who to blame for security anyway. They'll just blame the OS vendor or other random parties even if its the Application that is not secure.
reputational damage
Ah but the thing is that every company sucks so there is nowhere for customers to flee to.
"Cybersecurity" is why everything breaks on a daily basis at $DAYJOB. One random garbage packet from the internet can cause entire VLANs to go down, kicking people out of VPNs and Citrix. I get the need for security, but these security types cause more issues than they're trying to solve.
Care to elaborate? I've never seen a system that worked this way
We have a lot of Cisco switches, and most of our machines run anyconnect. The ones that don't have to have custom filters applied to them, but any time the switches see something they don't like, they bounce or shut off the port. Add to that having 20 border gateway switches and a complex checkpoint WAF, literally anything that strays from the norm breaks the network.
This seems like a poor choice of configuration that could easily be exploited by a malicious agent to conduct a denial of service attack
"Show me the incentives, and I'll show you the outcomes." - Charlie Munger
Right now, if you have a security breach, at least in the US, you send out a letter telling the person that their data could be God-knows-where and offer them two free years of credit monitoring. Victims aren't going to really use that because it's essentially useless. If they've got absolutely, positively nothing better to do with their time, I guess you could file a lawsuit. Who knows what the outcome would be. Probably not in their favor.
In other words, it's cheaper for them to overwork the InfoSec guys/gals and barely care about what is happening outside of day-to-day operations, than it is to really secure their stuff. So they don't spend that money.
If you saw corporate valuation-cratering fines being implemented - the kind that would end the c-suite's careers and bring shame to their family lines for seven generations - I bet that they'd start catering lunches for the InfoSec team.
New idea: AI tool to help generate legal letters to companies after they leak data to cause them maximum inconvenience.
The human speed legal system would become collateral damage.
You could also create an AI tool to help generate letters to lawmakers about how they need to make a real dent in this between reruns of Matlock in the retirement home.
> "Show me the incentives, and I'll show you the outcomes." - Charlie Munger
Also note that -like pharmaceutical companies- treatment is more profitable than cure for infosec consultants.
I don't think fines are enough of an incentive. They're too easy to evade and insufficiently consequential to the people who are actually shipping code. Moreover, making them enormous (as you put it well "valuation-cratering") unfairly punishes people who are not directly responsible for the failure. Instead, like in other engineering disciplines, Engineers need to be personally liable for the consequences of failure. Not necessarily every engineer--not every mechanical engineer needs to be a P.E.--but someone directly responsible for the quality of the work needs to stake their reputation on it, and suffer the consequences when it fails.
In practice this would mean that you need to show conformance to some kind of security process. The actual outcome of that process is of secondary importance as long as you can show that you’re compliant. Very carefully written process documents _can_ improve things, but my confidence in security processes is low for companies without intrinsic motivation.
I think one can reasonably argue that sufficiently large fines that don’t have a „but we followed iso-xyz“ loophole could produce better outcomes. The difficult part is making the companies care about existential tail risks.
Companies are already following a bunch of standards like SOX, SOC2, HIPAA, etc., and documenting their adherence to checking all of the boxes, but incidents still happen every week.
I say this all the time, corporate security is 100% a game. Unless you are a part of the small group of people literally working on exploit dev you are feeding the security delusion as a service apparatus. Also, contrary to how things started (phrack, L0pht, zines, etc.) your average corporate security drone is almost universally a dull-witted, uninspired specimen.
Yes, it'll generate a lot of super annoying paperwork. But, hopefully, it will also tighten up software engineering standards. It has worked well in other disciplines.
There already are areas where such standards exist, eg safety critical applications in aviation. Arguably the defect rate there _is_ lower, but I still think that this method for achieving this is quite inefficient. And I think that writing aviation software that doesn’t crash is a lot easier to define a process for than for writing software that is difficult to hack.
The missing piece is the requirement for a certified Professional Engineer to sign off on the system. That decouples the incentives from the corporate objectives, and makes it personal. We need that kind of professional accountability in software, otherwise it'll continue to be bad.
It is my understanding that personal responsibility already exists in safety critical software development.
I hadn't actually heard of this, I have never worked on safety critical systems, but doing some googling I found a reference to a "Designated Engineering Representative" in this article about the FAA's DO-178B: https://en.wikipedia.org/wiki/DO-178B
I wasn't able to find much information about U.S. P.E. certification for SWE's, although there is at least one state which offers it. I wasn't able to find any indication anywhere that a compliance process requires a P.E. to sign off on software. That doesn't mean it doesn't exist though!
One major problem is that now that software is "everywhere" it's escaping the boundaries of safety critical standards. Nobody will be killed directly by a bank getting hacked, but it could result in mortal harm to an individual who has their identity stolen. There are all kinds of systems that aren't labeled safety critical in the kinetic sense which are nonetheless very load-bearing. Software which runs on phones, for example. Surely people have died due to buggy phone software. Nobody is being held meaningfully accountable, so it will continue to happen.
To be clear, I'm not saying we should heap a whole lot more pressure onto security teams. Instead we need to find better ways to make security every engineer's professional ethical responsibility--either directly because they're signing off on the system or indirectly because their respected senior colleague is. I just don't see fines getting us there.
I understand your argument, but I feel that good development practices are essentially a company culture question. Putting additional burden on the engineers does not sit well with me as in my experience many engineers already care more about quality and security than their management would like to give them manpower to implement. If we make it easier to have some engineers be scapegoats for management failure this might actually backfire.
> Moreover, making them enormous (as you put it well "valuation-cratering") unfairly punishes people who are not directly responsible for the failure.
For better or for worse, that's how we've set up our system. The entire point of incorporation is to separate the people working at a company from the company, legally speaking. The most they can really do is fire you.
With respect to valuation-cratering, that's supposed to be considered fair in our system. If a bunch of shareholders elect a board that lets a C-suite operate a company with lax security culture, they're ultimately responsible for the losses they incur. It's only fair; they're the ones getting the profits, too.
I put emphasis on "supposed" because we don't really do this anymore. Instead of expecting shareholders to take the bath, we shift the loss onto the customers, who have to put up with the consequences of identity theft out of their own pockets.
In some professions incorporation doesn't shield you from malpractice claims. We have a software quality problem in tech, to the point where buggy, broken software has become completely normal. Security vulnerabilities are so commonplace and routine that companies hire entire departments to respond to security breaches.
We don't have bridges falling down left and right, airplane crashes aren't common, trains don't leave the rails every day. Software is all grown up now, and we need to grow up as a discipline whether we're ready or not. It'll get a lot worse before this happens, but I'm convinced it will happen eventually.
That would require us to have the equivalent of a bridge falling down.
I'd say some of that depends on the domain that the software is developed for. I've spent most of my 12 years developing software in healthcare IT. Typically, you don't see too many critical (meaning life-threatening) bugs in EMR/EHR software, which is one of those domains where you'd think it'd be easy to run into that sort of thing. Most of the problems in the domain have to do more with data access being granted or obtained by someone who shouldn't have it. You won't die or get seriously injured as a result of the software, but some guy in a dank basement outside Moscow might know you need your knee replaced.
A lot of that comes from the fact that software for systems that could have a bridge collapse-level of impact are already certified as a part of a larger regulatory scheme for the domain in which the software operates. Healthcare and avionics software instantly come to mind. A lot of people in the KC area make their living writing software for those domains, and while they aren't required to have an engineering license to do so, their wares have to be vetted enough that they have to work at the same level.
You'd need to convince lawmakers to set up a regulating body that tracks business and consumer software for security in the same way we do EHRs for patient safety risks.
It's more of a slow burn than any big event you can point to, so it's possible nothing will get done. Hopefully the political pressure will develop as the political class becomes more tech savvy. They will, over time, as younger people who grew up with technology replace the older generation. If the AI bubble pops in a way that causes a financial crisis I think we'll probably see a lot more scrutiny of the tech industry in general, which could lead to progress in quality. Maybe it won't happen, but the current state of things getting shipped before they're done, constant outages, everything being kind of broken all the time... it just doesn't seem sustainable.
> offer them two free years of credit monitoring. Victims aren't going to really use that because it's essentially useless
It's generally actively harmful, and the CRAs fight for this business from breaches because universally, to accept the free credit monitoring you have to sign up for their highest tier credit monitoring package (which can be up to $50/month), supply a credit card, and then hope to remember, a year later, to cancel at the end of the free period, because at that point they'll convert you to a paying customer.
Yep. I had a chance to go for a cybersecurity degree. And every time ive looked at that, the career path is basically an applied insurance job.
Cybersecurity does not make money. They do not raise the profit for a company. Instead, they are compliance, contractual, and legal defences to repel lawsuits and keep data boundaries clean.
And who's the first to go? Groups that dont make money. Like cybersec.
Cybersecurity certainly makes money. The good ones make a lot. I mean a lot.
But if you think you can just study for a year and get some security certificates and call it a day, you're going to be sorely disappointed in the compensation.
OP means internal to a company security ops don't generate add on revenue. Cybersec definitely adds revenue to services and provider companies.
This is true at companies that treat security as a checkbox driven by compliance.
Not true at top-tier tech companies building software in product categories that by their nature have very high security requirements, like Fintech (e.g. Coinbase, Mercury bank, Wise).
Man I have been in 3 data breaches this year and it's only April! My spam folder is going crazy. Security indeed.
[dead]
With Claude writing so much of the software in big companies, Anthropic is well-positioned to eat up SAST, DAST and a lot of the supply chain analysis. EDR and proactive security are still going to be massive businesses, however.