> the Tanstack compromise is very likely to have been the leak vector
....appears to have been backdoored to extract an API key with authorization to read the private codebase.
...
> immediately rotated all required tokens & credentials to prevent further incidents.
Rotating the API key doesn't quite put them in a position to "prevent further incidents" does it? The next PyPI/npm supply chain issue will just get the new key?
I suppose whatever they use that key for should be reviewed and re-scoped if possible?
Does github let you restrict where you can originate requests using a given API key? or are we just not there yet?
Funny to see this after I spent the morning troubleshooting and fixing my crowdsec install on my debian 13 vps. Apparently they stopped supplying a community blocklist to my machine because I'm running the old debian packaged version instead of directly from them (http 500). I had a LLM build a blocklist from publicly available sources rather than tie myself more tightly to their SaaS platform.
On the funny side, reading the website tagline, apparently they claim to know who is attacking you, they just happen to miss out on who attacked them.
Turns out they are not really a security company, just an aggregator of bad IPs. Ideally this kind of aggregator problem is best suited for a trusted not-for-profit company where providing the data needs some level of credibility and querying the data costs you nominal fee to keep the setup floating.
It wasn't the Github that was compromised, it was the access to their private repository that was compromised so somewhere down the line the security best practices are in question for sure. Self hosted repos available on public internet would have met the same fate, may be worse, given github does provides some level of security.
Even regarding the blast radius, I do not really believe any company is honest about it. They do not have tools to verify it, if the user information was accessed with leaked token or real token. The thing that works in their favor is that no one else can verify it either which absolves them from any responsibility. Any platform engineer knows that your CICD system has the keys to the kingdom.
The article you're commenting on says how they were compromised.
> The thing that works in their favor is that no one else can verify it either which absolves them from any responsibility
That's not how this works, at all. You need to have enough evidence that you can confidently demonstrate that there is no sign of a broader breach. If you get sued and can't do that, you're in trouble, because being unable to do that shows that you were acting negligently.
What, exactly, is the definition of a "security company" in your mind? Threat Intel companies definitely fall under that normally, so I'm curious what you think it is.
Also, the idea that this type of thing could just be stood up as a "not-for-profit" company and ran for peanuts is kind of silly. How would the nominal fee pay for the engineers and infrastructure? Or would this just be a volunteer effort and you'd like people to do this for free for you?
Any company where understanding of security practices has a direct impact on its revenue from early phases can be considered as a security company in my view.
From what I have seen a large chunk of internet exists and stands on the shoulder of folks who did the volunteer work cause they were passionate about it and enjoyed that part. Once built, the nominal fee for API to check IP address should cover the costs way easily for the servers.
Letsencrypt is a great example, it did took away the big money from all these commercial CA's, who used to issue blue, green and what not kind of checkmarks. Thats one big reason reason why the migration to HTTPS happened faster.
The basic software is open source, and the list is free if you're running the tool and contributing detections back. They do have some curated lists that you have to pay for.
It's quite a bit less expensive than most other commercial products of this kind that I've looked at.
So I guess your answer is yes, you would like someone to do this work for you for free, because others have done other work for free. And then once the free work is done, you will happily pay a nominal fee to gain the benefit of all the free work?
> Any company where understanding of security practices has a direct impact on its revenue from early phases can be considered as a security company in my view.
This could, quite literally, be any company on earth then? But not CrowdSec, because they had a single security issue? Every company on earth has had those, including every security company.
There is discussion and then there is argument. Discussion is more about sharing and learning where as argument becomes all about proving a point, specifically individuals targeted point of view.
My observation is generic and more about state of things rather than focusing on one entity.
Also lets not belittle the hard work by just labelling it out as free. If only money had been motivation for everyone then the world would have been a different place. And like a lot of people I have done my share of passionate work that provided satisfaction to me and money for others, thats way tangential though
fallacy and apologist point of view here. Shall we dig in?
fallacy is "Oh I guess you mean" .. insert large unsolvable and probably unpopular derailment. Bonus points for aggressive labeling of the opponent being opposed to money.
apologist - "Any company on earth" .. We all stand together, Every Company On Earth .. does this warrent serious replies?
Sounds like an exploit took the credentials needed to extract the code, makes me wonder if a Ubikey + SSL cert for git access would have prevented the entire leak.
We implemented CrowdSec for bot/scraping mitigation. The architecture is sound, but it ended up having an unacceptable false positive rate for us. This may be an issue with any kind of IP reputation approach. After a couple of months of work getting it ready to go I had to turn it off after a couple of days.
We had the main community blocklist and several of their pricey paid blocklists enabled in a PoC capacity. We had a lot of legitimate users end up blocked. In some cases these may have been VPN exit nodes, or users on CG-NAT, or devices on a shared network with some other compromised / bot device. I didn't get 100% of the details, just that we were inundated with support requests from real users that ended up blocked.
Given the number of residential proxies I see scraping the couple of sites I have responsibility for I don't find IP address-based blocking useful anymore. That ship has sailed.
Behavioral and client fingerprint analysis (ugh-- having to run Javascript just to view a static site) is the only way (at least until we get user "age" and identity attestation rammed down our throats).
Interesting, did you implement only IP reputation (via blocklist) or did you deploy the WAF as well? Regarding bot scrapping, you would probably want to try the new bot detection feature recently released
I have written my own honeypots to reduce the false positive rate. I simply have things like a VM with RDP and SSH open to the internet and any IP that tries to login gets banned at the firewall for x days. It works really well.
yet another security oriented company that doesn’t practice what they preach.
On the flip side, there was allegedly no PII leaked. But this event is still a red flag as it means their internal ops are absolutely shit. So it’s another vendor receiving a PNG flag.
I suppose whatever they use that key for should be reviewed and re-scoped if possible?
Does github let you restrict where you can originate requests using a given API key? or are we just not there yet?
Funny to see this after I spent the morning troubleshooting and fixing my crowdsec install on my debian 13 vps. Apparently they stopped supplying a community blocklist to my machine because I'm running the old debian packaged version instead of directly from them (http 500). I had a LLM build a blocklist from publicly available sources rather than tie myself more tightly to their SaaS platform.
Wait … that’s lame. If it’s in Debian repos I expect it to work with full functionality (up to the version number)
I can understand the myriad reasons for why a company would be inconvenienced by that.
But on the other hand, it seems like a reason for me to never use the vendor
On the funny side, reading the website tagline, apparently they claim to know who is attacking you, they just happen to miss out on who attacked them.
Turns out they are not really a security company, just an aggregator of bad IPs. Ideally this kind of aggregator problem is best suited for a trusted not-for-profit company where providing the data needs some level of credibility and querying the data costs you nominal fee to keep the setup floating.
I suppose CrowdSec isn't used to analyze GitHub's traffic, so they wouldn't have much info to go on.
If they had self-hosted their own repos, they might have had more luck.
It wasn't the Github that was compromised, it was the access to their private repository that was compromised so somewhere down the line the security best practices are in question for sure. Self hosted repos available on public internet would have met the same fate, may be worse, given github does provides some level of security.
Even regarding the blast radius, I do not really believe any company is honest about it. They do not have tools to verify it, if the user information was accessed with leaked token or real token. The thing that works in their favor is that no one else can verify it either which absolves them from any responsibility. Any platform engineer knows that your CICD system has the keys to the kingdom.
The article you're commenting on says how they were compromised.
> The thing that works in their favor is that no one else can verify it either which absolves them from any responsibility
That's not how this works, at all. You need to have enough evidence that you can confidently demonstrate that there is no sign of a broader breach. If you get sued and can't do that, you're in trouble, because being unable to do that shows that you were acting negligently.
GitHub Enterprise has at least some level audit log
What, exactly, is the definition of a "security company" in your mind? Threat Intel companies definitely fall under that normally, so I'm curious what you think it is.
Also, the idea that this type of thing could just be stood up as a "not-for-profit" company and ran for peanuts is kind of silly. How would the nominal fee pay for the engineers and infrastructure? Or would this just be a volunteer effort and you'd like people to do this for free for you?
Any company where understanding of security practices has a direct impact on its revenue from early phases can be considered as a security company in my view.
From what I have seen a large chunk of internet exists and stands on the shoulder of folks who did the volunteer work cause they were passionate about it and enjoyed that part. Once built, the nominal fee for API to check IP address should cover the costs way easily for the servers.
Letsencrypt is a great example, it did took away the big money from all these commercial CA's, who used to issue blue, green and what not kind of checkmarks. Thats one big reason reason why the migration to HTTPS happened faster.
The basic software is open source, and the list is free if you're running the tool and contributing detections back. They do have some curated lists that you have to pay for.
It's quite a bit less expensive than most other commercial products of this kind that I've looked at.
So I guess your answer is yes, you would like someone to do this work for you for free, because others have done other work for free. And then once the free work is done, you will happily pay a nominal fee to gain the benefit of all the free work?
> Any company where understanding of security practices has a direct impact on its revenue from early phases can be considered as a security company in my view.
This could, quite literally, be any company on earth then? But not CrowdSec, because they had a single security issue? Every company on earth has had those, including every security company.
There is discussion and then there is argument. Discussion is more about sharing and learning where as argument becomes all about proving a point, specifically individuals targeted point of view.
My observation is generic and more about state of things rather than focusing on one entity.
Also lets not belittle the hard work by just labelling it out as free. If only money had been motivation for everyone then the world would have been a different place. And like a lot of people I have done my share of passionate work that provided satisfaction to me and money for others, thats way tangential though
fallacy and apologist point of view here. Shall we dig in?
fallacy is "Oh I guess you mean" .. insert large unsolvable and probably unpopular derailment. Bonus points for aggressive labeling of the opponent being opposed to money.
apologist - "Any company on earth" .. We all stand together, Every Company On Earth .. does this warrent serious replies?
of course... Ask Fable if the private code is valuable or not...
Sounds like an exploit took the credentials needed to extract the code, makes me wonder if a Ubikey + SSL cert for git access would have prevented the entire leak.
We implemented CrowdSec for bot/scraping mitigation. The architecture is sound, but it ended up having an unacceptable false positive rate for us. This may be an issue with any kind of IP reputation approach. After a couple of months of work getting it ready to go I had to turn it off after a couple of days.
Not sure what you implemented.
They provide several IP blacklists. None of those seem to be false positives. You can also add custom 3rd party blocklists.
They also provide several different rulesets. It is up to you to choose which ones to use and fine tune. LLMs can be very helpful with that.
And there are 3rd party dashboards and tools that help you manage it more easily.
I use the free version as a simple WAF on multiple servers and it blocks a lot of bots. It did require some initial finetuning though.
Are there any better open source solutions?
We had the main community blocklist and several of their pricey paid blocklists enabled in a PoC capacity. We had a lot of legitimate users end up blocked. In some cases these may have been VPN exit nodes, or users on CG-NAT, or devices on a shared network with some other compromised / bot device. I didn't get 100% of the details, just that we were inundated with support requests from real users that ended up blocked.
Given the number of residential proxies I see scraping the couple of sites I have responsibility for I don't find IP address-based blocking useful anymore. That ship has sailed.
Behavioral and client fingerprint analysis (ugh-- having to run Javascript just to view a static site) is the only way (at least until we get user "age" and identity attestation rammed down our throats).
Interesting, did you implement only IP reputation (via blocklist) or did you deploy the WAF as well? Regarding bot scrapping, you would probably want to try the new bot detection feature recently released
This was just blocklist based. We had the main community list and a handful of the curated paid lists enabled.
wrt bot detection - this sounds very much like Anubis which we're also using with some success.
I have written my own honeypots to reduce the false positive rate. I simply have things like a VM with RDP and SSH open to the internet and any IP that tries to login gets banned at the firewall for x days. It works really well.
Not to be confused with CrowdStrike, a serious [0] security company
[0] https://en.wikipedia.org/wiki/2024_CrowdStrike-related_IT_ou...
Security firm hacked.
Security firm putting their source code on Github
A tale as old as time.
yet another security oriented company that doesn’t practice what they preach.
On the flip side, there was allegedly no PII leaked. But this event is still a red flag as it means their internal ops are absolutely shit. So it’s another vendor receiving a PNG flag.
CrowdSec. CrowdStrike.
I was going to register http://CrowdShite.com/ but somebody beat me to it.