In this episode of the Defensive Security Podcast, we discuss the theft of cloud credentials, the exploitation of SharePoint vulnerabilities, evolving malware techniques, and the importance of cyber due diligence for suppliers. They reflect on the challenges of managing secrets, the implications of auto-updates, and the need for robust risk management practices in the face of increasing cyber threats.
Delta’s Lawsuit, SEC Penalties, and Fortinet’s Zero-Day Exploit In this episode, hosts Jerry Bell and Andrew Kellett discuss current cybersecurity issues, starting with Delta Air Lines’ $500 million lawsuit against CrowdStrike over an IT outage and data breach. They explore SEC penalties imposed on tech companies for downplaying the SolarWinds hack’s impact, followed by an analysis of the Black Basta ransomware group’s new method of posing as IT support via Microsoft Teams. The discussion concludes with concerns about the exploitation of a zero-day vulnerability in Fortinet’s firewall manager, highlighting the need for transparency and timely communication from vendors.
In this episode of the Defensive Security Podcast, hosts Jerry Bell and Andrew Kalat explore several pressing cybersecurity topics as of October 2024. The discussion begins by addressing the rapid increase in vulnerability exploitation speeds, with a highlight that 70% of exploitable flaws in 2023 were zero-days, now being exploited within just five days. They stress the importance of effective patch management and prioritization tactics using tools like the CISA KEV list and Tenable’s Viper score. The episode also touches on the evolving nature of automated and targeted exploits, the critical role of timely patching, and the balance between production disruptions and security risks. The conversation broadens to include evolving endpoint security challenges, ransomware trends, and the need for vigilance in adapting to new threats. Additionally, the hosts discuss innovative ways to counter sophisticated attacks, such as leveraging more secure token-based authentication methods over SMS-based MFA. Lastly, the episode delves into how North Korean IT operatives infiltrate companies to steal sensitive data, the implications for remote work, and the importance of robust identity verification processes in hiring. Throughout, the focus remains on adapting to the dynamic threat landscape and continuous reassessment of security strategies.
00:00 Introduction and Casual Banter
00:41 Current Job Market Challenges
02:02 Cybersecurity Landscape Overview
02:20 Google’s Zero-Day Vulnerability Report
04:03 Importance of Patch Management
05:04 Trends in Exploitation Timelines
11:24 Strategies for Mitigating Vulnerabilities
20:03 Red Team Tool: EDR Silencer
26:52 Microsoft’s Ransomware Defense
27:25 Ransomware Attacks: A Decrease Despite the Increase
28:13 The Role of Unmanaged Devices in Cyber Attacks
28:39 Multi-Factor Authentication: Effectiveness and Adaptation
30:07 The Arms Race in Cybersecurity
30:49 The Importance of Phishing-Resistant MFA
32:11 The Rise of SIM Cloning in Ransomware
32:44 Challenges in Adopting Advanced Security Measures
36:46 North Korean IT Workers: A New Threat
40:50 The Future of Remote Hiring and Verification
Episode 282: Exploiting Trust in Cybersecurity Practices In episode 282 of the Defensive Security Podcast, hosts Jerry Bell and Andrew Kallett discuss several cybersecurity topics. They highlight a phishing attack outlined by Microsoft, where cybercriminals leverage file-hosting services like OneDrive and Dropbox to exploit trust and compromise identities. The episode also explores concerns about AI systems, like Grammarly sharing company confidential info, and emphasizes the growing need for well-defined governance policies. They touch on a cyberattack affecting American Water’s billing systems and the potential implications for OT systems. The final discussion surrounds Kaspersky’s decision to replace its software on US systems with Ultra AV, raising alarms over cyber responsibilities and government influence over IT.
In this episode of the Defensive Security Podcast, hosts Jerry Bell and Andrew Kalat discuss various cybersecurity events and issues. The episode opens with discussion on the recent weather impacts affecting Asheville and lessons for disaster preparedness in the security industry. A significant portion of the episode is dedicated to CrowdStrike’s recent Capitol Hill testimony, examining the fallout from their admitted testing failures and the implications of needed kernel access for security software. The hosts also explore an ongoing GDPR violation by Meta related to storing user passwords in plain text, and a hyped but less-critical-than-expected Linux vulnerability in the CUPS printing system. Finally, they delve into potential risks associated with AI systems like ChatGPT and the increasing need for security in OT and ICS environments. The episode concludes with a reminder about the essential nature of cybersecurity fundamentals.
In this episode of the Defensive Security Podcast, hosts Jerry Bell and Andrew Kellett delve into key cybersecurity topics. They discuss a recent statement by CISA director Jen Easterly on holding software manufacturers accountable for product defects rather than vulnerabilities, and the need for derogatory names for threat actors to deter cybercrime. The episode also covers Disney’s decision to ditch Slack following a data breach, and the impact of valid account misuse in critical infrastructure attacks. Additionally, they explore new tough cyber regulations in the EU under NIS2, and a Google security flaw from a Black Hat presentation concerning dependency confusion in Apache Airflow. The hosts share their thoughts on industry responses, regulations, and how enterprises can improve their security posture.
In episode 278 of the Defensive Security Podcast, Jerry Bell and Andrew Kalat discuss various recent cybersecurity topics. The episode starts with light-hearted banter about vacations before diving into the main topics. Key discussions include a new vulnerability in YubiKey that requires sophisticated physical attacks, resulting in a low overall risk but sparking debate about hardware firmware updates for security keys. Another key topic is Verkada being fined for CAN-SPAM Act violations and lack of proper security measures, including exposing 150,000 live camera feeds. The hosts also explore reports showing diverging trends in security budgets and spending, with some organizations reducing budgets while overall industry spending increases. They highlight the need for effective use of security products and potential over-reliance on third-party services. The episode also delves into the growing threat of deepfake scams targeting businesses, emphasizing the need for robust authentication policies and awareness training to mitigate risks. Finally, the hosts reflect on the broader challenges of balancing security needs with budget constraints in an evolving threat landscape.
Links:
https://www.bleepingcomputer.com/news/security/new-eucleak-attack-lets-threat-actors-clone-yubikey-fido-keys/
https://www.bleepingcomputer.com/news/security/verkada-to-pay-295-million-for-alleged-can-spam-act-violations/
https://www.cybersecuritydive.com/news/iran-cyberattacks-us-critical-infrastructure/725877/
https://www.theregister.com/2024/09/05/security_spending_boom_slowing/ vs https://www.cybersecuritydive.com/news/infosec-spending-surge-gartner/726081/ https://www.cybersecuritydive.com/news/deepfake-scam-businesses-finance-threat/726043/
Transcript
Jerry: All right, here we go. Today is Saturday, September 7th, 2024. And this is episode 278 of the defensive security podcast. And my name is Jerry Bell. And joining me today as always is Mr. Andrew Kalat.
Andrew: Good evening. Jerry, how are you? Kind sir.
Jerry: Doing fantastic. How are you?
Andrew: I’m great. Just got back from a little vacation, which was lovely. Saw a lot of Canada, saw some whales, saw some trains. It was
Jerry: Did you see any moose?
Andrew: Oddly we did not see a single moose, which was a bummer. We crossed from Toronto to Vancouver on a train and didn’t see a single moose.
I saw a metric crap ton of ducks though. I couldn’t believe literally in the thousands. I don’t know why.
Jerry: The geese are ducks. Cause
Andrew: We saw a
Jerry: geese are pretty scary.
Andrew: We were sealed away from them, so we were protected.
Jerry: I don’t know.
Andrew: hard to
Jerry: I don’t know. I w I wouldn’t I wouldn’t bet my life on that.
Andrew: But yeah, we saw a decent chunk of gooses, but mostly ducks.
Jerry: Good deal.
Andrew: Indeed. I’m good. Now, catching back up on work.
Jerry: And you’re back.
Andrew: And you are apparently the Southern Command Center.
Jerry: I am for another another day or two.
Andrew: Nice. Never sucks to be at the beach.
Jerry: It definitely does not. No, no bad days at the beach.
Andrew: Nice.
Jerry: All right. A reminder before we get started that the thoughts and opinions we express in the show are ours and do not represent those of our employers.
Andrew: Past, present, or future.
Jerry: That’s right. So our first topic or first story from today comes from bleeping computer. And this one was a bit of a, Oh, what’s the best, a bit controversial, best way to say it, controversial on on the social media sites over the past week. And the title is new leak. I’m not even going to try to pronounce that attack.
Let’s threat actors, clone, Yubikey, Fido keys.
Andrew: Shut down the internet. Shut
Jerry: Shut it down, just throw away your Yubikeys, it’s over.
Andrew: And apparently it can happen from 12 miles away with trivial equipment, right?
Jerry: No, actually, they the bad actor here actually has to steal it and it takes some pretty sophisticated knowledge and equipment. But apparently the equipment they allege are about, costs about 11, 000. However, the the YubiKey actually has to be disassembled, like they actually have to take the protective cover, protective covering off, and they have to instrument it and, and then they’re able to leverage a vulnerability in an Infineon chip that’s contained in these YubiKeys to extract the private key. And so it’s not a, it’s not a trivial attack. You have to lose physical possession of the token for some period of time. But if you were, The victim of this, it is possible for someone, some adversary, who was willing to put in the time and effort could clone your key unbeknownst to you, and then find a way to reconstitute Packaging and slide it back into your drawer, and you would be none the wiser.
Andrew: All seriousness, I think this has a very low likelihood of impacting the average listener to our show or the average person who cares about such things. But if you’re a very high profile target and, some sort of state intelligence service wanted to kidnap you and steal your YubiKey and then gain access to things before those sorts of permissions got revoked in some way, shape or form, I guess that could be viable, but this doesn’t seem like something that would happen to the average person.
Jerry: Oh, a hundred percent. And I still think, despite some of the the initial banter about this, you’re much better off using. I’m sure there are definitely certain use cases where you would be concerned about this, but for the average person, I think, like you said, it’s it’s really not a big deal.
So this does impact the YubiKey 5 series. And I think also the HSM 2 up through that was released, I think it was in May of 2024. The challenge is that you can’t actually update firmware on Yubikeys. That was a security decision.
Andrew: yeah, that seems like a wise security decision if you ask me.
Jerry: Yeah, it’s, I have observed quite a few people who who are now trying to find alternate. Security keys because they’ve been that they feel a little dejected by the fact that you can’t update the firmware on them. But I think it’s important to understand that. That actually is a very important security function, right?
The ability to not muck with the firmware on these keys is very important.
Andrew: right, otherwise a piece of malware could be doing that too.
Jerry: Exactly.
Andrew: Which not be all that happy
Jerry: No. Sad in fact.
Andrew: get the sort of knee jerk reaction to, I want to be able to update this to patch for flaws and such, but keep in mind that everything like that can be used by a bad actor just as easily, if not more easily.
Be careful what you wish for.
Jerry: Yeah. Now what’s interesting is this All of the hoopla around this is about Yubikeys, but the chip, the Infineon chip is actually used by multiple different types of security products, including some EFI. So the secure boot which, I guess at this point, it’s got his own problems already.
And then I believe even after, since this particular article has been written, that there are some other. Actual security keys, similar to YubiKeys that have been identified as also using this Infineon chip. So almost certainly going to be vulnerable in the same way
Andrew: But I guess, nothing to really panic about. But boy, this got a lot of press. A lot of social media traction.
Jerry: it really did. So anyway, I thought it was important to discuss because again, for most people, this is really not a big deal. YubiKey themselves rated the vulnerability as a A CVSS score of 4. 9 to give you an idea. And I think that, that seems right to me.
Andrew: Did it get a mascot?
Jerry: It did not get a mascot. There was some attempts some valiant attempts made.
Andrew: What about a jingle?
Jerry: I haven’t seen a jingle yet either, but it did get a name
Andrew: All right
Jerry: and it has a website. So
Andrew: geez. Okay, so mild panic then. If it’s got a name and a website, that equals mild panic. But got a mascot and a jingle, I’m full on panic.
Jerry: that, what else are you going to, what are you going to do? If it’s got a jingle, you gotta panic.
Andrew: what the tough part is, this is probably like getting traction, perhaps at executive levels who may not have the time or the knowledge to dig into the details and that they’re probably freaking out in certain C suites, but
Jerry: Yeah.
Andrew: send them our show. Tell them these two random guys on the internet said not to freak out.
Jerry: Yeah. I can’t put anything on the internet. That’s not true. That’s right. But, I was I was thinking it’s been a while since YubiKey or UBI has released a new version of the YubiKey. So
Andrew: So maybe this is driving an upgrade cycle. Maybe
Jerry: maybe
Andrew: it themselves. get people to buy new keys. Is that what you’re saying? Jerry,
Jerry: it could be just like how the antivirus companies are releasing all the viruses. Yes. That’s right.
Andrew: that’s some smart thinking right there.
That is, know what? That’s the kind of cutting edge analysis you get on this
Jerry: Thought leadership right there.
Andrew: man. to get out on this. All right, here’s the plan. Let’s spend 20 years making a company and then break our main thing. So people to buy new things.
Jerry: It’s a good idea. It’s solid. I don’t see any any faults in this plan.
Andrew: Hey, how’s that working out for CrowdStrike?
Jerry: We’ll find out soon.
Andrew: Indeed.
Jerry: All right. The next story comes from bleeping computer and title is Verkada to pay 2. 95 million for alleged CAN SPAM Act violations. So for those of you, not in the U. S. CAN SPAM was a law passed a couple of years ago, probably more than a couple of years ago at this point, that Unlike you, what you might expect does actually a permit spam in under certain particular circumstances.
It requires, for example, an unsubscribe link, which this company didn’t do. Verkata, by the way, creates security cameras. There were two big issues, not related here. First one is that their marketing team did what marketing teams do. They went wild with the with their prospect, people who enter, who expressed interest in the cameras, they started spamming the crap out of them without any way to opt out.
And so that was actually the genesis of the 2. 95 million fine. But on the other side, the company had been running around saying that they’re. Cameras are super secure and they are HIPAA compliant and they meet privacy shield requirements and a few other things. And at the same time they got hacked and lost.
Quite a lot of data, some sensitive data, actually video feeds from sensitive places like mental institutions and whatnot things, the kinds of video that you would not want to to have exposed. So in addition to that roughly 3 million fine for spamming, they also have now to appoint a and pay for.
A security overseer for the next 30, sorry, 20 years. And I think they have to report any data breaches to the financial or the federal trade commission within 10 days or you face additional sanctions. So I thought this one was interesting because we’re starting to see a definite trend. At least in the U.
of the government holding companies to account when they make what I’ll call false claims about the, in, in the aftermath, in retrospect, false claims about the security capabilities of their offerings. And so I thought this is interesting. It was really important for people to understand that, if you are going to make those claims, you better not have a problem like this because you’re going to end up in the crosshairs
Andrew: Yes, this wasn’t the SEC. So this didn’t matter if they were public or not. This was the FTC, the Federal Trade Commission. So that’s interesting. Some people would say if I’m not public, I don’t have to worry as much about this sort of liability, but guess what?
Jerry: you do.
Andrew: Looking at the details, 150, 000 live camera feeds were exposed. That’s impressive. And it looks like they didn’t know about the breach until AWS flags is a precious activity.
Jerry: Yeah.
Andrew: So kudos to AWS, but man,
Jerry: It’s not a good look.
Andrew: how would you like to be like that security overseer? How would you like that job? Just, working for the FTC, waiting for companies to come tell you stuff that they screwed up on.
Jerry: Don’t know. It. Could be could be good. Could be bad.
Andrew: Yeah. It’s. It’s interesting because the other thing that’s called out here is that Verkada did not implement basic security measures on its products, such as demanding the use of complex passwords, encrypting customer data at rest, implementing secure network control. So the complex passwords, it’s a little unclear to me if they mean this within their own environment or their customers. Requiring to use complex passwords, which gets back into that whole snowflake conversation we were having on previous episode of how much is a company liable for a customer’s poor use of their security features that are offered. Now, the other aspects here are obviously not, they’re very much for Cata’s choices and how they ran their environment and set up their, it and production environments and such, but doesn’t make me wonder are we going to see more things pushed on ensuring that. These companies are ensuring their customers are being somewhat safe with the use of the tooling.
Jerry: Yeah. I don’t know exactly where this particular issue sits, but I will say broadly speaking, I think the expectation is that as a, Technology provider, whether that’s a service provider or some kind of piece of technology, if you’re going to assert that it’s no quote secure, the expectation is that you have some sort of guardrails on them that feed, for example, mandatory multi factor authentication or mandatory password complexity, and if you don’t, and lots of customers, lots of your customers end up getting hosed because they’re using bad passwords, obviously they have a problem, but I think what.
And what we’re seeing increasingly is that you as the provider also have a problem, despite the fact that it’s based on the choices of the cost of your customers. Look, we can debate whether that’s the right approach or not, but I think that is in fact, what is happening.
Andrew: Yeah. Makes sense. Got to be careful with what you say. If you allege you’ve got good security, at least your marketing and sales people are. And you don’t, and you get bit, there are consequences.
Jerry: Yeah. Yeah, absolutely. All right. The the next story comes from cyber security dive and the title here is Iran linked actors, ramping up cyber attacks on us critical infrastructure. And I, There’s not a whole lot of technical Gorpy detail in here, but they do make reference to a number of different threat actor names like Pioneer Kitten.
And I’m always fascinated by the names that emerge with these threat actors, but it was more that the targets of these attacks are actual security products. So they’re Cisco firewalls, F5s, Palo Altos that are being attacked by these threat actors in a couple of different ways.
This one particular threat actors asserted to be associated with Iran and what they’re doing is they’re facilitating initial compromise, and apparently one of the ways they’re doing this is through exploitation of vulnerabilities in the security products, and then they’re using that access to basically sell that access to ransomware actors, and then taking a cut of any proceeds that those.
Ransomware actors end up getting. I think it’s an interesting approach because again they’re asserting that this is, this threat actor is associated with the the Iran Republican Guard IRA or IRG Who apparently is really in it for the money and not for other, other uses like data destruction or intellectual property theft and whatnot.
Andrew: Iran is a heavily sanctioned country, much like North Korea. They’re probably seeking hard capital. it makes sense. I, as I’ve often stated on this program I’m very skeptical of the veracity of some of these attribution reports. It’s tough to know how accurate they are because there’s no way to check easily. Somebody says it’s so and okay, based on what and how do you know and how certain are you and how do you know it’s not another actor trying to look like that actor? So I’m always quite skeptical. I’m just cautious about buying into this, but boy, is this such a focus of the industry of trying to attribute a certain actor and it’s interesting.
It makes for good fodder for conversation. And, at the executive suite, I think they like to know that, but I’ve said a few times, and I’ll probably say a few more times. As a defender, I don’t know that I care. I think I care about the TTPs. I think I care about the typical tactics and the typical techniques and typical approaches, but it’s coming from Iran or Billy down the street, really change my job too much, I just need to know what they’re up to and what I need to defend against. That’s not the point of your story. I know I just went off on a rant.
Jerry: I will say, so back to why I thought this was interesting when I, back when I was working. I had observed quite a few especially smaller organizations getting compromised and ransomware as a result of running older, vulnerable the edge protection devices like Cisco’s and Palo Alto’s and whatnot.
And it. It seemed to help in the spirit of quickly identifying or investigating the breach to understand what they were likely doing. And so if example, if you’re you’re Palo Alto firewall gets compromised while they’re You know, the first thing to look for is evidence that somebody’s trying to move laterally and deploy ransomware because that’s probably what’s going on.
But I’m broadly speaking, I’m with you, like you’ve got a, you have to be threat actor agnostic. You need to defend your environment regardless of who’s trying to get in. But I think, what I am. Really trying to impart in here is that I think as an industry and maybe not more sophisticated companies, but broadly speaking I think as an industry, we do a pretty bad job of maintaining certain really Key pieces of our infrastructure, like those firewalls, like I for whatever reason, I don’t know why, like we’ve, I have observed small companies keeping like their workstations patched and, their servers patched, but for whatever reason, It’s really common for their firewalls to be end of life.
Andrew: Yeah, it’s, I think there’s a lot of costs or a lot of reasons. I think cost is one, I think interruption of business and downtime perceptions of patching network gear is another, I think in general, network gear is not thought about as something needs to be patched very often, unlike general purpose computers. But I agree it’s a problem and we’ve seen, it goes Absent flows, and we get troughs and peaks of how often we see these sorts of devices have problems like this. And it seems like lately we’ve had a big spike in remote access VPN type technologies having pretty serious vulnerabilities. And what’s always scares me, what I think about is, okay, you’re 28 patches behind and something serious pops. That’s so much more disruptive to patch up than if you were on a more current patch level and you’ve got to Deploy a patch to fix a serious vulnerability So it’s there’s a hygiene aspect of keeping up to date on this stuff that I think makes your life easier in a crisis
And but also to your point It’s also very interesting that These types of actors get initial access and then broker that out and sell that and we’re not catching them at the initial access phase They can dwell without much notice You So that’s another interesting problem that we should get better at.
Jerry: Yeah. I think in part, it’s because the, these even well instrumented organizations don’t. They’re instrumented, especially at the edge to look for how best to describe it, to look for malicious things that are transiting through the firewall and not necessarily attacking the firewall.
I’m not saying that well, but And that my, my experience was a lot of these technologies don’t have unto themselves, things like EDR type capabilities to tell you that something is going horribly wrong on the firewall itself.
Blind to that.
Andrew: Yeah. Or they have log events that are obscure enough that your SIM wouldn’t know how to make heads or tails of it without custom alerting and. Yeah, when somebody is breaching something through some sort of vulnerability, how it reacts is highly unpredictable, hence it’s a vulnerability, hence it’s doing something it wasn’t designed to do. So it’s not that simple. We usually have to look for some sort of second order impact or or movement, like you mentioned, lateral movement to deploy ransomware to detect that intrusion often, sadly.
Jerry: Yes, indeed. All right, moving on to our next story, this one comes from the register and the title is security boom is over with over a third of CISOs reporting flat or falling budgets.
Andrew: Back it up. Let’s go home.
Jerry: Yeah, it was fun while it lasted which, by the way, I normally we don’t talk about that sort of thing, but I thought it was very interesting because it’s contrasted with another story from cybersecurity dive, which is titled InfoSpec spending to hit a three year growth peak, reaching 212 billion next year.
So the registered story is based on a survey from INs and the cybersecurity dive story is based on a report from Gartner.
Andrew: Which I’m assuming is also based on a survey from Gartner
Jerry: Yes,
Andrew: list of customers and whatnot.
Jerry: Now, when I was first contemplating this, because look, I, I’m pretty involved socially and I see a lot of people struggling to get work in the security industry right now. And we don’t have a story about it, but the US government recently announced that it’s going to Put a bunch of money into training people up to be cybersecurity workers, because there’s this, dearth of unfilled security jobs.
At the same time, we got a lot of people who can’t find work in the security industry. So I think that’s a, it’s an interesting dichotomy. And then, which kind of aligns with. The story from the register, but is antithetical or in opposition to the Gardner report. Now it occurred to me though, that they’re actually maybe saying the same thing.
Or at least they’re not mutually exclusive. I think what the IANS report is saying is that in general security budgets, budget growth is slowing some places it started to stop. And in some places it’s even starting to go down, particularly as it Pertains to hiring new people. And the Gartner report is really talking about industry spending.
So how much are you, how much are companies spending on security products and services? And it occurs to me that when you put those two things together, you have less money to spend, but the money that you do have, you’re spending on third party services and software means that you have less money To grow your team.
Andrew: Oh, CapEx versus OpEx. Although, gone to SaaS, so it’s all OpEx now these
Jerry: It’s not a lot of packs. Yeah. But I think, I, so what, one
Andrew: I could be
Jerry: of the concerns I had is a, as a CISO is, when you buy a thing, whether it’s a SAS thing or on prem thing, You actually have to do stuff with it. Like you have to have, it’s it’s an obligation. You’ve adopted a puppy. You have to care for and feed that puppy. And in the worst case, look at what happened to Target like many years ago now, probably almost a decade ago,
When they got hacked and all of the logs that indicated they had been hacked was like sitting there, but nobody was looking at them.
And so I’m concerned that as an industry, we’re becoming very enamored with technology and less so on the ability to actually use that technology. Now that’s a, maybe a naive and uninformed view, but that’s the benefit I give to the show.
Andrew: It’s fair. No, I, of my strong principles in the teams I run is try to develop mastery over your tools. to understand what’s normal, what isn’t normal, tinker with them, get to know them, play with them, know how to interpret what they’re saying. The challenge with that is you need a lot of free time and you need a lot of initiative and self starting mindset to go do that. And if you’re being pulled a thousand different directions, it’s hard. So you’re very reliant on the tool, raising the alarm, not sensing something is a little off based on familiarity with the tool or the environment or the situation. So I do think we’re also recovering still, or Adjusting to a higher interest rate environment.
I think when we were in zero interest rate environments, at least in the U. S., a lot of tech companies hired a ton of people. And so we’ve seen the slow motion layoff wave for the last year or so, or two years, as a lot of these companies. Probably overhired and with interest rates going up part of the goal those higher interest rates is basically slow down, industry growth and spending and make money more expensive and it’s working like companies slowing down. A little bit, and so they’re slowly laying off here and there and adjusting. And I wonder if that’s part of it to a lot of this is a number of companies with their budgets, taking a little bit of a pause as they’re adjusting, or their staffing levels come down a little bit, or. What not? The other thing that this reminded me of while I’m ranting on this is this consolidation push, which happens about every 5 to 10 years that I’m seeing. And then reference it a bit in one of these articles. Consolidation around certain tools or certain vendors and trying to do functionality in one, one particular solution for one particular vendor seems to be hot right now with executives. I don’t like about that is typically what will happen is a given solution or tool will be very good in one area. And then to gather and grab some of that consolidation money and increase their footprint, they’ll start expanding into other areas. But typically the offerings in those other areas are substandard.
They’re not great. They’re certainly not, of breed.
Jerry: Almost checkbox, right?
Andrew: Exactly. They are a checkbox coverage Of that other bit of functionality and usually sucks compared to the best of breed out there. But what is it’s enough to convince an executive. They don’t need a separate tool that they can gain efficiencies in that consolidation.
But I think we end up with of substandard tools. Then in these, offshoot areas that these other tools are growing into. Now they may get better over time, but typically I think that nuance is lost we are pushing to consolidate of, yeah, it says it does this, but how well does it do it in this other area?
That wasn’t the core purpose. We bought this particular tool or vendor to cover. I think some of that may be going on too. I see, I hear a lot of that push right now to consolidate and get down to less vendors and simpler platforms, but I think you risk losing some functionality and capability when you do that
Jerry: And I think that’s been a long term IT trend that is now crossing over into the security world. If you look at companies like HP and Oracle and IBM and others, they have a lot, they have a few best of breed things and they have a whole lot of checkbox crap and they’re, their sales tactic is to, convince company executives that look, you don’t have to go everywhere.
You can get everything here,
Andrew: and it works, which is helpful because our show has a couple of best read episodes and a bunch of checkbox episodes, so I’m. glad that’s acceptable to the industry. The other thing that I pulled out of one of these articles, I think it was the register one. I’m going to quote it. Quote, an encouraging sign also is that security spending as a proportion of the overall IT budget is on the rise up from 8.
6 percent in 2020 to 13. 2 percent this year. This trend looks set to continue, Kowalski opined, but still security spending was typically less than 1 percent of the revenue of those I always find that interesting. And I know what’s going to happen is a lot of CISOs and CFOs are going to look at that and go that’s what I should spend. 13. 2%. That’s my target.
Jerry: Yep.
Andrew: Regardless of their circumstance or their situation, companies love to try to measure against the averages around them and use, in theory, wisdom of crowds to measure what they should be spending on security.
Jerry: Yeah. That is the normal benchmark. That’s the normal playbook for you, whether you’re a CIO or CISO coming into a company, like that’s the gold standard is benchmarking your competition.
Andrew: It’s safest. It’s easiest. It’s an easy button. It’s, you can’t get criticized if you’re doing what’s average for the industry.
Jerry: Right,
Andrew: I think that’s a bit of a shortcut, a bit of a easy button, naive way to look at it. Like you don’t like your risk tolerance and your environment and your situation is very different from your competitor
Jerry: yeah, it
Andrew: got
Jerry: assumes text to text parity and lots of other
Andrew: Right.
Jerry: that are almost certainly not the case.
Andrew: So just something I pulled out just to riff on for a minute there that I worry about that being used as. Now the flip side is if you’re well below 13. 2 percent of your, your IT budget, you can go use this as evidence to, to fight for more budget.
Jerry: That’s certainly true. You’re not gonna get it in this market. It is I think if we take a big step back, The reality that we find ourselves in is that there’s a lot of pressure on spending, and I certainly felt it as a ciso. And I think that many people in that role feel the same and is born out in the in the ions report that, we’re being asked as an industry to do more.
With either the same or with less, the threat landscape is certainly not getting any better. It’s getting worse at a faster pace as we often talk about here. And from a, a personal liability standpoint, I think it’s also getting, At least in the U. S. getting more complex companies in their officers and their executives are now starting to be personally held liable in certain, edge case instances.
And so that’s there’s a lot of, I think, really fundamental changes afoot. But if you look at the big contour. There is an expectation of driving efficiency. And I think that what, just going back to my time as a CISO, it’s difficult for an organization to commit to hiring a person.
It’s easier for an organization to commit to spend, to signing a contract that lasts a year or two years, because, you can You know how much it’s going to cost. You can choose to not renew it. You can perhaps get out of the contract in, in the middle of it. And so that I think drives some of what has is being borne out in the Gartner report that we are starting to see companies trading off people for services, but as we’ve talked about in the past I still think that Especially if you talk, even like managed security services, I think it’s very difficult to take commodity services and have them be really effective in absence of some kind of an abstraction layer.
Like you, I think that. The place that we have to get to is in industries, figuring out how to optimally run vended services and our internal it I’m struggling with the right way to convey it, but I think we’ve got a long way to go, but my, Big concern is that we’re going to overshoot the mark, like that, that the, we’re going to cut too far
Andrew: Sure.
Jerry: and as security leaders, we’re going to end up getting exposed.
And as we started to see, that the employee, our employers are probably not going to have our backs.
Andrew: and the indicators you’ve cut too far are usually very lagging indicators that you’re not going to know right away. And it’s very difficult to know until you do a post mortem after a massive breach, why? And often those decisions make sense in the moment. And it’s a very tough job justifying a security ROI, which goes back to, Hey, what’s the average of the industry doing? Okay. That must be safe.
Jerry: I think even. I think even that is is a little perilous because, it’s, I guess it’s one thing if you feel like you’ve got all your bases covered and, you’re trying to decide if you want to, take the next step of maturity or what have you. But most companies have a big risk register.
And I always worry that those risk registers are gonna be exhibit A.
Andrew: Jerry, if it’s on the risk register, nobody can attack it.
Jerry: True.
Andrew: rule.
Jerry: I forgot about that. I’m sorry. You’re right.
Andrew: It’s out of bounds. If it’s.
Jerry: It’s out of bounds.
Andrew: No, you’re right. What’s worse having it on a risk register or not knowing your risk, I think is the first step.
Jerry: I completely agree. I guess my, my, my point was it’s hard. I think it’s very difficult, especially in some kind of a, litigation or regulatory situation to be able to justify having gnarly looking things on a risk register at the same time you’re cutting your budget. That’s my that was my point.
Andrew: Yeah, that’s fair. But just to play devil’s advocate as a CFO or a CEO. Hey, like I’m trying to keep my company afloat. I can’t just throw money at security.
Jerry: It’s certainly true. And I guess that kind of comes back to the risk tolerance, which, I guess I think one of the one of the things that’s really important. Problems that we have is what is, what does it mean to have a risk tolerance, right? Because
Andrew: Yeah.
Jerry: there seems to be like this
Andrew: And is it well understood and truly internalized by the executives making that decision?
Jerry: correct. And so in the past, okay like I, I’ve accepted a risk and no, gosh, something bad happened that. The thing that I accepted was exploited and we had a breach and, we got our hands slapped or we got some negative press. And then you move on, but now it’s okay.
And you’re going to get personally sued or perhaps go to jail or like that it’s the dynamics I think are changing. a bit. And so the, this concept of accepting the risk, I think is starting to take on a different flavor than it has in the past. And it’s probably candidly overdue. Because look, as society marches on and becomes more and more online and digital and reliant on, on on your Personal records and personal data, the sensitivity and the harm that can come from that information being stolen or maliciously used is it’s becoming much more perilous for the.
The people whose data that is exposed. And so it makes sense. But I’m not sure that we’ve broadly speaking, embraced that yet.
Andrew: No, it’s I agree. Problem. There’s always has been a tough problem and it’s constantly evolving and you go back to, okay, then what’s the framework I can use? What’s the standards I could measure against. How often are those finding the last set of problems? It’s. constantly see these sorts of approaches come into play that, whether it’s PCI, just as an example of, yeah, we’re completely PCI secure. Great. You still got hacked, right? We’re following the NIST cybersecurity framework. Okay, but how well? And there’s so much complexity.
It’s not easy. It’s really tough. getting the basics right is really tough. So it’s, I don’t know. It’s a tough problem. I, probably we should start caveating this show with the advice in the show is for entertainment purposes only so we don’t get sued.
Jerry: That’s a good point. It’s a very good point. Anyhow, but time marches on technically, the risk marches on and then the next story is a good example Of that. And this one also comes from cybersecurity dive. And the title is deep, fake scams, escalate hitting more than half of businesses.
Andrew: That was an amazing segue, by the way.
Jerry: I’m trying like I’m
Andrew: was, that’s years of work
Jerry: Right there. I’m done. Like I,
Andrew: Almost professional.
Jerry: yeah, a couple, like I say, just a few hundred more episodes and I’ll actually know what I’m doing here.
Andrew: Anyway
Jerry: It’s not been all that long that the CEO, the CFO, CEO, business email compromise concept has emerged. It started off probably 10, 10 ish, 15 years ago where, the CFO would send an urgent email to their accounting team saying I need you to wire a bunch of money to to this business email.
Bank because we’re buying a company and super secret. You can’t talk to anybody about it, but you know what, you got to do it immediately. And those for a period of time were pretty effective. And we it’s a community we adapted a bit. We built some processes and it still happens by the way, like companies are still falling victim to fairly unsophisticated attacks.
But, one of the things we said was like, you got to make sure that you know who you’re talking to. Is that email really from your CFO? And then that, that morphed into deepfake audio calls where the CEO, somebody would alter their voice. So it sounded like the CFO and for a period of time that was that was somewhat effective.
And now the next iteration of that is deepfake video where the CFO is getting onto a Webex or a zoom meeting with you. And. They’re face to face asking you to to transfer money and
Andrew: interactive video.
Jerry: right.
Andrew: Yeah, wild.
Jerry: And that’s what this is about. They’re saying that they interviewed a bunch of not IT or security people, they interviewed finance people.
And they said that of the 1500 people they interviewed, they said 85 percent view, this is an existential threat. They identified that about Roughly half of companies have I’ve been targeted and then of those that have been targeted, about 43 percent have fallen victim, which is a big number.
I don’t know how well that extrapolates out. Obviously they didn’t interview every company in the world, but still that seems like a big number. And when you tie that, by the way the Gartner report that we just talked about, the headline on that report was actually quite interesting. Because it, it said Gartner forecasts, I’m sorry, wrong headline.
Gartner predicts that by 27, 2027, so Three years from now, 70 percent of cyberattacks will involve generative AI.
Andrew: That’s wild. My toothbrush now has gender of AI, so sure, but
Jerry: I think what they’re, I think what they’re, I think what they’re saying is, it’s going to be things like this where they’re they’re using generative AI to create phishing lures and to create deep fakes and whatnot. So I don’t know that,
Andrew: sense.
Jerry: I don’t know that like they’re going to, I don’t think that they’re predicting that LLMs are going to start coming up with novel ways of hacking into your pal about the firewall, though, maybe that happens.
I don’t know. I think it’s more like coming up with creative ways to execute kind of old style attacks.
Andrew: You won’t have the badly translated poor English messages any longer.
Jerry: That’s true.
Andrew: in proper American English. It’ll sound normal and appropriate and you won’t have that trigger of, weird verbiage that sounds like a non native English speaker, which is usually a good tell. So that’s one thing that’ll make it a little harder for sure.
Jerry: So there was a report published in May. Yeah. By the, in this article makes reference to this report by the big four accounting firms, including Deloitte, that said that by 2027 fraud losses from generative AI, they expect to reach 40 billion. It’s
Andrew: That’s wild. And honestly don’t know the answer to this, and I probably should, and I feel bad that I don’t know this, but in a personal Fraud case. Let’s say somebody scams you into transferring money with Zelle, for instance, like a personal fraud, your personal homeowner’s insurance and your bank.
I’m like, no, sorry, we’re not covering that. You authorized that transaction. Yeah, you were scammed. It was and for us in the industry, we’re looking at that was a, that was basically stealing money. But right now Those banks and vendors and such are like, no, sorry, you were, you authorized it.
It’s an authorized transaction. We don’t own the liability. I wonder if that’s the case with cyber insurance. If you fell for, or were a victim, I shouldn’t say fall for, because that makes it sound like you’re at fault. If you were a victim of this sort of scam. Would there be coverage or would it be the same sort of story of no, you authorized it.
This is an authorized transaction. I don’t know. I really don’t.
Jerry: a good question. I I wonder if it would fall into I’m not a, not an insurance expert. But I do wonder if it would fall into more like errors and omission,
Andrew: Could be.
Jerry: Than cyber insurance, but I don’t know. That’s a, it’s a good question. Very good question.
Andrew: dozen or so listeners might know.
Jerry: Yeah. Hit us up. Let us know if you know the answer to that. So in any event, I. Now, this is certainly a rapidly evolving area. I know that there are a lot of people who are very bearish on generative AI, and there’s some valid ish reasons for that, but I don’t think it’s going away. The fact that you may not think it has a lot of utility and then it is unsustainable from a ecological perspective.
All that stuff can be true and it doesn’t go away. So I think that’s probably where we’re at and certainly from a, from the perspective of adversaries, I think we’re on what is likely to be a dramatic uptick in using this technology in adversarial fashion.
Andrew: Have some decent advice, which is the same thing I would ask you, which is educate your finance teams, your executive teams about the risk. Show them examples. Show, give them If there’s one thing to take away, it’s look for that sense of urgency. That’s what’s usually very common in these scams and it’s very effective psychologically, but if you can get them to understand, if someone is pushing you for a sense of urgency, be suspicious. the other thing is have processes that cannot be deviated from, which inquire, require. people to authorize and following a process. So if somebody is asking you to deviate from the process, that should be a red flag. And I think those are very common techniques that happen with these scams. So I think that’s a key. And I do think because there’s only a small number of people who can control finances in a company, can go a long way. It’s not like you’re trying to broadly target your entire population against, phishing attacks. You’re trying to get a few people control the purse strings know about this and be wary of this and be wise.
Now the flip side is it adds. It adds friction, right? You’re saying, if you have your boss legitimately coming to you and saying, I need you to do this. And you say, no, because our policy is X, Y, Z. That’s a little uncomfortable. That’s probably going to slow business down, but it’s one of the only ways probably to protect yourself against these types of scams.
Jerry: absolutely. Absolutely. I’m concerned about what other ways we’ll see this materialize.
Andrew: I actually am not here. This has been a deep scam
Jerry: You’re just an, you’re just an LLM.
Andrew: This is actually one of my cats and probably doing a better job than I can do.
Jerry: I I think. I was about to say that
Andrew: Look, hey, results, I’m results oriented. The show got done. Speaking of, where’s Betty? Stop hopping up on my desk. We’re not in video yet. But soon, we might be.
Jerry: getting close.
Andrew: random cats wandering through the shot.
Jerry: Yeah, it’ll be deep fake video.
We’ll be like we’ll pick different celebrities to portray us, but it’ll be okay.
Andrew: Ugh. What is happening? have no idea.
Jerry: Anyway I think this is one area to watch. I, I. I certainly think this particular area, like you said, there are some definite ways that we can avoid losses, but I am also concerned about ways that we maybe haven’t even thought of or seen exploited yet, for more broad consumption in, in, in other types of roles, for example, a lot of Organizations, even the ones, by the way, that work in person meet virtually.
And how long will it be before we start reading about you people sliding in adversary, sliding in and portraying someone and, stealing intellectual property or,
Andrew: And
Before the Zooms, the Google Meets, and the WebExes of the world offer an enhanced package with anti deepfake for a small, multi thousand dollar per month fee?
Jerry: Hey, there you go. We should patent that.
Andrew: We should.
Jerry: Brilliant.
Andrew: You know it’s coming. You know it is. It’s gotta be. It’s inevitable.
Jerry: Yeah. And it’ll be it’ll be LLM, generative AI based. Yeah. Two. So
Probably with some blockchain and cloud thrown in, I’m guessing.
Andrew: How could it not be?
Jerry: I don’t know.
Andrew: It’s a table stakes.
Jerry: We’ve I think we’ve hit terminal altitude and now we’re headed back down. So I I think that’s it for today. I certainly appreciate everybody’s time and attention.
Hopefully you found this interesting. If you like the show, give us some love on your favorite podcast app. We definitely like that. It helps other people find the show. If you. didn’t like it, you can also give us a positive rating because that still works. Like you, you don’t have to give us a bad rating.
You could give us a good rating and then just not listen again. That works too.
Andrew: It’ll encourage us to get even better.
Jerry: That’s right. Absolutely.
Andrew: We’re
Jerry: So you,
Andrew: positive reinforcement here.
Jerry: Not negative reinforcement, positive. All right. You can follow the show on our website at www. defensivesecurity. org. You can follow us on, you can download the podcast on just about every podcast platform there is out there now. I don’t know that we’ve missed any, although if we had, love to hear about it.
You can follow Mr. Callot on the social media at where
Andrew: I’m both on X, formerly known as Twitter, and InfoSec. Exchange, Jerry’s fine Mastodon service at LERG, L E R G, which someday I will explain, but not today.
Jerry: which is by the way, the best the very best in social media. So you can find, follow me on infosecular. exchange at. At Jerry at InfoSec that exchange. And with that, we bid you adieu. Have a great week, everybody.
In this episode, Jerry Bell and Andrew Kalat discuss various topics in the cybersecurity landscape, including the influence of cyber insurance on risk reduction for companies and how insurers offer guidance to lower risks. They touch upon the potential challenges with cybersecurity maturity in organizations and the consultant effect. The episode also goes into detail about issues surrounding kernel-level access of security tools, implications of a CrowdStrike outage, and upcoming changes by Microsoft to address these issues. They recount a case about a North Korean operation involving a laptop farm to gain employment in U.S. companies, posing major security concerns. The discussion highlights the pitfalls of relying on end-of-life software, especially in M&A scenarios, and how this could be a significant vulnerability. Lastly, they explore the massive data breaches from Snowflake and the shared security responsibilities between service providers and customers, emphasizing the importance of multi-factor authentication and proper security management.
Jerry: Here we go. Today is Saturday, August 24th, and this is episode 277 of the defensive security podcast. My name is Jerry Bell and joining me today as always is Mr. Andrew Kalat.
Andrew: Good evening, my good sir Jerry. How are you?
Jerry: I am awesome. How are you?
Andrew: I’m good. I’m good. I’m getting ready for a little bit of a vacation coming up next week So a little bit of senioritis. If I’m starting to check out on the show, you’ll know why
Jerry: Congrats and earned. I know.
Andrew: Thank you, but otherwise doing great and happy to be here as always
Jerry: Good. Good deal. All right. Just a reminder that the thoughts and opinions we express on this show are ours and do not represent anyone else or including employers, cats, relatives, you name it.
Andrew: various sentient plants
Jerry: Exactly. Okay. So jumping into some stories today. First one comes from cybersecuritydive. com, which by the way, has a lot of surprisingly good content.
Andrew: Yeah, I have enjoyed a lot of what they write. We’ve a couple good stories there
Jerry: Yeah. Yeah. So the title here is insurance coverage drives cyber risk reduction for companies, researchers say that the gist of this story is that there were two recent studies done or reports released one from a company called Omeda and another one from Forrester, which I think we all know and love.
And I’ll summarize it and say that they’re both reports indicate that companies which have cyber insurance tend to be better at quote, reducing risk more likely detect, respond, and recover from data breaches and malicious attacks compared to organizations without coverage. So I thought that was a little interesting.
On the other hand it to me feels like a bit of availability bias, so by that, what I mean is if you go and take a survey of people who go to the gym and work out at the gym on their diet, you will probably will find out that Eat a healthier diet than the public at large.
Andrew: But I go.
Jerry: you just go.
Andrew: I, look,
Jerry: I’m not saying, I’m not saying everybody, right?
Andrew: least I show up, right? And I’ve been told showing up is half the battle.
Jerry: It is half the battle, that’s right. Knowing is the other half.
Then doing is the other half.
Andrew: I will say, speaking of G. I. Joe quotes, I thought catching on fire was going to be a far bigger problem in my life than it turned out to be.
Jerry: That and quicksand.
Andrew: I, we were
Lot about that as children of
Jerry: quick, quicksand.
Andrew: Heh.
Jerry: QuickSand was, I, I lived in fear of QuickSand, but it turns out it’s really not that big of a concern.
Andrew: For as much as I heard stop drop and roll done it
Jerry: Yet.
Andrew: That’s true. The day is young. Anyway back to your story. I think you’re right I will also say having worked with a number of these companies do interestingly have their own towards trying to keep you from getting hacks. They have to pay out So they do push certain things like and I’ve seen myself and I won’t say it You know, it doesn’t matter where, when, but if you have things like one of the well known EDR tools well deployed, they might cut you a rate on or a break on your rates. Because they have their actuarial table saying, Hey, if you’re using certain bits of technology that lowers your risk of usually ransomware, right? So they
Jerry: Sure.
Andrew: seems to me, my opinion is that these insurance companies feel that some of the well known EDR brands in a Windows environment It is very effective or decently effective at stopping ransomware, therefore they’re less likely to pay out, therefore they lower your rates. So there might be some of that too. They do to give companies guidance on what they see across their industry to reduce risk.
Jerry: I think that, that makes sense. I’ll say, on, on one hand, like I was saying before, I think companies that buy cyber insurance are probably maybe more mature, more invested in, protecting their environment than others. But I think that there’s also this consultant effect when when you want to drive change and whether whatever kind of change that is, reorganizing revamping your security program, justifying additional expenses for anything outside guidance, typically Carries a lot more weight than something that comes from internal.
Andrew: Sad but
Jerry: and so I think, yeah, anybody who’s been in the industry for a long time or really any amount of time knows that, especially this is a, the CISO trick, right? When you come into a new organization as a CISO, the first thing you do is you go off and you hire a, a big name consultant.
You burn a half a million bucks on a consulting engagement. And at that point, it’s not you telling the company, Hey, we’ve got to spend a bunch of money to improve our security program. It’s some, hard to argue with independent third party who is making that assessment. And to some extent you argue with that at your own peril, right?
Because now it’s it’s a, it’s an assessment that becomes exhibit a, if something goes wrong and which is, both a blessing and a curse. But my experience is it certainly helps a lot. And I think that this cyber insurance and their somewhat prescriptive guidance and expectations around the kinds of controls and technologies you need to have in place is a very similar kind of thing, right?
If you’re engaging with them, they’re going to be opinionated on what you should and shouldn’t be doing and and then like a consulting engagement. It’s a third party giving you that guidance. And so I think that tends to carry a lot more weight.
Andrew: Agreed on all points. The only caveat I would say to that is sometimes these recommendations that come from some insurance companies are not customized typically to your particular risk environment or situation. They are very broad approaches to reducing risk across many different types of environments with many different types of risk profiles. Technology stacks and all that sort of stuff. So they’re very somewhat generic recommendations, I think.
Jerry: I think you’re probably right. In any event, it’s I thought it was I thought it was quite interesting. Certainly having that insurance can help. I will tell you in my time as a CISO in dealing with customers and to some extent business partners, there was a I would say a growing expectation that you have to have cyber insurance.
Actually, I experienced firsthand quite a few customers actually writing into contracts. That you have now, I don’t know how far and wide that permeates the industry, but I think it’s probably becoming a lot more common these days because, companies have this interdependence and so it’s not necessarily just like a cloud service provider where that kind of thing can manifest, look at over the, what now, 12, 13 years we’ve been doing the show.
How many times have we talked about a company like, let’s say, Target or Home Depot getting hacked as a result of something happening with one of their suppliers? And so I think, as time goes on, we’re going to see that becoming kind of table stakes to, to have these business relationships, especially with larger and more mature companies.
Andrew: Why do you think that is, what do you think that the third party is assuming that you will get from that insurance? Just so you have the ability to recover from an incident and sustain As a going concern or that they assume that if you have insurance, it’s coming with requirements that level up the maturity of your program or what value do you think that third party sees in their business partner having cyber insurance?
Jerry: That’s a great question. I think it’s both, actually. I think there is this, naive view that if, if something bad were to happen this insurance would, provide that buffer. It would make sure that, the company didn’t go out of business, but the reality is that, especially, if you look at some of the really large hacks.
can happen with relatively small organizations who are, I would say fairly highly leveraged, at least in terms of their insurance policy. So yeah, it’s great. They may have a 5 million insurance policy, but if they hit if they’re, let’s say a, a hundred million dollar company and they get hit with a, 50 million in breach fees, their 5 million in insurance coverage, isn’t really going to go very far.
So I don’t know that it’s extraordinarily useful in terms of protecting customers from harm. I think there’s a facade that it provides. And I also think it does give some, at least a segment of, roles at companies gives them this warm, fuzzy feeling that somebody else is looking over their shoulders.
In that respect, it’s not different than like a sock to or an ISO, SOA or what have you.
Andrew: I wonder if there’s some sort of implied, Hey, you’ve ransomware you can recover faster. The other thing I think about is the perverse incentive. So when we look at an insurance in general, it’s to shift risk. It’s to shift
Jerry: I
Andrew: risk to a third party. So is there the risk that a executive committee will say, Hey, we don’t need to invest in much in cybersecurity because we have insurance if something bad happens.
Jerry: mean, I would love to sit here and say no, that’s that would never happen. But I don’t think it happens that every organization, but I definitely expect it happens more than it should.
Andrew: Yeah, it’s interesting. It’s interesting interplay of competing priorities. When you start to introduce these sorts of things and how what sort of behavioral economics comes into play
Jerry: Yeah, absolutely. All right. Anyway, go go talk to your insurance carrier and it might might help you with your internal program and justify additional improvements to your program. Our next story comes from Ars Technica and the title here is Crowdstrike.
Unhappy with shady commentary from competitors after outage
Andrew: I’m shocked. I say shocked
Jerry: Totally surprised by this so we’ve talked about this Several times and i’m sure we’ll talk about it several more times CrowdStrike obviously had a pretty devastating Snafu With one of its products that caused probably the largest single meltdown of I.
T. in history and a lot of their competitors have been capitalizing on that outage. And so now this story is talking about in the wake of some of the back and forth tit for tat. mudslinging that’s been going on. I think they call out Sentinel one in particular. CrowdStrike is, I think, getting a little peeved at how their competitors are behaving, basically saying, hey, this could have happened to anybody.
And I think there’s a lot of differing opinions in the industry based on my experience and exposure to different, to, the industry. I don’t think everybody’s on that bus. I think there’s a lot of people who think that, no, this really would be a lot less likely with other companies. Although it is interesting that SentinelOne is, is one, I think one of the more aggressive mudslingers, but they also, by the way, as far as I can tell, do use they do access windows The kernel.
And in fact, the next story we have actually talks directly about that.
Andrew: Yep they do and this goes back to something that I’ve I don’t have expertise in so I’m just dancing around and pontificating at something I can’t be authoritative on but I think what I keep seeing is that most security tooling feel that they need to be in the Windows kernel to be effective on the way Windows is architected today. it’s interesting when they talk about they being various competitors of CrowdStrike talk about safer methodologies, whatever that means, and I think somewhat that implies perhaps not operating at the kernel level. However, safer in terms of not causing an outage per se, but are they as effective at spotting and stopping malware? I don’t know. I, my assumption is there’s always some sort of trade off. If we’ve got most of the industry wanting to operate at the kernel level, and we’ve got another story that talks about this a little bit, and Microsoft themselves is talking about maybe we can find ways to make this effective. seems to me as not a, not having worked at those companies that, but Operating at the kernel level allows these security tools to be more resilient against malware trying to shut them down, and in theory be faster and more effective, and if they are operating at the user level or in user space, the implication that I’m getting from these articles is malware could Shut down the anti malware tool and do whatever it wants to do. And that appears to be harder at the kernel level. That it’s better able to protect itself and spot things at a deeper level in the operating system. I don’t know if that’s true, but it seems to be most of these companies operate that way. And in fact, there was even an implication we talked about it on a previous show. From Alex Stamos, who’s the newly appointed CSO over at or c SSO or cso, one of the two over at Sentel
Jerry: CTO. CTO.
Andrew: Now it says Chief Information Security Officer in this particular
Jerry: Oh, okay.
Andrew: Alright. Anyway, he talked about, Hey, we’ll back outta using the kernel if all of our competitors will as well. there’s clearly some advantage to being there. And I don’t know that anybody really wants to talk about that.
Jerry: I think there’s, I think there, that we are talking about this as if it’s like one, monolithic choice of, you’re either there or you’re not there.
Andrew: Yeah.
Jerry: that’s probably not the right way to think about it. I suspect that there’s the nuances that there are certain kinds of functions that you don’t need to perform in the kernel.
And as an example, you don’t need to, parse your file, your definition files in the kernel. You could do that in user mode and then. pull it into the kernel module. I suspect that there’s some of that. It certainly adds a lot of complexity and I’m not going to argue that, but I think that’s where, I don’t know if, I don’t know anything about SentinelOne and their technology, but I’m going to guess what, when they say, when they’re trying to throw rocks at CrowdStrike, I think what they’re probably saying is we do.
more things outside of outside of the kernel. But like you said, if they were to completely move out of the kernel their ability to function would be impaired. And so that kind of dovetails into the second story about this, which is from CNBC. And the title there is Microsoft plans September cybersecurity event to discuss changes after a CrowdStrike outage.
And there was I don’t know if this was directly an outgrowth of the comment that Ed Bastian, the CEO of Delta, made to CNBC, gosh, right after the outage. He said something like, we don’t see this sort of problem with Apple. And, and he was. really talking about, and by the way I’m assuming that Ed had heard that from, his security people hey, Apple just doesn’t allow this, which is true, right?
They don’t they are much stricter about this access. Microsoft will certainly point to the the decision that was rendered against them in the EU that forced them to open this up because by the way, Microsoft is a direct competitor to both SentinelOne and CrowdStrike when it comes to endpoint security products.
Like they have their own, they have their own product, which is in contrast to Apple. Apple doesn’t really have, an equivalent thing like Microsoft Defender, right? They,
Andrew: a single brand around security. They do things a little differently. This is a tough comparison, and I struggle I hear this, and I get frustrated because it’s In a vacuum that isn’t looking at market share. It isn’t looking at
Context. It isn’t looking at all the tradeoffs Microsoft had to make to run an open hardware ecosystem and all the backward compatibility choices they made Apple, to just say Apple just does it better.
It’s not and I’m not an anti Apple fan. I use Mac all day long, but it is Microsoft Mechanics www. microsoft. com. I disingenuous to say it outside of the context of everything else around that ecosystem that has contributed to this. And all the frustrations people have with Apple being so hardcore about, Oh yeah, you, sorry, this hardware no longer supported, go away. Or where is Apple’s surfer ecosystem or, there, there’s just, he’s not wrong, but it’s also like one 10th of the story.
Jerry: Sure. You
Andrew: like that. there’s, it’s not just, it’s not just Microsoft is stupid. It’s, and Apple is smart. There’s so much more that goes into this. It’s my frustration.
Jerry: know, as a society, though, we’ve boiled everything down to 15 second soundbites, it’s just the way of the world.
Andrew: You’re not
Jerry: lost our, we’ve lost our tolerance for nuance.
Andrew: But that’s
Jerry: But,
Andrew: Jerry. To bring the nuance back.
Jerry: That’s exactly right. Microsoft on September 10th, which is coming up fast. is going to have this summit with endpoint security providers. And I think what they’re trying to establish is a set of best practices around what is and is not done in the kernel so that they can avoid catastrophes like this going forward.
And then also, as we’ve talked about in the past, they’re going to start trying to encourage these companies to use the eBPF interface, which is, an alternative way of hooking into the kernel that has less ability. It provides probably not the exact same level of control and visibility, a very, very substantially similar without all some of the downside.
So I think I think Microsoft’s eBPF implementation is rapidly maturing as, relative to what’s been out there for Linux for some time, but it is, it’s not something that it might my experience that the security industry is really embraced yet, but I think this is probably going to be the forcing function that really drives us in that direction.
And this, by the way, may be the thing that It ultimately enables Microsoft to say, no more access to the kernel. If you want to do this, you’ve got to do it through this particular feature like eBPF. I think that’s how I see this playing out. I think if you were to look five years, five years down the road, I don’t think companies are going to be linking and hooking into the kernel.
I think they’re going to be forced through a function like eBPF. It’s just Jerry’s wild ass speculation though.
Andrew: It makes sense. One quote that I do want to mention from this article that kind of goes to what we were saying in the previous story, which is, quote, Software from CrowdStrike, Checkpoint, SentinelOne, and others in the endpoint protection market currently depend on kernel mode. Such access
Jerry: Right.
Andrew: quote, monitor and stop bad behavior and prevent malware from turning off security software, end quote, a spokesperson said. So that kind of
Jerry: Yeah.
Andrew: what we were saying earlier that there’s clearly some
Jerry: Yes.
Andrew: sure that they’re trying to get, like you mentioned, new methodology to give them that same capability without being deep in the kernel.
Jerry: But I think the, I think both customers of these endpoint products and the manufacturers of them too are going to say, okay, fine. But then tell me how you’re going to help us not get, not our endpoint security product, not get killed by. The ransomware companies the adversaries who are pretty adept at stopping things, even that are running in the kernel today, it’s a lot easier to do with user mode processes.
So I, I think that there’s going to be a bit of a meet me in the middle here where Microsoft is going to say, get your crap out of the kernel. And those companies are going to say then, give us something. Enable us because they’re, I don’t think they’re fully enabled today. Yes,
Andrew: wouldn’t be fair. And I think that’s where the EU is coming out with their ruling years ago that started all this. So there’s, make no mistake, I guess is what I’m saying here, that there is absolutely competitive standoff going on here. These companies are frenemies in this situation, so they don’t want to back out and be, leave Windows, their competitor with the only capability to do something that they can’t do, and then they can’t compete.
Jerry: Exactly. And by the way, my Microsoft we have, we talked about it before. We have seen instances where Microsoft has shot themselves in the foot too, right? This is, it’s not unprecedented that Microsoft’s own updates have caused outages. Outages. So I’ve got to, I’ve got to believe like they’re thinking about this too.
It’s so great. They they kick all the other endpoint security products out of the kernel. And then suddenly they are, they’re the cause of the next CrowdStrike scale outage because of their security product is now the only one that’s in the kernel. How bad would that be for them? Holy crap.
Anyway I think that’s what we’re going to see. Agreed.
Andrew: on you there.
Jerry: No, all good. So our next our next story is also from Ars Technica, and this is a bit of a follow up from the Know Before story that we talked about, I think, two or three episodes ago. The title is, title here is, Nashville Man Arrested for Running Laptop Farm to Get Jobs for North Koreans.
So the, if you recall,
If you recall the security awareness company Know Before published a blog post about how they had hired what turned out to be a North Korean agent. And the way that went down was they interviewed and selected and hired a a person who turned out to be a North Korean citizen. And they shipped the laptop to a laptop farm.
And I actually had some questions about how that worked. This actually explains how this goes down. It’s very enlightening. So this this person named Newt, his last name is Matthew Isaac Newt, who lived in in Nashville. So he had a relationship with a set of people in North Korea, where he would facilitate and it looks like it was a fairly sophisticated operation where he would help track down identities that they could use.
He provided a place for North Koreans who were being hired by, unknowingly hired by the way, by U. S. companies to have their laptop sent to. He would receive the laptops, put it on his home network. install remote access software and allow the North Koreans remote access into the laptop so they could quote, do their work.
And it’s just a, it’s a fascinating thing. He I guess they, they said that they were making each one of the North Korean employees was making about 250, 000 over the roughly year period that this was going on. The allegation by the US government is that the money these people were earning was in turn being used by North Korean to fund their weapons program.
So obviously not not super awesome. And they do go on to say in the article, by the way, this is not a one off situation. They refer to another person named Christina Marie Chapman. Down in Arizona who basically did the very same thing So it’s a fascinating I didn’t realize this was as large of a problem as it is, but Apparently this is a fairly Industry becoming an industrial scale operation run by, individuals.
I’m I’m surprised.
Andrew: Yeah it’s interesting. It’s a little different to the know before because in this case, it looks like the North Korean it employees are doing legitimate work. They’re not immediately installing, malicious software and that they’re trying to earn their wage as it were
Activity in this case.
But I’m also very curious how the money was moved over to North Korea. And, was this, paid, I’m sure it’s paid to a US bank account. And then what does it look like to get it over North Korea, which is somewhat non trivial. I don’t know if they use Bitcoin or something like that. But that’s a big part of the charges here is the wire fraud, that sort of thing.
But the other thing I think about is, if as an employer don’t allow your random employee to install software, it would stop a lot of this. I get that’s a big cultural taboo and there’s a lot of gnashing of teeth around that topic, but if they couldn’t install remote access tools or, you as an IT department or security department don’t monitor for those remote access tools, it certainly would stop a lot of this.
You’d be, It just wouldn’t work, unless some other methodology is found. A way to fight this it’s one more reason to not necessarily let local users have full admin rights.
Jerry: Even if you do, I think it’s very prudent to actively look for and block and investigate people who are installing remote access tools because like remote access tools. Whether it’s RDP or TeamViewer or, any of the myriad other software, that stuff has been the source of so many security incidents over the years.
And in fact, it’s one of the common ways that frauds, like just garden variety fraud is perpetrated where, the Windows help desk scam, where they’ll call you up and ultimately install some sort of remote access software to get into your system. So I think this is really important.
Now I will say, I think this particular set of scams was reliant on them installing this remote access software. But I think, a sophisticated, if they were if they were sophisticated, network KVMs are pretty cheap these days. So it’s not necessarily a home run to say we’re like, we’re fully protected because we don’t allow that we would certainly detect it.
But it is not, there are other ways around that. It’s another step. And I can imagine it would make, perhaps the would increase the cost and complexity of hosting these but probably not prohibitively. And, it goes back to you’ve got to, especially in a remote working situation, you’ve got to have good diligence on and good awareness of who you’re hiring.
And by the way, that I say that is somebody like full throated supportive remote work.
Andrew: Yeah, certainly, but I also feel like you’ve spent a lot of time thinking about this. Is this your new post retirement career? Are you setting up laptop
Jerry: But my, my kids have moved. My oldest son has moved out and someday soon my youngest will move out. So I’m trying to figure out, do I go the Airbnb or do I go the, hosting laptop farms? I don’t know yet.
Andrew: That explains the eight new air conditioning units you just added onto your house.
Jerry: Yeah. Then the Bitcoin mining, like you gotta diversify.
Andrew: You having free time is dangerous. I don’t know that this is a good thing at all.
Jerry: What is the saying about idle hands?
Andrew: Indeed.
Jerry: All right. The next next article comes from Dark Reading, and the title here is Why End of Life for Applications is the Beginning of Life for Hackers. This is a big problem. So the gist of this story is that end of life applications is a boon for threat actors. It’s they make reference to 35, 000 applications moving to end of life status over the course of the next year.
I think that’s probably optimistic. I think it depends on how you define application for sure, but I think the look, in my, my, my personal experience. This is probably one of the larger problems we have in IT. It just, we talked last time about patching and how nobody wants to, nobody wants to patch stuff.
But there are so many issues that come along with using end of life software. Not the least of which, by the way, is that Most vulnerability management programs are built around, subscribing to vendor alerts to understand that a patch was, needs to be applied. When you’re using end of life software, that doesn’t happen anymore.
Like it’s, it goes quiet. Certainly.
Andrew: end of life for vulnerabilities. Much less issue patches.
Jerry: Yeah, exactly.
Andrew: Yeah.
Jerry: Yeah, most of the time you’re. Your vendors, vendor notifications are not vulnerability based, they’re patch based. They announce the availability of a patch to fix a vulnerability. And so now you know that you have a piece of work that has to be done because a patch was released and you got to go and apply that patch.
There’s no patch. A lot of vulnerability scanners work in a similar way, especially as it pertains to maybe less less well known applications. Now, certainly if you’re using a tenable Nessus and you’re running an out of support version of Linux or Windows, like it, it flags that itself as a critical vulnerability, but you don’t have any granularity about what the actual technical vulnerabilities are because they don’t know.
It’s just like it’s end of life. Who knows what. What sort of vulnerabilities there are and I think it gets It starts to descend into obscurity after that when you get into like open source Components and whatnot. You just don’t know you’re unaware that they’re not being maintained anymore And that becomes a big problem I will also say, one of the things that they talk a little bit about how companies can get into the Or defend against letting this, letting things go end of life.
But I will say my, in my experience, it’s easy. It’s a trap to fall into when something goes end of life, right? Because, Hey it’s end of life, but there’s no known vulnerabilities and we have this other thing that needs to be done and it’s super high priority. It’s going to make a billion dollars, blah, blah, blah.
And at the time, that’s true, right? It’s true. It’s a low, it’s a low risk thing. Your your version of WordPress is out of date. There’s no known vulnerabilities. But then, you start to collect these things. And suddenly, you’re buried under too much technical debt. And and it’s hard to, really hard to get out from under and you end up in this position where you’ve got so much of this debt that you have a hard time even understanding, not only what is, what all is end of life, but, are there actually vulnerabilities?
Like at the time you made that risk acceptance to allow this end of life thing happen. You knew that there wasn’t, but are you actually keeping up with the vendor and with the industry to know whether that has changed? And I think, if you’re talking about one thing, one application, it’s fairly easy to manage.
But once you start accumulating a lot of these, it becomes really unmanageable.
Andrew: Yeah, you echoed a lot of the notes I had as well, which is once you get so far behind, it’s that much more difficult to get caught back up. And then it becomes that much more of a fight comparing against other higher priority things to work on patching or massive upgrading, as opposed to just keeping things up to date bit by bit. other thing I think about, I don’t have my notes here, is If something is that far end of life, it’s not just a security thing. You don’t know necessarily if other interactive or interrelated components are supporting that version anymore, or tested against that version. And you might start seeing some weird buggy artifacts as a result. and you brought up the open source thing. That’s. Most of these sort of end of life checks are usually coming from some sort of end of life policy statement from a commercially supported application or operating system. The problem with a lot of these open source dependencies and third party packages, they don’t have that.
They don’t have any sort of published end of life, end of support methodology. So how do you know when something goes end of life when it’s open source? It hasn’t been updated in three years. Is that just because it’s just really stable or because it’s been abandoned and what is the criteria you want to use there?
Do you have a criteria that says, Hey, we need to use currently maintained third party dependencies in our code. Okay. How do you define what that is? Something that has had an update in the last X amount of months we see all the time that. Open source packages just become abandoned ware without any notification, without any understanding of that.
It’s just in hindsight you go, oh yeah, that guy stopped working on that three years ago. knows.
Jerry: Yeah, it’s not certainly not universally true. There are plenty of open source, especially the larger ones that have a defined roadmap. But I think you’re spot on. If you don’t, if you don’t see that there’s been an update on a particular piece of open source, is it because there’s nothing wrong with it or, or is it on you to go look at its GitHub repository and see that like people have been.
jamming the issue log with requests to fix some vulnerability that are falling on deaf ears. And, when you start to think about the many tens of thousands or even hundreds of thousands of open source components, some applications use that, it becomes really challenging. And I think that’s where some of the open source management tools Like mend and others can start to help but you know Even those aren’t infallible and even so you have to have as an organization Some amount of discipline and capacity to make sure that you’re staying up to speed
Andrew: Yeah.
Jerry: The other thing by the way that I wanted to hammer on because I hate it.
I hate it With the burning passion and it has happened every time i’ve been involved in it And it’s something that I feel like as an industry, we have to do something about. And that is in the aftermath of an acquisition, accounting systems. Can we just talk about those for a second? Because I don’t know how many acquisitions I’ve been, on the acquired side and I’ve been on the acquire side a bunch of times.
And it happens every single goddamn time. The acquired company has an accounting system. And what happens when, why do companies acquire other companies? One of the main reasons is this thing called synergy, right? And the synergy basically means that we don’t want to run duplicate HR systems and accounting and whatnot.
If we consolidate all that backend stuff. And continue to make money and the products they sell, then like we have, we’ve increased the value that the whole is more than the sum of the parts. And that’s why a lot of companies
Andrew: Is,
Jerry: other,
Andrew: yeah. Sorry,
Jerry: right?
Andrew: gonna say the back office overhead
Jerry: And
Andrew: Yeah.
Yeah.
Jerry: exactly. So want to. When a company acquires another company, one of the things they want to do very quickly, as quickly as possible, is start to realize those savings. And one of the first things that happens is, putting out to pasture the accounting system. And this could also be the HR system, although, nobody does HR internally anymore.
It’s like running your own voicemail system these days. But accounting systems seem to be still very much insourced. And, every single time I, without exception, every single time I’ve seen this happen, company gets acquired. We stopped, they stopped paying for maintenance on the accounting application.
And. Ends up running on some old ass operating system that is out of date. It can’t be that you can’t update the operating system because the accounting system won’t work on the new operating system. You can’t update the app, the accounting system because like they’re out of business or you don’t have a license to upgrade anymore and it would cost millions of dollars and it’s not in the business case, my, oh my God.
And it. wouldn’t provide any like accretive value to the company if you did upgrade it because it’s not going to be used anymore it but you know what if you shut it off like people will go to jail at least that’s what i’ve been told over and over again and so you end up with this thing sitting in the corner which is as best i can describe it an attractive nuisance because like everything about it is terrible it’s all out of date and it can’t go away because you Joe from accounting says that somebody will go to jail if you turn it off.
Anyway, I may be a little angry about this. Ah,
Andrew: If you just ran your accounting on Excel, like God intended, you wouldn’t have this problem.
Jerry: it’s
Andrew: No
Jerry: so true.
Andrew: than what can be done in an Excel sheet.
Jerry: Amen. I, the only thing I can say is in, in those instances, and I look I’m going to just be forthright and say, I’ve never seen an effective. counter to this. It’s been a problem every single time I’ve seen it happen. The only thing I can say is, as an idea, have a, have a way I don’t think any company is going to be I shouldn’t say any company, but I think most companies will not be effective in changing those facts.
It’s going to happen, your acquired company is going to have an accounting system. There’s not going to be an appetite to update it but there it is, and so you have to you have to mitigate the risk of it, and I think that having a defined approach to doing that, whether it’s like a separate VLAN that has no access, or like you have to, do multi factor authentication to get in and out of that network, it could be pretty simple and dirty, but have a plan, because it just, It happens no matter how mad it makes me, it, it happens.
And and so I think we’ve got to recognize that there are cases where that will be and come up with, relatively workable mitigations around it, but it can’t be the rule.
Andrew: What’s your ideal use case? That they just migrate all the data off the old system to the new system and just kill it?
Because I’m
Jerry: I don’t know. I
Andrew: system is maintained, is because they need a system of record for the last seven years or whatever, for tax purposes or government regulatory purposes. And I’m guessing that’s why, I’m assuming.
Jerry: typically, yes. In, in, frankly, I think the best course would be to figure out the different types of reports that are needed and to do, to run exports and have those exports exist in a spreadsheet. Now, I don’t know if there, I’m not an accountant, I’m not a,
Andrew: right out of the See, look, it all comes back.
Jerry: But I don’t know if there’s some like statutory requirement that data, that system of record has to be there because if, like the sec or, the department of justice or some other legal Authority came to you and wanted to investigate like why did you claim? Why did you say you made x dollars in, seven quarters ago before you were acquired?
You’ve got to be able to go back and replay that maybe that’s why and maybe that can’t be done through exports I don’t know but I In every instance, the accounting folk have insisted that system has to be available. It’s not enough to just dump the data.
Now I don’t know if that’s laziness or what. The other problem I have, and while I’m beating on this drum, Over time, the people who are familiar with that system go away
Andrew: Certainly.
Jerry: and so like at one of the, one of the things you have to watch out for is that eventually what, like when that system’s usefulness is done, there isn’t anybody left to say, Hey, now it’s time to turn it off.
Andrew: And then who wants to take the risk of being the one who makes the wrong call? So they say people go to jail. Can we talk about who might go to jail and if that’s really a bad thing?
Jerry: I like where this is going. I,
Andrew: just weighing the outcome. What are my options?
Jerry: I think I’ve beat that one to death. The the last story we have today, it’s also from the cyber security dive. And the title is after a wave of attacks, Snowflake insists security burden rests with customers. Now, Snowflake had a a large problem. And this happened earlier in, in 2024 before we got back to podcasting.
But I would say that I don’t think it’s an overstatement to say that the security breaches or data breaches associated with Snowflake will probably go down as the largest in history. It, bigger than anything there ever was, and perhaps bigger than anything there ever will be again, maybe it’s a huge numbers of customers.
Lost lots of data. Now, the point of this story is to say, Hey, like Snowflake was not, snowflake is saying, Hey, it wasn’t us. It wasn’t us. It was. Our customers.
Andrew: Because they
Jerry: they’re not
Andrew: single factor login, was easy to find in other dumps of passwords bad actors did a widespread campaign to, and upward force, password test, all of those passwords against Snowflake user accounts and lo and behold, a bunch of them worked. And that’s, they’re
Jerry: Yes.
Andrew: shouldn’t have relied on single factor and reusing of passwords.
And that’s on you because you didn’t take appropriate security measures.
Jerry: Correct. Yeah we just provided you with an account and a place to store your data or process your data. It was on you, the customer, to pick a good password. And that’s basically what they are saying here. They’re saying our systems, our Snowflake systems didn’t get hacked. We, you know Our infrastructure is fine.
Everything worked as designed. The fact that bad guys got your password and stole all your data is a horrible thing, but it’s not our fault. It’s your fault because, it was your password that they got. We don’t know how they got your password. Was it the same password you used on LinkedIn or on, Ashley Madison?
I don’t know. Who, who knows? And that’s, they’re saying it’s not our problem, not our fault. And they basically, of course they do give lip service to the fact that, hey, there are customers and of course we care about them and we’re in it together as they say, but not our fault.
By the way they have since implemented some snowflake, I should say, has since implemented some changes, which require mandatory multi factor authentication for new customers, and it also gives customers the ability to require or enforce multi factor authentication for all of their users or for specific roles in their account.
So I should, by the way, I should have said for those of you not in the know, Snowflake is a what I would call like a managed storage managed database provider. They do lots of. Value added services around data analytics and whatnot. So the kinds of data that you would have stored in Snowflake are the kinds of data that you wouldn’t want to get compromised.
And so I think this was a central place, one stop shopping for some adversaries to go and do their password stuffing. And it looks like they they got somewhere in the neighborhood of 150 or 200 different customer accounts and pulled all that data out. And it was, and still is by the way, we’re still, even now hearing about net new companies that were hacked or had their data stolen.
And I don’t know if that’s because they’ve, been quote, responding and investigating to the incident, or if they just recently realized that this has happened, but this is a big problem and I think more interesting than, this itself, because I don’t think that Snowflake is all that, commonly used in the industry is the concept that, these service providers are.
They have a hard line of demarcation of what they’re responsible for. And so when we as consumers of these services decide what we’re going to go use, what do we do? Like we look at their SOC 2 and we look at their PCI report and we look at their ISO certification and we look, we look for all these things, but how often do we look to see What the capabilities of their services from a security perspective do you know, do you require multi factor authentication as by default, how many customers, I can say this authoritatively, I don’t know, in my time that I ever saw any customers asking about that, and they should.
And this is, the, because the, look go ahead.
Andrew: no I think this absolutely is a product management decision. And I
Jerry: Yes.
Andrew: the implication here is security causes friction. Friction
Jerry: Yes.
Andrew: make us uncompetitive. So if I, let me bring it back to a bank as an example. A bank knows how to secure your accounts. A bank could easily force multifactor authentication.
They could force tokens. They could force all sorts of things. But they know there’s a certain percentage of customers who will move away from them as a result. That friction will be more complicated than they want to deal with. And they will not value the security They will look at this as a detriment to that service and go someplace else. So that is a decision that every company and every product manager makes about what security they built into their tool. So in my mind obviously I have no insight, insider insight to Snowflake, but the idea that those admins could allow single user passwords and static passwords is a product design choice to reduce friction and usability of to make the usability of the platform easier. Whether it’s for whatever reason, as opposed to like multi factor authentication is a very mature known. Solved problem. So if it’s not being put in place, it’s because somebody made a choice and that choice typically is competitive in nature. So to your point,
Are not demanding it, it’s not going to show up. The other thing that this occurs to while I’ve stolen the floor for a moment is with these rise of all of these SaaS vendors, typically the administrators for these are no longer it professionals or security professionals. They are. users of the data professionals. So they may not even understand or have any insight into the implications of the administration of these tools around security aspects like this. All they know is, Hey, I need to make an account. Okay. I made an account. They may not have the background or the guidance to know single factors bad, and here’s why. Like you would, if a. More traditional IT or security team were administering these tools. So I think we’ve also got a problem with these SaaS tools that have become so ubiquitous and easy to use that we’ve somewhat enabled less technical staff to administer them and I think things like this become oversights that come back to bite us.
Jerry: I couldn’t have said it better. I think that’s exactly what’s going on. And so it’s not friction on IT and security departments that a product might require multi factor authentication. It’s friction on the business users. And so when the bit when it’s the business users that are specifying and deciding what services to use.
You know that. It’s not hard to imagine that they’ll pick one that is easier to log into even though it could have a devastating effect, like a lot of the customers of Snowflake here. And so it’s a complicated thing because if all of this, if all of the providers were requiring multi factor authentication, that wouldn’t be a, there wouldn’t be a difference.
It wouldn’t be a differentiation. I guess the business users would have kind of a similar experience across all of the different providers, but we know that’s, we know that’s not true. But I think that if you zoom out, the concern I have as an industry is that we’re we’re not broadly speaking.
We’re not. deeply aware of the responsibilities that we are picking up for properly managing those services. We are, to some extent, doing what we think is due diligence, looking at those providers and saying they’re a reputable company, they’re secure, they have these certifications, but then that’s like the end of it.
We don’t think about What our obligations are. And this manifests itself in so many different ways. Like how many times we talked about open S3 buckets. It’s another permutation of that. We had the big Capital One breach in AWS. Also misconfiguring how they set up their IAM.
Like it. The devil is in the details in how we manage these software as a service systems and, that the companies are not that these providers aren’t going to come to us and hit us on the back of the head and say, you big dummy, like you should have, you, we saw that you don’t have multi factor authentication turned on and you really need to go do that.
Now, maybe. snowflake will start doing that because of the reputation damage that they’ve incurred as a result of this breach, which they assert isn’t their fault. It is having an impact on them. It is, I think it is probably having a negative impact on them.
It’s attracting attention that I’m sure they don’t want to have. And I’ll, another way of thinking about it is like, there’s no such thing as bad PR, but I think You don’t want to have your name associated with the largest data breaches that have been around. But again,
Andrew: podcast like this with tens of listeners
Jerry: tens of listeners. And I’m not blaming, I’m not disagreeing by the way, I’m not disagreeing with the premise of Snowflake’s comments that it was, their customers were responsible. This was not intended to be like a Bash on Snowflake segment. It was more like we have to understand how companies like Snowflake are viewing their relationship with their customers.
You as the customer are responsible for ensuring that you are properly securing your stuff.
Andrew: So I go back to maybe we’ve got non technical running this, these tools in companies, maybe what we should be doing is allocating GRC’s time to go auditing how they’re running these tools
Jerry: Yes.
Andrew: and clean them up.
Jerry: I I don’t have a better, I don’t have a better option or better idea. Yeah,
Andrew: out there for and such that sort of are meant to be, allow you to apply policy, and those are technical controls of the same problem, which is proper policy should be applied. And I think the challenge is we have such a sprawl of SAS tools that it’s really difficult to stand on top of and ahead of. And I’ve worked at try to mandate as best they can. Hey, if you want to buy a SaaS tool, it’s got to integrate to single sign on or MFA. you could do that and then 10 minutes later, the admin sets up another username and password that you don’t have purview into. And they don’t necessarily know they’re doing anything wrong.
Jerry: exactly. They have a business objective to meet. They’re trying to solve a business problem.
So
Andrew: this is I think one of the unintended consequences of the, Democratization of admin capabilities through the SaaS and Cloud Sprawl, that is making life more difficult for security teams.
Jerry: Yes. And I do wonder, by the way I don’t know that we’ll ever have clarity on this, but I do wonder of the, I think it’s 165 at last count of the 165 companies who’s data was breached from from their Snowflake account. How many of those companies found their IT and security departments found the fact that they were using Snowflake in that way a surprise or that they didn’t have multi factor authentication turned on?
How many times was that a surprise? And I think it’s going to be an unfortunately high number. Yes,
Andrew: And what I unfortunately foresee happening is executives will just say, fine, security, IT, go fix it. Without allocating enough resources.
Jerry: yes you failed in your job, which, I guess it’s not a it’s not a completely unfair statement. But on the other hand, I think we have to be we have to be enabled in our jobs. And I’m not sure that always happens.
Andrew: Yeah. Yeah.
Jerry: So anyway, That is that is the stories for today. Oh, go ahead. Sorry. I think there’s a lag now.
Andrew: Yeah. We might be,
Jerry: I
Andrew: stepped each other a little bit this this show, but we’ll figure it out.
Jerry: think there’s a I don’t know if it’s because of the way we’re recording or what, but I think there’s a bit of a lag. So in any event that is the show for today. I appreciate everybody’s attention and hope you found this useful. And if you like the show, you can. Go listen to back episodes.
Everything is available on our website at www. defensivesecurity. org or on your favorite podcast player. If you do the show, we would we would love, love, love for you to give us a five star review that that helps make sure that other people are able to find us and and get us.
And by the way, if you don’t like the show. You can also still give us a five star review.
Andrew: Even more
Jerry: And just not listen. Yeah.
Andrew: That’s true.
That it’s free five stars is free
Jerry: Yeah, and then you just don’t listen anymore.
Anyway.
Andrew: just play it for your cats. That’s what I do.
Jerry: Yes.
I’m not even going to continue down that road.
Andrew: That’s fair, it’s going off the rails.
Jerry: All right. You can find a Mr. Kellett. On the social media where
Andrew: I’m on Twitter slash X at atlerg L E R G and on InfoSec Exchange on the Fediverse, also atlerg L E R G.
Jerry: Awesome you can find me on the fediverse at jerry at infosec. exchange and with that We will talk to you again very soon. Thank you all. Have a great week
Check out the latest Defensive Security Podcast Ep. 276! From cow milking robots held ransom to why IT folks dread patching, Jerry Bell and Andrew Kalat cover it all. Tune in and stay informed on the latest in cybersecurity!
Summary:
In episode 276 of the Defensive Security Podcast, hosts Jerry Bell and Andrew Kalat delve into a variety of security topics including a ransomware attack on a Swedish farm’s milking machine leading to the tragic death of a cow, issues with patch management in IT industries, and an alarming new wormable IPv6 vulnerability patch from Microsoft. The episode also covers a fascinating study on the exposure and exploitation of AWS credentials left in public places, highlighting the urgency of automating patching and establishing robust credential management systems. The hosts engage listeners with a mix of humor and in-depth technical discussions aimed at shedding light on critical cybersecurity challenges.
00:00 Introduction and Casual Banter
01:14 Milking Robot Ransomware Incident
04:47 Patch Management Challenges
05:41 CrowdStrike Outage and Patching Strategies
08:24 The Importance of Regular Maintenance and Automation
15:01 Technical Debt and Ownership Issues
18:57 Vulnerability Management and Exploitation
25:55 Prioritizing Vulnerability Patching
26:14 AWS Credentials Left in Public: A Case Study
29:06 The Speed of Credential Exploitation
31:05 Container Image Vulnerabilities
37:07 Teaching Secure Development Practices
40:02 Microsoft’s IPv6 Security Bug
43:29 Podcast Wrap-Up and Social Media Plugs-tokens-in-popular-projects/
Jerry: Today is Thursday, August 15th, 2024. And this is episode 276 of the defensive security podcast. My name is Jerry Bell and joining me tonight as always is Mr. Andrew Kalat.
Andrew: Good evening, Jerry. Once again, from your southern compound, I see.
Jerry: Once again, in the final time for a two whole weeks, and then I’ll be back.
Andrew: Alright hopefully next time you come back, you’ll have yet another hurricane to dodge.
Jerry: God, I hope not.
Andrew: How are you, sir?
Jerry: I’m doing great. It’s a, it’s been a great couple of weeks and I’m looking forward to going home for a little bit and then then coming back. How are you?
Andrew: I’m good, man. It’s getting towards the end of summer. forward to a fall trip coming up pretty soon, and just cruising along. Livin the dream.
Jerry: We will make up for last week’s banter about storms and just get into some stories. But first a reminder that the thoughts and opinions we express are those of us and not our employers.
Andrew: Indeed. Which is important because they would probably fire me. You’ve tried.
Jerry: I would yeah. So the the first story we have tonight is very Moving.
Andrew: I got some beef with these people.
Jerry: Great. Very moving. This one comes from security affairs and the title is crooks took control of a cow milking robot, causing the death of a cow. Now, I will tell you that the headline is much more salacious than the actual story that the. When I saw the headline, I thought, oh my God, somebody hacked a robot and it somehow kill the cow, but no, that’s not actually what happened,
Andrew: Now, also, let’s just say up front, the death of a cow is terrible, and we are not making light of that. But we are gonna milk this story for a little while.
Jerry: that’s very true.
Andrew: I’m almost out of cow puns.
Jerry: Thank God for that. So, what happened here is this farm in Sweden had their milking machine, I guess is a milking machine ransomware and the farmer noticed that he was no longer able to manage the system, contacted the support for that system. And they said, no, you’ve been ransomware.
Actually, the milking machine itself apparently was pretty trivial to get back up and running, but apparently what was lost in the attack was important health information about the cows, including when some of the cows were inseminated. And because of that, they didn’t know that one of the pregnant cows was supposed to have given birth, but actually hadn’t.
And so it. What had turned out to be the case is that the cow’s fetus, unfortunately passed away inside the cow and the farmer didn’t know it until they found the cow laying lethargic in it stall, and they called a vet. And unfortunately, at that point it was too late to save the cow.
This is an unfortunate situation where a ransomware attack did cause a fatality.
Andrew: Yeah, and I think in the interest of accuracy, I think it was in Switzerland,
Jerry: Is it switzerland? Okay. I knew it started with a S W.
Andrew: That’s fair. You’re close. It’s Europe.
Jerry: It’s all up there.
Andrew: But yeah, I guess in this theory that if they had a better tracking date when the cow had been inseminated, they would have known that the cow was in distress with labor and could have done something more proactively to save cow and potentially the calf. And unfortunately, because I didn’t have that data, because it was in this ransomwared milking robot machine we ended up with a dead cow and a dead calf.
Jerry: So not without grilling the farmer too much. I was I was thinking, that,
Andrew: Wow!
Jerry: I’m sorry. I was thinking that, they clearly had an ability to recover. And what they thought was the important aspect of that machine’s operation, which was milking, they were able to get that back up and running pretty quickly.
But it seemed to me like they were unaware that this other information was in kind tied to that same system. I don’t fully understand. Seems like it’s a little more complicated than I’m, than I’ve got it envisioned in my mind. But very clearly they hadn’t thought through all the the potential harm.
A good lesson, I think for us all.
Andrew: I feel like we’ve butchered this story.
Jerry: The the next story we have for today comes from register. com and the title is patch management still seemingly abysmal because no one wants the job can’t stop laughing. All right.
Andrew: A cow died! That’s tragic!
Jerry: I’m laughing at your terrible attempts at humor.
Andrew: I couldn’t work leather in there. I tried. I kept trying to come up with a leather pun.
Jerry: We appreciate your efforts.
So anyhow. This next story talks about the challenge that we as an IT industry have with patching. And basically that it is a very boring task that not a lot of people who are in IT actually want to do. And so it, it highlights the importance again of automation and.
This in the complimentary story which is titled misguided lessons from CrowdStrike outage could be disastrous from cybersecurity dive. I put these two together for a reason because one of the, one of the. I think takeaways from the recent CrowdStrike disaster is we need to go slower with patching and updates and perhaps not rely on automatic updates.
And these 2 articles really point out the folly in that. Number 1, this. Article from the register is pointing out that relying on manual patching is a losing proposition because really nobody wants to do it and it doesn’t scale. It’s, it’s already, it’s IT operations is already a crap job in many instances, and then trying to expect people to to do things manually is a problem.
The second article points out the security issues that come along with Adopting that strategy, which is, you’re exposing your environment unduly unnecessarily. And in fact the improvements in. Your security posture and the let the reduction in likelihood of some kind of an attack far outweigh the remote possibility of what happened.
Like we saw with CrowdStrike. Now there is a kind of an asterisk at the bottom. They point out the importance of doing staged deployments of patches, which I think is one of the central lessons of the, at least for my Perspective, one of the central lessons of the CrowdStrike disaster is that go fast, but stage it.
Andrew: yeah it’s an interesting problem that we’re struggling with here, which is how many times have we saved our own butts without knowing it by automate or rapidly patching? It’s very difficult to prove that negative. And so it’s very difficult to. Weigh the pros and cons empirical data showing where automatic patching or rapid patching solved a problem or avoided a problem versus when patching broke something.
Cause all we know about is when it breaks, like when a Microsoft patch rolls out and breaks and that sort of thing. And it’s one of those things where it has to be perfect every time is the feeling from a lot of folks. And if it, if every time we have a problem, we break some of that trust. It hurts the credibility of auto patching or, rapidly patching. The other thing that comes to mind is I would love to get more IT folks and technical operations folks and SREs and DevOps folks, with the concept of patching as just part of regular maintenance. That is, just built into their process. A lot of times it feels like a patch is an interrupt driven or toil type work that they have to stop what they’re doing to go work on this.
Where, in my mind, at least the way I look at it from a risk manager perspective, unless something’s on fire or is a known RCE or known exploited, certain criteria. I’m good. Hey, take patch on a monthly cadence and just catch everything up on that monthly cadence, whatever it is. I can work within that cadence.
If I’ve got something that I think is a higher priority, we can try to interrupt that or drive a different cadence to get that patched or mitigated in some way. But the problem often is that, Okay. Every one of these patches seems to be like a one off action if you’re not doing automatic patching in some way, that is very Cognitively dissonant with what a lot of these teams are doing and I don’t know how to get Across very well that you will always have to patch it was all this will never stop So you have to plan for it.
You have to build time for that. You have to build Automation and cycles for that and around it and it’ll be a lot less painful It’s it feels like pushing the rock up the hill on that one.
Jerry: One of my observations was
an impediment to fast patching is the reluctance for downtime and, or the potential impacts from downtime. And I think that dovetails with what you just said, in part, that concern stems from the way we design our IT systems and our IT environments. If we design them in a way that they’re not patchable without interrupting operations, then my view is we’ve not been successful in designing the environment to meet the business.
And that’s something that, that I tried hard to drive and just thinks in some aspects I was successful and others I was not. But I think that is one of the real key things that, that we as a it leader or security leaders really need to be imparting in the teams is that when we’re designing things, it needs to be, Maintainable as a, not as a, like you described it as an interrupt, but as an, in the normal course of business without heroic efforts, it has to be maintainable.
You have to be able to patch, you have to be able to take the system down. You can’t say that gosh, this system is so important. Like we can’t, we take it down. We’re going to lose millions of dollars ever. Like we can’t take it down. Not a good, it’s not a good look. You didn’t design it right.
Andrew: That system is gonna go down. Is it gonna be on your schedule or not? The other thing I think about with patching is not just vulnerability management But you know Let’s say you don’t patch and suddenly you’ve got a very urgent Vulnerability that does need to be patched and your four major versions and three sub versions behind now you have this massive uplift That’s probably going to be far more disruptive to get that security patch applied, as opposed to if you’re staying relatively current, n minus one or n minus two, it’s much less disruptive get that up to date.
Not to mention all of the end of life and end of support issues that come with running really old software. And don’t even know what vulnerabilities might be running out there, but just keeping things current as a matter of course, I believe. It makes dealing with emergency patches much, much easier. all these things take time and resources away from what is perceived to be higher value activities. So it’s constantly a resource battle.
Jerry: And there was like, there was a quote related to what you just said in, at the end of this article, it said I think it mostly comes down to quote, I think it mostly comes down to technical debt. You explained it’s very, it’s a very unsexy thing to work on. Nobody wants to do it and everyone feels like it should be automated, but nobody wants to take responsibility for doing it.
You added the net effect is that nothing gets done and people stay in the state of technical debt. Where they’re not able to prioritize it.
Andrew: That’s not a great place to be.
Jerry: No, there wasn’t another interesting quote that I often see thrown around and it has to do with the percent of patches. And so the, I’ll just give the quote towards the beginning of the article. Patching is still notoriously difficult for us to principal analyst. Andrew Hewitt told the register.
Hewitt, who specializes in it ops said that while organizations strive for a 97 99 percent patch rate, they typically only managed to successfully fix between 75 and 85 percent of issues in their software. I’m left wondering, what does that mean?
Andrew: Yeah, like in what time frame? In what? I don’t know. I feel like what he’s talking about maybe is They only have the ability to automatically patch up to 85 percent of the deployed software in their environment.
Jerry: That could be, it’s a little ambiguous.
Andrew: It is. And from my perspective, there’s actually a couple different things where we’re talking about here, and we’re not being very specific. We’re talking about I. T. Operations are talking about corporate I. T. Solutions and systems and servers. For an IT house, I work in a software shop, so we’ve got the whole software side of this equation, too, for the code we’re writing and keeping all that stuff up to date, which is a whole other complicated problem that, some of which I think would be inappropriate for me to talk about, but, so there’s, it’s doubly difficult, I think, if you’re a software dev shop to keep all of your components and dependencies and containers and all that stuff up to date.
Jerry: Absolutely. Absolutely. I will also say that A couple of other random thoughts on my part, this, in my view, gets harder or gets more complicated, the larger in larger organizations, because you end up having these kind of siloed functions where responsibility for patching isn’t necessarily clear, whereas in a smaller shop.
You may have an IT function who’s responsible end to end for everything, but in large organizations, oftentimes you’ll have a platform engineering team or who’s responsible for, let’s say, operating systems. And then you may have a, that, that team is a service provider for other other parts of the business.
And those other parts of the business may not have a full appreciation for what they’re responsible for from an application perspective, and especially in larger companies where, they’re want to reduce head count and cut costs, the, those application type people in my, my experience, as well as the platform team are are ripe targets for reductions.
And when that happens. You end up in this kind of a weird spot of having systems and no clear owner on who’s actually responsible. You may even know that you have to patch it, but you may not know whose job it is.
Andrew: Yeah, absolutely. In my perfect world, every application has a technical owner and every underlying operating system or underlying container has a technical owner. Might be the same, might be different. And they have their own set of expectations. Often they’re different and often they’re not talking to each other. So there could be issues in dependencies between the two that they’re not coordinating well. And then you get gridlock and nobody does anything.
Jerry: So these are pragmatic problems that in my experience. They present themselves as salt is a sand in the gears, right? They make it very difficult to move swiftly. And that’s what in my ex in my experience drives that heroic effort, especially when something important comes down the line, because now you have to pay extra attention because that something is not going to, that there isn’t a well functioning process.
And I think that’s. Something as an industry, we need to focus on. Oh, go ahead.
Andrew: I was just gonna say, in my mind, some of the ways you solve this, and these are usually said difficult to do, but proper. I should define that. Maintained asset management, I. T. asset management is key. And in my mind, you’ve got to push your business to make sure that somebody has accountability to every layer of that application. And push your business to say, hey, if we’re not willing to invest in maintaining this, and nobody’s going to take ownership of it, it shouldn’t be in our environment. must be well owned. This is, it’s like when you adopt a dog. Somebody’s got to take care of it. And you can’t just neglect it in the backyard. So we run into stuff all the time where it’s just, Oh, nobody knows what that is. Then get rid of it. attack surface. That’s a single thing out there is something that could be attacked. If it’s about being maintained, that becomes far riskier from an attack surface perspective. So I think that, and I also think about, Hey, tell people before you go buy a piece of software, do you have the cycles to maintain it? Do you have the expertise to maintain it?
Jerry: The business commitment to fund its ongoing operations, right?
Andrew: Exactly. I don’t know. It gets stickier. And now we have this concept of SaaS, where a lot of people are buying software and not even thinking about the backend of it because it’s all just auto magic to them. So they get surprised when it’s, Oh, it’s in house. We’ve got to actually patch it ourselves. Yeah,
Jerry: The other article in cybersecurity dive had a, another interesting quote that I thought lacked some context and the quote was. There are, there were 26, 447 vulnerabilities disclosed last year and bad actors exploited 75 percent of vulnerabilities within 19 days.
Andrew: no, that’s not right.
Jerry: Yeah, here’s, here is the missing context.
Oh, and it also says one in four high risk vulnerabilities were exploited the same day they were disclosed. What now, the missing context is this report linked, or this quote is referring to a report by QALYS that came out early. At the beginning of the year and what it was saying is that about 1 percent of vulnerabilities or are what they call high risk and those are the vulnerabilities that end up having exploits created, which is an interesting data point in and of itself, that only 1 percent of vulnerabilities are what people go after.
Patching our goal is to patch all of them. What they’re saying is that 75 percent of the 1%, which had vulnerability or had exploits created, had those exploits created within 19 days.
Andrew: That makes, that’s a lot more in line with my understanding.
Jerry: And 25 percent were exploited within this the same day. So I, and that’s the important context. It’s a very salacious statement without that extra context. And I will say that as a as a security leader, one of the challenges we have is, again, that there, there were almost 27, 000.
Vulnerabilities. I think we’re going to blow the doors off that this year,
not all that they’re not all equally important. Obviously they’re rated at different levels of severity, but the real, the reality for those of us who pay attention, that it’s not just the critical vulnerabilities that are leading to. being exploited and hacked and data breaches and whatnot.
There’s plenty of instances where you have lower severity from a CVSS perspective, vulnerabilities being exploited either on their own or as put together but the problem is which ones are important. And so there’s a whole cottage industry growing up around trying to help you prioritize better with which which vulnerabilities to go after.
But that is the problem, right? We, like we, we, I feel like we have quite Kind of a crying wolf problem because 99 percent of the time or more, the thing that we’re saying the business has to go off and spend lots of time and disrupting their their availability and pulling people in on the weekends and whatnot is not, Exploited, it’s not a targeted by the bad guys, you only know which ones are in that camp after the fact.
So if you had that visibility before the fact, it’d be better, but that’s a that’s a very naive thing at this point.
Andrew: Yeah. If 1%.
Jerry: If I could only predict the winning lottery numbers.
Andrew: The other thing, and the debate, this opens up, which I’ve had, Many times in my career is ops folks, whomever, I’m not the bad guys. They’re just asking questions, trying to prioritize. Prove to me how this is exploitable. That’s a really unfair question. I can’t because I’m not hacker who could predict every single way this could be used against a business.
I have to play the odds. I have to play statistically what I know to be true, which is that some of them will be exploited. One of the things I could do is I could prioritize, Hey, what’s open to the internet? What’s my attack service? What services do I know are open to anonymous browsing or not browsing, but, reachability from the internet. Maybe those are my top priority. And I watched those carefully for open RCEs or likely exploitable things or, and I prioritize on those, but at the end of the day, not patching something because I can’t prove it’s exploitable. that I can predict what every bad guy is ever going to do in the future or chain attacks in the future that I’m not aware of.
And I think that’s a really difficult thing to prove.
Jerry: Yeah, a hundred percent. There, there are some things that can help you, some things beyond just CVSS scores that can help you a bit, certainly if you look at something and it is worm able , right? Remote code, execution of any sort is something in my estimation that you really need to prioritize the the CISA agency, the cybersecurity infrastructure security agency, whose name still pisses me off.
All these years later, because they has the word security too many times in it, but they didn’t ask me. They have this list they call Kev. It’s the known exploited vulnerabilities list, which, in, in previous years was a joke because they didn’t update it very often. But now it’s actually upgrade updated very aggressively.
And so it contains the list of vulnerabilities that the U S government and some other foreign partners see actively being exploited in the industry. And so there’s a, that’s also a data point. And I would say. My perspective is that shouldn’t be the thing that you say that’s, those are going to be what we patch then it’s your, my view, your approach should be, we were going to patch it all, but those are the ones that we’re not going to relent on.
There’s always going to be a need. There’s going to be some sort of There’s going to be an end of quarter situation or what have you, but these are the ones that, that you should be looking at and saying, no, like these can’t wait they have to, we have to patch those.
Andrew: Yep. 100%. And a lot of your vulnerability management tools are now integrating that list. So it can help you right in the tool know what the prioritization is. But bear in mind, there’s a lot of assumptions in that, that those authorities have noted activity, have noted and shared it, understood it, and zero days happen.
Jerry: Somebody had to get, the reality is somebody had to get hacked.
Autologically, somebody had to get hacked for it to be on the list.
Andrew: right, so don’t rely only on that, but it is absolutely a good prioritization tool and a good focusing item of look, we have this, know we have this is known exploitable. We’re seeing exploits in the wild. We need to get this patched.
Jerry: Yeah, absolutely. So moving on to the next story, this one is from a cybersecurity consulting company called Cybernary. I guess it’s how you would say it.
Andrew: I’d go with that. That seems reasonable.
Jerry: The title is, I’m sure somebody will correct me if I got it wrong. Title here is what’s the worst place to leave your secrets. Research into what happens to AWS credentials that are left in public places. I thought this was a fascinating read, especially given where I had come from. I’ve been saying for some time now on this, on the show, API keys and whatnot are the next big horizon for attacks.
And in fact, we had been seeing that, we’re actually, I think on the upswing in the past, in my former role, we saw a lot of that manifesting itself as attackers using those to mine crypto, like they would hijack servers or platforms. Or containers or whatever to mine cryptocurrency.
But I think over time, we’re going to see that morph into, more data theft and perhaps less overt actions. I’m sure it’s, it is already happening. I’m not, I don’t mean to say that it isn’t happening, but I think it’s in the periphery right now where a lot of the activity, at least A lot of the voluminous activity tends to be what I’ll call more benign, like again, crypto mining.
But anyway, this the approach that this organization took here was pretty interesting. There’s a company called thinkst that has this kind of like. Concept of canary tokens and canary credentials. And they are exactly what they sound like. It’s a set, it’s a set of secrets that you can create through this company and watch how they’re used.
You can see, you can get an alert when somebody tries to use them. And that’s exactly what they did here. They created, I think it was 121, 121. That’s the number of total attempts. I don’t know exactly how many credentials they created. They created a number of credentials and they spread them around and they they used a number of different services.
Let’s see, they had GitHub, GitLab, Bitbucket, Docker hub. They created their own FTP web server. And blog, they put them on pastebin jfiddle. They put them into the NPM JS repository in the PyPI repository, which we just talked about. They put them in the various cloud storage buckets. And then they just waited to see how and when they were accessed.
And unsurprisingly to me this. Was pretty successful. It was a very successful test. I am a little surprised at some of the results. So the one of the, one of the charts they give is how long it took for different credit credentials planted at different places to be used.
Andrew: It’s insane.
Jerry: So the NPM JS was less than 60 seconds.
Andrew: So that means, from the time of posting, somebody somehow picked it up with some kind of scanner and could turn around and test using it in less than a minute.
Jerry: Yes.
Andrew: That’s insane.
Jerry: Pi was 120 seconds. So just shy of two minutes. GitHub was 127. 127 seconds, just just over two minutes. Pastebin was 50 minutes their own web server, 47 hours and docker hub 6. 9 days.
Andrew: Man, what’s going on with Docker Hub? Just nobody cares?
Around to it.
Jerry: nobody cares. I think it’s a lot more involved. It’s not as it’s not as readily scannable, I would say.
Andrew: I can tell you from my own experience in previous roles, we used to get reports all the time for Hey, you’ve got the secret out here. Hey, you’ve got the secret out here people looking for bounties. I still want to know what tools are using to find this stuff so rapidly because it’s fast.
Jerry: Yes.
Andrew: And
Jerry: Like a GitHub, GitHub will, we’ll give you a, an API so you can actually subscribe to an API. Again, it’s not perfect because it’s obviously they are. Typically relying on randomness or, something being prefixed with password equals or what have you. But it’s not a, it’s not a perfect match, but there’s lots of lots of tools out there that people are using.
The one that I found most interesting and it’s more aligned with the Docker Hub one, but not. And I think it’s something that is a much larger problem that hasn’t manifested itself as a big problem yet. And that is, with container images you can continue to iterate on them.
You can and by default when you spin up a container, it is the end state of a whole bunch of what I’ll just call layers And so if you, let’s say included credentials at some point in a configuration file, and then you later deleted that file, when you spin up that container in a running image, you won’t find that file.
But it actually is still in that container image file. And so if you were to upload that container image file to, let’s say Docker hub and somebody knew what they were doing, they could actually look through the history and find that file. And that has happened I’ve seen it happen a fair number of times you, you have to go through some extra steps to squash down the container image so that you basically purge all the history and you only ended up with the last intended state of the container file, but not a lot of people know that, like how many people know that you have to do that?
Andrew: well, including you, the six people listening to this show, maybe four others.
Jerry: So there’s a lot of there’s a lot of nuance here. So I thought, The time the timing was just. Fascinating. That, that it was going to be fast just based on my experience. I knew it was going to be fast, but I did not expect it to be that fast. Now in terms of where most of the credentials were used.
That was also very interesting. Hello. Was a little, in some areas, some respects, not what I expected. So the most the most targeted or the place where the most credentials was used was Pastebin, which is interesting because Pastebin also had a relatively long time to detect. And so I think it means that people are more aggressively crawling it.
And then the second most common is a website. And I think that one does not surprise me because crawling websites has been a thing for a very long time. And I think there’s lots and lots of tools out there to help identify credentials. So obviously it’s a little. Dependent on how present them.
If you have a password. txt file and that’s available in a index in directory index on your webpage. That’s probably going to get hit on a lot more.
Andrew: I’m, you know what?
Jerry: Yeah, I know. You’re not even going to go there. Yep. You’re I’ll tell you the trouble with your mom. There you go. Feel better.
Andrew: I feel like she’s going to tan your hide.
Jerry: See, there you go. You got the leather joke after all. Just like your mom.
Andrew: Oh, of nowhere,
Jerry: All right. Then GitHub was a distant third.
Andrew: which surprises me. I,
Jerry: That did surprise me too.
Andrew: Yeah. And also I also know GitHub is a place that tons and tons of secrets get leaked and get labs and similar because developers do have, it’s very easy for them to accidentally leak secrets in their code up to these public repos. And then you can never get rid of them.
You’ve got to rotate them.
Jerry: I think it. So my view is it’s more a reflection of the complexity with finding them, because in a repository, you got to search through a lot of crap. And I don’t think that the tools to search for them is as sophisticated as let’s say, a web crawler, hitting paste in the website.
Andrew: Which is fascinating that the incentive is on finding the mistake by third parties. Yeah. got better tooling then. Now, to be fair, all of these, like if GitHub for instance, has plenty of tools you can buy, both homegrown at GitHub or third parties that in theory will help you detect a secret before you commit it, but they’re not perfect and not everybody has them.
Jerry: Correct. Correct. And I also think it’s more in my experience. It’s much more of a common problem from a, from a likelihood of exposure from from the average it shop, you’re much more likely to see your keys leak through GitHub than you are from people posting them on a website or on pastebin.
But, knowing that if they do end up on pastebin, like somebody’s going to find them is I think important to know, but my Experience it’s, it’s Docker hub in the code repositories, like PyPy and MPM and GitLab and GitHub. That’s where it happens, right? That’s where we leak them.
It’s interesting in this, in this test, they tried out all the different channels to see which ones were more, more or less likely to get hit on. I think you get hub in my experience, GitHub and Docker hub and whatnot. are the places that you have to really focus and worry about because that’s where they’re, that’s where they’re leaking.
Andrew: Yeah. It makes sense. It’s a fascinating study.
Jerry: Yeah. And it
sorry, go ahead.
Andrew: I would love for other people to replicate it and see if they get similar findings.
Jerry: Yes. Yes. I, and this is one of those things that, again the tooling is not there’s not a deterministic way to tell whether or not your code has a password or not in it. There are tools, like you said, that will help identify them. To me, it’s. And it’s important to create a I would call the three, three legs of the stool approach.
One is making sure that you have those tools available. Another would be making sure that you have the tools available on how to store credentials securely, like having. Hash a car vault or something like that available. And then the third leg of the stool is making sure that the developers know how to use those.
Know that they exist and that’s how you’re, how they’re expected to actually use them. Again, it’s not perfect. It’s not a firewall. It’s, you’re still reliant on people who make mistakes.
Andrew: Two questions. First of all, that three legged stool, would that be a milking stool?
Jerry: Yes.
Andrew: Second, plus a question, more comment. I would also try to teach your teams, hey, try to develop this software with the idea that we may have to rotate this secret at some point.
Jerry: Oh, great point. Yes.
Andrew: and try not to back yourself into a corner that makes it very difficult to rotate.
Jerry: Yeah. I will also, I’ll go one step further and say that not only should you do that, but you should at the same time implement a strategy where those credentials are automatically rotated on some periodic basis. Whether it’s a month, a quarter, every six months, a year, it doesn’t really matter having the ability to automate it or to have them automated, have that change automated, gives you the ability in the worst case scenario that somebody calls you up and says, Hey, like we just found our key on on GitHub, you have an ability to go exercise that automation, but Without having to go create it or incur downtime or whatnot.
And that the worst case is you’re stuck with this hellacious situation of I’ve got to rotate the keys, but if I rotate the keys, the only guy that knows how to, this application works is on a cruise right now. And if we rotate it, we know it’s going to go down and we, so you’re end up in this That’s really bad spot.
And I’ve seen that happen so many times.
Andrew: And then the see saw ends up foaming at the mouth like a mad cow.
Jerry: Yes, that’s right. Cannot wait for this to be over. All right. The the last story mercifully is Mike is also from the register. com. And the title is Microsoft patches is scary. Wormable hijacked my box via IPv6 security bug and others. It’s been a while since we’ve had one that feels like this. So the the issue here is that Microsoft just released a patch as part of its patch Tuesday for a a light touch pre authentication, remote code exploit yeah, remote code exploit over the network, but only for IPv6,
Andrew: which to me is holy crap, big deal. That’s really scary.
Jerry: incredible.
Andrew: and either, I don’t know, I feel like this hasn’t gotten a ton of attention yet. Maybe because there wasn’t like a website and a mascot and a theme song and a catchy name.
Jerry: Yes. And
Andrew: But, so if you’ve got IPv6 running on pretty much any modern version of Windows, zero click RCE exploit, over, have a nice day. That’s scary. That’s a big deal.
Jerry: the better part is that it is IPv6. Now I guess on the on the downside it’s IPv6 and IPv6 typically is isn’t. Affected by things like NAT based firewalls. And so quite often you have a line of sight from the internet straight to your device, which is a problem. Obviously not always the case.
On the other
Hand, it’s not widely adapted.
Andrew: but a lot of modern windows systems are automatically turning it on by default. fact, I would wager a lot of people have IPv6 turned on and don’t even know it.
Jerry: Very true.
Andrew: Now you’ve got to have all the interim networking equipment, also supporting IPv6 and for that to be a problem, but it could be.
Jerry: So there, there’s the the researcher who identified this has not released any exploit code or in fact, any details other than it exists. But I would say now that Apache exists I think it’s fair to say every freaking security researcher out there right now is trying to reverse those patches to figure out exactly what changed in hopes of finding out what the.
Problem was because they want to create blogware and create a name for it and whatnot. I’m sure. This is a huge deal. I think it is for alarm fire, you’ve got to get this one patched like yesterday.
Andrew: Yeah. It’s been a while since we’ve seen something like this. Like you said, at the top of the story, it’s, Vulnerable, zero clickable, RCE, just being on the network with IPv6 is all it takes. And I think it’s everything past Windows 2008. Server, is vulnerable. Obviously patches are out, but it’s gnarly. It’s a big deal.
Jerry: As you would say, get ye to the patchery.
Andrew: Get ye to the patchery. I’ve not used that lately much. I need to get back to that. Fresh patches available to you at the patchery.
Jerry: All right. I think I think we’ll cut it off there and then ride the rest home.
Andrew: Go do some grazing in the meadow. As you can probably imagine, this is not our first radio.
Jerry: Jesus Christ. Where did I go wrong? Anyway, we I I sincerely apologize but I also find it. I also find it weird.
Andrew: I don’t apologize in the least.
Jerry: We’ll, I’m sure there’ll be more.
Andrew: Look man, this is a tough job. You gotta add a little lightness to it. It can drain your soul if you’re not careful.
Jerry: Absolutely. Now I Was
Andrew: But once again, I feel bad for the cow and the calf. That’s terrible. That’s, I don’t wish that on anyone.
Jerry: alright. Just a reminder that you can find all of our podcast episodes on our website@wwwdefensivesecurity.org, including jokes like that and the infamous llama jokes way, way back way, way back. You can find Mr. Clet on X at LER.
Andrew: That is correct.
Jerry: Wonderful, beautiful social media site, InfoSec. Exchange at L E R G there as well. And I am at Jerry on InfoSec. Exchange. And by the way, if you like this show, give us a good rating on your favorite podcast platform. If you don’t like this show, keep it to yourself.
Andrew: Or still give us a good reading. That’s fine.
Jerry: Or just, yeah, that works.
Andrew: allowed,
Jerry: That works too.
We don’t discriminate.
Andrew: hopefully you find it useful. That’s all we can that’s our hope,
Jerry: That’s right.
Andrew: us riffing about craziness for an hour and hopefully you pick up a something or two and you can take it, use it and be happy.
Jerry: All right. Have a lovely week ahead and weekend. And we’ll talk to you again very soon.
Jerry: Today is Wednesday, August 7th, 2024. And this is episode 275 of the Defensive Security Podcast. My name is Jerry Bell and joining me tonight as always is Mr. Andrew Kalat.
Andrew: Good evening, Jerry. How are you? Good, sir.
Jerry: I am amazing. It is blistering hot at the beach, but it’s awesome.
Andrew: recording from your southern compound.
Jerry: I am.
Andrew: Nice.
Jerry: Yeah, Bell Estate South.
Andrew: And Debbie was not an issue.
Jerry: Debbie not here. We got probably 45 minutes worth of rain.
Andrew: Yeah, it seems, at this point, in real time, stalled out over South Carolina
Jerry: Yeah, it looks several feet of rain hitting like Savannah and That is nuts. But no, it was not a big issue here. I was pretty worried. I packed up all my Milwaukee batteries with lights and whatnot in preparation for the worst got extra tranquilizer for my dog who hates storms.
But no, it’s been absolutely amazing here.
Andrew: So you took the tranks instead? Is that what I’m hearing?
Jerry: Absolutely. You gotta sleep somehow.
Andrew: That’s fair. I’m glad it was a non event, at least for your little neck of the
Jerry: Yeah, it was Nice you could actually see some of the storm clouds off in the distance. And that was the best way to watch a hurricane is when it’s far away.
Andrew: That’s true. That
A few I’ve been through. Stuck on islands, but
Jerry: Yeah, that’s right. since I’ve been here, I have been in the building for two hurricanes, and the building’s been hit by three tornadoes. And then there was also a unsuccessful base jump.
Andrew: So we’re saying you are cursed. Is that what we’re saying?
Jerry: am the human equivalent to a plastic flamingo.
which attracts tornadoes for those who don’t know. Anyway.
Yeah.
Andrew: after that meteorological update,
Jerry: Yeah. just a reminder that the thoughts and opinions we express on the show are ours and do not represent those of our employers past, present, or future.
Andrew: maybe even our
Jerry: Or our pets. my pet is licking me right now and she says, nope, it’s not her opinion.
Andrew: fair,
Jerry: Okay I would say that this is going to be a CrowdStrike heavy episode.
Andrew: three weeks in a row.
Jerry: Yeah, it continues to get more and more interesting. Obviously the main event itself is largely behind us and now we are in the lawyer up phase of the party.
Andrew: the blamestorming
Jerry: blamestorming has indeed begun. The first topic we have to talk about here is the actual formal full root cause analysis was released yesterday by CrowdStrike and it is a 12 page long document. It has lots of marketing fluff in it.
And only I would say a little bit of substance. I don’t think there’s anything that is remarkably telling or revolutionary in the document, but it does indicate technically what went wrong. And it gives some indications of the, potential improvements for their quality assurance, which I think is where a lot of this went wrong.
So the, I’m not going to go through the details in uber technical specificity, but the net is that this channel file update is for this inter process communication agent, for lack of a better term, I’ll call it. And that agent, expects configuration files that have
20 parameters, but through some unfortunate
bad planningtheir test harness actually was Marking the 21st as a catch all, as an asterisk. It was effectively being marked as not used. And so in this particular update, they actually started using it, and that ended up causing their parser to perform what ultimately ended up being an out of bounds read.
Because that parser wasn’t set up to actually read it. And so when that read attempted to happen in kernel space, it tried to access memory. It wasn’t allowed to access, wasn’t allocated. And that caused the blue screen. And because the same thing happened every time it booted up.
You just had this endless boot loop until that particular file got removed. I think the more substantive issue, and that’s the kind of thing that can happen,
Andrew: So let me restate that to make
The application was expecting. a file that had 21 fields in it, and it got a file with 20.
Jerry: Yes.
Andrew: And where it went to read that 21st, it wasn’t allowed to read, and the way that systems protect themselves to do a kernel panic and shut down if you’re trying to read something you’re not allowed to
Jerry: Yes.
Andrew: If you’re in
Jerry: Windows basically says something is horrifically wrong. This should not happen.
Andrew: If I went by that criteria, I’d shut down every day.
Jerry: And so if that were to happen in user space, the application that performed that read would crash. But when it happens in kernel space, Windows attempts to protect itself and it blue screens.
And so the challenge is that testing harness was built assuming that 21st parameter was always set up as a catch all and so effectively was being ignored.
And I think there were really two issues here. One was they didn’t have a very thorough, their testing harness obviously wasn’t, Properly designed, but then they also did not have staged deployments. Like they, what they have a process where once it goes through that test harness and passes it, it goes out far and wide.
There is no staged, deployment ring concept that you have in, let’s say, Microsoft Windows updates and whatnot. And because of that, it, it blasted out. Everybody implicitly trusted CrowdStrike updates and those got applied to pretty much as, fast as they were delivered and the rest is now history.
Andrew: I think it’s a very complicated series of events that led to this. And I think just reacting to a lot of the zeitgeist in the social media world around this, there’s a lot of angry finger pointing some of which is probably well warranted, but it’s interesting to see how the inner chain came together.
And going back to another area I know a little bit about is like aviation incident investigation and things like when space shuttles explode or fall apart, I’ve read a lot of books on those and that sort of thing. Anyway, it’s interesting how there’s very rarely one root cause is where I’m going with this. Usually series of events, an air chain that led to this situation. If one of these situations have been slightly different, this would have been caught and all the Swiss cheese holes lined up just right this situation to happen, not absolving or apologizing for it. It’s just interesting how complex these situations truly are. Compared to how a lot of people will knee jerk their opinions on things, usually based on their own bias around what they care about.
Jerry: When I was reading it, it reminded me of the show. I think you’ve probably watched it too called engineering disasters. And the history or the learning in each one of those episodes that a sequence of disconnected things all lined up in just the right way. for that disaster to happen.
Andrew: right.
Jerry: And I think that is definitely what happened here.
Andrew: For everybody involved, but there’s a part of me that finds these things fascinating to watch play out.
Jerry: I still think, for me, what is most, troublesome, because this is not unprecedented, right? Obviously the amount of systems that were impacted, is unprecedented, but that’s probably more a function of how interconnected and dependent we are on computers than any point in time.
But what’s interesting is that this sort of thing has happened in the past, right? This has happened with Symantec and McAfee and Microsoft probably about five or six different times and several others that I’m probably missing. But one of the things that, distinguishes this from those is that those were much less impactful because they did stage rollouts.
And so when it happened, it was devastating to the people who were among the first, the canaries that they had the problem. But this is a different thing. I think that the fundamental Coding and architecture errors are hard to foresee.
They’re easy to see in hindsight, right? this is like the signal and the noise thing. The failure is easy to identify after the fact, because it’s obvious. Like you can’t, duh, it’s so obvious that this was going to happen, but it’s only obvious after the fact.
Andrew: Certainly.
Jerry: weren’t looking at it beforehand and saying, Oh, we’re just going to accept the risk. They just, it wasn’t in their mind. And so, that part I find less, obviously it is the thing that caused it, but what I find most problematic is the fact that they hadn’t adopted what I would call the industry standard practice of the tiered rollouts.
Andrew: I’m sure that was an intentional decision. I obviously don’t know for sure. I have no idea about decisions that go on a cross track. I’ve never worked there. However, those in their mind, I would imagine a value in not doing, in
Jerry: sure,
Andrew: So dig, if you will, the picture, they are using these sensors, not just to stop things, but also together
these sensors are reporting back to CrowdStrike all the time about potentially malicious or confirm malicious activity. And then they use that data to shotgun this out. And this is basically a reject expression. They were trying to get out to the world about something they thought was malicious. they’re. Go to market strategy is to stop breaches specifically around ransomware. So their business model is really geared towards a ransomware. That’s in my opinion, they’re, I’m not speaking for them. It’s just my read of it. And they care greatly about stopping ransomware and windows environments. So they know that ransomware spreads quickly and rapidly, and I’m sure. In their mind that the value of getting these updates out as quickly as possible to as many sensors as possible a valuable thing to do. So let’s say they know about a new attack type and they pick it up at one customer and they start staging this and they stage it to 10 percent of the customers and 30 percent of the customers and then 50 percent of the customers.
Meanwhile, you’re at a customer who hasn’t gotten staged yet and you get hit by it. Are you happy with their staged deployment at that point?
No. End. So I’m just playing devil’s advocate a little bit on, it probably was an intentional decision to not do staged rollouts. It’s not like they just were unaware of that concept. I think they felt that there was value in updating as rapidly as they possibly could to as many sensors as they could.
Jerry: Likely, I would say that is definitely the proof is in the pudding, right? That is exactly what they were doing.
Andrew: A, in hindsight, it’s a bad decision, but it might not have been a bad decision at the time.
Inputs as to why they made that call, I guess is what I’m saying.
Jerry: I’m sure there is. And by the way, I’m not advocating for, something like what with Microsoft where the rollout happens over the course of days or weeks, I’m more talking about. Hours that, potentially in my mind, at least is a happy medium where you could conceivably have canaries that go an hour before the rest of the world, right?
So yeah, you miss an hour. And I’m sure that potentially exposes some number of customers and maybe this is such a rare thing that one hour difference is something that their customers feel important, to accept the risk on. I don’t know. it’s a fair question.
Andrew: They are saying in their root cause analysis that one of the things they will do as part of the mitigation is provide customers more control over the deployment of rapid release content updates. So to me that means maybe choose when and where rapid release content updates are deployed is what they’re saying. Like they want to go down that path of, hey, if you want to be as safe as possible from ignoring operational risk, or I should say, minimizing your concern about operational risk, deploy as quick as you can. If you are concerned about operational risk, here’s these tools, right? But the trade off you might be More exposed to malware risk.
Jerry: Correct. And, from a customer perspective, it does give you the opportunity to do your own soak testing. But I think you probably also, again, depending on your risk tolerance and size of your environment and exposure and whatnot, would do well to do your own sort of testing.
Before you push it out far and wide. But again, that takes resources. It takes time. It takes people, it takes labor. And if it drops on,after work, right before a long weekend, are you going to, you’re going to have the appetite to do that?
Andrew: and how consistent is your fleet?
Are you on the same patch level? Are you in the same build? Are you on the same hardware? There’s a whole lot of things that are coming to play. And if you’re only testing on a couple of examples,
Jerry: Yeah.
Andrew: yeah, I don’t disagree. I know I’m being a little contrarian on this. It’s more a matter of, again, people are saying things like, if you only just, and I’m like think that through a little, and you’re going to find out that there’s still holes in your planning and that you may have to accept some risk for the benefit of the tool. And have a way to solve that for you. There’s no perfect answer.
It’s you’ve got to figure out that trade off for your own organization.
Jerry: Yeah, I think that’s fair. The other, again, I don’t know a whole lot about their release pipeline process, but it, I find it fascinating that they clearly don’t as part of their release process, verify that it doesn’t blue screen a system like there, there’s a very different distinction between, this.
channel file update causes CrowdStrike to crash or for it to not detect things and then require a subsequent update. to fix it versus the agent is now creating an unbootable situation. And it seems like there’s a very utilitarian, very specific set of tests. And I think deterministic tests that they could put in line in their release pipeline when you deploy it onto.
X number of different windows systems. Does it crash? And if it doesn’t, then you release it with the assumption that if it’s not a catastrophic problem, any other problem could be fixed with an update.
Andrew: Yeah. Or if it’s not crashing on the largest deployed population set example, any crashes you have will be limited to relative small percentage of systems.
Jerry: All right. So anyway, it’s an interesting read. Um, there’s obviously I think it’s 12 or 13 pages long, a lot more technical details about the actual nature of the crash and whatnot. So I invite you to read it if you’re interested in that sort of thing. Moving on to the next story, which is related, this one I’ll take a step back and say for those of you who aren’t aware, there’s a bit of a war of words between Delta Airlines CrowdStrike and Microsoft.
So Delta CEO, Ed Bastian is very famously now said that this outage has cost Delta Airlines about half a billion us dollars in losses and that he feels compelled on behalf of his customers and shareholders and employees to try to recoup some of those losses. And so he’s signaled that he’s going to be suing.
both CrowdStrike and Microsoft. And we’ve talked about this a little bit in previous shows, but now CrowdStrike and Microsoft have separately responded to those legal threats. I don’t think they’ve actually materialized as filed lawsuits yet, but the first one is from the register and the title is CrowdStrike Unhappy About Delta’s Litigation Threat Claims Airline Refused Free On Site Help.
So this particular story is about a open letter that one of CrowdStrike’s lawyers sent to the legal counsel that Delta retained. And I think that legal counsel is pretty some fairly high profile legal team that was involved in in prior high profile cases. In this, I’ll summarize this letter is basically saying, you, Delta should be, cautious about what you ask for.
And by the way, if you are going to proceed with this, we would expect you to retain a certain set of information that we think will be useful in this litigation. And there was one quote, I think summarizes the whole thing very well. And it says, Should Delta pursue this path, meaning the lawsuit, Delta will have to explain to the public, its shareholders, and ultimately a jury, why CrowdStrike took responsibility for its actions swiftly, transparently, constructively, while Delta did not.
Now, there is an interesting adjunct to this that both Microsoft and CrowdStrike had offered assistance to Delta. And in the industry, there’s lots of, armchair quarterbacking about how Silly that is, because what would they really be able to do, right? Like in this particular instance, you need hands and feet going and sitting in front of computers to do the thing, to get the systems back up.
And so what were either CrowdStrike or Microsoft, really going to do? And I think that’s a fair characterization, but It’s not highlighted in this particular letter, but Delta, to both Microsoft and CrowdStrike basically said, no we’re good.
We don’t need your help. And so now both CrowdStrike and Microsoft are throwing that back in their face saying, if it was so horrible, why? Why didn’t you accept our offer of help? But I think that does beg the question, what help could they really have been? And I actually don’t have an answer for that.
Andrew: it’s interesting because these lawyers for CrowdStrike and Microsoft also go into this whole, and this is clearly for public, and posturing
Jerry: Oh yeah.
Andrew: lawsuits, but saying, Hey, what you Delta clearly were different in the way you approach this and your competitors. So what’s up with your organization that makes you different?
When why didn’t you want our help? What were you hiding? And, implying that. dirty laundry would come out on Delta’s side, and they’ve got skeletons in the closet that, Microsoft and CrowdStrike are aware of, that they are brandishing of, hey, if you want to drag us into court. You’re not going to come out of this unscathed, which I’m sure this was going to come up great during negotiations for renewal on CrowdStrike with Delta. I’m sure that sales team is really happy right now.
Jerry: Oh
Yes.
Andrew: this is wild. I also, and we’ve talked about this before, I don’t know what sort of legal leg Delta really has to stand on.
I guess this is what the court system will explore, but yeah it’s an ugly one. And it was pretty aggressive on both Microsoft and CrowdStrike’s lawyers parts to be like, Delta. Yeah, we screwed up, but you screwed up a lot too.
Jerry: So the Microsoft response. And there’s a similar article from the verge titled Microsoft says Delta ignored Satya Nadella’s offer of CrowdStrike help. And Microsoft similarly wrote a letter to Delta’s lawyers, but this one actually shines a light on a, perhaps a question or line of questioning that I hadn’t thought of before, and it does highlight again that Microsoft obviously did reach out and offer help, and they were told that, no, they Delta have it under control.
And what Microsoft and the reporter on the verge seemed to be insinuating is that the problem for Delta was not actually with windows. Obviously the problem started with CrowdStrike. Causing their window systems blue screen. But what they seem to be asserting is that when that happened, it created downstream problems in their legacy infrastructure, which is not based on windows and that then had to go get fixed.
And that is what took the place. All the time. Now that is an interesting point, although I will say it stains a little bit in contrast to some of the images that we saw of contractors standing up on ladders and airports rebooting. Display terminals, I think well after the, like up to a week after the outage, it’s an interesting point, what Microsoft is asserting here is that Delta has chosen not to modernize its it.
And when this incident happened, because of the fragility of their system, they were able to get their Windows systems back up and running relatively fast, as evidenced by their refusal. the problem they had that took all the time was with their old aging IT infrastructure.
Now it’s an interesting thing because legally, It almost doesn’t matter, because even if that’s true, and the court fines in favor of Delta, the fact that Delta didn’t invest doesn’t necessarily absolve either Microsoft or CrowdStrike of anything.
Andrew: Yeah. And Southwest is still running on the Commodore 64 and they were fine.
Jerry: true.
Andrew: counterpoint,
Jerry: That’s very true.
Andrew: it’s almost do you want the court to, Find what is due diligence and what is appropriate level of I. T. modernization. And you’ve heard this point a couple of times. You don’t think this is ever going to
Jerry: No, this is they’re trying this in the court of public opinion. This is never going to go to trial. This is going to get settled out of court. None of these parties, maybe Microsoft, I don’t know. They may be okay, but certainly CrowdStrike and Delta don’t want.
They don’t want, they’re not going to want it to go through the process of discovery and having all of this shit about them in their it programs and development programs laid bare for the public to see, they’re just not going to want that. So I think it’s going to get settled out of court. And right now this is about PR and damage control and everybody’s trying to make sure that their own we don’t have a story about it, but CrowdStrike’s shareholders have filed a lawsuit against CrowdStrike for for loss of shareholder value.
Andrew: Indeed.
It’s interesting. So I have one other topic on the CrowdStrike thing before I move on, if you’ll indulge me.
Jerry: And I think this is the LinkedIn posts, right? And I’ve got, I’m going to put these links in the show notes for you.
Andrew: Okay. So, Alex Stamos, who’s well known and, was at all sorts of interesting organizations recently joined SentinelOne Now SentinelOne is a direct competitor of CrowdStrike, so keep that in mind. And full disclosure. I happen to use Sentinel 1, so I know it better, but I don’t have any, strong opinions one way or the other, but I just want to be transparent about that. Anyway, so about a week ago, Alex put a post out on LinkedIn talking about this outage. So keep in mind it’s coming from a competitor. And he’s basically alleging it is false to say this could have happened to anybody. It’s clear that CrowdStrike made intentional architectural engineering and process decisions that led to this global catastrophe. All right. And he has a bunch of points, but there’s one specifically that I want to drill into here, which is 0.
4 in his post. And this addresses why are people running a kernel level? And why do they have kernel for these tools? And a lot of people have said, Hey, you shouldn’t do that. You don’t need to do that. You don’t do that on Mac. You don’t do that on other systems. Why are you doing in Windows? Point four, quoting, Additionally, there are models for architecting EDR with minimal kernel access, and the team at Sentinel One is, quote, willing to work with Microsoft on exploring these models, assuming Microsoft holds their own products to the same standards, end quote. So as I think this is a very key point to this entire discussion and this entire debate, because what this is saying, and this goes back to some things that Microsoft 2009, the EU through antitrust settlement Microsoft, who was competing at a, with a security tooling. other vendors to give other vendors the exact same access as Microsoft had to the kernel for the efficacy of their security tooling. So what Microsoft is saying, look, we didn’t have a choice. The government, the EU made us give folks access to this kernel. And what I’m reading here is saying, we also have access to the kernel because we have to compete with Microsoft’s tooling. So if Microsoft is willing to not be in the kernel with their security tooling, we’ll do the same. And that is calling attention, I think, to there’s obviously some benefit to running your security tooling at the kernel level if they’re worried about competing with Microsoft who has kernel access and they wouldn’t
Jerry: When I read that, whole thing is interesting because. It’s coming from a competitor of CrowdStrike. And so I come back to my new favorite legal term, which is corporate puffery,
Andrew: indeed.
Jerry: And I see corporate puffery. Obviously there’s some well founded points in here, but.
I will say that the first point and that fourth point stand in contrast to each other. Because on the one hand, they seem to be saying nobody else is doing it that way. CrowdStrike is standing alone in how they do this. And by the way we look forward to working with Microsoft to figure out how we can also not do it.
Andrew: Right?
Jerry: I don’t know. it’s an interesting point. there is associated video where Alex talks a bit more about his thoughts in depth. I think it fundamentally isn’t wrong. if there is an alternative way to do this, that is safer outside of the kernel and we don’t lose visibility we should be doing that.
Andrew: Yeah. speed. Severity of impact of the agent. There’s multiple things there. And I just think that there’s more at play here than just, we chose to do it because it was easier. I think there’s a competition issue here with Microsoft.
Jerry: Yeah. There was one point in here that I wanted to talk about. I picked up on his second point, which is quote, it is dangerous to claim that any security product could have caused this global outage because you’re telling CEOs, CIOs, and boards around the world that it is highly risky to deploy advanced security products.
In the long run, that makes it the world harder to defend and less secure. in any event so I don’t disagree conceptually. Like we shouldn’t be doing things that make our job harder, but we also shouldn’t be sugarcoating things and telling our senior leadership that, there is no risk
Andrew: The other thing I find interesting about this point that is not ever included in these conversations is the cost of the software.
So I’m making up numbers here. Don’t quote me, but I’m in the right area. Let’s say for the average company, average CrowdStrike agent per host. I, again, I’m making up numbers, but just. Work with me, 50 bucks a head, right? So it’s probably more than that, but let’s just say it’s 50 bucks a unit. What if I offered you a thousand bucks a unit, but I promise you it’ll never crash. Is it worth it to you? I’ve got to apply so many more resources and so much more time and so much more energy to do that extra checking and I’m going to develop slower and I’m going to be behind everybody else, but it’s more stable. Is it worth it to you? There has to be that trade off. There has to be that question of speed versus cost versus stability.
Jerry: So I,
Andrew: And we never
Jerry: yeah I think it just becomes unaffordable, right? It’s not a, do I want it? Do I think it would be, is it necessary? at that price point you’re basically. Probably spending two or three times more than your overall budget just on a single tool, not even including, what it would require to run the tool.
So I think it would be awesome to have, aviation, life safety grade stuff, but it’s really not. It’s not practical, but I guess my,
Andrew: that’s my point. It’s do we acknowledge that ever? That, hey, look, there are folks who develop things that run aircraft, that cost, many multiples of standard software development because of life, grade. I used a good term that I’ve blanked on. That isn’t affordable for the average generic system. But is it then fair to assume that they would have the same stability that highly refined, or however you want to say it, system?
Or are we just deluding ourselves?
Jerry: no I strongly suspect most organizations would not say that it’s worth it to them, right? They would trade off some amount of downtime or the chance of some amount of downtime for. To be able to a, not spend that exorbitant amount of money, and also to have generally a reasonable amount of protection in place, which I think is the implicit or the tacit trade off that we’ve made.
The thing that I guess is concerning to me is that this particular bullet seems to be asserting that, we shouldn’t. Be, upfront about the potential downside, right? I think what Alex and again, I didn’t talk to Alex. I don’t know, Alex, but I think what he’s trying to say is we have a hard enough time as it is.
We have a hard enough time allocating money for security programs as it is. And if we go and we tell our senior leadership team that, hey, not only do we have to spend all this money, but by the way, there’s, a non significant chance that it’s going to cause, a global outage for us because, it happens to everybody.
I think that’s what he’s railing against, but on the other side, I think we, at our own detriment, tell our boards that, there’s no risk in running CrowdStrike or SentinelOne or any other program
Andrew: let’s take it back to password managers, right? When somebody has a problem like LastPass did, for instance, We all came out and said, look, don’t throw password managers out.
Jerry: right?
Andrew: Like you’re still better off running a password manager statistically and everything else.
You’re still better off using unique complex passwords on every site. You manage through password manager. It’s still better even with these sorts of circumstances. So I feel that’s a similar conversation. Yeah, there’s a risk, but there’s no zero risk here. You can’t be in business and have zero risk. So got to, I think Alex knows better, frankly, than to try to imply that there’s a zero risk option here. There’s always some trade
Jerry: Yeah I think he’s two, two things. One is he works for a vendor and so he needs to make sure that companies still keep buying XDR software. But I think more broadly, he is hoping that we don’t shoot ourself in the foot by talking our senior leadership team out of allocating money for these important products.
Andrew: fair. And by the way, I don’t know Alex yet. I think I’ve met him once or twice, but I certainly am not trying to, Know what he’s saying. I think there’s a, it’s somewhat disingenuous to say that there’s zero risk in running software like this,
Jerry: All right. So moving on to more exciting things.
Andrew: From security week. The title here is , thousands of devices wiped remotely following mobile guardian hack. So mobile guardian is anMDM solution, mobile device management solution, who focuses on the education sector.
Jerry: they had an incident on August 4th that resulted in some number of their iPads managed through their platform, being deregistered in the process of being deregistered that caused the devices to be wiped. And there’s actually some fairly sad looking pictures of kids in front of stacks of broken iPads, because they’ve been deregistered.
wiped. this particular article doesn’t go into it, but it seems like this company mobile guardian has actually had a run of some security related issues leading up to this one. This one apparently isn’t related to the prior ones, but it does give me.
Concerns about the health of their offering. right now, in fact, they’ve actually got their offering deactivated and any of the systems that are managed by them are currently not functional until they. fix whatever issues they have. I think they took a proactive step of disabling their central management infrastructure.
So that’s rendered the functionality of those devices pretty significantly.
Andrew: Yeah, it’s an interesting attack factor. Somebody gets into your centralized management software and could just wipe all your devices. That’s tough to recover from. Not one I’d thought about before, to be honest, from an MDM
Jerry: Yeah. The it tools have long concerned me where you have central management of your infrastructure. In this particular case, it’s outsourced to a third party company, but we have lots of different flavors, whether that’s like Ansible or salt or Active Directory or many others that serves as a concentration point, like it is a place that an adversary can go in with one, you area cause untold chaos for your organization.
And so it really needs to be well protected. And, in this instance, the country of Singapore was pretty significantly impacted by this. And so they have actually decided to move off of a mobile guardian. Going to guess they won’t be the last one.
Andrew: Yeah, this is an interesting corollary of, is it worth it to have the protection? They’ve decided, nope, it
Jerry: Yeah I think
Andrew: need
Jerry: my guess is they’ll
Move to something else, but
Andrew: I should say not even just the protection, but the manageability. Which yeah, you’re right. They’re going to go to a competitor. I’m sure. But the capabilities offered by the centralized manageability is what can be used against you, which is what you were saying.
Jerry: Yeah,
Andrew: but I don’t know how you’d manage a modern large infrastructure without it. you just got to
Jerry: I don’t think you can, but I also don’t think that many it shops spend enough time thinking about how to protect those orchestration and management platforms. And I also don’t think that we necessarily don’t do enough due diligence on. companies like Mobile Guardian.
And again, I don’t know, this is the first time I’ve ever heard about them. I don’t know anything about them. They could be an absolutely fine company, just having a bad run. I don’t really know, but I think it’s a, to me, a highlight on an issue that I think is going to continue to grow in prominence.
So now on to, the other topic I want to talk about, which is really focused on targeting I. T. people. So the first story is from Bleeping Computer. The title here is Stack Exchange Abused to Spread Malicious PyPy Packages as Answers. And it, again, it’s not a, like a massive attack here, but I thought the technique was super interesting and worth talking about.
There’s a couple of different blockchain platforms, one of them called radium and I forget the name of the other one. Radium doesn’t have any Python modules. And so what the adversary here did was they created some malicious. Repositories in the PiPi repo. And then they created a whole bunch of fake accounts on stack exchange and they started answering questions and they started basically talking up how to use these bogus packages that they.
Uploaded into the PyPy repo and it worked, they got almost 2, 100 downloads.
Andrew: that’s actually really what we’re up to with the show. It’s a long con to start telling people to go install malware once we’ve gotten over 5, 000 episodes out and people trust us. That’s what we’re doing
Jerry: we’ll be there sometime around 2050.
Andrew: but
Jerry: amazing.
Andrew: They’re just, They’re abusing the reputational trust that comes with that platform of valid appropriate answers and Oh yeah, it’s like the easy button to go find an answer and yeah, I trust it. You get some malware, have a nice day.
Jerry: Yeah. So this particular one was, capturing credentials and other sensitive information out of big crypto wallets Telegram and other instant messaging software. And then also information out of browsers. But what I thought was quite interesting was that it was, focused on developers.
It was targeted at developers and As I guess now the former CISO, this causes me a lot of concern, right? That you have a population of developers who are, I would argue what makes an effective developer these days is figuring out how to get efficient answers to your technical problems. you can’t hire developers that know everything, that are completely prescient and omnipotent.
the average IT person is an effective IT person because they know where to go to get the answers to the problems they have. Whether that’s programming, or system administration, or helpdesk, or anything like that. And this worries me. And I think we have to be very cognizant of this trend going forward.
And I think in the near term, we at least have to be incorporating something about this into our security education for developers. But, I think this is probably going to come up more broadly. And that, by the way, dovetails into The other way that it manifests, or has manifested, and I’m sure there’s lots more ways, this one also comes from Bleeping Computer and the title is Ransomware Gain Targets IT Workers with New Sharp Rhino Malware.
And this is more of a watering hole slash
It’s, watering hole, SEO, domain hijacking, where they are purporting to be the angry IP scanner. They’re doing typo squatting or malicious ad buys, trying to divert you. The IT person to a malicious website that they control that looks like the legitimate source of the angry IP scanner for you to download.
And somebody who’s downloading that in the context of an IT or a security person in a company, probably more often than not, has some sort of elevated privileges because they’re probably trying to get that to do some important IT work.
Andrew: Indeed.
Jerry: Between the two, I think we should expect an increase focus on targeting it workers. I think in I think we’ve seen it people being targeted by more sophisticated adversaries for quite some time. What I think is becoming more novel is that this is really becoming a commodity type.
tactic where you’ve got, like lower tier ransomware actors and people trying to steal crypto wallets, actually targeting IT people. So I think we’re almost certainly going to see more and more criminal elements coming in and using these sorts of tactics to try to infiltrate, we know that entry as a service or, that initial entry point into an organization is part of the industrialization of the attack chain where You can, as a ransomware actor or as somebody trying to steal intellectual property, go and buy access.
So you’ve got bad actors who are trying to find ways of Infiltrating companies. And this I think is going to become a much more prominent and effective way in the future, because we, it people hold ourselves in high regard in our ability to not fall for stuff.
Yeah.
Andrew: Believe me, it happens. And the bad guys can keep getting more sophisticated and keep trying different tactics and they start to hit on things that work for a while or, catch certain people at a certain time in a certain set of stressors or certain circumstances that just, it’s, I was curious, going back to the malicious PyPy packages. I do wonder if any static code analysis tools would have picked that up. I don’t know. Maybe I’ll play with that.
Jerry: This is
Andrew: Or that sort of
Jerry: the they were not detected by. Like when the normal anti malware, but I don’t know if they would have been detected by, your aquas and and whatnot.
Andrew: And is it worth running? Having a process of any third party code? That you bring in and scan. I don’t know. I’m just musing out loud here. I honestly don’t know. And I think for a while, wasn’t Google, I think a couple people were trying to do like a safe package repository where, it was vetted and secured.
And I don’t know how well that’s been adopted, but it might be another idea. Don’t just get random stuff, like only go to the safe
Jerry: There’s,
Andrew: But that may slow you
Jerry: yeah.
Andrew: may not be able to answer all the questions you need that way. And, you may not find what you need there.
Jerry: lot, there’s a lot of. focus on supply chain. And I think we’re seeing a lot more focus on supply chain security as it pertains to open source? But my observation is that tends to focus more on Products where we’re, to some extent becoming compelled to care about this, because for example, the U S government has created some, regulatory hoopla where you now have to create S bombs and you have to make attestations about the integrity of your software development practice and whatnot.
So I think. in certain contexts that make sense. But there’s a lot of companies that have a lot of developers that don’t make products for sale or offer services for sale. We have huge numbers of people creating internal applications and whatnot. And I don’t expect, I think it’s a very long time before we see that kind of maturity being applied to purely internal operations.
Andrew: fair. And, these are also expensive sort of things to ask an organization to do it. You’d have to be pretty mature and pretty well staffed and well funded to, to do some of these things too.
Jerry: I definitely agree, but I think we have to be mindful about the ecosystem that we operate our, what I’ll call privileged users in, right? it goes back to that quote, with great power comes great responsibility. If you’re going to have IT people with elevated privileges, they should have to operate within a tighter set of parameters.
And whether that’s, the context of what Microsoft used to call their red forest, I forget what they call it these days. I’m getting old. But I think it’s very difficult to combat all of the different ways these things can manifest. And in particular, a lot of companies aren’t going to have the tolerance, to, stop their employees from leveraging open source that way.
But that doesn’t mean that we get to throw up our hands and say it’s just too hard. I think it means that we need to take a different look at how we construct our environment in a way that is more tolerant to the problems that can come from this sort of thing.
Andrew: Yep. I would agree. It makes me think also somewhat similar, like if they’re a high value target, Similar to how executives are a high value target. It’s only a reason they shouldn’t break the rules or have exceptions to the rules because the consequences of them being hacked in some way are so much higher than the average employee.
Jerry: Oh, a hundred percent. Absolutely. All right. That is the show for today. I appreciate everybody’s attention. Thank you for working through my noisy cat and our technical problems. If you like the show, you can find all of our back episodes on our website at defensive security. org. You can also find us on your favorite podcast platform.
I think we’ve got them all now, if I’m not mistaken, pretty much all.
Andrew: Unless there’s ones out there we don’t know about.
Jerry: Yeah,
Andrew: We’ll
Jerry: absolutely. You can follow me on the Fediverse at Jerry at infosec. exchange. You can follow Andy where?
Andrew: Also on the Fediverse at lurg, L E R G, at infosec. exchange. Also, I’m still hanging around the Twitter slash X world. Also at lurg, L E R G.
Jerry: All right. I look forward to talking to everybody again real soon. I hope you have a great week.
