Paul Asadoorian is joined by Chase Snyder and Vlad Babkin to unpack FortiBleed, a large-scale Fortinet credential-harvesting campaign, and what it reveals about network edge security. Welcome to episode 77 of Below the Surface . Paul Asadoorian sits down with Chase Snyder and Vlad Babkin to examine FortiBleed, a multi-phase campaign targeting Fortinet FortiGate and FortiOS SSL-VPN environments. The discussion begins with the leaked attacker infrastructure and quickly moves into a deeper question: what happens when the firewall itself becomes the foothold? The episode is less about a single vulnerability than about the chain of assumptions defenders rely on. Patching matters, but FortiBleed shows why patching alone may not be enough when attackers have harvested credentials, cracked old hashes, created backdoor access, or used network devices as passive sniffers. Eclypsium’s own analysis of FortiBleed describes it as a campaign that cracked administrative credentials for a large share of internet-facing FortiGate firewalls and showed why persistence below the operating system can outlast ordinary remediation. Paul, Chase, and Vlad use the campaign to explore broader lessons for defenders: credential hygiene, auditability, firmware integrity, cloud authentication risk, Zero Trust inside the network, and the operational difficulty of securing appliances that often sit outside EDR visibility. The conversation also turns toward AI-assisted vulnerability discovery, Mythos, and why faster vulnerability finding does not automatically mean faster remediation. Key Topics Covered FortiBleed as a campaign, not a single CVE: The hosts frame FortiBleed as a credential-harvesting and access-brokering operation rather than a conventional vulnerability event. The campaign combined historical Fortinet leaks, infostealer logs, brute-force cracking, credential validation, and post-compromise sniffing. Why the firewall becomes the foothold: A compromised FortiGate is not just another asset. It sits in the path of authentication traffic, VPN traffic, administrative sessions, and service-account activity, which makes network devices unusually valuable to attackers. Eclypsium describes network infrastructure as difficult to defend because these devices often cannot run EDR agents and operate outside traditional endpoint monitoring. Open directory exposure and attacker tooling: Paul explains how researchers gained insight into FortiBleed after a threat actor exposed an HTTP server with an open directory. The files reportedly included credential databases, scanning tools, sniffing tools, deployment scripts, Hashopolis job logs, and victim metadata. Credential abuse over malware deployment: One of the striking points in the episode is that much of the FortiBleed activity did not require malware on the Fortinet appliance itself. Attackers used legitimate FortiOS functionality, stolen credentials, backdoor credentials, and living-off-the-land techniques. Weak password hashes and cracking at scale: Vlad explains the FortiOS password-storage history, including older AK1 and SH2 hashes, PBKDF2 migration, and cases where older hashes remained available in configuration backups. The hosts discuss how cloud GPU access lowers the barrier for cracking SHA-256-based password hashes. The “old password” problem: The episode spends significant time on Fortinet’s old-password field and why preserving prior hashes for downgrade compatibility creates risk. The hosts argue that hidden security state is dangerous because administrators cannot easily audit or act on what they cannot see. Firmware updates are necessary but incomplete: Paul emphasizes that defenders should still update the operating system and firmware on network devices, but FortiBleed shows that version status is not the same as integrity. Eclypsium’s FortiBleed analysis makes the same point: a device can report as patched while still carrying artifacts of compromise. Passive sniffing and internal credential exposure: The hosts discuss FortiGate Sniffer as an orchestrator that uses access to Fortinet devices to capture credentials from traffic crossing the device. Vlad argues that defenders should assume a compromised firewall can observe internal traffic and respond by encrypting internal services, not just protecting the perimeter. Initial access brokers and monetized credentials: Chase asks whether anyone has claimed responsibility. Paul explains the SantaAd attribution and describes the likely business model: collect validated access, enrich it by organization and revenue, and sell it to ransomware operators, espionage groups, or other threat actors. Honeypot detection and attacker refinement: Paul notes that the attackers appeared to filter aggressively for real FortiGate systems and used simple but effective honeypot detection, such as treating multiple successful credential guesses in a row as suspicious. Network edge incidents are hard to scope: Chase focuses on the defender’s problem: the team responsible for network appliances often lacks the telemetry, access, and tooling needed for rapid incident response. That creates a gap between knowing a campaign exists and being able to determine whether a device is clean. Logging, auditing, and external log storage: The hosts discuss the value of FortiOS logging, especially for detecting packet-sniffing commands or unexpected administrative activity. Vlad stresses that logs should be stored outside the appliance so attackers cannot easily erase evidence from the device they control. Layered security and internal encryption: Vlad pushes back against the idea that any security appliance can be a single control point. If an attacker compromises the firewall, internal HTTPS, certificates, PKI, secure authentication protocols, and credential rotation become the next layers of defense. AI vulnerability discovery and the patching bottleneck: Chase connects the conversation to Mythos and Project Glasswing, noting that companies may find more vulnerabilities with AI, but still need engineering, testing, release processes, and customer deployment. Eclypsium’s Mythos analysis similarly argues that faster discovery does not automatically solve patch deployment. Auditability as the long-term lesson: Vlad argues that security appliances need to become more auditable. If attackers can reverse engineer firmware and inspect device internals, defenders need more transparency, not less. Timestamps 00:00 – Setup, livestream checks, and episode introduction 01:06 – Paul introduces Below the Surface episode 77 with Chase Snyder and Vlad Babkin 01:28 – Eclypsium resources and setup for the FortiBleed discussion 02:00 – FortiBleed as a credential-harvesting and access-brokering operation 03:30 – A brief detour into AI models, writing, and research workflows 04:49 – Open directory exposure, leaked attacker files, and AI-assisted analysis 07:30 – Credential databases, custom tools, Hashopolis logs, and victim sorting 08:30 – Why the campaign relied heavily on credentials and living off the land 10:08 – FortiOS hashes, old-password fields, and large-scale brute forcing 12:24 – Verizon DBIR trends, network device exploitation, and breach reporting 13:45 – Scale of affected FortiGate devices and why the numbers matter 14:44 – Vlad explains AK1, SH2, PBKDF2, encrypted passwords, and legacy hash behavior 16:54 – Configuration backups, downgrade compatibility, and hidden old hashes 19:33 – Secure-by-default failures and hash mismanagement 20:28 – CVE-2018-13379, prior Fortinet leaks, and the Belsen Group configuration dump 21:30 – Why Fortinet is not the only vendor with appliance password-storage problems 23:08 – Security through obscurity, auditability, and assumptions in password-system design 24:51 – Multi-factor authentication, cloud authentication, and single points of failure 26:14 – SSH keys, certificate-based authentication, and management complexity 27:40 – What better FortiOS hash migration might have looked like 29:04 – FortiGate Sniffer and passive credential capture across protocols 30:15 – SantaAd attribution and the initial access broker model 32:13 – Internet scanning, Shodan filtering, and honeypot detection 33:44 – SSL certificates, company enrichment, revenue, and access pricing 36:24 – How defenders should classify incidents involving old vulnerabilities and leaked credentials 37:56 – Privileged access requirements for auditing and fixing network devices 39:11 – Why removing sniffing capability is not a complete defense 39:46 – Internal encryption, Zero Trust, TLS, and PKI as compensating controls 41:05 – Modeling what happens if the firewall is compromised 43:12 – FortiOS logging, sniffer-command detection, and early indicators 43:53 – External log storage, firewall inventory, version tracking, and credential rotation 44:44 – Firmware updates, symlink persistence, and malware that survives patching 46:48 – Why this pattern applies beyond Fortinet to PAN-OS, Junos, MikroTik, Ubiquiti, Cisco, and others 48:52 – Mythos, AI vulnerability discovery, Project Glasswing, and patch deployment limits 52:46 – Cases where there is no clean patch and configuration becomes the mitigation 54:15 – AI, reverse engineering, obscurity, and the need for more open security models 55:17 – Closing thoughts Links & References Core References FortiBleed: You Can’t Patch Your Way Out of This – Eclypsium’s analysis of the FortiBleed campaign Eclypsium Network Device Security – Background on defending VPNs, firewalls, routers, switches, and other network edge devices Firmware Security for Enterprises – Eclypsium resource on firmware visibility and integrity Protecting Your Fortinet Devices With Firmware Security – Earlier Eclypsium analysis of Fortinet risk and firmware security Fortinet Under Fire: Network Edge Attacks Start Strong in 2026 – Eclypsium post on Fortinet authentication bypass and network edge attacks Firewalls Are A Growing Security Problem – Eclypsium post on firewall exploitation, persistence, and edge-device risk Mythos Finds Vulnerabilities. But Can Anyone Patch Fast Enough? – Eclypsium post discussed near the end of the episode Vulnerabilities and Campaign References CVE-2018-13379 – FortiOS SSL-VPN path traversal flaw discussed as a source of prior credential exposure Belsen Group FortiGate configuration dump – Mentioned as another source of FortiGate configuration files and hashes SantaAd – Threat group attribution discussed in the episode Hashopolis – GPU-cluster job-management system referenced in connection with hash cracking Shodan – Used in the discussion of internet-facing FortiGate discovery and filtering Verizon DBIR – Referenced by Chase in the discussion of vulnerability exploitation and network device breach trends Tools, Platforms, and Technical Concepts Project Glasswing Fortinet FortiGate FortiOS FortiOS SSL-VPN FortiCloud FortiGate Sniffer SHA-1 / AK1 SHA-256 / SH2 PBKDF2 John the Ripper Hashcat LDAP RADIUS Active Directory OpenVPN WireGuard SSH keys TLS / HTTPS PKI Mythos Transcript Paul (01:06.282): Welcome to Below the Surface. This is episode number seventy-seven, being recorded on June 26, 2026. I’m Paul Asadoorian, joined by Mr. Chase Snyder. Chase, welcome. Chase (01:17.25): Hey Paul. Paul (01:19.414): What’s going on? Mr. Vlad Babkin is here with us. Vlad, welcome. Vlad (01:25.119): Hello? Paul (01:28.396): This is a weird lag—problems always arise once you start recording. This is the rule of podcasting. Now it seems better. Hopefully it keeps up for the remainder of the show. Before we get started, Below the Surface listeners can learn more about Eclypsium by visiting eclypsium.com/go . There you’ll find the Ultimate Guide to Supply Chain Security, an on-demand webinar I presented called Unraveling Digital Supply Chain Threats and Risk, and a paper on the relationship between ransomware and the supply chain. Vlad (01:40.316): Let’s hope so. Paul (01:57.406): And a customer case study with DigitalOcean. If you want to see our product in action, you can sign up for a demo. All that at eclypsium.com/go . This week we probably have some other topics too that we might flesh out once we talk about FortiBleed. FortiBleed was the big one that happened recently. We put out a blog about it, we put out a lot about it, and we’ve got detections in the works for some of the password hashes we’ll talk about—decisions Fortinet made to prevent administrators from being locked out that ended up causing some unintentional exposures, as I’ll call them. To set the stage, I generated some AI show notes—not to publish or cheat, but because the three of us have learned a lot about FortiBleed and I wanted some talking points. I have to say, the opening paragraph generated by Claude Sonnet 4.6 is pretty awesome. You could read it almost like a movie trailer: “FortiBleed: It’s not a zero-day, it’s not a single CVE. It’s the name given to a long-running, multi-phase, industrial-scale credential harvesting and access brokering operation targeting internet-exposed Fortinet FortiGate firewalls and FortiOS SSL-VPN gateways.” FortiBleed coming soon—actually, already coming to a network near you. Chase (03:25.543): Right. Chase (03:30.813): In a world… Do you choose Sonnet 4.6 for writing specifically? Do you choose it because of cheap tokens? Why are you on that older model? Paul (03:38.026): Yeah, I think it’s cheap. In terms of the token price combined with its effectiveness for writing and doing basic research, I find it to be pretty good. I probably need to go back and try some other models, but that seems to be the one I like. I reserve Opus for my coding and projects, and I try not to use it to generate content. Chase (03:52.62): Okay. Chase (04:04.427): Yeah, good ratios. There’s always something going around about which models are the best; everyone has different benchmarks. But I heard a fairly compelling argument that Gemini 1.5, maybe, is really great at writing. Which would be a huge get for the Gemini models. I feel like I’m off-topic already. I can’t stop talking about AI models, but the— Paul (04:12.48): And that changes. Yes. Paul (04:21.024): Mm-hmm. Paul (04:28.532): That’s okay. Chase (04:32.375): The Google models are really great at everything except for coding. I would not use Gemini for coding, but images, video, and writing, apparently. I haven’t actually been using it for that, but maybe I’ll try. Those are also good things. Paul (04:49.824): Yeah, and that’s pertinent to this topic because there’s a lot to FortiBleed—there’s a lot to the campaign. It says in the notes that due to an open directory, which is how we learned about it, a security researcher, Volodymyr (aka Bob Diachenko), discovered that a threat actor had accidentally exposed an HTTP server running an open directory, and they grabbed 319 files. This is where AI helps us. Someone did the analysis of most of those files and what they contained, and there were multiple publications. As security researchers working for a cybersecurity company for defenders, you have to consume all of that and figure out what it means to you. That’s a good use case for AI. I heavily used AI to help query what was being publicly published on this, identifying the best sources. If I found a source, I fed that in, like, “Hey, take a look at this.” I don’t know if we link to the report, but there was a great report—well, the Hudson Rock one is the one that gets a lot of press, but there was another one that published an analysis of it, and I can’t remember which one that was. That was a PDF—I think it was a SOCRadar PDF or one of those companies. Consuming and making sense of something this large certainly requires—AI is really good at it. Those 319 files included a credential database, custom scanning and sniffing tools, and deployment scripts. The Hashopolis job logs—which was the GPU cluster they were renting to crack the hashes—and victim data sorted by industry, revenue, and headcount, which most of us inferred the threat actors were collecting in order to sell. What’s interesting is that, to my knowledge, there was no malware deployed to a Fortinet device. This was 100% credential abuse, coupled with living off the land on a Fortinet device. The malware and tooling ran on Windows and Linux systems and other places. They weren’t necessarily using malware against the Fortinet devices, which, from an attacker’s perspective, is great because it gives defenders fewer indicators to go on. For cybersecurity people like us trying to help customers detect this stuff, it really sucks because they’re living off the land. And so, you can glean a lot from your logs. I can talk about that later. Certainly, having good log telemetry, even though I’m not a huge fan of relying on logs for some of these indicators, can help. Essentially, this campaign revolved around credentials and collecting credentials in ways that maybe a lot of us hadn’t put together in a single campaign. It’s interesting as people ask us questions about this; they’re like, “Well, if my Fortinet device isn’t exposed to the internet, I’m fine, right?” I’m like, “No, you’re not,” because they were collecting Fortinet credentials from infostealers on Windows systems. If you were an admin logging into a Fortinet device and you had an infostealer on your machine, that’s one way they grabbed your credentials. Once they got into a Fortinet device via some credential, they also used credentials known to be backdoor credentials on Fortinet devices from previous campaigns. Threat actors would use an exploit for a vulnerability on an internet-exposed Fortinet device, implant a backdoor, and then those backdoor credentials were used to attack the entire internet of Fortinet devices. But then it leapfrogged. The way they built their credential database is they essentially turned every Fortinet device into a passive sniffer, listening for specific protocols using a built-in FortiOS command to sniff packets. This means you really just need one compromised Fortinet device, Paul (09:39.832): and any traffic going through it, attackers were looking for credentials—not just for your Fortinet devices, but also other things. They were building large datasets of credentials based on this. For the attackers, that’s pretty smart. If Vlad and I were to construct this attack, putting our evil hats on, it might look pretty similar; it makes sense. Vlad (10:04.722): Yeah. Vlad (10:08.584): Yeah, they pretty much harvested a lot of credentials across the board. They logged into devices and harvested credentials. As far as I’ve read in write-ups, they also took into account the ‘old password’ field, which FortiOS for some reason decided to keep—even if you rotated your hashes to be more modern—so that they could brute-force them. Then they just ran a massive GPU cluster to try to brute-force all of the hashes they got. This is like— Paul (10:29.536): Yeah, so they would— Vlad (10:37.494): Literally just scaled-up credential harvesting. Paul (10:37.494): Yeah, it’s interesting. They were preying upon SHA-256 hashed passwords. I don’t know where we stand as an industry as to how much of that was a problem before now. We knew that there were weak—I don’t want to say weaknesses, but computing—yeah. Vlad (10:56.645): It is a problem. Vlad (11:00.72): It’s called weak hashing, and it’s a common weakness. If you use SHA-256 for a password, it’s a weak hash. This is defined in, I believe, MITRE’s CWE—I don’t remember the exact numbers, but there is a specific weakness that identifies it like this. Paul (11:20.204): Okay. I think we almost got away with it until you just needed a credit card and a 40-GPU cluster to crack SHA-256 hashes, which is exactly what the threat actors did. They realized these Fortinet devices have passwords hashed with SHA-256, so they spun up 40 GPUs in a cluster and started cracking them. We’ve progressed to a point where the threat actors can do that easily. Previously, building and setting that up was a barrier. That barrier is much lower now, because all you need is a credit card, which is interesting. Vlad (12:03.07): Mm-hmm. Vlad (12:07.305): If you look at the movie *Terminator* and how many teraflops Skynet needed to run, a single RTX 4090 has, I believe, twice that amount. Modern GPUs are powerful for not a lot of money. If you buy a whole cluster of those, you can just go and crack hashes. Paul (12:19.54): Mm-hmm. Paul (12:24.277): Right. Chase (12:24.621): This is good. I’m glad to see us— Chase (12:29.559): —returning to the foundational benchmarks of our industry: the amount of teraflops required for Skynet. It’s the comparison against which all other systems are judged. Something interesting to me is a trend we’ve been tracking, which Verizon also tracks in their Data Breach Investigations Report (DBIR). Paul (12:41.056): I like how we’re measuring things. Yeah. Paul (12:47.53): Yeah. I would still— Vlad (12:48.016): This— Chase (12:57.673): It’s the rise of vulnerability exploitation in general as an initial access vector, and specifically the presence of network devices in breaches. It feels like this FortiBleed event could meaningfully influence both of those numbers. It happened recently enough that it’s not reflected in the 2026 Verizon DBIR report, but I’m going to be curious if they report numbers about this next year. If they include these Fortinet devices as network edge devices present in a breach, it’s a large number of devices. It’s one big campaign that could really shift things. Paul (13:45.814): Well yeah, because if you combine Fortinet devices that were or are vulnerable, or have a credential gathered through all the means we described previously, we find that—it says it right here—they had credentials for approximately 73,000 to 86,000 internet-facing Fortinet FortiGate firewalls. That was roughly 50% of all internet-facing Fortinet devices because of the way they structured the campaign so precisely to compromise these platforms. The password thing is interesting. I’m going to explain it one more time, and then I swear I don’t ever want to explain how Fortinet stores passwords like this ever again. The question keeps coming up because it’s confusing on the surface. If you don’t dig into the details, you’re left wondering what is happening. Chase (14:19.735): Those are crazy numbers. Paul (14:44.596): I don’t have the specific versions, but up until a certain version of FortiOS, passwords are stored in a SHA-256 hash. Or what’s the other one, Vlad? AK1? Vlad (14:59.358): There are four hashes. Let me explain this in detail. There is the AK1 hash, which is based on SHA-1, and it’s not very hard to compute. There are public modules in John the Ripper and Hashcat, so you can just find the implementation and implement it in Python if you want a proof of concept. There is the SH2 hash, which is based on SHA-256, and again, it’s Paul (15:02.038): So there are four. Yeah, go ahead, Vlad. This is part of Vlad’s life now, too. Paul (15:11.276): Yeah. Vlad (15:27.354): almost the same as AK1; it just replaces one hash function with another. There is a third hash type from modern versions—I don’t remember the exact update when it was introduced, but it is based on the actually secure PBKDF2 function. There is another hash type used by older FortiOS versions where the password was not stored hashed but encrypted, so you could decrypt it straight if you got the key out of the firmware. There is one other caveat. Paul (15:51.03): Mm-hmm. Vlad (15:57.531): If you migrate from SHA-256 to PBKDF2—and I believe migrating to SH2 is probably the same—FortiOS stores the so-called ‘old password’ property inside user accounts. This is not normally visible when getting the configuration. But if you make a configuration backup, you get that old password. Even if you rotated your hash to PBKDF2, but never told FortiOS— Paul (16:23.51): You have to log in. Vlad (16:25.19): No, it’s not just logging in. If you log in, you will get the updated hash, but the old password still stays. Paul (16:35.828): Right. Until the administrator logs in. Vlad (16:38.878): No, until you remove it forcefully. It serves to allow downgrading. Paul (16:41.6): I thought the Fortinet docs say it wipes it out when you log in. Is that not true? That’s interesting. Vlad (16:47.1): Maybe. I believe I read a document saying it doesn’t. Regardless, you needed to take action to wipe it. Paul (16:54.25): Yes. It’s interesting—you can’t just go change these hashes on your own as far as I know. But if you upgrade from an older version that was hashing them with a weak algorithm, when you upgrade to a certain version or later, it replaces the hashing function with PBKDF2 and stores the old hash. But you can’t see your old hash if you do a show config or the equivalent in FortiOS. You need the full configuration backup. For those of us managing even IoT routers, you go into the management interface and do a full configuration backup; in that backup is where you’ll see your old password hashes. They did that in case—and here’s where you find configuration backups because Vlad (17:25.618): Yep, you need the backup. Paul (17:50.568): if you’ve ever administered these devices and you’re upgrading from one version to another—let’s say from version one to version two—when you’re on version one, the documentation almost always states to do a full configuration backup and then upgrade. So that configuration backup is there. If you have problems, let’s say and you need to revert to version one, that old password hash gets restored so you can still log in. If they wiped out the hashes during the upgrade and you had to revert, it would lock the administrator out. That’s why the Fortinet docs state they preserve that old password hash. It’s kind of weird they preserve it only in the full configuration backup. I guess it’s one way to do it. When a new administrator logs in, we know everything is good. Although I’m curious: if you upgrade, everyone logs in, those old hashes go away, and then you revert, what happens? Are you locked out? Potentially. Vlad (19:04.84): Yep, that’s what I read. If you set a specific option, old hashes are wiped out and administrators are locked out. But if you don’t set this option, old password hashes are stored, so if you revert, you can still log in. This is not obvious and completely hidden—a design choice to hide that the old hash still exists from the administrator, requiring a config backup to check if it’s there. Paul (19:07.498): Yeah. Paul (19:24.043): Yes. Vlad (19:33.734): And then they have a whole separate advisory to explain it. Fortinet, are you a security company or not? Because if you are, you’re not doing a good job at security. You don’t do this with hashes. Paul (19:44.798): It’s interesting that threat actors figured this out. Even if you upgraded to a new version, they were recovering your old password hashes and cracking them. That’s how we learned about all this. Vlad (19:52.796): Yep, this is hash mismanagement at its finest. If you update the hash, either prompt the administrator to commit to the upgrade, or tell them it’s still insecure and show the old hash in the configuration so they can check easily without digging through tons of documentation. There’s a secure-by-default principle. If you aren’t secure by default, it’s going to fail, and it’s failing right now. Paul (20:28.406): Yeah, and the other way they got some of these hashes was through previous leaks. CVE-2018-13379 was a FortiOS SSL-VPN path traversal flaw that exposed plaintext credentials, and I believe someone collected around 500,000 accounts from that. They had access to that previous leak. Vlad (20:51.902): Mm-hmm. Paul (20:56.534): but never wiped out your old hashes, an attacker could still get in, recover those old hashes, and compromise other accounts on that system. There was another leak, too—the Belson Group dropped 15,000 FortiGate configuration files, and they started cracking all of those passwords. They were building a giant password database for FortiGate firewalls based on multiple sources, including infostealer logs, which is just amazing. Vlad Babin (21:30.729): Yeah, and while we are singling out Fortinet, they aren’t the only ones with this mess in password storage. A lot of devices actually have this. A surprising number of security appliances cannot even manage their own hashes. This is a paradox: you buy an appliance to make your network secure, but at the same time, it becomes less secure. At this point, we should start calling them ‘insecurity appliances.’ Paul (21:38.848): Yeah, agreed. Paul (21:42.764): Yeah. Paul (22:07.552): Yeah, the bar is not very high with a lot of these things. You’re right, Vlad, this underscored the need to audit your password hashes, especially on devices where you don’t have great visibility into what can happen when weak password hashes are in play. Vlad Babkin (22:29.086): To give a bit of a silver lining, at the very least FortiOS is explicit about the kind of password storage they use. Many devices are not. They store the password somehow, and you never know, because you don’t have any documentation. In many cases, the password is reversible. I don’t know what’s up with MikroTik, for example. Paul (22:37.276): Mm-hmm. Paul (22:47.552): Yep, and you don’t know what hash it is. It’s an important point for our audience that Fortinet, when it stores its password hashes, includes a prefix that tells you which type of hash it is. That’s not uncommon, but it makes it much easier for us to create checks for it. Vlad (22:58.876): Mm-hmm, that’s normal. Vlad (23:08.582): This is security through obscurity. When you design a system for hashing passwords, you have to assume the attacker already has all of your source code and knows exactly how the system works, and it must still remain secure. That is the main mistake FortiOS made—they didn’t assess it this way. They hoped that some level of obscurity would save them with this old password hash being hidden, assuming nobody would figure it out. But they will. You can make your hashes identifiable; just make them hard to crack. In our own product, when we build a password management system, we build it around the assumption that anyone can know exactly how the hashes are stored, and it must still be complicated to actually get to them. For example, we employ techniques like storing the hash of the password—not the password itself—encrypted. If you compromise the database, you first have to steal the encrypted hashes, and then you have to steal the key from another location that is not the database. We are explicit about this. All devices, especially security appliances, should start becoming auditable. There must be a push from the industry to make them auditable, where you can check their internals, because the situation we’re in right now—where appliances don’t let you into the OS shell, while attackers get RCEs on them and can walk around invisibly—is ridiculous. Paul (24:51.542): The other mitigating factors they put in place include enabling multi-factor authentication on many of these devices. You can also use a cloud account—like a Google-authenticated account or, in Fortinet’s case, a FortiCloud account—to authenticate to those devices. But keep in mind, there was a heinous vulnerability a while back with FortiCloud. If configured with FortiCloud, it exposed a vulnerability that let attackers log in. All of this is an attack surface, and there are no silver bullets in security. Vlad (25:32.639): Yep. So—Call me old-fashioned, but I don’t like the cloud authentication systems we have. Remember *Battlestar Galactica*? It highlighted a valid point: if all of your systems are networked and operate with a singular login system, that’s a single point of failure. Break that, and your entire system falls apart. You don’t want to attach security appliances to a cloud system, because if it breaks, it breaks basically every customer FortiOS has. I want my device to authenticate fully locally; I don’t want to go through the cloud to log in. Paul (26:14.55): Right. With SSH, I haven’t tested this, but I’m fairly certain you can put an SSH key on most of these devices, Fortinet included, and remove password-based reliance on SSH. But that doesn’t work on the web interface. Those credentials are for the web interface, which doesn’t benefit from key-based SSH authentication. Vlad (26:33.352): Mm-hmm. Vlad (26:42.994): Yes, unless they do certificate-based authentication, and even then, it will not save them from credential harvesting. If you steal the certificate from an admin’s browser, along with the password for the certificate, you’ve got a credential. Though, it will make breaking into a device much harder. If you break in and find a lot of public keys—congrats, try brute-forcing those. Paul (26:52.756): Right. Paul (27:10.825): Mm-hmm. Vlad (27:10.888): This would make for a very strong authentication system, but on the other side of the spectrum, we have complexity with certificate management. You have to rotate them, you have to have a PKI. This might be doable for large companies, but medium and smaller companies are not going to do that; it’s too much complexity. The real answer is fixing the mess with hash management. Paul (27:20.843): Yes. Vlad Babkin (27:40.477): Imagine a perfect world where FortiOS did everything right. In this case, the old password would be visible, so administrators would know the risk. They would see their old password still stored there—look at the config, see the weak hash, change the password, or remove it forcefully. They could lock out accounts using the old hash so that until someone changes their password, they cannot log in. If FortiOS did it like this, it would still be possible to brute-force the passwords, but brute-forcing PBKDF2 is exponentially slower than SHA-256. It would require more hardware and effort from the attacker. Attackers would not have an easy time recovering credentials. Even if they had the leak and SHA-256 hashes, they would still have to crack PBKDF2. There is a lesson here to learn. Paul (28:11.212): Mm. Paul (28:22.11): Right. Paul (28:43.264): Yeah, that’s what made the scope of this attack exponentially larger—the weak password hashing issue in previous leaks and infostealers. Vlad (28:52.286): It’s not just problems with the previous hash, but also mismanagement of how you store the old hash. Paul (29:04.906): Yep. This culminated in FortiGate Sniffer, which is interesting because it’s not malware on the FortiOS device. FortiGate Sniffer is like an orchestrator you run on your workstation; you give it access to a bunch of Fortinet devices, it deploys a sniffer to them, and then collects the credentials the sniffers catch. They basically ran a FortiOS command to sniff credentials over 24 different protocols. It wasn’t limited to a few protocols; whatever traffic was passing through, if it matched those 24 protocols, the sniffer on your workstation extracted those passwords. We saw screenshots of a dashboard where they used this utility they created to collect, sort through, and manage those credentials, which is pretty advanced. Chase (30:15.915): We have some live listeners, guys, so we’re getting comments on the LinkedIn livestream. One person asked a question I’ve been chewing on: has anybody claimed responsibility for this attack yet? I know it’s being attributed to SantaAd, which is an initial access broker group. To me, this doesn’t seem like the kind of attack someone claims responsibility for. It seems very financially motivated, and it’s not even one individual attack. Someone was sniffing and harvesting credentials to sell them for other attacks. Usually, when someone claims responsibility for an attack, it’s either politically or ego-motivated, Paul (30:19.052): Mm. Chase (30:45.409): which I’ve never heard of with this group. Has anyone claimed responsibility, or is there just a speculative attribution from the discovery? Paul (31:15.69): No, there are just indicators. A Russian-speaking multi-operator threat group, which Unit 42 dubbed SantaAd. I agree with the classification: this is likely an initial access broker based on the indicators we have. These are folks collecting initial access credentials for the purpose of selling them, not using them to monetize directly. These are specialized subgroups within the threat actor ecosystem. This group focuses purely on initial access. They get a bunch of credentials and sell them to ransomware operators, espionage groups, or whoever. Chase (31:23.306): Mm-hmm. Chase (32:05.719): Yeah, it’s sort of a middleware-type role in the overall ecosystem. Paul (32:13.6): Yep. This campaign is similar to other identification campaigns. They are scanning the internet, and actually, Vlad, it’s funny—it’s similar to how we design systems for defenders to identify devices, determine the device type, and take specific actions based on that. The threat actors created just that. If you read through some of the detailed documents analyzing their tradecraft, once they scanned the internet and Shodan, they refined that to specifically filter out anything that wasn’t a FortiGate firewall. They also had indicators to detect a honeypot. One indicator was: if we attempt to guess three credentials in a row and all three are successful, that’s likely a honeypot that accepts any credential. They had honeypot detection in there, and a lot of refinement to weed out things that were not Fortinet FortiGate firewalls. Paul (33:34.636): Which I thought was pretty clever—hat tip to them. Chase (33:38.241): Yeah, that’s a clever tidbit—if you get it within three, it’s probably a honeypot. Really interesting. Paul (33:44.704): Yeah, it’s basic detection. They also used the certificate. They retrieved the SSL certificate associated with the device and looked at the metadata fields to see what organization it belongs to. That’s how they filled out the company, industry, revenue, and headcount. They looked at your Fortinet device, saw the certificate belonged to some large publicly traded company, identified who the firewall belonged to, and enriched that data with industry, revenue, and headcount. When they sell the data, they can set the price tag based on who they’ve compromised. Vlad (34:37.438): Imagine the scale of this harvesting. Considering how users manage their passwords, those credentials were likely useful across multiple systems. Depending on who the user is, those credentials might be more valuable than just FortiOS access for their company—imagine getting into a company’s Salesforce as a CEO or a high-level admin. Paul (35:04.779): Yes. Vlad (35:06.748): And suddenly you have a ton of intelligence out of nowhere. Paul (35:10.666): Yeah, that’s what they do. Once they get access based on those credentials, other groups will go in and, as you said, Vlad, log into your Salesforce, dump all the data, and export whatever they can based on user permissions. Then they take that to another group—I think it’s LeakBase or a similar data leak forum—hand over hundreds of gigabytes or terabytes of data, and have them sift through it to place a value on it because they’ve gathered too much to analyze themselves. That’s how they are operating today, which is interesting. Paul (36:16.752): What else do we want to talk about with FortiBleed, Chase? Chase (36:24.157): When thinking about how we categorize and report on this kind of incident, the fact is that older vulnerabilities were part of the mix. Returning to the Verizon DBIR, they talk about the number of vulnerabilities that get exploited and the number of edge devices that participate in breaches. This is a middle ground where a vulnerability from a prior breach led to the leak of credentials or hashes. How do you count that? I really feel for the defenders—whether it’s the security team, ops, or the infrastructure team—who are responsible for trying to protect these network edge devices or do incident response, because whoever is responsible probably doesn’t have the access, tools, and telemetry they need to discover, scope, and fix this activity quickly. They are probably in an ongoing world of pain because the machinery to respond to an incident like this inside of an organization with a lot of FortiGates is likely not super well-oiled. Paul (36:40.865): Yep. Vlad (36:43.23): Yeah. Vlad (37:39.454): Mm-hmm. Vlad (37:56.915): Yep, doing something like this—even detecting if you have an old hash—requires a high-level credential account so you can dump the configuration. I don’t know if it needs a super admin account specifically, but it is definitely high-level. To fix it, you need a super admin. There is a reason why our network device sensor requires a high-privilege account, even though we understand it’s dangerous, and we have tools in place to prevent us from becoming the next SolarWinds incident. I don’t want to single out companies, just incidents that started the modern storm of attacks against network devices. It’s not about the company anymore; maybe they fixed their processes by now. But in this case, the interesting part is that we have to recommend yet again: don’t expose your management interface to the public. The public should only ever see your VPN interface—specifically OpenVPN, WireGuard, or what have you. Nothing else should be sticking out. Paul (38:04.166): Yeah, it needs super admin. Paul (38:32.394): Yeah, for sure. Paul (39:11.594): Right. But even in this case, the scope and breadth of this attack is concerning. Sometimes it wouldn’t matter if you don’t expose it to the internet; if attackers have a foothold in your network through another mechanism, they can find these devices and turn them into passive sniffers. Sometimes my brain goes to, “Can’t we just disable or remove the passive sniffing capability on the Fortinet devices?” but then attackers would probably just put it back. I don’t think that helps. It’s not a great countermeasure. Vlad (39:16.85): Mm-hmm, yep. Vlad (39:46.877): Removing sniffing capability won’t help. Eventually, an attacker will find an RCE and bring their own sniffing capability. This device is de facto sitting on top of your network traffic, so whether you like it or not, it can sniff it. The only thing you can do is go Zero Trust and implement TLS for your internal web services. Don’t just rely on FortiOS to protect you—have internal encryption. Get certificates, get your own PKI, get something. It’s not complicated to get a certificate for a web server. Managing certificates for users is complicated, but giving a web server a certificate is not hard. Even if an attacker gets on top of FortiOS, if your traffic is encrypted with HTTPS and well-configured, they only get encrypted traffic. They can sniff it all they like, but they aren’t getting credentials. This comes down to network-level defenses. Paul (40:02.837): Mm-hmm. Paul (40:58.272): Right. Paul (41:05.908): Yeah, I agree. It’s an encompassing thing you need to think about and model: what if an attacker gains control of my firewall? What are our defenses? One defense is making sure you use secure protocols and secure password hashing algorithms, like with Active Directory. Have you configured Active Directory such that it’s hard for someone, even if they sniff the traffic, to get and crack those hashes? There are multiple levels of protection, even within Active Directory. Every protocol and system has its own way of protecting traffic and credentials. Oftentimes we treat internal security as a lower priority, but if an attacker gets on your firewall or router, it’s game over. Vlad (41:18.718): Yeah. Vlad (41:58.687): Get your systems authenticated, give them certificates, and establish trust in those certificates. It might be complicated for a home network user, but for a company with an administrator, even without a dedicated security department, you can ask them to set up certificates and credentials. Give your admin the breathing room to do this. Allow them to disrupt the process a little bit to make it secure; it’s going to pay off in the long term. The only security that works is layered security. You don’t get a magic appliance box that fixes all of your problems. Even if FortiOS did everything perfectly right, it’s impossible to write perfectly secure software. It’s just not a thing. Eventually, someone is going to find a remote code execution vulnerability on that device. Now what? If your entire company is not prepared for this, I have bad news for you. It’s not just Fortinet who is in pain here—it’s the whole industry. Paul (42:37.302): Yeah. Paul (43:12.48): Yeah. There are ways—if you enable auditing and logging on your FortiOS devices, I believe you can determine if someone is running commands to sniff packets. While some might say that is too late, if it’s an early indicator and you realize your firewall is compromised, you can shut it down early, minimize damage, and change your passwords. It limits your exposure if you can detect that none of the administrators ran the sniffer command, meaning someone else did. That’s bad. There is a logging aspect to this as well. Vlad (43:53.16): Yep. Store your logs outside of the device. Again, obvious advice. It’s not doable for a home user, but definitely doable for a small-to-medium-sized company, let alone a large one. It’s going to save you a lot of pain. Also, inventory your firewalls. If you know about your FortiOS devices, monitor their versions, and update them regularly, that’s going to save you from a lot of pain. If you must, rotate admin credentials from time to time. If you’re afraid someone harvested your admin credentials, make sure to rotate them every few months. For admin accounts, that’s not too hard of an ask. Paul (44:44.414): Yeah. I would never tell people not to update the operating system or firmware on these devices; that certainly helps. But as we’ve described, you can still be vulnerable and compromised. Symlink persistence is what allows attackers to persist even after a firmware update. There is that, and there are the backdoor accounts we talked about. Vlad (44:52.158): It has problems. Vlad (45:10.974): Yeah, this is— Paul (45:13.58): There are malware implants on Fortinet and other devices that persist through firmware updates. You should still upgrade to the latest version, but there are a lot of other things you need to do to monitor these devices beyond just patching. Vlad (45:13.65): Mm-hmm. Vlad (45:26.014): Yeah, it’s not the only thing to do. The mindset should be: “What if someone broke into our platform?” This is what we do at Eclypsium, by the way. If a really bad day comes for Eclypsium and someone breaks into our platform, we plan for it. We don’t just assume our secure processes will protect us and that it won’t happen. That is not how it works in this industry. Paul (45:54.57): Not how it works. Yep. Vlad (45:55.963): It happened. What do you have to mitigate the impact? We have mitigation mechanisms for different scenarios. For example, we try to add extra protection to prevent jumping from our platform to our customers. If we have a bad day, we don’t want our customers to also have a bad day. There is a whole lot of stuff you can do if you start to think about the problem. Paul (46:16.844): Mm-hmm. Vlad (46:23.152): And not all of this is complicated. Some of it is just properly setting up your certificates for web servers. It might take a couple of weeks, but if you have an administrator, they can do this. It’s not that time-consuming. Paul (46:42.112): There was a lot to this attack. It took most of the show. It’s a big one. Chase (46:45.085): It’s a big kahuna. Vlad Babkin (46:48.318): Yeah, it’s just so encompassing. While we are bashing on FortiOS, this isn’t FortiOS-specific. Replace FortiOS with another vendor name for a security appliance like PAN-OS, Junos, MikroTik, Ubiquiti, or Cisco. Add any vendor that makes a firewall or router, and you get the exact same reasoning we just did. It’s not specific to one vendor. Paul (46:57.781): No. Paul (47:11.786): Yeah, I agree. Similar campaigns have targeted Cisco ASA and FTD devices, as well as Ivanti devices. The TTPs are very similar: compromise the device, implant the device in some way, maybe hook the login function—it takes different forms, but attackers are targeting these devices from multiple vendors to collect credentials, and then use that to pivot or sell them so someone else can pivot into your environment. This is very much what is happening today. We got lucky that this tradecraft leaked, giving us deep intelligence as to what happened. Vlad (47:23.23): Mm-hmm. Vlad (47:58.953): Yeah, in many cases you don’t get information about this. You might get some surface-level acknowledgement, but you don’t always get a full log of how many credentials were stolen or how the attacker operated. In this case, we know exactly about the hash-cracking. They are not the first attackers to do this. The industry can keep insisting we can make things better without any change, but no. Real change needs to happen for administrators, who need to start thinking: “Okay, it got broken, now what?” You need something beyond just relying on the device itself. When you have four, five, or six mitigating controls, even if attackers break into your firewall, they still have to face HTTPS encryption and other authentication layers. Paul (48:19.478): Right. Paul (48:52.342): That’s a great point, Vlad. In that context, Chase, you wanted to talk about this with Mythos. Let’s say we have great AI technology to identify vulnerabilities—and we do. Hopefully we can use that same technology to create patches and fixes. They still have to be rolled out, and customers have to apply them. But even if you do that, we talked about other things you must do to be more resilient that go above and beyond relying on the vendor to find and fix vulnerabilities, or even applying those patches. You could still be in a vulnerable state, as is the case with storing old password hashes, or having malware that persists through firmware updates. There are other insights and intelligence you have to gather and act on, even if you are running fully patched versions. There’s always the possibility of a zero-day or a leaked credential, even if you’ve hardened and upgraded your network edge devices to the latest versions. You’re never done; it’s a constant process. Chase (48:58.252): Yeah. Vlad (49:17.918): Mm-hmm. Chase (50:14.187): Yeah, this is a great example. Companies that got access to Mythos as part of Project Glasswing immediately started talking about how they had found and patched many vulnerabilities. Many of those have not hit—perhaps none have hit—the 90-day disclosure window yet, so we don’t even know what they were. But companies like Mozilla are saying they found and fixed 271 vulnerabilities thanks to Mythos. If you dig deeper, they found the vulnerabilities, but Mythos couldn’t write the patches; they had to go patch it themselves. At a further layer, when you look at vendors like Fortinet, Palo Alto Networks (who was in Project Glasswing, I believe), or Cisco, they can find the vulnerabilities and build the patch, but it’s still going to take a long time for that patch to make its way out and get rolled out to the end user. It’s highly possible that by the time the vulnerability is disclosed, even if a patch is available, there is a significant window of time where the adversary knows about it just as well as the defender, but it hasn’t been patched yet. It has to go through slow machinery because patching carries operational risk. Paul (51:38.366): Mm-hmm. Chase (51:42.892): Patching fast creates a risk of downtime and other second-order effects that security and operations teams are rightly wary of, so they don’t want to roll out patches too quickly. As we get tons of vulnerabilities and patches, the release cycle for these big companies accelerates. But for the defenders inside these companies, their capacity or speed to patch is not going up; it’s going down. The number of patches they successfully roll out and the speed at which they do so are getting worse, according to that Verizon report. There’s a lot of messaging about how Mythos is affecting cybersecurity, and it’s going to create a hard time for many teams who are trying to keep up. Paul (52:23.212): Yeah. Paul (52:46.25): As we point out, Chase, there isn’t always a patch. Sometimes a vulnerability or misconfiguration is identified and, like the one Apple had recently, they can’t patch it. We see that in a lot of firmware-based systems where a specific vulnerability is tied to the hardware and cannot be fixed. There’s also the Arista case, and Microsoft does this too, where they say they won’t patch something because it could have too many negative effects. The solution is that you need to configure your systems in a specific way to be resilient. The answer isn’t always just to apply a patch; it might be reconfiguring your systems so they aren’t vulnerable, which can be as difficult as, or worse than, applying a patch. Chase (53:36.427): Yeah, it’s way less clear-cut as to what to do and how to do it. The impact of AI on vulnerability management and vulnerability disclosure programs is still shaking out and will continue to for a long time, but I don’t think it’s going to be an unalloyed good. Paul (53:42.547): Mm-hmm. Paul (53:59.476): No. Because in this case, yes, you should have applied a firmware update, but you also need to do other work to ensure your system is configured to be resilient. Sometimes the answer is both a patch and configuration. Chase (54:14.805): Mm-hmm. Vlad (54:15.398): What will happen is that AI will start forcing a lot of companies trying to stay obscure into the open, because it will make no sense to stay obscure. If an attacker can dump firmware, reverse-engineer it with AI within a day, and get all of the nitty-gritty details more regularly than they do now, then any kind of obscurity is gone. At this point, it is much better to come out fully in the open and operate in the open. You get feedback from security researchers instead of just getting messed with by attackers. That is the only way forward. It’s not just magical models leading to vulnerability discoveries; it’s engineering effort and a lot of knowledge behind setting this up. A raw model will not reverse-engineer a binary on its own; it needs tools. If you don’t set up the tools properly, you’re not getting anything out of the model, even if it has the capability. Paul (54:22.828): Mm-hmm. Paul (55:17.452): Well, cool. I think that is a good place to wrap up. I want to thank Chase and Vlad for appearing on the show today, and thank everyone for listening and watching this edition of Below the Surface. Thanks, everyone. We’ll see you next time. Chase (55:32.066): Thanks, Paul. Thanks, Claude. The post BTS #77 - FortiBleed Uncovered: How Attackers Harvest Credentials from Fortinet Devices appeared first on Eclypsium | Supply Chain Security for the Modern Enterprise .