AI Confidence: It's a Trap! (LIVE in San Francisco)

May 05, 2026

CISO Series Podcast
AI Confidence: It's a Trap! (LIVE in San Francisco)

Early research indicates that AI delivers clear productivity gains, but the ROI is harder to quantify. Almost all LLMs will confidently produce seemingly high-quality output. But are we setting ourselves up for failure when we mistake confidence for accuracy?

This week’s episode is hosted by David Spark, producer of CISO Series, and Mike Johnson, CISO, Rivian. Joining is Sara Madden, CISO, Convera. This episode was recorded in front of a live audience at BSidesSF 2026.

Click here to see more photos from our live podcast recording and CISO Series game show at BSidesSF 2026.

Listen to the full episode here.

Playing vendor roulette

The typical vendor selection process is a setup for regret. Richard Stiennon of IT-Harvest describes the pattern well: evaluate a handful of vendors, pick the one with the best demo, sign a three-year contract. The problem is you then rip it out eighteen months later when the vendor gets acquired, stops shipping, or turns out to be the marketing leader rather than the market leader. This doesn't just burn time and money. You waste political capital defending the original call. The fix starts before you even build a shortlist. Peers who've solved the same problem are a better resource than analysts or search results. Instead, begin by writing down your requirements, run a real proof of concept, and don't lock in long-term until you know what you're buying.

Confident and wrong

Most organizations report seeing significant AI productivity gains, with only a small minority seeing clear ROI. That gap points to something specific for GigaOm CEO, Howard Holton, mistaking confident output for good output. AI produces polished formatting, an authoritative tone, and a coherent structure, making it feel trustworthy even when it's wrong. Teams that accept whatever the model produces, without developing a real sense of what "good" looks like, aren't using AI as a force multiplier. They're outsourcing their own judgment. The smarter approach treats AI the way security teams already treat vulnerability scanners, a useful first pass that still needs a human to sort signal from noise.

Making conferences count

Advice on getting value from security conferences usually boils down to "just network." This isn't wrong, it's just not helpful. A thread on the cybersecurity subreddit reframes the question more practically: who in this room could you call in a crisis? Moving from collecting contacts to building a bench of people who can fill real gaps changes how you move through an event. Vendor floors are optimized to capture email addresses, not to build useful relationships. The engineers worth talking to are usually found at talks and in hallway conversations. For teams, the investment only pays off if attendees bring something back.

The stakes problem in tabletops

Incident response tabletops are lying to you. Not because the scenarios are unrealistic, but because of the incentives, claims Joshua Copeland of Crescendo. In a real breach, authority is the first point of failure. People slow-roll decisions not because of a lack of training, but because escalation is political. You don't hear "legal is reviewing the language" in a tabletop, but you will in an incident. A tabletop with no consequences won't surface those fault lines. Bringing in an outside firm helps. They don't share your team's assumptions, and spending real money on the exercise creates its own pressure. The value builds over time, especially when cross-functional teams are included and allowed to fail without someone quietly steering them back on track.

Listen to the full episode on our blog or your favorite podcast app, where you can read the entire transcript. If you haven’t subscribed to the CISO Series Podcast via your favorite podcast app, please do so now.

Thanks to Craig George of GuidePoint Security for providing our "What's Worse" scenario.

Huge thanks to our episode sponsors, Nudge Security, QuilrAI, and Zenity

Subscribe to CISO Series Podcast

Please subscribe via Apple Podcasts, Spotify, YouTube Music, Amazon Music, Pocket Casts, RSS, or just type "CISO Series Podcast" into your favorite podcast app.

Security You Should Know
Verifying Identities with Trusona

In this episode, Ori Eisen, founder and CEO at Trusona, makes a case for getting out of the AI detection arms race entirely. He argues that trying to catch AI-generated fakes with AI detection is the antivirus playbook, and we know how that ends.

Trusona instead anchors verification to authoritative sources, DMV records and physical-world signals, things AI can mimic on screen but can’t actually own. No pre-registered devices required. And it works in both directions: attackers calling your help desk, and attackers calling your employees while pretending to be IT.

Joining him are Eduardo Ortiz, VP and Global Head of Cybersecurity at Techtronic Industries, and Mandy Huth, SVP and CISO at Ultra Clean Technology.

Want to know:

  • Why do MFA and SSO still leave gaps attackers walk right through?

  • How Trusona verifies identity with no pre-registered devices or tokens?

  • Why building AI detection on top of AI fakes is a losing strategy?

  • How is a false rejection rate of zero achievable without locking out real employees?

  • What deployment actually looks like, and how fast you can be live?

  • Which departments beyond IT need identity verification, and where do you start?

  • How to measure the business value of this beyond just counting blocked account takeovers?

  • Why is a solid help desk protocol still not enough on its own?

Read more and listen to the full episode for the answers you need.

Thanks to our podcast sponsor, Trusona

Subscribe to Security You Should Know

Please subscribe via Apple Podcasts, Spotify, Amazon Music, Pocket Casts, RSS, or just type "Security You Should Know" into your favorite podcast app.

Biggest mistake I ever made in security…

“Saying, "I told you so." There's nothing wrong with mistakes as long as it doesn't have a material impact on the business. I love the saying, "There's no losses, there's only lessons," but when I think back across my career and I think about the mistakes that haunt me at night, it's the times that I have bent to the pressure of business and I went against my better judgment and I put myself in a position of you can see around corners and you know it's going to happen. And when that does and you say, "I told you so," it's never a good feeling because it's not our jobs to say I told you so, it's our jobs to manage risk and never have that happen. “ - Sara Madden, CISO, Convera

Listen to the full episode of "AI Confidence: It's a Trap! (LIVE in San Francisco)”

How Do You Know If Your Backups Will Survive a Ransomware Attack?

"The biggest lie in cyber resilience is that backups equal recovery. Active Directory is not an IT system. It's a control plane for your entire business, and without it, you can't even operate." - Heath Renfrow, co-founder, Fenix24

Listen to the full episode of “How Do You Know If Your Backups Will Survive a Ransomware Attack?”

Subscribe to our newsletters on LinkedIn!

CISO Series Newsletter - Twice every week

Default-Deny as a Practical Zero Trust Control

SPONSORED ARTICLE

Default-deny sounds like you're shutting down the business. You're not.

In this "best of" compilation, cybersecurity leaders break down what "deny by default" actually means, why it works, and how to make it palatable to the business. From blocking execution over chasing CVEs, to controlling what trusted tools like PowerShell are allowed to do, the insight here is practical and pointed.

Huge thanks to our sponsor, ThreatLocker.

Cybersecurity Headlines - Department of Know

Our LIVE stream of The Department of Know happens every Friday at 4 PM ET / 1 PM PT with CISO Series producer Richard Stroffolino, and a panel of security pros. Each week, we bring you the cybersecurity stories that actually matter, and the conversations you’ve been having at work all week long.

Friday’s episode will feature Jason Elrod, CISO, MultiCare Health System and Jonathan Waldrop, CISO, Acoustic. Join us on YouTube and catch up on what shaped the week in security.

Thanks to our Cybersecurity Headlines sponsor, Vanta

Super Cyber Friday
Join us Friday for “Hacking the End of Compliance”

Join us on Friday, May 8, 2026, for Super Cyber Friday: “Hacking the End of Compliance: An hour of critical thinking about the security benefits of moving toward continuous monitoring.”

It all kicks off at 1 PM ET / 10 AM PT, when David Spark will be joined by Roland Cloutier, former Global CSO TikTok & ByteDance, ADP, and EMC, and Dale Hoak, CISO, RegScale, for an hour of insightful conversation and engaging games. And stick around for our always-popular meetup, hosted right inside the event platform.

Thanks to our Super Cyber Friday sponsor, RegScale

Participate! Add our live shows to your calendar

Learn more about all of the fun ways you can participate, and add our events to your calendar.

Cybersecurity Headlines - Daily News Shorts

Subscribe to the CISO Series YouTube channel, for daily shorts videos from CISO Series reporter, Rich Stroffolino. You can find all of the stories he’s covered, plus new content every weekday, at the Cybersecurity Headlines Shorts YouTube playlist.

Thank you for supporting CISO Series and all our programming

We don’t just say we appreciate your feedback; we incorporate it into our programming. Learn more about all of the fun ways you can participate.

We love all kinds of support: listening, watching, contributions, What's Worse?! scenarios, telling your friends, sharing on social media, and most of all we love our sponsors!

Everything is available at cisoseries.com.

Interested in sponsorship, contact me, David Spark.