Defense in Depth
What's the Most Efficient Way to Rate Third Party Vendors?

Organizations don't have the time to do in-depth vetting on every third-party. This leaves them turning to "better than nothing" security rating vendors. These might be fine for liability, but do these vendors actually help improve your understanding of third-party risk? 

Check out this post from Paul Valente of VISO TRUST for the discussion that is the basis of our conversation on this week’s episode co-hosted by David Spark, the producer of CISO Series, and Geoff Belknap. Joining them is Steve Knight, former CISO, Hyundai Capital America.

Listen to the full episode here.

Streamlining vendor evaluations 

A strategic approach to vendor assessment can dramatically reduce workload while maintaining security standards. "Maybe everyone else already does this, but we've implemented a few filters that cut down on the amount of vendor evaluation we do," said Christine Chalmers of Netflix. "Vendors are tiered based on the data they process, and we include those tiers in annual security awareness training. We have a checklist of the top few reasons vendors fail our assessments (usually because they're expensive to implement, like encryption at rest), and we train everyone to ask those questions when they're first evaluating options." This proactive filtering approach helps teams avoid lengthy back-and-forth negotiations by addressing common deal-breakers upfront. However, Mark Dunaisky of D3 Risk Management Group cautioned against oversimplifying third-party risk programs. "How many people actually know how to read a SOC 2 and get value out of it? How useful are, and where is the best use of, security risk ratings platforms?" he asked. "There are many companies that are only using security risk rating platforms as their entire cyber third-party program because 'they're better than nothing.' As a practitioner, to me, unless it's a first step on one's maturity roadmap, this is not reasonable."

Moving beyond compliance theater

True third-party risk management requires a fundamental shift from checkbox exercises to genuine risk assessment. "Most third-party risk management programs don't actually manage risk," said Simon Marvell of Acuity Risk Management. "Policy/controls-based due diligence questionnaires provide point-in-time self-assessments which are impossible to verify at scale and even where accurate tell you nothing about the risk you are exposed to and whether it is tolerable." He argued that security ratings solutions, while helpful, only capture externally visible vulnerabilities and don't measure actual risk exposure. Instead, Marvell advocated for "some old fashioned risk management where we identify vendors and risk scenarios that could cause (truly) material impacts on achievement of business objectives." This approach focuses on working directly with vendors to assess specific risks, agree on targeted controls, and establish metrics that provide ongoing assurance, rather than relying on one-time snapshots. Sam Reddy of InterSec echoed this holistic perspective, emphasizing that "metrics/scorecard/specific report is not a silver bullet." He stressed the importance of integrated risk management assessments that examine how third parties impact the organization's business, operations, and technology, giving CISOs "a single pane of glass to see through to manage the risk." 

The scorecard skeptics

Security rating platforms are facing growing criticism from practitioners who question their value proposition. Val Dobrushkin of AVA Compliance Solution dismissed security scoring services as "a scam" that provides no true measure of security, due to their reliance on external scanning without internal validation and their tendency toward false positives. John Overbaugh of Alpine Investors shared his frustration as both a user and target of these services, noting that "these third-party scans and reports are wholly misleading" and comparing the correction process to credit bureaus, where "buying a membership makes it easier, which sounds to me a lot like reputation-based ransom." Ira Winkler of CYE highlighted a key transparency issue, pointing out that while SOC 2 reports reveal their methodologies, scorecard vendors use "proprietary algorithms and companies have to pay to have them address the shortfalls in their findings."

Finding the right balance

Security ratings can serve as a helpful starting point, but practitioners emphasize they shouldn't be the end of the conversation. Vladimir Yakovlev of Higher Intelligence LLC sees value in ratings as an initial feasibility check, noting they "at the very least, allow companies to decide if it is even feasible to consider a given vendor's offering." He also highlighted their utility for ongoing monitoring, providing "active notifications in case of a rating drop" that can trigger remediation requests or alternative vendor considerations. Yakovlev mentioned discussing with rating companies the possibility of listing downstream dependencies, which could help organizations better understand common exposures across their vendor ecosystem. Subarna Bhowmik of Deloitte advocated for a more comprehensive approach, emphasizing that organizations should first "determine the length and breadth of vendors present within the organization" before applying any assessment methodology. While acknowledging that risk sensing and monitoring reports serve a limited purpose, Bhowmik argued for a multi-layered strategy combining questionnaires, interviews to validate control effectiveness, and independent reports review to achieve a truly holistic third-party risk assessment.

Please listen to the full episode on your favorite podcast app, or over on our blog where you can read the full transcript. If you’re not already subscribed to the Defense in Depth podcast, please go ahead and subscribe now.

Huge thanks to our sponsor, Formal

Subscribe
Subscribe to Defense in Depth podcast

Please subscribe via Apple Podcasts, Spotify, YouTube Music, Amazon Music, Pocket Casts, RSS, or just type "Defense in Depth" into your favorite podcast app.

In observance of July 4th in the United States, we won't be releasing an episode of Cyber Security Headlines - Week in Review

Join us in two weeks, Friday July 11, for “Hacking the Resilience Mindset”

We have no live shows this Friday, but join us on Friday, July 11, 2025, for Super Cyber Friday: “Hacking the Resilience Mindset.”

It all kicks off at 1 PM ET / 10 AM PT, when David Spark will be joined by Liz Morton, field CISO, Axonius, and Nick Vigier, CISO, Oscar Health, for an hour of insightful conversation and engaging games. And at 2 PM ET / 11 AM PT, stick around for our always-popular meetup. This time, it will be hosted right inside the event platform.

Register now on LinkedIn

Remember to add it to your calendar via LinkedIn or on Airmeet link in the invite.

Thanks to our Super Cyber Friday sponsor, Axonius

Cyber chatter from around the web...
Jump in on these conversations

“What's your secret sauce for security awareness?” (More here)

“How are people finding jobs right now?” (More here)

“Have 5+ years as a SIEM using EDR/XDR using Security Engineer? Which of these questions seems unanswerable for you personally in an interview?” (More here)

Thank you!
Thank you for supporting CISO Series and all our programming

We love all kinds of support: listening, watching, contributions, What's Worse?! scenarios, telling your friends, sharing in social media, and most of all we love our sponsors!

Everything is available at cisoseries.com.

Interested in sponsorship, contact me, David Spark.