Automating compliance evidence collection with AI agents

ByNicolás Barberis
Trust Operations Manager at Synthesia

The request

A customer sent us a security questionnaire as part of their renewal process. This is not unusual. What was unusual was the scope: 70 top-level questions, each with between three and seven sub-items asking for specific evidence. Screenshots of configurations. Proof of policy enforcement. System outputs demonstrating that controls are active. In total, somewhere north of 300 individual evidence items, that ended up being over 900 files.

Turning a problem into infrastructure

The first instinct was to power through it manually. Open each system, take multiple screenshots, export reports, write an explanation, upload it to our GRC platform. Repeat over 70 times. This is what most teams do, and it's why evidence collection is one of the most time-consuming parts of any compliance program.

Instead, we decided to invest the time differently. We uploaded the customer's questionnaire into Vanta as a custom framework, then mapped each question to our existing controls, documents, and automated tests. This gave us a structured view of what was already covered (most of it) and what gaps needed manual evidence.

Then we built an agent to close those gaps.

The evidence collection agent

The tool is a Claude agent skill — a set of instructions and scripts that an AI coding agent follows autonomously. When invoked, it runs a pipeline:

1. Query Vanta for pending evidence.
2. Understand the ask.
3. Research how we actually do it.
4. Navigate and capture.
5. Detect access issues.
6. Generate an explainer document.
7. Upload to Vanta.

How it works in practice

A typical run looks like this:

/get-vanta-evidence soc2

→ Checking prerequisites... credentials OK, Chrome CDP OK
→ Verifying SSO session at hub.synthesia.io... authenticated
→ Found 61 pending documents for SOC 2

#  Title                                 Status
1  Application User Listings             Needs doc
2  Confidential File Access Permissions  Needs doc
3  Data in Transit Encryption Settings   Needs doc
...

→ Processing: Application User Listings
→ Auditor asks: "Provide user access lists for critical applications"
→ Navigating to Okta admin console...
→ Capturing: 01_okta_user_directory.png ✓
→ Capturing: 02_okta_group_memberships.png ✓
→ Generating explainer PDF...
→ Uploading to Vanta (draft)... ✓

→ Processing: Data in Transit Encryption Settings
→ Navigating to AWS Certificate Manager...
→ Auth wall detected: "Please log in to AWS Console"
→ [User handles login, confirms]
→ Capturing: 01_aws_acm_certificates.png ✓
→ Capturing: 02_aws_alb_tls_policy.png ✓
→ Uploading to Vanta (draft)... ✓

Summary:
✓ 48 items completed (uploaded as draft)
⚠ 13 items need manual attention

Review uploaded evidence in Vanta before submitting.

The architecture

The skill consists of four components:

1. A Vanta API client that handles OAuth authentication, queries documents and tests with status filters, uploads evidence files, and auto-paginates through results.
2. A screenshot capture script built on Playwright that connects to Chrome via the DevTools Protocol.
3. An evidence report generator that produces clean, professional PDFs using ReportLab.
4. A SKILL.md file that orchestrates the entire pipeline.

What we learned

1. Custom frameworks in Vanta are underrated. Uploading the customer questionnaire as a custom framework let us map their questions to our existing controls and tests.
2. SSO is the unlock for automated evidence collection. By routing through Okta SSO tiles, a single authenticated session propagates to dozens of downstream systems.
3. Human-in-the-loop is not a limitation, it's a feature. The agent uploads everything as draft. A human reviews the screenshots, reads the explainer, and decides whether to submit.
4. The 70-question questionnaire was a gift. It forced us to build tooling that now serves every audit cycle, every customer security review, and every framework renewal.

What's next

We're extending the agent in several directions:

• Notion and Linear integration for evidence that lives in documentation rather than dashboards.
• Scheduled evidence refresh to keep time-sensitive documents current without manual intervention.
• Multi-framework deduplication so evidence captured for SOC 2 automatically maps to ISO Standards and customer questionnaires that ask for the same thing.

Through audits, Vanta, and certifications, we surface the enormous amount of work that is actually happening behind the scenes every day.