Case Study · Secure Regulatory Data API

Regulatory filing data, opened as an API - without opening anything else.

The client turns insurance rate filings into analytics its customers price and plan against. Letting partners and machines reach that data meant building an entry point that could survive being public. We scoped a serverless filing API on Google Cloud: three layers of defence before any logic runs, machine-to-machine authentication with short-lived tokens, query functions isolated in a private VPC, secrets the administrators cannot read, and every resource provisioned as code.

InsurTech · Rate filing analytics
Serverless on Google Cloud
Terraform-provisioned
List · search · view filings
How a filing request is served
scoped
An admin issues access
service account · ClientID · private key
The caller signs a JWT, gets a short-lived token
ClientID as sub · expiry enforced
01
Load balancer
02
Cloud Armor & WAF
03
Gateway authz
Three layers reject before anything runs
DDoS · OWASP · payload limits · rate limiting
A Cloud Function queries inside a private VPC
Python · filters & search · secrets fetched at call time
Filtered filings returned, usage counted
attributed to the calling identity
As scopedsigned off
0

Steps from key issue to API response

0

Platform components in the Phase 1 scope

0

Layers of defence before any logic runs

0

Design principles the architecture answers to

The client

An InsurTech rate-filing analytics company whose platform turns insurance rate filing data into visual analytics and generated reports for pricing and strategy teams. Identity withheld at the client's request.

The challenge

That data had to become a public API for partners and machine consumers - without giving the internet a route to the database, or to the credentials behind it.

The build

A serverless filing API on Google Cloud: API Gateway behind a load balancer and Cloud Armor, Python Cloud Functions in an isolated VPC, secrets held outside admin reach, all defined in Terraform.

What it delivers

List, search and view filings over an authenticated API - plus an admin dashboard for issuing access and watching usage per account.

01The client

An analytics company whose raw material is other people's paperwork.

The client is an InsurTech company built on regulatory rate filings - dense, slow-moving public documents that insurers, reinsurers and analysts need to read comparatively rather than one at a time. The platform turns that material into visual insight and generated reports, so a pricing or strategy team can see how a market is actually moving instead of working through filings in sequence.

That product already existed and already worked. What did not exist was any way for anyone else to reach the underlying data programmatically. A customer who wanted filings inside their own pricing model, a partner building a service on top of the dataset, a machine consumer pulling on a nightly schedule - every one of them needed an interface, and an interface means a public endpoint.

Focaloid was engaged to scope that endpoint: a secure backend that streamlines the retrieval, filtering and monitoring of regulatory insurance data, and can be handed to a third party without handing over anything else.

Rate filing analytics
Comparative market insight
Report generation
Partner & machine consumers
Regulatory data at scale
Per-account access control

The shape of the problem

The client's data has commercial value precisely because it is curated and complete. That is also exactly what makes uncontrolled access expensive - this was a security requirement arriving in the shape of a product decision.

Two things the API had to be at once

The structural tension that shapes every decision downstream.

An open door

Reachable, predictable, self-service to call

  • Reachable by partners and machines over the public internet
  • Documented and predictable enough to build against
  • Filterable and searchable, not a bulk dump
  • Callable on a schedule, without a human in the loop

A closed system

Nothing exposed but the endpoint itself

  • No network path from the endpoint to the database
  • No credential readable by anyone who doesn't need it
  • Every call attributable to a known identity
  • Every account's usage visible and limitable

Between them sits the requirement the whole architecture answers - the entry point has to be public, and everything behind it has to be unreachable.

02The challenge

The data was ready. Everything about exposing it was not.

Publishing an API over valuable data is not a routing problem. Six distinct requirements had to be satisfied before a single filing could be served to an outside caller - and each one fails differently.

01

A public endpoint is an invitation

The moment filing data is reachable over the internet it is also reachable by everything else on the internet: volumetric traffic, malformed payloads, and probing requests that never intended to be valid in the first place.

02

Machines cannot log in

The consumers are scripts and services, not people. Passwords, sessions and interactive consent do not apply - the API needed a machine-to-machine credential model designed in from the start, not retrofitted around a human login.

03

Keys have to be issued and shared

Someone has to create the account, generate the key and get it to the right person. Every step in that chain is a place a credential can leak, or the wrong account can quietly be granted access to a paid dataset.

04

Secrets cannot sit with the people issuing them

Private keys have to be stored somewhere the query functions can reach and the humans administering accounts cannot. That is a specific isolation requirement with a named boundary - not a general instruction to keep things safe.

05

The database must never sit next to the edge

Query logic needs the warehouse; the public endpoint must not be able to reach it. Without deliberate network separation, one compromised entry point is one hop from the data it was supposed to be protecting.

06

Usage has to be visible, per identity

Without per-account attribution and analytics there is no way to spot abuse, price access, or answer the basic operational question: who called this, how often, and for what.

Why it mattered

An API is a growth decision that arrives as a security decision. The value of the dataset is exactly what makes uncontrolled access costly - commercially as well as technically. The scope had to make the endpoint safe enough that opening it up became a straightforward product move rather than a risk the business had to weigh every time.

03The mandate

Three non-negotiables, fixed before a line of code.

Focaloid engaged to design and scope the API ecosystem end to end. Three constraints were set as hard architectural requirements rather than good intentions - each one is visible in the diagram, not just in the document.

Non-negotiable · 01

Isolation by default

Nothing in the architecture can reach anything it does not need. Query functions sit in their own VPC. Secrets are readable by the functions and by nobody else - including the administrators who create them.

Non-negotiable · 02

Nothing invalid reaches the logic

Malformed, oversized and unauthenticated requests are rejected at the network edge, before a Cloud Function ever runs. The gateway is not the first line of defence - it is the third.

Non-negotiable · 03

Infrastructure as code, no exceptions

Every GCP resource is provisioned through Terraform. No console clicks, no configuration drift, and no environment that exists only in one engineer's memory of how they built it.

The decision that shaped everything

Serverless, so there is no server to secure. Choosing Python Cloud Functions over a running service removes a whole category of attack surface - no long-lived host, no patch cadence on an instance, no idle capacity sitting there to be compromised - and lets capacity follow request volume instead of a forecast. Every decision downstream assumes short-lived, isolated execution.

A gated path from requirement to provisioned estate

P1

Edge & entry

Global HTTPS load balancer, Cloud Armor with OWASP policies, WAF rules and payload size restrictions - designed to reject cheaply, before anything is inspected in depth.

P2

Identity & tokens

Service accounts with per-account private keys, a JWT signed with the ClientID as its sub, exchanged at the gateway for a short-lived access token.

P3

Isolated delivery

Python Cloud Functions inside a private VPC, filtering and search parameters on the query, and secrets fetched from Secret Manager at the moment they are needed.

P4

Control & code

A React admin dashboard for accounts, keys and usage analytics - and the whole GCP estate declared in Terraform so it can be reviewed, versioned and rebuilt.

Scope discipline as a safety feature

What Phase 1 deliberately does not do. Key rotation is not self-service, there is no developer portal, and users cannot mint their own credentials - all three arrive with Apigee in Phase 2. Phase 1 issues keys through an administrator, which is slower and materially harder to get wrong. For an API that has never been public before, the smaller surface is the point, and the scope document says so explicitly rather than leaving it implied.

04How it works

A key is issued once. Every call after it is checked four times.

An administrator creates the account and hands over a private key and a ClientID. From then on the caller works entirely against the API: sign a token request, exchange it for a short-lived access token, and query filings until the token expires. Between the caller and the data sit four separate checks, in four separate places.

What every request has to satisfy

Rejected at the edgeValidated at the gateway
edge
Is this request even well-formed?
edge
Is the payload within its size limit?
edge
Does it trip an OWASP policy or WAF rule?
edge
Is this traffic pattern volumetric?
gateway
Is the JWT signed by a key we know?
gateway
Is the ClientID a live account?
gateway
Has the access token expired?
gateway
Is this account within its usage parameters?

Watch a filing request become a filtered response.

The console replays a representative call: the token exchange, the three edge checks, the isolated query, and the response - narrating each step as it happens.

  • The key is issued, never self-served. An administrator creates the service account, generates the private key and shares it with the ClientID - so every credential in circulation has a known owner and a known issue date.
  • The token is short-lived by design. The private key signs a JWT; the JWT buys an access token with an expiry. A leaked access token stops working on its own, without anyone having to notice it leaked.
  • Three layers stand in front of the logic. Load balancer, then Cloud Armor and WAF rules, then the gateway's own authorization and rate limiting. A Cloud Function only ever runs on a request that survived all three.
  • The data path never touches the public one. Functions query the warehouse from inside a private VPC and pull their credentials from Secret Manager at call time - nothing is baked into code, and nothing is reachable from the edge.
filing-api · request trace↻ replay
token
JWT presented, signed with account private key · sub matches a live ClientID
token
Access token issued - short-lived, scoped to this account
edge · 01
Global HTTPS load balancer - request accepted, TLS terminated
edge · 02
Cloud Armor - OWASP policies evaluated, payload within limits, WAF clean
edge · 03
API Gateway - token valid, not expired, usage parameters within allowance
route
GET /filings?state=TX&line=personal_auto&from=2025-01-01
function
Filing Discovery Service invoked - Python, cold start absorbed by the runtime
secrets
Warehouse credentials fetched from Secret Manager · nothing cached in code
vpc
Query executed from inside the private VPC - no route back to the edge
200
Filtered filing set returned · call attributed to the ClientID and counted
reject
A second call arrives with an expired token - refused at the gateway, no function invoked, no compute spent
Illustrative trace with representative parameters. In the delivered API every check, filter and count runs against the real request and the real filing corpus.

05Architecture

Six components, arranged so that no two of them trust each other more than they must.

A public API gateway as the only entry point. Python Cloud Functions carrying the discovery logic. A private VPC around the data path. Secret Manager holding credentials outside human reach. An IAM controller enforcing who can do what. And Terraform underneath all of it, so the arrangement is a definition rather than a memory.

Insurance Filing API · Google Cloud Platform
every resource Terraform-provisioned
Consumers
External clients & partners
HTTPS with signed JWT
Admin
React dashboard · accounts, keys, usage
Edge · security
Global HTTPS Load Balancer
minimise surface · DDoS mitigation
Cloud Armor
OWASP policies · WAF rules · payload limits
Public entry point
Cloud API Gateway
authorization · rate limiting · throttling
Security & IAM controller
authn / authz policy enforcement
Private VPC · access isolation
Cloud Function - list & search
Python · filter parameters
Cloud Function - view filing
Python · single-record retrieval
BigQuery - filing corpus
analytical scans · no public route in
Terraform - infrastructure as code
every resource above, declared · versioned · reproducible across environments
Isolated secrets
GCP Secret Manager
Database credentials · per-account private keys · configuration.
Fetched by the Cloud Functions at call time. No admin read path.
Admin surface
Dashboard & analytics
Create accounts, generate keys, watch API usage over time through visualisations.
Phase 2
Apigee - not in this scope
Admin management portal, developer portal and self-service key rotation arrive later. Key management moves here.
Request path · internal HTTPS
Secret fetch · functions only
Terraform-provisioned

The life of a request, step by step

STEP 01

An account is created

An administrator creates a user service account for the API consumer on the dashboard. Accounts are never self-registered in Phase 1.

STEP 02

A key is generated

A private key is generated for that account and written straight to Secret Manager - not to the dashboard, and not into any application code.

STEP 03

The key is shared

The administrator passes the access key and the ClientID to the user. This is the one deliberately manual step in the whole flow.

STEP 04

A token is obtained

The user signs a JWT with the ClientID as its sub, presents it in the authorization header, and receives a short-lived access token in return.

STEP 05

Usage is validated

The gateway checks the token and the account's usage parameters. Only then is the request authorized to reach anything behind it.

STEP 06

The database is queried

A Cloud Function inside the private VPC queries BigQuery using the caller's filter and search parameters, with credentials fetched at call time.

STEP 07

The response is returned

The filtered filing data is returned to the caller, and the call is counted against that account for the usage analytics the admin sees.

THE POINT

Seven steps, one manual

Only step three involves a human. Everything else is a policy the platform enforces the same way on every call, whether anyone is watching or not.

06Inside the build

The decisions that make a public endpoint defensible.

Anyone can put a function behind a URL. The engineering that makes it safe to hand the URL to a stranger lives in a handful of deliberate choices - each one below is specified in the scope, not inferred from it.

Decision A - Edge posture

Reject before you inspect.

The cheapest request to handle is the one that never reaches your code. So the architecture spends its first two layers doing nothing but refusing traffic: a global HTTPS load balancer to minimise the attack surface and absorb volumetric attacks, then Cloud Armor enforcing OWASP policies, WAF rules and payload size restrictions. Requests that do not conform to the expected structure are rejected outright - before any authorization logic runs, and long before a Cloud Function is invoked.

"Is this volumetric?"
→
"Is the payload oversized?"
→
"Does it trip an OWASP rule?"
→
"Is the structure even valid?"
→
"Is the token real and live?"

Three layers is a cost decision as much as a security one. On a serverless platform, an invalid request that reaches a function is a request you paid for. Refusing it at the edge means abuse traffic never becomes an invoice line - and never becomes a cold start in front of a real caller either.

Decision B - Machine identity

A private key, a signed assertion, and a token that expires on its own.

The consumers are machines, so the credential model is built for machines. Each account holds a private key issued by an administrator. To call the API the client signs a JWT with that key, putting its ClientID in the sub claim, and presents it in the authorization header. On success the platform returns a short-lived access token, which the client uses until it expires and then re-signs for another. Nothing long-lived ever travels on a request.

The authentication function is deliberately isolated from the public endpoint: when an invalid token is presented, the failure is contained in the auth path rather than reaching anything behind it.

# exchange a signed assertion for a short-lived token
POST /token
Authorization: Bearer <JWT signed with the account private key>

// the assertion the client signs
{
  "sub": "<ClientID>",
  "aud": "filing-api",
  "iat": 1738252800,
  "exp": 1738253100
}

// what comes back
{
  "access_token": "…",
  "token_type":   "Bearer",
  "expires_in":   900
}

Why an expiring token rather than a long-lived API key? A static key is only as safe as every place it has ever been copied. A short-lived token limits the blast radius of a leak to its own expiry window, and the identity in every request stays traceable to a named account - which is what makes per-account usage analytics possible at all.

Decision C - Secrets

The administrator issues the key and cannot read it back.

Credentials have to be reachable by the code that needs them and unreachable by everyone else - including the people who created them. The scope names that boundary explicitly, and the architecture enforces it in three beats.

1

Written on creation

When an account is created on the dashboard, its private key is written straight to Secret Manager alongside the database credentials and configuration.

2

Isolated from admins

Administrators have no direct access to Secret Manager. The dashboard can create and revoke; it cannot read a stored secret back out.

3

Fetched at call time

Only the Cloud Functions retrieve secrets, and only at the moment a request needs one. Nothing is baked into code or held in a config file.

It is a deliberately awkward arrangement, and that is the value. An admin who cannot read a key cannot leak one, cannot be phished for one, and cannot hand over every customer's credential in a single compromised session. The cost is that a lost key has to be replaced rather than re-sent - one extra step for one fewer catastrophic failure mode.

Decision D - Network isolation

The data path and the public path never meet.

The Cloud Functions carrying the discovery logic run inside their own VPC, configured specifically to maintain access isolation. The warehouse is reachable from inside that network and from nowhere else. The public endpoint terminates at the gateway; it has no route onward to the database, which means a compromise at the edge is a compromise of the edge rather than of the corpus behind it.

A

Scalable and lightweight

Short, read-only, bursty work is exactly what serverless handles best. Capacity follows request volume with no instance to size or keep warm.

B

Isolated from everything else

The functions are separated from the other infrastructure components by design, so their blast radius is the query they are running.

C

Managed secrets, narrow access

The functions are the only consumers of the secrets they need - an access policy in IAM, not a convention someone has to remember.

Decision E - Infrastructure as code

If it isn't in Terraform, it isn't provisioned.

The whole API infrastructure is provisioned through Terraform. This is the least glamorous decision in the scope and probably the one with the longest tail - five distinct properties follow from it, and every one of them is a thing you cannot retrofit cheaply.

  • Automated updates without human intervention. An infrastructure change is an applied definition, not a sequence of console actions someone has to perform correctly.
  • Reuse across environments with no extra overhead. The same design instantiates staging, production and anything after them, from one definition rather than three parallel histories.
  • Versioned and tracked over time. Every change to the estate has an author, a diff and a date - the same accountability code already has.
  • Effective team collaboration. The code is the single source of truth, so "why is this setting like that" has an answer that does not depend on who is still at the company.
  • A path to CI/CD. A declared estate is something a pipeline can act on. The scope notes this as future work; the choice that makes it possible is being made now.
Decision F - The admin surface

One dashboard, three jobs, no read path to a secret.

The React admin dashboard is where access is granted and usage is watched. It does three things, and its restraint about a fourth is what makes decision C hold.

01

A dedicated admin surface

Separate from the consumer-facing API entirely - a different audience, a different threat model, a different set of permissions.

02

User access management

Create the account, generate the key, share the ClientID, revoke when access ends. The full credential lifecycle, minus reading.

03

Usage monitoring & analytics

API usage over time, visualised per account - the operational view that makes abuse visible and access priceable.

07Defence in depth

Stated as layers, not as intentions.

"Secure" is not a property you can review. Layers are. Each one below has a named job, a named place it runs, and a specific class of thing it stops - so the posture can be audited rather than trusted.

ControlWhat it stopsWhere it runsWhy it exists
Global HTTPS load balancerVolumetric and DDoS traffic
edge
Minimises the attack surface before anything is inspected in detail
Cloud Armor & WAF rulesOWASP-class exploits, malformed requests
edge
Policy enforcement that does not depend on application code being correct
Payload size restrictionsOversized and abusive request bodies
edge
Cheap rejection of a common and inexpensive attack pattern
Gateway authorizationUnauthenticated, forged and expired tokens
gateway
Identity is established before any compute is spent on the request
Rate limiting & throttlingAbuse from inside a legitimate identity
gateway
A real account can still behave badly, deliberately or by accident
Private VPCAny direct route from the edge to the warehouse
network
The database is never adjacent to the public endpoint
Secret Manager isolationHuman read access to private keys
IAM
The people who issue credentials cannot read them back
JWT machine-to-machine auth
Short-lived access tokens
OWASP policies at the edge
Private VPC data path
Secrets isolated from admins
Per-account usage analytics
Comprehensive logging & monitoring

Being unable to help is sometimes the design working.

Can the dashboard show me a user's private key? They've lost theirs and I want to re-send it.

No - and that is the point of the arrangement. The key was written to Secret Manager when the account was created, and the dashboard has no read path to it. What it can do is revoke that credential and issue a new one. One extra step for the administrator, and one fewer way for every key on the platform to leak from a single compromised session.

Issue a replacement key

Auditable, by design principle

The scope names auditable as one of five design principles, delivered through comprehensive logging and monitoring across the stack - and the admin analytics view is the human-facing end of it. A control you cannot observe is a control you cannot prove, which is the difference between a security posture and a security intention.

08Infrastructure

Provisioned as code, or not provisioned at all.

Four technologies carry the whole build, and one of them exists purely so the other three can be rebuilt. The difference between an estate that was clicked into existence and one that was declared shows up the first time somebody needs a second environment.

Without IaC - clicked into existence

A configuration nobody can reconstruct

  • Resources created in the console, one environment at a time
  • No record of why a given setting is the way it is
  • Staging and production drift apart quietly
  • A change is a person, remembering correctly
  • Rebuilding an environment becomes an archaeology project
  • Nothing to review before a change lands

With Terraform - declared and versioned

A definition anyone can apply

  • Every GCP resource defined in Terraform
  • The code is the single source of truth
  • Environments reproduced from one definition
  • Updates applied without human intervention
  • A rebuild is an apply, not an investigation
  • CI/CD has something concrete to hook into

The same architecture, twice - the only difference is whether it exists as a definition or as institutional memory.

Python · Cloud Functions

The Filing Discovery Service - the logic that lists, searches, filters and returns insurance filings on demand.

Why: the work is short, bursty and read-only. Serverless means no host to secure, no patch cadence, and no idle capacity to pay for or defend.

React · Admin dashboard

Account creation, key generation, and the usage analytics view that visualises API consumption over time.

Why: the admin surface is a real application with real state, and it is the one place the platform needs a considered interface rather than an endpoint.

BigQuery · The filing corpus

The regulatory filing data and the queries the discovery service runs against it.

Why: columnar and built for analytical scans, which is the shape of every filing query - filter wide, narrow hard, return a slice.

Terraform · The estate

Every resource above, plus the gateway, the load balancer, Cloud Armor, the VPC, Secret Manager and the IAM policy.

Why: reproducibility and review. An infrastructure change earns the same scrutiny as a code change, and any environment can be rebuilt from the definition.

09What ships

What Phase 1 delivers, and what it leaves for Phase 2 on purpose.

The figures below are architectural facts from the signed-off scope - components, layers, steps. No performance percentage appears anywhere on this page, because none has been earned yet. The before-and-after is a description of what changes about the work, not a claimed metric.

The scoped platform, in countable facts

As scopedsigned off
0

Platform components in the Phase 1 architecture

0

Workflow steps, key issue to API response

0

Independent layers of defence at the edge

0

Core technologies in the stack

0

Design principles the architecture answers to

Before - no programmatic accessAfter - the scoped filing API
Filing data reachable only through the client's own productList, search and view filings over an authenticated API
No mechanism to grant a partner or a machine access to the datasetAn administrator issues a service account, a private key and a ClientID
No credential model designed for non-human consumersJWT signed with the account key, exchanged for a short-lived token
A public endpoint would sit directly in front of the warehouseThree edge layers, then a function inside a private VPC
Credentials would live wherever whoever created them decided to put themSecret Manager - readable by the functions, not by administrators
No visibility into who is calling what, or how oftenPer-account usage analytics, visualised over time on the dashboard
Environments built by hand and reconstructed from memoryThe whole estate declared in Terraform, versioned and reviewable

Access becomes a product

Filing data can be partnered, integrated and priced, because there is finally a controlled way to hand it to someone outside the company.

Security is structural

The isolation lives in the network topology and the IAM policy - not in a rule someone has to remember to follow on a busy afternoon.

Change stays cheap

A Terraform-defined estate can be extended, duplicated and reviewed without anyone re-deriving how it was assembled in the first place.

10Phase 2

Built so the administrator can be taken out of the credential path.

Phase 1 keeps a human in the middle of key issue deliberately. Phase 2 removes them - and the scope names exactly how, so the interim arrangement is documented as interim rather than discovered as debt later.

Where it goes next

  • Apigee admin management portal - usage monitoring, access management and per-user rules, with fine-grained API management configurable for each account.
  • Apigee developer portal - users log in, create their own secret keys, monitor their own usage and rotate credentials themselves, without an administrator in the loop.
  • Key management moves to Apigee - the Phase 1 Secret Manager arrangement is explicitly an interim, and the scope records that it changes when Apigee lands.
  • CI/CD over the Terraform estate - the IaC choice was made partly to make this cheap; the pipeline itself is the next step rather than a rewrite.
  • Measured outcomes - latency, throughput and per-account consumption figures, once the API is live and carrying real traffic worth measuring.

The reusable pattern

This generalises.

A three-layer edge, machine identity with short-lived tokens, compute isolated from the entry point, secrets isolated from the people who issue them, and the whole estate as code - the pattern transfers to any dataset with commercial value that therefore has to be handed out carefully.

Regulatory data APIs
Partner data access
Machine-to-machine auth
Serverless read APIs
Monetised datasets

The stack, end to end

PY
Python
RE
React
TF
Terraform
BQ
BigQuery
CF
Cloud Functions
AG
API Gateway
CA
Cloud Armor
LB
Load Balancer
SM
Secret Manager
VPC
Private VPC
IAM
Cloud IAM
GCP
Google Cloud

Let's build

Have data worth opening up - and good reasons to be careful about it?

We design and ship cloud platforms where the security posture is in the architecture rather than in the documentation. Isolated by default, provisioned as code, and reviewable before anything reaches production.