Case Study · Secure Regulatory Data API
The client turns insurance rate filings into analytics its customers price and plan against. Letting partners and machines reach that data meant building an entry point that could survive being public. We scoped a serverless filing API on Google Cloud: three layers of defence before any logic runs, machine-to-machine authentication with short-lived tokens, query functions isolated in a private VPC, secrets the administrators cannot read, and every resource provisioned as code.
Steps from key issue to API response
Platform components in the Phase 1 scope
Layers of defence before any logic runs
Design principles the architecture answers to
The client
An InsurTech rate-filing analytics company whose platform turns insurance rate filing data into visual analytics and generated reports for pricing and strategy teams. Identity withheld at the client's request.
The challenge
That data had to become a public API for partners and machine consumers - without giving the internet a route to the database, or to the credentials behind it.
The build
A serverless filing API on Google Cloud: API Gateway behind a load balancer and Cloud Armor, Python Cloud Functions in an isolated VPC, secrets held outside admin reach, all defined in Terraform.
What it delivers
List, search and view filings over an authenticated API - plus an admin dashboard for issuing access and watching usage per account.
01The client
The client is an InsurTech company built on regulatory rate filings - dense, slow-moving public documents that insurers, reinsurers and analysts need to read comparatively rather than one at a time. The platform turns that material into visual insight and generated reports, so a pricing or strategy team can see how a market is actually moving instead of working through filings in sequence.
That product already existed and already worked. What did not exist was any way for anyone else to reach the underlying data programmatically. A customer who wanted filings inside their own pricing model, a partner building a service on top of the dataset, a machine consumer pulling on a nightly schedule - every one of them needed an interface, and an interface means a public endpoint.
Focaloid was engaged to scope that endpoint: a secure backend that streamlines the retrieval, filtering and monitoring of regulatory insurance data, and can be handed to a third party without handing over anything else.
The shape of the problem
The client's data has commercial value precisely because it is curated and complete. That is also exactly what makes uncontrolled access expensive - this was a security requirement arriving in the shape of a product decision.
The structural tension that shapes every decision downstream.
An open door
A closed system
Between them sits the requirement the whole architecture answers - the entry point has to be public, and everything behind it has to be unreachable.
02The challenge
Publishing an API over valuable data is not a routing problem. Six distinct requirements had to be satisfied before a single filing could be served to an outside caller - and each one fails differently.
The moment filing data is reachable over the internet it is also reachable by everything else on the internet: volumetric traffic, malformed payloads, and probing requests that never intended to be valid in the first place.
The consumers are scripts and services, not people. Passwords, sessions and interactive consent do not apply - the API needed a machine-to-machine credential model designed in from the start, not retrofitted around a human login.
Someone has to create the account, generate the key and get it to the right person. Every step in that chain is a place a credential can leak, or the wrong account can quietly be granted access to a paid dataset.
Private keys have to be stored somewhere the query functions can reach and the humans administering accounts cannot. That is a specific isolation requirement with a named boundary - not a general instruction to keep things safe.
Query logic needs the warehouse; the public endpoint must not be able to reach it. Without deliberate network separation, one compromised entry point is one hop from the data it was supposed to be protecting.
Without per-account attribution and analytics there is no way to spot abuse, price access, or answer the basic operational question: who called this, how often, and for what.
Why it mattered
An API is a growth decision that arrives as a security decision. The value of the dataset is exactly what makes uncontrolled access costly - commercially as well as technically. The scope had to make the endpoint safe enough that opening it up became a straightforward product move rather than a risk the business had to weigh every time.
03The mandate
Focaloid engaged to design and scope the API ecosystem end to end. Three constraints were set as hard architectural requirements rather than good intentions - each one is visible in the diagram, not just in the document.
Non-negotiable · 01
Nothing in the architecture can reach anything it does not need. Query functions sit in their own VPC. Secrets are readable by the functions and by nobody else - including the administrators who create them.
Non-negotiable · 02
Malformed, oversized and unauthenticated requests are rejected at the network edge, before a Cloud Function ever runs. The gateway is not the first line of defence - it is the third.
Non-negotiable · 03
Every GCP resource is provisioned through Terraform. No console clicks, no configuration drift, and no environment that exists only in one engineer's memory of how they built it.
The decision that shaped everything
Serverless, so there is no server to secure. Choosing Python Cloud Functions over a running service removes a whole category of attack surface - no long-lived host, no patch cadence on an instance, no idle capacity sitting there to be compromised - and lets capacity follow request volume instead of a forecast. Every decision downstream assumes short-lived, isolated execution.
Global HTTPS load balancer, Cloud Armor with OWASP policies, WAF rules and payload size restrictions - designed to reject cheaply, before anything is inspected in depth.
Service accounts with per-account private keys, a JWT signed with the ClientID as its sub, exchanged at the gateway for a short-lived access token.
Python Cloud Functions inside a private VPC, filtering and search parameters on the query, and secrets fetched from Secret Manager at the moment they are needed.
A React admin dashboard for accounts, keys and usage analytics - and the whole GCP estate declared in Terraform so it can be reviewed, versioned and rebuilt.
Scope discipline as a safety feature
What Phase 1 deliberately does not do. Key rotation is not self-service, there is no developer portal, and users cannot mint their own credentials - all three arrive with Apigee in Phase 2. Phase 1 issues keys through an administrator, which is slower and materially harder to get wrong. For an API that has never been public before, the smaller surface is the point, and the scope document says so explicitly rather than leaving it implied.
04How it works
An administrator creates the account and hands over a private key and a ClientID. From then on the caller works entirely against the API: sign a token request, exchange it for a short-lived access token, and query filings until the token expires. Between the caller and the data sit four separate checks, in four separate places.
The console replays a representative call: the token exchange, the three edge checks, the isolated query, and the response - narrating each step as it happens.
sub matches a live ClientIDGET /filings?state=TX&line=personal_auto&from=2025-01-0105Architecture
A public API gateway as the only entry point. Python Cloud Functions carrying the discovery logic. A private VPC around the data path. Secret Manager holding credentials outside human reach. An IAM controller enforcing who can do what. And Terraform underneath all of it, so the arrangement is a definition rather than a memory.
An administrator creates a user service account for the API consumer on the dashboard. Accounts are never self-registered in Phase 1.
A private key is generated for that account and written straight to Secret Manager - not to the dashboard, and not into any application code.
The administrator passes the access key and the ClientID to the user. This is the one deliberately manual step in the whole flow.
The user signs a JWT with the ClientID as its sub, presents it in the authorization header, and receives a short-lived access token in return.
The gateway checks the token and the account's usage parameters. Only then is the request authorized to reach anything behind it.
A Cloud Function inside the private VPC queries BigQuery using the caller's filter and search parameters, with credentials fetched at call time.
The filtered filing data is returned to the caller, and the call is counted against that account for the usage analytics the admin sees.
Only step three involves a human. Everything else is a policy the platform enforces the same way on every call, whether anyone is watching or not.
06Inside the build
Anyone can put a function behind a URL. The engineering that makes it safe to hand the URL to a stranger lives in a handful of deliberate choices - each one below is specified in the scope, not inferred from it.
The cheapest request to handle is the one that never reaches your code. So the architecture spends its first two layers doing nothing but refusing traffic: a global HTTPS load balancer to minimise the attack surface and absorb volumetric attacks, then Cloud Armor enforcing OWASP policies, WAF rules and payload size restrictions. Requests that do not conform to the expected structure are rejected outright - before any authorization logic runs, and long before a Cloud Function is invoked.
Three layers is a cost decision as much as a security one. On a serverless platform, an invalid request that reaches a function is a request you paid for. Refusing it at the edge means abuse traffic never becomes an invoice line - and never becomes a cold start in front of a real caller either.
The consumers are machines, so the credential model is built for machines. Each account holds a private key issued by an administrator. To call the API the client signs a JWT with that key, putting its ClientID in the sub claim, and presents it in the authorization header. On success the platform returns a short-lived access token, which the client uses until it expires and then re-signs for another. Nothing long-lived ever travels on a request.
The authentication function is deliberately isolated from the public endpoint: when an invalid token is presented, the failure is contained in the auth path rather than reaching anything behind it.
Why an expiring token rather than a long-lived API key? A static key is only as safe as every place it has ever been copied. A short-lived token limits the blast radius of a leak to its own expiry window, and the identity in every request stays traceable to a named account - which is what makes per-account usage analytics possible at all.
Credentials have to be reachable by the code that needs them and unreachable by everyone else - including the people who created them. The scope names that boundary explicitly, and the architecture enforces it in three beats.
When an account is created on the dashboard, its private key is written straight to Secret Manager alongside the database credentials and configuration.
Administrators have no direct access to Secret Manager. The dashboard can create and revoke; it cannot read a stored secret back out.
Only the Cloud Functions retrieve secrets, and only at the moment a request needs one. Nothing is baked into code or held in a config file.
It is a deliberately awkward arrangement, and that is the value. An admin who cannot read a key cannot leak one, cannot be phished for one, and cannot hand over every customer's credential in a single compromised session. The cost is that a lost key has to be replaced rather than re-sent - one extra step for one fewer catastrophic failure mode.
The Cloud Functions carrying the discovery logic run inside their own VPC, configured specifically to maintain access isolation. The warehouse is reachable from inside that network and from nowhere else. The public endpoint terminates at the gateway; it has no route onward to the database, which means a compromise at the edge is a compromise of the edge rather than of the corpus behind it.
Short, read-only, bursty work is exactly what serverless handles best. Capacity follows request volume with no instance to size or keep warm.
The functions are separated from the other infrastructure components by design, so their blast radius is the query they are running.
The functions are the only consumers of the secrets they need - an access policy in IAM, not a convention someone has to remember.
The whole API infrastructure is provisioned through Terraform. This is the least glamorous decision in the scope and probably the one with the longest tail - five distinct properties follow from it, and every one of them is a thing you cannot retrofit cheaply.
The React admin dashboard is where access is granted and usage is watched. It does three things, and its restraint about a fourth is what makes decision C hold.
Separate from the consumer-facing API entirely - a different audience, a different threat model, a different set of permissions.
Create the account, generate the key, share the ClientID, revoke when access ends. The full credential lifecycle, minus reading.
API usage over time, visualised per account - the operational view that makes abuse visible and access priceable.
07Defence in depth
"Secure" is not a property you can review. Layers are. Each one below has a named job, a named place it runs, and a specific class of thing it stops - so the posture can be audited rather than trusted.
The load balancer minimises the exposed surface and absorbs volumetric traffic before anything downstream is aware of it.
Cloud Armor applies OWASP policies, WAF rules and payload limits - enforcement that lives outside the application code entirely.
The gateway validates the token, confirms the account is live, and checks usage parameters before spending any compute.
The private VPC and Secret Manager boundaries mean a breach of any earlier layer still does not reach the data or the credentials.
| Control | What it stops | Where it runs | Why it exists |
|---|---|---|---|
| Global HTTPS load balancer | Volumetric and DDoS traffic | edge | Minimises the attack surface before anything is inspected in detail |
| Cloud Armor & WAF rules | OWASP-class exploits, malformed requests | edge | Policy enforcement that does not depend on application code being correct |
| Payload size restrictions | Oversized and abusive request bodies | edge | Cheap rejection of a common and inexpensive attack pattern |
| Gateway authorization | Unauthenticated, forged and expired tokens | gateway | Identity is established before any compute is spent on the request |
| Rate limiting & throttling | Abuse from inside a legitimate identity | gateway | A real account can still behave badly, deliberately or by accident |
| Private VPC | Any direct route from the edge to the warehouse | network | The database is never adjacent to the public endpoint |
| Secret Manager isolation | Human read access to private keys | IAM | The people who issue credentials cannot read them back |
No - and that is the point of the arrangement. The key was written to Secret Manager when the account was created, and the dashboard has no read path to it. What it can do is revoke that credential and issue a new one. One extra step for the administrator, and one fewer way for every key on the platform to leak from a single compromised session.
Issue a replacement keyAuditable, by design principle
The scope names auditable as one of five design principles, delivered through comprehensive logging and monitoring across the stack - and the admin analytics view is the human-facing end of it. A control you cannot observe is a control you cannot prove, which is the difference between a security posture and a security intention.
08Infrastructure
Four technologies carry the whole build, and one of them exists purely so the other three can be rebuilt. The difference between an estate that was clicked into existence and one that was declared shows up the first time somebody needs a second environment.
Without IaC - clicked into existence
With Terraform - declared and versioned
The same architecture, twice - the only difference is whether it exists as a definition or as institutional memory.
The Filing Discovery Service - the logic that lists, searches, filters and returns insurance filings on demand.
Why: the work is short, bursty and read-only. Serverless means no host to secure, no patch cadence, and no idle capacity to pay for or defend.
Account creation, key generation, and the usage analytics view that visualises API consumption over time.
Why: the admin surface is a real application with real state, and it is the one place the platform needs a considered interface rather than an endpoint.
The regulatory filing data and the queries the discovery service runs against it.
Why: columnar and built for analytical scans, which is the shape of every filing query - filter wide, narrow hard, return a slice.
Every resource above, plus the gateway, the load balancer, Cloud Armor, the VPC, Secret Manager and the IAM policy.
Why: reproducibility and review. An infrastructure change earns the same scrutiny as a code change, and any environment can be rebuilt from the definition.
09What ships
The figures below are architectural facts from the signed-off scope - components, layers, steps. No performance percentage appears anywhere on this page, because none has been earned yet. The before-and-after is a description of what changes about the work, not a claimed metric.
Platform components in the Phase 1 architecture
Workflow steps, key issue to API response
Independent layers of defence at the edge
Core technologies in the stack
Design principles the architecture answers to
| Before - no programmatic access | After - the scoped filing API |
|---|---|
| Filing data reachable only through the client's own product | List, search and view filings over an authenticated API |
| No mechanism to grant a partner or a machine access to the dataset | An administrator issues a service account, a private key and a ClientID |
| No credential model designed for non-human consumers | JWT signed with the account key, exchanged for a short-lived token |
| A public endpoint would sit directly in front of the warehouse | Three edge layers, then a function inside a private VPC |
| Credentials would live wherever whoever created them decided to put them | Secret Manager - readable by the functions, not by administrators |
| No visibility into who is calling what, or how often | Per-account usage analytics, visualised over time on the dashboard |
| Environments built by hand and reconstructed from memory | The whole estate declared in Terraform, versioned and reviewable |
Filing data can be partnered, integrated and priced, because there is finally a controlled way to hand it to someone outside the company.
The isolation lives in the network topology and the IAM policy - not in a rule someone has to remember to follow on a busy afternoon.
A Terraform-defined estate can be extended, duplicated and reviewed without anyone re-deriving how it was assembled in the first place.
10Phase 2
Phase 1 keeps a human in the middle of key issue deliberately. Phase 2 removes them - and the scope names exactly how, so the interim arrangement is documented as interim rather than discovered as debt later.
The reusable pattern
A three-layer edge, machine identity with short-lived tokens, compute isolated from the entry point, secrets isolated from the people who issue them, and the whole estate as code - the pattern transfers to any dataset with commercial value that therefore has to be handed out carefully.
Let's build
We design and ship cloud platforms where the security posture is in the architecture rather than in the documentation. Isolated by default, provisioned as code, and reviewable before anything reaches production.