CSCapacity decision support
A global engineering group had bought more cores the year before, and it had worked: queue times fell by three quarters while the workload grew. The open question was how long that would hold. We built a simulation on their own job logs, proved it by having it predict a year it had never seen, and turned the next hardware decision into a single number the business owns.
A global engineering group running large-scale simulation workloads - structural and crash analysis, fluid dynamics, and the overnight batch. Identity withheld at the client's request.
Engineering was asking for more cores. Average utilisation read 54%, which made the case look weak. Nobody could price the next purchase.
A discrete-event simulation that replays the real job trace under a chosen growth rate and core count, on a medallion pipeline inside the client's own warehouse.
A Power BI report the committee drives itself, a companion screen that plays a scenario out visually, and a published record of where the model has and has not been tested.
The client
A global engineering group running large-scale simulation workloads for its design teams.
Structural and crash analysis, fluid dynamics, and the overnight batch that has to finish before the next morning's design review.
This case study covers the cluster carrying that work. Every job it handles - submission, start, completion, cores requested, queue - lands in the group's Snowflake warehouse, and the analysis below is built on those records at job level.
Early in 2026 the group added roughly 2,500 cores to the cluster, taking it from around 3,000 to around 5,500. It worked. Across two comparable six-month windows the queue shortened dramatically while the workload grew.
That is the position this engagement started from: a recent, successful purchase, and a leadership team wanting to know when the next one falls due.
Where this starts
The previous purchase was a success, and this page says so before it says anything else. What was missing afterwards was a way to price the next one.
Before
Average utilisation 77%. A 95th-percentile wait of 10.25 hours, and a worst individual case of 107.7 hours.
After
Average utilisation 54%. A 95th-percentile wait of 2.56 hours, on 36% more jobs.
The challenge
Engineering was asking for more cores. Leadership wanted the reasoning behind a number before approving one.
The obvious objection sat right on the dashboard: average utilisation was 54%, which reads as a machine sitting idle half the time. The case for more hardware looked weak on that number alone.
Daily requested core-hours repeatedly touch the available capacity line. A 54% average includes days running at or above 90%, and those are the days engineers notice.
Below roughly 60% utilisation, waiting is negligible and flat. Above 70% the curve turns sharply: in the 89% utilisation band, 95th-percentile wait reaches 122 minutes. A small increase in demand near that knee produces a large increase in wait.
Headline percentiles stay calm long after the longest-running jobs have degraded badly. The roughly eight-hour turnaround engineers were describing lives in those worst cases, which a mean or even a 95th percentile smooths over.
The step up is where the cores were added, and so is how quickly the peaks rose to meet it. This is the chart that answers the "utilisation is only 54%" objection. Capacity levels and the two window means are the engagement's figures; the daily series is drawn at the right shape for this comp and gets replaced by the delivery team's chart.
Flat to roughly 60%, then the sharp turn above 70%. The 60% knee and the 89% band at 122 minutes are the engagement's figures; the intermediate bands are drawn at the right shape for this comp.
A highway at half capacity flows freely. The same highway completely full is a traffic jam. Nothing changed about the road, and nothing had changed about the hardware. Judging a cluster by its average is like judging a highway by its 3am traffic.
average utilisation, the number that made the case look weak
utilisation on the days the average hides
95th-percentile wait once the fleet reaches the 89% band
the utilisation level above which waiting stops being flat
Utilisation measured per hour from job-level records. Wait measured from submission to start.
The mandate
Q1
Why do jobs wait when average utilisation looks low?
Q2
If workload grows by X%, what happens to queue time?
Q3
How many extra cores hold queue time inside the service target?
Q4
Does the model match what actually happened?
The service target
A 95th-percentile wait of five hours. Every sizing answer on this page resolves to holding that line.
Delivery constraint
Everything runs inside the client's existing Snowflake estate under existing permissions. No new platform, no data movement, no separate access model to maintain.
Scope
Diagnosis, simulation, validation, and delivery. Delivery is a Power BI report for the committee and a companion screen that plays a scenario out visually. The first release of that screen reads prepared data snapshots, a deliberate choice to keep it fast and portable in a meeting.
The solution
Under conditions that have not happened yet.
The system takes the real sequence of submitted jobs from the job log, every arrival and every run time exactly as it occurred, and replays it against a chosen workload growth and a chosen core count. It reports what the queue does: the 95th percentile, the 99th, the average wait of the longest-waiting jobs, how many jobs queue at all, and what utilisation the cluster settles at.
Nothing is sampled from an assumed distribution. The past is replayed under different conditions.
| Question | Output |
|---|---|
| What happens to turnaround if workload grows X%? | 95th and 99th percentile wait, plus the longest-waiting jobs |
| How many cores hold turnaround inside the target? | Cores to add, and what that is as a share of current capacity |
| When does today's headroom run out? | The growth point at which the service target is crossed with no purchase |
| When should we stop buying? | The point on the curve where added cores stop buying any time at all |
| What does over-buying cost? | Average utilisation under each option, the idle-hardware side of the trade |
Inputs
Arrivals and run times come straight from the job logs. No assumed arrival distribution, no fitted curve, no synthetic workload.
Growth
A workload increase compresses the gaps between real submissions, so a busier year keeps the shape of the real one: the same clustering around design reviews and batch windows.
Trade
Every purchase option shows the queue time it buys and the utilisation it gives away.
Estate
Inside the existing warehouse, under existing permissions.
Ringed points are the engagement's own figures: 2.56 hours today, the five-hour target crossed at roughly 12% growth with no purchase, and the 25% growth column at 9.65 / 5.25 / 2.65 / 2.00 / 2.00 hours. The curve between them is drawn for this comp and gets replaced by the delivery team's chart.
Architecture
Bronze
Raw ingest: flattened job logs and usage history.
Silver
Deduplicated records, queue-rule-corrected wait times, hourly usage rollups.
Gold
Status summary, by-queue and by-rule breakdowns, weekday and weekend splits, wait distributions.
Simulation
Two output sets: a retrospective grid used for validation, and forward scenarios that feed the report and the app.
Schema and table names are stripped: they identify the client.
Layer 2
A purpose-built discrete-event simulation. Running jobs sit in a min-heap ordered by completion time; waiting jobs sit in a first-come-first-served queue. Arrivals and run times are replayed straight from the real job logs, and a growth scenario compresses the gaps between real submissions by a factor of 1/(1+growth). Single resource pool, first-come-first-served discipline.
Layer 3
Raw simulated wait percentiles carry a known bias, so each receives an additive correction derived from real historical reference points and interpolated by utilisation level. The correction is deterministic and inspectable.
Layer 4
Power BI dashboards read the Gold tables directly. A companion web app plays a single scenario out visually.
Why it matters
A capacity model that needs its own platform is a second thing to govern. This one is a set of tables in a warehouse the client already runs.
Inside the build
Decision A
The alternative is to fit a statistical arrival pattern and generate jobs from it, which produces a tidy average week and loses the thing that causes queuing. Submissions cluster around design reviews and overnight batch windows, then go quiet.
Replaying the actual sequence keeps that clustering intact for free, along with the multi-day jobs that occupy cores everyone else queues behind.
Decision B
A growth scenario shortens the intervals between real submissions. A year with 25% more work keeps the shape of the real one: same peaks, same quiet windows, closer together.
Decision C
Every scenario returns the 95th percentile, the 99th, the average wait of the five longest-waiting jobs, and the count of jobs that queue at all. The tail metrics are there because that is where the complaints originate.
The gap between them is large. Projected to 35% workload growth with no additional cores, the headline percentiles stay comparatively calm while the five longest-waiting jobs average 96.8 hours. Adding 500 cores brings that to 72.5. The queued-job count is there for a simpler reason: fewer jobs waiting means fewer escalations.
+35% growth, no purchase
Average wait of the five longest-waiting jobs.
+35% growth, +500 cores
The same measure, on 9% more hardware.
Decision D
Every purchase option also reports the average utilisation the cluster lands at. Projected to 35% growth, adding 500 cores holds utilisation at 66%, while adding 5,000 drops it to 33.8%: queue-time relief bought at the price of idle hardware. Putting both on the same screen is what stops a recommendation reading as a blank cheque.
Large purchases drop average utilisation into the thirties. The +500 and +5,000 columns are the engagement's figures; the two between are drawn at the right shape for this comp.
Queue time is set by behaviour at peak. The budget conversation had only ever looked at the average.
Validation and trust
Before being shown it.
A capital request does not get signed off because a model sounds confident. It gets signed off when someone senior can see it called something correctly that it had no way of knowing.
So the model was fitted on January to June 2025 alone. It was then given the two things that actually happened in the following year, demand up 15.05% and roughly 2,500 cores added, and asked what the 95th-percentile wait would be.
The same method, refitted on the 2026 data, produces every forward scenario on this page.
Model's prediction
What actually happened
95th-percentile wait, January to June 2026.
The 2025-fitted curve with the 2026 outcome marked on it. Prediction and reality land on the same point, which is why there is only one marker. The marked point and the growth and core figures behind it are the engagement's; the rest of the curve is drawn for this comp.
predicted, on a year the model had never seen
what actually happened
the number of times it has been back-tested
| Condition | Status |
|---|---|
| The growth path that actually occurred | Validated, exactly |
| Core-reduction scenarios | Not yet run |
| Calibration where the reference points sit at near-identical utilisation | Known edge case: the worst-case figure can overshoot |
Publishing where a model has been tested, and where it has not, is what lets a recommendation be argued on its merits.
Back-test fitted on January to June 2025 job logs only, projected forward under the workload growth and core count that subsequently occurred, and compared against the January to June 2026 observed 95th-percentile wait.
The handover is a report anyone on the committee can drive. Every chart shares one x-axis, how much the workload grows, and one line per purchase option. The committee argues about the growth number, and the core count follows from it.
Report controls
Report pages
Metrics on every scenario
| Cores added | Share of current capacity | 95th-percentile wait |
|---|---|---|
| None | - | 9.65 hrs |
| 500 | +9% | 5.25 hrs |
| 1,000 | +18% | 2.65 hrs |
| 1,500 | +27% | 2.00 hrs |
| 2,000 | +36% | 2.00 hrs |
Against a five-hour service target, 500 cores brings a 25% growth scenario to 5.25 hours, within touching distance of the line, for 9% more hardware. The curve then flattens completely: 1,500 cores and 2,000 cores both land at 2.00 hours, so the last 500 buy nothing at all.
Where today's headroom ends
With no purchase at all, the 95th-percentile wait crosses the five-hour target at roughly 12% workload growth. That single number converts an open-ended hardware argument into a question the business can answer: how much growth are we planning for?
Also delivered
A browser-based screen that plays a single scenario out visually: jobs arriving, cores filling, the queue building and draining. Built for the room where the decision gets made, so people can watch why a plan breaks. This first release reads prepared data snapshots, which keeps it fast and portable for a meeting; a live warehouse connection is on the roadmap.
Figures from the 2026-fitted simulation at 25% workload growth. Core percentages expressed against current capacity.
Outcomes
Two comparable six-month windows, measured the same way.
| Measure | Jan-Jun 2025 | Jan-Jun 2026 |
|---|---|---|
| Average cores | 3,010 | 5,510 |
| Average utilisation | 77% | 54% |
| Jobs submitted | 84,300 | 114,500 |
| 95th-percentile wait | 10.25 hrs | 2.56 hrs |
| Worst individual case | 107.7 hrs | 29.1 hrs |
More work, absorbed faster. Demand rose 15% in core-hours and 36% in job count, and delays roughly halved.
95th-percentile wait, year on year
more jobs absorbed over the same window
more cores answers a 25% workload increase
the core count beyond which more hardware stops buying time
Cores required to hold 95th-percentile wait inside the five-hour target.
| Workload growth | Wait if nothing is bought | Cores to add | Share of capacity | Resulting wait |
|---|---|---|---|---|
| Today | 2.56 hrs | 0 | - | 2.56 hrs |
| +12% | about 5 hrs | 0 | - | about 5 hrs |
| +25% | 9.65 hrs | 500 | +9% | 5.25 hrs |
| +50% | about 24 hrs | about 1,500 | +27% | about 5 hrs |
The analysis converts a hardware argument into a growth assumption. Management owns the growth number; the model owns the core count that follows.
At 25% growth, 1,500 cores and 2,000 cores produce the same 2.00-hour wait, while average utilisation keeps falling. Over-buying is a measurable waste, and it can now be shown.
The model has been back-tested once. Each new quarter of job logs is another chance to check it, and the report rebuilds from live data.
The real output was a defensible size for the spend, and a page showing exactly where the answer has been tested. That page is why the recommendation survived the room.
What's next
Everything delivered so far is deterministic: real job traces, replayed, with an arithmetic correction. That was the right call for a model a committee has to trust with a capital decision. The next phase is where a trained model extends the pipeline.
| Opportunity | Technique | What it replaces |
|---|---|---|
| Workload forecasting | Prophet, SARIMA, exponential smoothing | Growth scenarios specified by hand |
| Wait-time regression | Gradient boosting | The two-point calibration correction |
| Job runtime prediction | Regression on job features | One-to-one replay of historical run times |
| Change-point detection | PELT, Bayesian online changepoint | A manual threshold for spotting capacity changes |
| Workload clustering | k-means, hierarchical | New capability, no equivalent today |
The reusable pattern
Diagnose peak against average, replay the real arrival trace, report the tail alongside the headline, prove the model against a period it has never seen, and publish where it has not been tested.
We start with the logs you already have.