Medium

By Violetta Pidvolotska and Naveen Mareddy Stratum is Netflix’s internal serverless media processing platform, shaped by over a decade of experience developing large-scale distributed systems. It powers a wide range of media workloads — including encoding, packaging, inspection, and quality scoring — behind playback, advertising, studio workflows, AI embeddings, and more. Engineers develop Stratum functions by packaging application code and OS dependencies into OCI images. Stratum then schedules these containers as batch jobs on Netflix’s Titus compute platform, with upstream services triggering function executions. Every day, Stratum manages hundreds of thousands of containers across bursty, high-throughput, and latency-sensitive workloads. As Netflix’s core media processing platform, this load needs to scale as new content formats enter the catalog, such as podcasts, live events, sports, ads, or regional ingests like TF1 in France. To request resources for these functions, owners specify their CPU, memory, disk, and network limits as YAML configuration, as shown below: functionName: VideoEncoder.encodeV1 resources: numCpus: 8 memoryInMB: 18000 diskSizeInMB: 50000 networkInMbps: 1000 Maintaining optimal values manually is challenging amidst evolving Netflix infrastructure, EC2 instance upgrades, and application-level changes. Consequently, configs are often over-provisioned upfront or slowly drift away from their ideal values over time. Mis-sized configurations are a universal pain of large-scale serverless platforms, causing additional cognitive load for function builders and operational costs for platform owners. In this post, we share how we built the Stratum Resource Tuner (SRT) — an automated system that safely right-sizes running containers in production without human intervention. We will walk through real-world examples, our safety architecture, and what we learned when an automated change went wrong. ### The Challenge: Over-Provisioning When engineers write a new Stratum function, they typically copy a resource YAML from an existing function or size it for the absolute worst-case scenario. Once running in production, engineers rarely revisit these configurations until a failure occurs, prompting them to over-provision resources out of an abundance of caution. Manual resource tuning does not scale for the following reasons: - High Operational Friction: Downsizing requires days of manual metric monitoring and trial-and-error testing, diverting valuable engineering cycles away from other high-impact work. If a cut goes too far, engineers face production incidents — making manual tuning a high-risk, low-reward task. - Continuous Infrastructure & Codec Evolution: Cloud providers regularly upgrade their hardware. A resource allocation that was optimal on an older hardware generation quietly becomes inefficient on newer hardware. At the same time, Netflix continuously advances media standards and improves encoding algorithms (such as AV1, AV2, and spatial audio). An allocation tuned for one codec might not be optimal when an improved algorithm is released. - Dynamic Workload Evolution: As Netflix expands into new media types — such as streaming video podcasts, sports, interactive AI-adapted ad formats, and major broadcaster integrations like TF1 in France — media processing footprints can change unpredictably, rendering static container resource limits imprecise. - Rising Industry Hardware Costs: Driven by massive global AI infrastructure demand, server hardware and DRAM prices continue to climb, making idle container memory significantly more expensive to waste. - Compounding Platform Scale: At Netflix scale, even minor over-provisioning multiplies exponentially across hundreds of thousands of daily container executions, compounding wasted capacity across the entire Stratum platform. To understand the scale of the challenge, we analyzed historical usage across Stratum’s workloads. The telemetry confirmed our hypothesis: over-provisioning was frequent, wasted compute was compounding, and the sustainable path forward required a fundamental re-evaluation of how resource configurations are managed. ### Meet the Stratum Resource Tuner To address the challenges listed above, we built the Stratum Resource Tuner (SRT) to automatically right-size Stratum containers. Our north star is to ensure that users never have to manually specify the resource configurations. We started with a handful of high-throughput workloads with relaxed latency requirements, which account for the majority (70%–80%) of resource usage. As a quick note on terminology, we use “right-sizing” and “tuning” interchangeably in this post. “Downsizing” means decreasing resource limits, while “upsizing” means increasing them. Since our goal is to optimize for cost, this post focuses on downsizing. Automatic right-sizing is not enabled by default. Function owners must manually opt in on a per resource type basis (e.g., CPU, memory, network, or disk). Once enrolled, SRT operates without a human in the loop across five steps: Figure 1: SRT Workflow High-Level 1. Collect resource usage metrics across all executions of that function. 2. Generate a recommendation for each resource type (e.g., CPU or memory). 3. Verify the recommendation through experiments. 4. Apply or reject the recommendation based on the results of experiments. 5. Monitor post-apply metrics, watching for anomalies after applying the recommendation. “Recommendation” is a key construct in SRT and serves as the primary unit of operation. As SRT executes its steps, it promotes the recommendation through several stages: Figure 2: Recommendation State Machine - CREATED: A scheduled workflow generated the recommendation. - EVALUATING: Recommendation verification is running, and SRT is analyzing the results. - APPLIED: All experiments passed, and SRT automatically rolled out the recommended configuration. - REJECTED: At least one experiment failed, leaving the function configuration untouched. - MONITORED: SRT monitors the recommendation during a post-apply soak period to verify stability. The sections below examine each of these stages in detail. To illustrate the flow, we will follow the real-world lifecycle of the VideoEncoder.encodeV1 function — a high-throughput video encoding function whose over-provisioned memory and CPU footprints were successfully tuned. Note that all operational thresholds and safety limits discussed below are fully configurable per function to accommodate varying workload characteristics and risk tolerances. ### Metrics Collection To make these recommendations possible, we need a solid metrics collection mechanism. Every Stratum container reports its function execution metadata, infrastructure dimensions, and resource utilization (CPU, memory, disk, and network) at sub-minute resolution. These metrics flow through Keystone, Netflix’s real-time event streaming platform, and land in two separate datastores: Figure 3: Metrics Collection - Execution summaries store: Stores execution-level summary metrics indexed in Elasticsearch for fast point lookups, serving as our short-term datastore for querying and inspecting individual runs. - High-resolution long-term store: Stores raw, high-resolution usage metrics in Iceberg tables. While slower to query, it serves as our long-term datastore, holding months of history for resource utilization trend analysis. One record is written per execution, which is represented as a span. Here is how these records look for a single span of our VideoEncoder.encodeV1 function. In Elasticsearch, the record captures the high-level summary of the completed span, identified by tags.stratum.spanId: { "tags.stratum.spanId": "38536313f65f060a", "tags.functionName": "VideoEncoder.encodeV1", "tags.stratum.phase": "run", "tags.stratum.success": true, "tags.stratum.retriedWithMoreResources": false, "tags.stratum.function.execDurationSecs": 26, "tags.cpuCount": 8, "tags.memoryInMB": 18000, "tags.diskInMB": 50000, "tags.networkInMbps": 1000, "tags.titusTaskId": "06e1a7c0-2502-4a54-905f-b13a1e786107", "tags.instanceType": "r7a.24xlarge", "tags.nf.region": "us-east-1", "ts": 1780743028000 } In Iceberg, the record contains the same metadata, but adds the sub-minute resource utilization timeseries tracked during that span: { "span_id": "38536313f65f060a", "function_name": "VideoEncoder.encodeV1", "phase": "run", "success": true, "start_epoch_secs": 1780743002, "end_epoch_secs": 1780743028, "runtime_secs": 26, "cpu_requested": 8, "memory_mb_requested": 18000, "disk_mb_requested": 50000, "network_mbps_requested": 1000, "titusTaskId": "06e1a7c0-2502-4a54-905f-b13a1e786107", "instance_type": "r7a.24xlarge", "tags.stratum.retriedWithMoreResources": false, "env": "prod", "resource_usage": [ { "cpu_usage": 1.13, "memory_mb": 3701.37, "disk_mb": 20120.4, "net_in_mb": 221.09, "net_out_mb": 0.12 }, { "cpu_usage": 2.25, "memory_mb": 3584.09, "disk_mb": 25410.1, "net_in_mb": 0.02, "net_out_mb": 0.05 }, { "cpu_usage": 4.16, "memory_mb": 6750.79, "disk_mb": 30400.5, "net_in_mb": 165.90, "net_out_mb": 0.38 }, ... ] } ### Recommendation Generation A scheduled Conductor workflow runs daily, reading the past month of usage data from Iceberg and generating recommendations for every function and resource type. The workflow also generates recommendations for functions that have not opted into automatic right-sizing to preview potential savings. Each recommendation is saved in the recommendations datastore, enabling us to query and filter recommendations by function, resource type, or status on an ad hoc basis. Figure 4: Recommendation Generation Sequence For each function and resource type, the workflow applies the following generation logic: - Filter by workload type: We only tune high-throughput workloads. This focuses our efforts on right-sizing, which yields the most meaningful fleet-wide savings. - Aggressively tune CPU and network: We target p99 usage with no extra headroom. CPU and network are compressible resources; if a container occasionally hits these limits, the function might execute slightly slower. - Conservatively tune memory and disk: We target p99.9 usage and include a safety buffer. Running out of physical memory or disk space causes hard failures — such as Out of Memory (OOM) or Out of Disk (OOD) crashes — making a protective margin essential. - Apply safeguards to changes: We discard recommendations if the absolute reduction is too small to justify operational effort. For memory and disk, we clamp the maximum reduction allowed in a single cycle to 15% — a safety limit introduced after learning from early production runs. The workflow creates new recommendations in the CREATED state. If a recommendation already exists in this state for a given function and resource type, SRT overwrites it with updated calculations. Once a recommendation advances past the CREATED state, its recommended_value and stats fields become immutable. Subsequent daily runs generate a new recommendation with a unique ID instead of modifying the existing record. Here is how these recommendations look across resource types in their initial CREATED state for our VideoEncoder.encodeV1 function: - CPU recommendation { "function_name": "VideoEncoder.encodeV1", "env": "prod", "resource_type": "CPU", "current_value": 8, "recommended_value": 4, "status": "CREATED", "resource_cost_usd_per_hour": ***, "stats": { "usage": { "p99": 4.16, "p999": 4.57 }, "execution": { "total_fn_executions": 748762182, "success_fn_executions_pct": 99.75, "total_fn_exec_runtime_hr": 88102723 }, "window": { "start": "2026-05-05T00:00:00Z", "end": "2026-06-05T00:00:00Z" } }, "experiment_inputs": null, "experiment_workflow_execution_id": null, "outcome_message": null, "finalized_at": null } - Memory recommendation (15% reduction in one step) { "id": "2edb734a-6184-4f5e-8ffa-23004bd665d7", "function_name": "VideoEncoder.encodeV1", "env": "prod", "resource_type": "MEMORY_MB", "current_value": 18000, "recommended_value": 15300, "status": "CREATED", "resource_cost_usd_per_hour": ***, "stats": { "usage": { "avg": 5296.43, "p99": 6750.8, "p999": 7820.3 }, "execution": { "total_fn_executions": 748762182, "success_fn_executions_pct": 99.75, "total_fn_exec_runtime_hr": 88102723 }, "window": { "start": "2026-05-05T00:00:00Z", "end": "2026-06-05T00:00:00Z" } }, "experiment_inputs": null, "experiment_workflow_execution_id": null, "outcome_message": null, "finalized_at": null } As the recommendation moves through its lifecycle, SRT populates the remaining parameters. When proceeding to the verification stage, SRT transitions the recommendation status to EVALUATING, records the target experiment settings, and links the active workflow execution tracking the experiment. Finally, once verification runs complete, the status changes to APPLIED or REJECTED, storing the outcome details and completion timestamp for analysis. ### Retry with More Resources We take a conservative approach to generating recommendations, prioritizing system stability over extreme efficiency. Because recommendations rely on historical usage data, they cannot always anticipate sudden spikes or novel workload profiles when functions encounter heavy, unseen requests. To handle these unexpected spikes, we developed a fail-safe mechanism. A watcher process running in each tuned container tracks memory and disk usage during function execution. If the container exceeds the pre-set thresholds, 85% for non-cached memory usage or 95% for disk usage, the watcher flags the execution as high risk, cancels execution, and re-queues the task onto a larger container: new_container_size = original_container_size * 1.5 This 1.5 multiplier only applies to memory and disk dimensions. Figure 5: Retry with More Resources This “retry with more resources” mechanism serves as our safety net: larger container configurations are applied only when retrying tasks to self-heal from edge cases. Although these infrequent retries incur additional compute costs and latency, they allow us to keep day-to-day configurations highly optimized without having to size for rare, costly outliers up front. For our VideoEncoder.encodeV1 function, if a heavy run causes memory usage to exceed the 85% threshold (15,300 MB) within its current 18,000 MB container, the system aborts the execution and reruns it on a 27,000 MB container. While we prefer to trigger retries based on actual OOM or OOD events rather than pre-set thresholds, technical constraints in our current architecture make true event detection challenging. Using pre-set thresholds serves as a reliable stopgap while we develop improved detection capabilities. SRT automatically enables the retry mechanism for any function opted into memory or disk right-sizing. Each Stratum container reports retry events, which flow into Elasticsearch and Iceberg alongside standard performance telemetry. ### Recommendation Verification Every recommendation must pass a two-stage verification process managed by a Conductor workflow before rollout. During this EVALUATING stage, SRT executes two distinct experiment types: 1. Synthetic experiment: A low-cost, low-risk test to catch obvious regressions before production rollout. 2. Production experiment: A live evaluation that validates changes across diverse production workloads to increase confidence. A recommendation advances to production testing only after passing the synthetic experiment. When an opted-in function has multiple pending recommendations, SRT prioritizes and evaluates the candidates with the highest estimated savings first. Figure 6: Evaluating Stage Synthetic experiment: - Runs in a test environment isolated from production traffic. - Sets up two clusters: one for the current and one for the recommended config. - Replays identical request sets across both clusters simultaneously without splitting traffic, exposing both configurations to 100% of test requests. - Typically runs for 1–2 hours. - Runs to completion without early termination checks. Figure 7: Synthetic Experiment Flow Production experiment: - Runs in production on real user traffic, split three ways: ◦ ~90% remains on the main cluster (current production config). ◦ ~5% goes to a baseline cluster (current production config). ◦ ~5% goes to an experimental cluster (recommendation config). - Evaluates recommendations against a similarly sized baseline cluster rather than the main cluster to preserve comparable traffic profiles. - Typically runs from one day to one week, depending on traffic volume. - Fails fast: cancels the experiment early if regressions are detected. Figure 8: Production Experiment Flow The Stratum platform executes other experiments alongside SRT experiments. To avoid over-splitting traffic, SRT experiments run at lower priority and yield resources to active, higher-priority experiments. Traffic arrives in irregular bursts separated by multi-day or multi-week lulls. The verification pipeline accommodates these variable arrival patterns rather than expecting steady traffic. Back to our VideoEncoder.encodeV1 memory recommendation (18,000 MB → 15,300 MB), verification is triggered with: { "recommendationId": "2edb734a-6184-4f5e-8ffa-23004bd665d7", "syntheticExperiment": { "numberOfExecutions": 3000 }, "productionExperiment": { "enabled": true, "maxContainers": 20, // max containers shift to baseline & experimental clusters "upGateTimeoutMinutes": 10080, // time to wait for traffic to start flowing "maxRunDurationMinutes": 1240, // max time to run the experiment once traffic starts "minSamples": 500 // min executions needed on both clusters before evaluating } } Both verification stages evaluate recommendations against two sequential checks, terminating immediately if either fails: - Retries with more resources: Restricts memory and disk downsizes to a maximum 1% retry threshold over the experiment duration. - Execution time: Compares median runtimes to prevent outlier distortion, permitting up to a 10–15% slowdown relative to baseline, depending on workload, and is configurable by function. If a recommendation passes both verification stages, SRT updates its status to APPLIED and rolls out the configuration to the main production cluster. If a check fails at any point, SRT marks the recommendation as REJECTED and preserves the existing configuration. SRT sends notifications for both outcomes. For VideoEncoder.encodeV1, both stages passed cleanly: { "outcome": "APPLIED", "syntheticExperiment": { "result": "SUCCESS", "samples": { "baseline": 3000, "experimental": 3000 }, "medianExecutionSecs": { "baseline": 59.0, "experimental": 59.0 }, "slownessIncreasePct": 0.0, "retryRatePct": { "baseline": 0.0, "experimental": 0.0 } }, "productionExperiment": { "result": "SUCCESS", "samples": { "baseline": 50361, "experimental": 49790 }, "medianExecutionSecs": { "baseline": 196.0, "experimental": 198.0 }, "slownessIncreasePct": 1.0, "retryRatePct": { "baseline": 0.0, "experimental": 0.0 } } } The recommendation moved to the APPLIED state, permanently setting the memory allocation for VideoEncoder.encodeV1 to 15,300 MB. We observed similarly clean results for the CPU recommendation, which reduced CPU allocation from 8 cores to 4. ### Recommendation Verification Passing verification experiments does not mark the end of the recommendation lifecycle. Updated function code, modified input parameters, or underlying changes in the compute environment (Titus/EC2) can alter production behavior over time. To ensure configuration safety and foster feature adoption, we instituted continuous post-apply monitoring for applied changes. We monitor metrics for 90 days across two rolling time windows: a 30-minute fast window and a 24-hour slow window. Each window maintains specific alert thresholds: a 3% retry rate over 30 minutes or a 2% rate over 24 hours triggers an automated notification. At steady state, workloads exhibit baseline retry rates around 0.1%, with rates up to 2%–3% falling within acceptable operational variances. Most applied recommendations remained stable over the past three months. VideoEncoder.encodeV1 has operated on its tuned configuration for months without breaching thresholds. However, VideoEncoder.encodeV2 presented a challenging edge case: a 50% memory reduction passed all verification checks and ran stably for two weeks before an unexpected traffic shift drove retry rates above 60%. Figure 9: Unexpected Retries Spike Because early versions lacked automated rollback capabilities, engineers manually triaged exception cases. This incident highlighted the need for automated mitigation, prompting us to construct self-healing rollback functionality. Today, the two auto-revert windows look like this: - Last 30 minutes: 3% retries warn us; 10% triggers an auto-revert. - Last 24 hours: A 2% retry rate triggers both an alert and an automatic revert. Breaching an auto-revert threshold immediately restores the function configuration to its pre-recommendation baseline. Figure 10: Recommendation Auto-Revert We intentionally maintain straightforward revert logic — restoring the last known stable configuration rather than attempting dynamic resource adjustments during an incident. The VideoEncoder.encodeV2 incident also reshaped our recommendation generation rules for memory and disk allocations. Rather than applying full target reductions immediately, SRT now applies smaller, incremental 15% step changes, advancing only after confirming stability at each step. Additionally, we introduced a global kill switch to disable automated right-sizing across all functions in case of unexpected issues or side effects. Additionally, we enforce a mandatory cooldown period — typically one week — before initiating new experiments on a function. Pausing adjustments allows us to observe performance across diverse production traffic patterns, identifying potential risks before applying subsequent recommendations. ### Where Are We Today Oversized containers and config drift don’t have to be an accepted tax of running serverless workloads at scale. By building the Stratum Resource Tuner, we transformed manual, risk-averse resource sizing into an automated, self-healing pipeline. Through real-world workloads and hard lessons from unexpected spikes, we’ve proven that continuous right-sizing can safely deliver significant gains in infrastructure efficiency. We initially focused on right-sizing high-throughput video encoding workloads, aiming for an initial 5% reduction in overall Stratum compute costs. For instance, VideoEncoder.encodeV1 achieved a 50% reduction in CPU usage and 40% in memory usage without compromising performance or retry rates. However, edge cases like VideoEncoder.encodeV2 highlighted key lessons: large single-step reductions can introduce hidden risks, and manual incident response slows recovery. These insights led us to enforce stricter safety guardrails in the Stratum Resource Tuner, including capping memory and disk reductions at 15% per cycle and adding automated rollbacks when retry thresholds are breached. Guided by these safeguards, re-tuning VideoEncoder.encodeV2 yielded a stable 20% memory reduction with zero subsequent incidents. Looking ahead, our key priorities include: - Expand SRT beyond high-throughput encoding workloads to support a wider variety of functions and workload types. - Further tune the resources in regions with constrained resource availability. - Move from reactive alerts to proactive, automated upsizing for under-provisioned functions. - Experiment with intentional upsizing for latency-sensitive workloads to evaluate performance gains and reduce function execution time. - Upgrade our failure signals for OOM/OOD events from lagging indicators to leading indicators to improve tuning precision. If solving problems like this sounds exciting, we are hiring in our Warsaw office. Check out open roles here. ### Credits Special thanks to Olof Johansson, Jakub Karczewski, Adam Mazur, Ethan Mathieu, Rosanna Lee, Ameya Vasani, Jean Barkhuysen, Paul Hieromnimon, and Adithya Prakash.

By Emma Yanyang Kong, JJ Tan, Ishan Gupta, Lars Olds, Claire Campbell, David Fagnan, Ratna Kavuri, Rohan Gosain, Veli Balin, Minsu Jang, Louis Garcia ### Introduction Any team generating text with an LLM at scale hits the same problem. You cannot evaluate everything manually. You don't know which outputs are great, which are mediocre, and which are harmful, because you simply cannot read them all. The standard solution is LLM-as-a-Judge, a second model that scores the first model’s output as a proxy for human raters. It is fast, cheap, and easy to deploy. However, it introduces a new and critical question: Who evaluates the evaluator? Furthermore, if the judge drifts, quietly degrading and starting to approve bad output, nothing downstream will flag it. This post is about how we solve that problem, by building, aligning, and continuously monitoring LLM judges so they stay trustworthy over time. Our core principle is simple: An LLM judge must never operate alone. It has to be anchored in human judgment. We keep a human-in-the-loop (HITL) at every phase of the lifecycle. Humans define what “good” means, create the labeled benchmark datasets to train and validate the judge, monitor its behavior while it runs in the system, and review and approve any updates before they’re rolled out. We believe: A reliable LLM judge is not a static artifact. It has a lifecycle. It needs to be built against trusted human labels, aligned with human preference and reasoning, run under human-set policy, and maintained as the data and usage pattern shift. Each of these stages has human owners. In the sections below, we walk through these four phases, the key design decisions in each, and the lessons we’d share with a team building their own LLM judge, using one concrete task as our running example: recommendation explanations. A recommendation explanation is a short natural-language message that accompanies a recommended item, and it shapes how users perceive and trust a recommender system. In our work, an explanation is human-readable evidence for why a title was recommended based on what the member has previously watched, shown on the title’s detail page, which is the screen a member reaches by selecting a title from the homepage. We focus on similarity-based explanations, which connect a recommended title to one reference title the member has interacted with via shared attributes such as genre and tone. For example, A funny, heartfelt holiday romance about love and new beginnings, much like “My Secret Santa.” Quality matters here because a misleading or poorly grounded explanation erodes the very trust the explanation is meant to build. Everything that follows comes out of a series of controlled, user-facing online experiments we ran on the explanation feature, including a month-long experiment on the Netflix mobile app. In these tests we used an LLM to generate explanations, hundreds of thousands a week across a constantly changing catalog. Manually reviewing them at this scale isn’t feasible, so we deployed a second LLM to evaluate them. What counts as a good explanation is specific to our task. However, the way we build, align, and monitor the LLM judge for evaluating explanations should be reusable and generalizable across different tasks. ### The Lifecycle at a Glance Figure 1 summarizes the four phases of our LLM-judge lifecycle and the data flow connecting them. Figure 1. The four-phase lifecycle of an LLM judge, and the data that flows between the phases. We start with Birth: Benchmark Data Creation, where human experts craft hard or near-boundary examples and human raters label them with pass/fail decisions and written rationales. We augment this with LLM-generated gray-area examples to build a richer benchmark dataset. Next, in the second phase of Training: LLM Judge Development, we tune the LLM judge to align with human labels and rationales from the Birth phase, using self-reflection to identify mismatches between the judge and humans, and updating the rubrics based on those mismatches and anchor examples. We iterate in this phase until our metrics show that the judge is sufficiently aligned with human raters. In the third phase, Deployment, we integrate the judge into the explanation generation system as both a gate and a critic. The LLM generator writes explanations, and the LLM judge scores them and either passes them through or blocks them and supplies a rationale that the generator can use for self-refinement and retries. Passed explanations are served to members, and low-quality explanations are gated. Finally, in the last phase, HITL Alignment & Drift Monitoring, we keep the judge aligned with humans over time. “Human-in-the-loop” here means a person is directly involved in the decision, not just reviewing outcomes after the fact. We benchmark the judge against trained raters, and a human approves every change before it is rolled out to the system. During each test we sampled a set of explanations each week (both passed and gated, with a bias toward newer items), had human raters review them with labels and rationales, and compared the judge against these human ratings. If drift exceeded a threshold, we triggered retraining and benchmark data augmentation. This closes the loop by feeding new human-labeled data and updated signals back into the benchmark and development phases. #### System components Four automated components appear across all phases, alongside the human roles that define, validate, and supervise them. - Generator: produces the explanation text, optionally using prior judge rationales and traces as part of its prompt for revision. - Judge: evaluates an explanation and returns a pass/fail label together with a written rationale. In the deployed pipeline, it acts as both a guardrail (gating bad outputs) and a critic (providing feedback for self-refinement). - Reflector: revises the Judge’s rubric when the Judge makes mistakes. It updates the rubric based on mismatched examples, human rationales, and anchor examples. It tunes the rubric and does not directly evaluate explanations. - Meta-Judge: checks whether the Judge failed an explanation for the right reason by comparing the Judge’s rationale against a human rater’s rationale. - Human experts and raters: the experts who define the quality criteria; the raters whose labels and written rationales the Judge is tuned and measured against; the reviewers who monitor the Judge while it is running and explicitly approve any changes before they were rolled out. ### Phase I, Birth: Establishing Ground Truth Everything downstream depends on knowing what “good” means, so this is the most human-intensive phase. Our internal writing experts define the must-have criteria an explanation must satisfy, author detailed labeling guidelines, and hand-craft tricky adversarial examples. Those same guidelines do double duty. They’re both the rulebook for human raters and the seed rubric for the LLM judges we tune later. #### Why we built our own benchmark Off-the-shelf LLM-judge benchmarks focus on general-purpose text evaluation and miss what makes our task specific. Explanations are short, item-specific, and context-dependent (each one is tied to a target title and one or two reference titles the member has watched). We also care about alignment to our own editorial bar, not just generic language quality. As a result, we built our own custom benchmark from three complementary sources: - Expert-crafted adversarial cases: hand-authored pass/fail examples with known failure modes, the kind of edge cases nothing else reliably produces. - Boundary examples: LLM-synthesized explanations that are deliberately pushed near the pass/fail boundary, then rated by humans. These capture the ambiguous cases that sampling from live traffic rarely surfaces on its own. - Real sampled explanations: explanations sampled from the output of our LLM generator during test runs. #### Two details that mattered - The benchmark is not frozen. A monitoring pipeline (Phase IV) appends a fresh batch of human-rated explanations on a weekly cadence, so over the course of a test the benchmark tracked the current catalog and new usage patterns instead of going stale. - Every “fail” label carries a free-text rationale explaining why the explanation failed. This looks like a small bookkeeping detail, but it is actually critical for the next phase, since those rationales are what we use to tune and audit the judge’s reasoning. ### Phase II, Training: Aligning the Judge to Humans #### The failure mode: right answer, wrong reason There is a failure mode that does not show up in label-agreement metrics at all. A judge can agree with humans on the label, passing when they would pass and failing when they would fail, while reaching the right verdict for the wrong reason. On a dashboard it looks perfectly aligned. But as soon as you ask it to explain a rejection, which our pipeline does to steer self-revision, it can return misleading guidance. We therefore care about not just label agreement but also reasoning alignment. In our setup, the Judge is always asked to output: - a label (e.g., PASS/FAIL), and - a short natural-language rationale for that label. Here is an illustrative example. The member watched a musical, a family fantasy with a young heroine and a well-known ensemble cast. We recommend an adventure movie that is also a family fantasy with a young heroine, which is why the recommender paired them. But it shares neither the songs nor the ensemble cast. The label (FAIL) is correct, and the stated reason is reasonable at first glance. But it turns out to be incomplete in a way that simple label agreement cannot detect, as we show below. #### Reasoning-Aligned Rubric Tuning We tune each criterion’s LLM Judge with a procedure we call Reasoning-Aligned Rubric Tuning (RART). As introduced in the system components above, the Judge, Meta-Judge, and Reflector are distinct components. The Judge scores an explanation against a rubric and gives a rationale, the Meta-Judge compares that rationale against a human-written one, and the Reflector edits the rubric text itself. RART is how the three are combined into a training loop. Every Judge is a fixed prompt template parameterized by a rubric, so tuning it does not mean retraining the underlying model. Instead we iteratively refine the rubric text until the Judge’s behavior matches human labels and human rationales. At iteration i, the Judge is conditioned on the current rubric text and maps an explanation to a pair, a label and a rationale. R_i is the current rubric text, y_J(x) the label, and r_J(x) the rationale. At a high level, each RART iteration proceeds as follows. 1. Score the training data with the current rubric. 2. Evaluate weighted alignment metrics on a held-out validation set. If all per-criterion metrics exceed their thresholds, stop. 3. Otherwise, construct an error-driven focus set of misaligned examples and hand it to the Reflector, which proposes a revised rubric. The key difference from a standard loop is how we define a “mistake.” Most self-improvement loops look only at label disagreements. RART adds a second class of error. The Meta-Judge examines the cases where the Judge and a human both said fail and compares the Judge’s stated reason against the human’s rationale. When the two diverge, that is a right answer, wrong reason error, and it is also fed to the Reflector. Concretely, the focus set handed to the Reflector is the union of two error classes: - Label mismatches: examples where the Judge’s label differs from the human’s. - Right label, wrong reason: examples where the Judge and the human both said FAIL, but the Meta-Judge finds their two rationales don’t match. So the Reflector revises the rubric from both ordinary label errors and right label, wrong reason errors. That second term is the core of RART. The Judge is pushed not only to match human decisions but to justify them in a way the Meta-Judge recognizes as aligned with human reasoning. Returning to the earlier example, the LLM Judge said FAIL and a human said FAIL, so a label-only check would register agreement. Placing the two reasons side by side shows what that check misses. RART routes it into the focus set, so the next rubric revision has to account for it. Since the Meta-Judge is an LLM too, we checked it against humans first. Trained human raters labeled a sample of agreed-fail rationale pairs, and the Meta-Judge matched their verdicts on nearly all of them. #### Alignment metrics We run one Judge per criterion, each with its own rubric, tuned and scored on its own. So everything below is measured per criterion on held-out data. For each one we track three metrics. - Specificity (fail-recall): how often the Judge correctly rejects bad explanations. This is the one we care about most, because a bad explanation that slips through the gate is a trust hazard. - Recall (pass-recall): how often it correctly passes good explanations, which keeps coverage high and revision cost low. - Reasoning agreement: on the explanations both reject, how often they reject for the same reason. Each criterion’s Judge gets its own three, and one can be healthy while another slips. That is also why the gate is strict. An explanation has to pass every must-have criterion to be eligible to serve. #### Ablation: does the reasoning signal help? To isolate the impact of the reasoning signal, we compare RART to an otherwise identical loop that uses only label disagreements. We call this baseline vanilla: same Reflector, same Judge, same data, but no Meta-Judge and no use of human rationales. Figure 2. Improvement over the starting rubric, per criterion, for reasoning-aware tuning (RART) and label-only tuning (vanilla). Figure 2 shows the result. On two of the three must-have criteria, vanilla and RART land in the same place, because the starting rubric was already near ceiling and the metrics had very little room to gain. On the third criterion, the vanilla (label-only) method degrades. The Reflector in the vanilla loop rewrites the rubric into degenerate forms, and specificity and reasoning agreement fall with every iteration, ending up worse than the rubric it started from. RART stays stable on that same criterion and lifts both, with a small recall gain as well. The reason is that a vanilla (label-only) optimizer can reach the right label through unsound reasoning, and unsound reasoning doesn’t generalize to new data. Anchoring the signal on the rationale prevents that. Reasoning alignment helps where there is headroom and doesn’t hurt where there isn’t. ### Phase III, Deployment: Putting the Judge to Work Once deployed, the judge sat inside the generation pipeline, gating and critiquing explanations as they were produced during our tests. #### Two roles for one judge Once we align the LLM Judge with humans, we reuse it in two distinct roles for every generated explanation: - Role 1, the gate. The Judge scores each explanation against the must-have criteria and rejects anything that fails. - Role 2, the critic. When an explanation fails, we don’t just throw it away. The Judge’s free-text reason gets appended to the Generator’s prompt, and the Generator tries again. The second role turns the pipeline into a generate → judge → revise loop, with the Judge’s critique steering each revision. #### Bounded retries, and why we drop rather than serve We cap the retries. Each revision costs another round of generation and judging, and the gains flatten quickly. Figure 3 shows what the loop actually buys, on 1,000 generated explanations run through as many as 12 revisions with four generators. Every curve climbs, which is the Judge’s critique doing its job, but the steep part is early, and the gap between generators is obvious. Revision amplifies a capable generator rather than rescuing a weak one. In our test runs we used a budget of 3 retries with our strongest generator, and an explanation still failing after that was dropped rather than served. Figure 3. Cumulative share of explanations that clear the Judge as we allow more revisions, across four generator models. That “drop rather than serve” rule is deliberate. The two outcomes aren’t symmetric. A bad explanation is a trust hazard, while a missing explanation just forgoes an opportunity. When in doubt we’d rather show nothing, and most explanations cleared the bar within the retry budget. #### Do judge-approved explanations help members? Passing the Judge proves an explanation is good by our standards, but it doesn’t prove that judge-approved explanations are useful to members. To test that connection, we ran a one-time controlled experiment on the Netflix mobile app, explanations vs. no explanations on the title detail page, over roughly a month, with the pipeline generating and judging hundreds of thousands of explanations a week during the period. Members who saw explanations shifted their viewing toward titles they hadn’t watched before, and more of their browse sessions ended in a meaningful play. Both effects were statistically significant, and we saw no member-initiated takedowns or escalations about explanation quality during the test. The result worth carrying over isn’t the size of that effect. It’s that offline agreement with our raters was necessary for these explanations to work but never sufficient to show they help members. Only the online A/B test could establish that judge-aligned explanations are not just quality-compliant but genuinely useful. ### Phase IV, Monitoring: Keeping the Judge Aligned A judge that’s well aligned at launch won’t stay that way. The catalog shifts, and so does what counts as high quality. The lifecycle has to close the loop. #### The weekly human review For each week a test was running, we pulled roughly 300 generated explanations for human review, stratified across the Judge’s decisions (served, revised, dropped) and weighted toward newer catalog items, where drift is most likely. The stratification stayed fixed week to week, so successive weeks were comparable. A panel of at least three raters reviewed each explanation, and the majority label was our ground truth. Those labels did two jobs. They told us whether judge-human agreement was slipping, and they grew the benchmark. #### Setting the bar with human disagreement The panel also tells us how much the raters disagreed with each other, which turns out to be the right yardstick for the Judge. Rather than hold it to a fixed threshold, we score the Judge and each rater against the same majority label and ask that the Judge fall within the raters’ own spread. Take one criterion and one of its metrics, M. That week’s sample gives one value M(J) for the Judge and a set of per-rater values M(H) for the three or more raters on the panel. We require so the Judge must score no worse than two standard deviations below the average rater. A harder week widens human disagreement and widens the band with it, so the Judge isn’t penalized for difficulty the humans also felt. Falling below the band on any metric-criterion pair is a drift event, which triggers re-tuning (back to Phase II) on the augmented benchmark. A re-tuned rubric never goes live automatically. Someone on the team reads the proposed changes and signs off before it reaches the running pipeline, and we keep the previous rubric for instant rollback. #### What human review catches that metrics can’t Beyond asking whether the Judge still agrees with human raters, the weekly review answers a second question. Are the current guidelines still asking for the right things? In the weekly review process, reviewers surfaced issues that did not show up in Judge–rater agreement. An explanation can use accurate attributes and still overstate how similar two titles really are. A single vague adjective (“exciting”, “fun”, “emotional”) can technically satisfy the rubric while saying almost nothing. Sometimes the reference itself is the problem, something the member wouldn’t think of as a comparable title at all, so the comparison lands oddly no matter how the attributes line up. Some genres show the same problem at scale. Stand-up specials can share similar metadata tags but differ completely in tone and cultural context, so a comparison may look justified on paper, not be bad enough for a rater to fail, and still land poorly with members. And some explanations meet every must-have criterion while still reading like generic “word soup” that does not help the user much. When recurring patterns kept appearing, we updated the Phase I guidelines, which seed both the raters’ instructions and the judges’ rubrics, so one revision updates both sides. Other times we added product rules outside the Judge, for example by blocking certain title types from being used as references. Drift detection catches a judge that has stopped agreeing with people. Only the continued human review tells us when our definition of a “good explanation” is too narrow or incomplete. We need both. - Drift detection keeps the Judge aligned with the current rubric. - Continuous human review keeps the rubric aligned with what actually works for members. ### What We Learned Five lessons we think transfer to any team shipping an LLM judge: 1. Invest in the benchmark before the judge. A modest, rationale-annotated benchmark beats a large label-only one. Those written rationales are exactly what make reasoning-aligned tuning possible. You can’t correct a judge’s reasoning if you never wrote down the human’s. 2. Tune on reasoning, not just labels. A judge can reach the right verdict for the wrong reason, which looks fine on a dashboard but gives misleading feedback and doesn’t hold up on new data. In our ablation, the label-only loop degraded a rubric that the reasoning-aware loop kept stable. 3. One aligned judge can serve many roles. Reusing the same judge as both gate and critic amortizes the alignment cost and keeps production behavior consistent across roles. Alignment is the expensive part, so get one judge right and use it more than once. 4. Plan for monitoring from day one. A recommender platform’s items and members never stop changing, so an LLM judge, like any ML model, has to be watched and re-tuned if needed, not shipped and forgotten. 5. Keep people in the loop by design, not by exception. Automating evaluation is not the same as delegating responsibility for it. In our lifecycle, humans set the standard (Phase I), supply the rationales the judge is aligned to (Phase II), decide the conservative default of dropping rather than serving a doubtful output (Phase III), and review a standing sample of live decisions and approve every rubric change before it ships (Phase IV). These checkpoints are much harder to retrofit once a judge is already making decisions on its own. The standing review also tells us what our rubric is missing, not just when our judge has slipped. Taken together, the judge extends the reach of our reviewers, and those reviewers are the basis for trusting the judge. That relationship is what makes it responsible to run this system at scale. An automated evaluator that no one audits provides no assurance about the decisions it is making. ### Acknowledgements This post summarizes our research paper, “The Lifecycle of LLM-as-a-Judge for Large-Scale Recommendation Explanations,” which has the full methodology, metrics, and ablations. Our thanks to the internal human writers and review experts who defined what a good explanation is and keep those guidelines current; to our Human Data Operations team and the raters they train and calibrate, whose labels and written rationales are what the judges are aligned to; and to the engineers across the team who built and operate the pipeline; and to our UI engineering partners, who built the member-facing surfaces that bring these explanations to the homepage.

By Prashanth Gedde Narayanaswamy As one of Netflix’s largest and critical data pipelines, member-sessionizer consolidates tens of billions of hourly client events, encompassing user taps, playbacks, and engagement signals, into individual viewing sessions per profile. A quick two-week tuning effort spiraled into two months when standard approaches hit a wall. Because the failures only happened at full production scale, each test took hours, limiting me to just three or four attempts daily. The turning point was changing the goal: instead of trying to speed up a single run, I built a way to run and monitor multiple tests at once and track every result. That mindset led to an unconventional setup: kicking off five Spark jobs at once and monitoring them over spotty inflight Wi-Fi during a day-long trip home, while cloud servers handled the actual heavy lifting. This post covers how I set up that loop, along with the Spark bottlenecks and memory quirks I uncovered along the way. ### The Scale Dilemma: Small Samples Don’t Catch Scale Bugs Standard engineering wisdom says that you should isolate bugs on small datasets to iterate quickly. That works for standard software bugs, but it fails completely for big data performance bottlenecks. While upgrading our sessionization pipeline from a legacy logging architecture to a modern platform, the new job hit critical failures, including executor Out-Of-Memory (OOM) errors, severe shuffle spillage, and thousands of stage retries. None of these issues could be reproduced using sampled data. The failures only triggered at full production scale because of data skew: an idle Netflix tab left streaming overnight generates tens of thousands of telemetry events, while a typical session only produces a handful. A standard 1% sample misses these extreme long-tail sessions entirely. To catch the real bugs, every test run had to execute against a full hour of unsampled production data, along with all the deployment overhead that came with it. ### The Strategy: Branch-Driven Isolation The biggest win came from a feature in Dataflow, Netflix’s command-line tool for building, testing, and deploying data pipelines. It supports branch-driven development: our workflows run on Maestro *(*Netflix’s workflow orchestrator), which schedules and executes the jobs, and they already use a sanitized branch name (the ${dataflow.sanitizedbranch} placeholder). So if I push a branch like pgn/cap-experiment, then the deployed workflow, its tables, and its run namespace all get tagged with that branch name. This eliminates collisions with team members or between my own concurrent runs. With that isolation in place, the model was simple: one branch is one independent experiment. I parameterized the scheduler (cap size, shuffle partitions, heavy-device threshold, feature flags) and gave each variant its own branch and output table. Instead of queuing five sequential tests over 10 hours, I fired off five variants simultaneously against the same time window. Total execution time dropped from the sum of all runs to just the length of the single slowest job. Managing per-branch tables added zero overhead. Dataflow provisions and updates tables automatically from declared schemas without manual DDL. Whenever I modified the event bag structure between experiments, the table schema updated seamlessly, no CREATE TABLE or ALTER TABLE required. The only real constraint was cluster contention. Because these heavy Spark jobs shared resources with the rest of the company, peak hours sometimes limited me to two concurrent runs instead of five. But even running two or three jobs in parallel was a massive improvement over single-threaded debugging. The key takeaway: don’t focus on making a single multi-hour run faster, build a setup where you can run many experiments simultaneously without them stepping on each other. While the previous chart shows the time savings, the architecture that enables this isolation looks like below: ### How an AI Agent Helped Running five parallel experiments gets messy fast. Tracking instance IDs, configs, memory spills, and stage failures across variants makes it easy to lose track. This is where an AI agent stepped in, not to write clever code, but to handle operational grunt work. I described variants in plain English, and the agent updated scheduler configs, deployed to the branch, triggered jobs, and polled metrics. Once complete, it compared spill and failure counts across variants in a summary table, allowing me to ask, “Which run is closest to OOM-free at peak load?” and get an immediate answer. The value wasn’t deep Spark expertise, but operational memory. It turned tedious Spark UI cross-checks into a single prompt. Eventually, I packaged this workflow into a reusable agent skill (.agents/skills/sandbox-deploy-and-run/) run via /sandbox-deploy-and-run <module> <scheduler-file> [PARAM=value]. It handles the full lifecycle—preflight, deploy, background polling, auto-diagnostics, and re-entrant resume <instance_id> support—turning multi-hour experiments into a single command. ### Why tmux Mattered All of this ran on Workbench, Netflix’s cloud dev environment, using my laptop as a thin terminal. Without session persistence, an SSH drop or sleeping laptop would kill multi-hour Spark runs mid-flight. tmux running directly on Workbench solved this. Every experiment lived in a named cloud session: I could close my laptop, reattach hours later, and pick up where I left off. Combined with branch isolation, this made "start five and walk away" possible. This is now my default setup: a local terminal (Ghostty), dedicated git worktrees per branch, and persistent tmux sessions on Workbench. The local laptop is disposable—closing it or losing network never interrupts the jobs. That setup turned a long travel day into active iteration time. I kicked off batches at the gate, checked results via AI agent over spotty inflight Wi-Fi, and squeezed 4–5 full experiment cycles into a single 24-hour flight. ### What All Experiments Uncovered The root cause came down to a single operator: collect_list. Because Spark buffers each full event bag in memory, long-session OOMs were driven by individual bag size, not total dataset volume. The legacy pipeline handled identical row counts without crashing, proving that each event in the new bag had grown significantly heavier. Two factors inflated event weight: - Field types and Option boxing: The session ID changed from an 8-byte Long to a UUID string. Worse, wrapping fields in Scala Option forced Spark to deserialize typed Datasets into JVM objects (Some wrappers), driving up object allocation and GC pressure across tens of thousands of events per session. - Bloated string attributes: Low-value fields (e.g., referrer URLs, test allocations) added several hundred bytes per event. Multiplying these strings across collect_list wasted terabytes per hour, even though downstream tasks rarely used them. The Fixes: 1. Unboxed primitives: Removing Option wrappers from non-nullable fields dropped peak executor memory from 112% to 58% of the slot with zero disk spill. 2. Slimmed event bags: Dropped unused per-event attributes and hoisted session-invariant metadata (app version, country) up to the session header. Takeaway: Reducing individual event weight delivered far better stability than throwing more cluster memory at the problem. ### Two Experiments That Changed My Mind Out of dozens of runs, two findings completely overturned my assumptions about Spark memory management: - Finding 1: Narrowing rows can trigger faster OOMs. Slimming our table dropped its size below Spark’s auto-broadcast threshold, prompting the query planner to switch from shuffle joins to broadcast joins. Because broadcast joins pin build-side tables directly into executor memory, narrower rows actually increased total memory pressure. - Finding 2: Broadcast hash joins were silently draining memory. Adaptive Query Execution (AQE) rewrote nine sort-merge joins into broadcast hash joins at runtime. While each join stayed under our 400 MB broadcast limit individually, AQE pinned ~3.1 GB of cumulative data simultaneously in a single executor, pushing containers over the edge. The Surgical Fix: Instead of disabling broadcast joins globally, I lowered only AQE’s runtime threshold (spark.sql.adaptive.autoBroadcastJoinThreshold) to 10 MB while leaving the compile-time threshold untouched. That single change freed up ~3.1 GB of executor memory without impacting initial query planning. Takeaway: Broadcast joins speed up queries, but pinned memory isn’t free. Watch accumulated broadcasts in your execution plan, not just dataset sizes. ### Keeping a Single Experiment Log When running dozens of parallel tests, the biggest risk isn’t job failure, it’s losing track of what you’ve already tried or confusing which JAR and config produced a result. To solve this, I scaffolded the AI agent to append a row to a repository Markdown table after every run: - Run # | Jar ver | Change vs. Previous Run | Config | Key Metrics (Spill, Failed Tasks, Peak Memory, Duration) If a run yielded a key finding, the agent added a short note explaining what the result proved or disproved. Key Takeaways: - Single source of truth: It created an auditable history for the entire investigation. - Forced discipline: Requiring “Change vs. Previous Run” forced me to isolate variables. Whenever I bundled multiple changes, I couldn’t isolate the root cause and had to rerun the test — a mistake the log format immediately exposed. A secondary doc (events-join-spark-tuning.md) captured the distilled findings as a quick "symptom-to-knob" lookup, so teammates didn't have to wade through the full chronological history. ### A Mistake I Learned From: The Fix That Only Worked Off-Peak. To cap session size, I added row_number().over(Window...) and filtered for the first N events. It passed off-peak tests cleanly, but crashed with OOMs at peak load. The issue was structural: Spark’s Window operator buffers an entire session in memory before evaluating row numbers and filtering. The cap trimmed output downstream, but it didn't limit memory usage during buffering, so a single skewed session overwhelmed executor memory. The Fix & Lesson: We moved truncation ahead of the Window stage to cap events before buffering occurs. Takeaway: Test fixes against worst-case input skew. Off-peak runs hide structural bottlenecks that peak scale will immediately trigger. ### The final fix Pulling it together, four changes shipped, each isolated to a single experiment before it landed: - Unboxed the never-null fields from Option to plain primitives. Peak executor memory dropped from about 112% of the slot to about 58%, with zero spill. - Slimmed the event bag. Dropped per-event fields nobody used, and hoisted session-invariant fields (versions, country, connection type) to the session level so they are stored once per session, not once per event. - Capped session size before the Window stage. Dropped oversized sessions ahead of the buffering, and truncated the bag to the cap rather than dropping whole sessions. - Lowered AQE’s runtime broadcast threshold. Set spark.sql.adaptive.autoBroadcastJoinThreshold to 10 MB, leaving the compile-time threshold untouched. ### The Unresolved Bottleneck While these fixes made the pipeline run cleanly, they didn’t solve the underlying scaling issue: aggregating an entire session’s event bag into a single row has a hard ceiling. As our upstream telemetry source evolves and adds new attributes, each event in the bag becomes heavier, meaning this memory pressure will eventually return without a single line of code changing on our end. Slimming and unboxing bought us headroom, but they didn’t change the fundamental data shape. The biggest takeaway from this effort is that tuning buys time, not infinite scale. If your underlying data model allows unbounded row growth, tuning alone won’t save you , eventually, the architecture itself has to change. ### Summary - If the bug only appears at scale, stop trying to sample. Build a way to run many full-scale runs in parallel instead. - If your scheduler can use the branch name, then one branch becomes one isolated experiment, and running many at once becomes easy. - Use an AI agent to run the operation, not only to write code. Deploy, poll, pull metrics, compare variants: this is where most of the time goes. - Once a loop repeats, package it as a skill. The deploy-run-watch-diagnose cycle becomes one command. - Use tmux for long runs. The run must survive you closing the laptop. - Keep one single experiment log, and make every entry state the one change from the previous run. The format itself forces you to change one variable at a time. - For memory problems, look at the per-row width first, not only at the total data volume. Bigger field types, Option boxing, and fat unused strings all add up inside a collect_list bag. - Tuning buys headroom, not a new ceiling. If the data shape can grow without a bound, the design has to change eventually, tuning alone will not save you.