Netflix Bridges AWS IAM and Internal Identity with Workload Attestation

Netflix has developed a workload attestation mechanism that allows Apache Spark jobs running on Amazon EMR to exchange their AWS execution identity for a short-lived internal workload identity. This approach connects AWS IAM with Netflix’s internal Metatron PKI, enabling managed compute workloads to authenticate to internal services without manually injecting credentials.

Dhruv Pratap, a Big Data Platform Engineer at Netflix, described the design in a LinkedIn post, stating that on infrastructure built by the company, users can bootstrap their own identity as needed.

Netflix operates two identity systems: AWS provides managed workloads with an IAM execution role, while internal services utilize Metatron, a private PKI that issues short-lived X.509 certificates for mutual TLS. Data Projects serve as the ownership and identity boundary for data access, requiring Spark jobs on EMR to establish that their AWS identity corresponds to the internal identity linked to their Data Project.

The design maps each Data Project to a dedicated IAM role. Given that Netflix anticipates Data Projects to number in the tens of thousands, these roles are distributed across a pool of dedicated AWS accounts instead of being concentrated in a single account.

When a job is submitted, Netflix’s control plane validates the request, resolves the Data Project’s IAM role, and creates a signed workload metadata payload that identifies the intended workload. The control plane is the only service authorized to create these claims.

The Spark driver uses credentials associated with its execution role to create a pre-signed request to the AWS Security Token Service GetCallerIdentity endpoint. Netflix’s Identity service retrieves this request from AWS STS to determine which IAM role signed it, providing an independently verifiable statement of the workload’s AWS identity.

The Identity service then compares the AWS response with the signed metadata from the control plane. The AWS response confirms possession of the expected IAM role, while the signed metadata identifies the intended workload. Their intersection allows the service to issue a Metatron certificate.

This architecture also addresses Spark’s executor fan-out. Instead of independently attesting thousands of executors, Netflix has the driver attest once and distribute credentials through authenticated and encrypted Spark RPC. This method reduces attestation traffic proportional to executor count while making the driver an additional credential distribution trust boundary.

Source: InfoQ