OPA memory usage considerations and lessons from our transition to Kyverno

Running multi-tenant Kubernetes clusters requires robust governance and policy enforcement to ensure security, compliance, and consistent resource usage for all tenants. 

Open Policy Agent (OPA) was the backbone of SCHIP, our Kubernetes platform, policy enforcement for a long time—even as we began migrating to Kyverno.

As we deployed many clusters and tenants, it became nearly impossible to write policies without factoring in the state of other objects in the cluster. OPA offers a powerful feature to sync external data allowing us to leverage additional context from various Kubernetes resources. However, turning on this feature also introduces additional resource overhead, particularly in terms of memory consumption. 

In this article, I’d like to share key considerations when enabling OPA’s data sync capabilities, how it can impact memory usage and why you need to balance these benefits with the resource costs. While the goal here is not to discourage you from using advanced OPA features, it’s crucial to be aware of their implications.

Finally, I’ll also share lessons learned from our transition to Kyverno, including how we prioritised the migration of rules based on their resource impact. This article should help you make more informed decisions about your policy management and potential migration paths.

A simple OPA policy

A simple OPA policy can work with the context provided by the object being validated itself. The example policy below checks the host field under the ingress object being validated