Deploying containerized apps to production in Kubernetes with rigorous security practices requires minimizing what is installed on a container image and what can be done in a running container. But doing so can make it challenging to diagnose elusive issues since the containers won’t have tools installed that could help diagnose the problem. This article demonstrates a technique for enabling diagnostics for a containerized .NET application while maintaining a high-security posture during normal operations.

Deploying Securely to Kubernetes

Standard practices for securing containers deployed to Kubernetes include:

  • Building containers with a minimal footprint reduces the attack surface, adding only what is needed to run the application.
  • Running the container as a non-root user.
  • Running with a read-only root filesystem so that the operating system and critical files cannot be tampered with.

However, doing so limits options for debugging and diagnostics when something goes wrong.

.NET Diagnostic Tools

.NET includes a suite of tools for diagnostics. But adding them to every container you run in production increases the attack surface.
Another catch: .NET applications run with diagnostics enabled by default. But if you have tried running a .NET container with the Docker –read-only flag, your app will crash with this error:

Failed to create CoreCLR, HRESULT: 0x80004005c

This prevents running the containers in Kubernetes with a read-only root filesystem.

However, it turns out that we only need to create a writable /tmp directory for .NET diagnostics to work. In Kubernetes deployments, we can do this with a writable emptydir mount on /tmp. (There needs to be more documentation on this point).

Solution: On-Demand Diagnostics Sidecar

We want to run our application containers in production with a minimal attack surface and the smallest footprint possible while still being able to debug when necessary. Our approach will be to use a sidecar container and enable .NET diagnostics on-demand when needed for diagnostics.

  • Helm is a common way to manage Kubernetes deployments, and with Helm, we can templatize our Kubernetes YAML to pass a value at deployment time to switch diagnostics on or off. When diagnostics are enabled, the Helm chart will:
  • Enable .NET diagnostics by setting the DOTNET_EnableDiagnostics environment variable to 1.
  • Add a sidecar container to our application pods with the diagnostics tools installed.
  • Sets the shareProcessNamespace property on the pod to true so that the sidecar container can initiate dumps on the application container.

With this approach, we can enable diagnostics on-demand when creating a .NET process dump for in-depth debugging while keeping our normal runtime footprint minimal.

Full Sample

See the details and everything you need to try it out in the open-source k8s-dotnet-diag GitHub repo.