Komodor is an autonomous AI SRE platform for Kubernetes. Powered by Klaudia, it’s an agentic AI solution for visualizing, troubleshooting and optimizing cloud-native infrastructure, allowing enterprises to operate Kubernetes at scale.
Proactively detect & remediate issues in your clusters & workloads.
Easily operate & manage K8s clusters at scale.
Reduce costs without compromising on performance.
Guides, blogs, webinars & tools to help you troubleshoot and scale Kubernetes.
Tips, trends, and lessons from the field.
Practical guides for real-world K8s ops.
How it works, how to run it, and how not to break it.
Short, clear articles on Kubernetes concepts, best practices, and troubleshooting.
Infra stories from teams like yours, brief, honest, and right to the point.
Product-focused clips showing Komodor in action, from drift detection to add‑on support.
Live demos, real use cases, and expert Q&A, all up-to-date.
The missing UI for Helm – a simplified way of working with Helm.
Visualize Crossplane resources and speed up troubleshooting.
Validate, clean & secure your K8s YAMLs.
Navigate the community-driven K8s ecosystem map.
Who we are, and our promise for the future of K8s.
Have a question for us? Write us.
Come aboard the K8s ship – we’re hiring!
Discover our events, webinars and other ways to connect.
Here’s what they’re saying about Komodor in the news.
Join the Komodor partner program and accelerate growth.
Most teams running Azure Kubernetes Service at scale don’t have a metrics problem. They have a correlation problem.
Container Insights is collecting data, Prometheus is scraping everything that holds still long enough, and Grafana dashboards are multiplying faster than namespaces.
What’s missing is the ability to connect cluster health, workload behavior, and actual spend across environments fast enough to do something useful before the next incident or the next budget review.
Incidents take three times longer to resolve because no one can confirm which cluster is the source, FinOps reviews where the platform team can’t break down the Azure bill by team or workload, and a steady stream of cost questions is routed to SREs who have access but not answers.
This guide covers AKS monitoring best practices for teams operating multiple clusters in production, with a focus on what to measure, what to cut, and how to turn cluster data into decisions.
When you’re running three or four AKS clusters, the monitoring setup that worked for your first cluster still feels manageable. Each cluster has its own Container Insights workspace, its own Prometheus instance, maybe its own Grafana deployment.
The pain arrives gradually, then suddenly, when you’re debugging a latency spike , and you’re toggling between four dashboards, two Log Analytics workspaces, and a Slack thread that’s aged three hours.
The issue is that nothing is stitched together into a view that crosses cluster boundaries, and no single person has the full picture when something goes wrong.
Microsoft’s own guidance on monitoring Azure Kubernetes Service treats it as a multi-layered concern: platform metrics from the control plane, workload metrics from containers and nodes, and logs from both the application and infrastructure layers.
For a single cluster, this is already a lot to manage. For five clusters spread across regions or business units, the same architecture produces five siloed data sources, five sets of alert rules, and five places to check when you’re trying to answer a question that spans all of them.
The other thing that breaks down at scale is cost attribution. AKS cluster cost analysis is deceptively hard because Azure bills at the infrastructure layer, meaning VMs, disks, load balancers, egress, while your actual spend drivers live at the workload layer: which namespace, which team, which deployment is consuming what.
Bridging that gap requires tagging discipline, chargeback tooling, and a consistent labeling strategy across all clusters.
Most teams have some version of this, and most versions are partially broken: tags that got applied when a cluster was created and never updated, namespaces that map loosely to teams but not to cost centers, and node pools that serve multiple workloads with no way to split the bill.
The result is a monthly Azure invoice that no one can explain in detail, and a Kubernetes cost optimization conversation that keeps getting deferred because the data isn’t clean enough to act on.
When you monitor Azure Kubernetes Service across multiple clusters, the first layer to get right is platform-level metrics from Azure Monitor.
These cover control plane health like API server latency, etcd availability, scheduler queue depth – the metrics that tell you whether Kubernetes itself is struggling, before you start blaming your application.
Azure exposes these through the Diagnostics settings on each cluster, and they feed into Log Analytics by default.
The mistake most teams make here is that they enable the diagnostic settings during cluster setup and never tune the alert thresholds, which means they either generate alert noise immediately or go completely silent until something is genuinely on fire.
For multi-cluster environments, the right approach is to configure a single centralized Log Analytics workspace or a small number of workspaces aligned to the environment, be it production or staging, rather than the cluster, and route platform metrics from all clusters into it.
This makes cross-cluster queries possible and gives your on-call engineer one place to start an investigation rather than four.
Azure Resource Graph and Azure Monitor Workbooks both support multi-cluster queries against a shared workspace, which is where you can start building the unified view that individual cluster dashboards can’t give you.
Container Insights is the managed monitoring layer for AKS and covers node-level resource usage, pod performance, and container logs without requiring you to deploy Prometheus yourself.
It’s genuinely useful, particularly for teams that aren’t ready to operate a full self-managed metrics stack, and the integration with Log Analytics means you can write Kusto queries against performance data without touching a YAML file.
The catch is that Container Insights is expensive at scale because log ingestion costs accumulate fast when you’re collecting at the default verbosity across multiple large clusters.
The fix is almost always to tune collection frequency and log filtering before you’re surprised by your Azure bill.
Container Insights supports configurable collection intervals (the default is 60 seconds, and for many workloads, 120 or 300 seconds is perfectly adequate), and you can suppress collection for specific namespaces, typically system namespaces like kube-system that you don’t need to monitor at the same granularity as production workloads.
A data collection rules (DCR) configuration that drops high-cardinality but low-value log streams can significantly cut Container Insights costs without losing the signals that actually matter for incident response.
For workload-level observability like request rates, error rates, latency distributions, and custom application metrics, most mature AKS environments run Prometheus alongside Container Insights rather than instead of it.
Azure Managed Prometheus, which is part of the Azure Monitor managed service for Prometheus preview, removes the operational burden of running your own Prometheus and supports remote write from multiple clusters into a single Azure Monitor workspace.
For teams that want control over their recording rules and alert rules without managing Prometheus infrastructure, this is worth evaluating seriously.
For teams already running self-managed Prometheus, the multi-cluster challenge is federation. Prometheus federation lets you run per-cluster instances that scrape local metrics and forward a subset to a central Prometheus or Thanos instance.
Thanos in particular solves the multi-cluster problem well: it provides global query capability across all your Prometheus instances and handles long-term storage more gracefully than Prometheus alone.
Grafana then connects to the central query layer and gives you dashboards that can answer questions like “which cluster is running the most over-provisioned pods?” without requiring you to log into each cluster separately.
The question teams get stuck on is which combination of these tools to run, and the answer depends more on your operational maturity and cluster count than on which tool has the better feature list.
If you’re running fewer than five clusters and don’t already have a Prometheus operator deployed, start with Container Insights and Azure Managed Prometheus.
You get meaningful coverage with minimal operational overhead, and you avoid inheriting the maintenance burden of a self-managed metrics stack before you have the team bandwidth to run it well.
When you hit the point where your recording rules are too custom for the managed service, your retention requirements exceed what Azure Monitor workspaces handle cost-effectively, or you need cross-cloud federation because you’re running clusters outside Azure, that’s when self-managed Prometheus with Thanos becomes the right call.
For log aggregation and control plane metrics, a centralized Log Analytics workspace is the right answer at almost any scale.
The query capability and Azure integration justify the cost, and fighting it in favor of a self-managed Loki stack is rarely worth the effort unless your ingestion volume is high enough that the Log Analytics pricing becomes a real problem.
The biggest source of avoidable spend in most AKS environments is node pool configuration, specifically VM SKU selection and cluster autoscaler behavior, which haven’t been revisited since the cluster was first deployed.
Teams tend to pick a VM size during initial setup based on what seems reasonable, and then let the autoscaler handle demand by adding more of the same node type.
What they often end up with is a fleet of general-purpose VMs that are 60-70% utilized on CPU and 40% utilized on memory, because the workload mix has evolved, but the node pool design hasn’t.
AKS cluster cost optimization at the node pool level starts with understanding actual resource consumption versus requested resources.
A workload requesting 2 vCPU and 4GB memory that consistently uses 0.4 vCPU and 1.5GB memory is wasting allocated resources and making the whole node less efficient, because the scheduler treats requests as guaranteed reservations.
Across a fleet, this kind of systematic over-provisioning compounds into meaningful overspend, and Kubernetes rightsizing at scale is one of the highest-return levers available before you start renegotiating reserved instance commitments.
The Azure Monitor metrics cpuUsagePercentage and memoryWorkingSetPercentage at the node level, correlated with kube_pod_container_resource_requests from Prometheus, give you the raw data to quantify this. The analysis is manual unless you’ve built or adopted tooling to automate the comparison at a workload scale.
AKS cluster cost analysis that stays at the node level tells you what you’re spending, but not who’s responsible. The attribution layer requires mapping Azure infrastructure costs back to Kubernetes namespaces and teams, which is where most cost programs stall.
Azure Cost Management supports tag-based filtering, but Kubernetes workloads don’t automatically inherit Azure tags.
You need a consistent labeling strategy applied at the namespace and deployment level, and then a tool that bridges the gap between Kubernetes resource consumption and Azure billing data.
The practical approach is to use Kubernetes labels to map namespaces to cost centers or teams, enforce label requirements through admission controllers (OPA Gatekeeper works for this), and then use a cost allocation tool like OpenCost or Kubecost to calculate per-namespace and per-workload cost based on actual resource consumption and node pricing.
This gives you a chargeback model that’s grounded in what workloads actually consume rather than an even split of the cluster bill, which is both more accurate and more likely to drive behavior change from application teams.
One of the most effective AKS cluster optimization levers available on Azure is the combination of spot node pools for fault-tolerant workloads and reserved VM instances for baseline capacity.
Spot VMs on Azure can run 60-90% cheaper than on-demand pricing, with the caveat that Azure can evict them with 30 seconds’ notice when capacity is needed elsewhere.
For batch jobs, CI runners, stateless workers, and other workloads that can tolerate interruption, this is usually an acceptable trade. For stateful workloads, latency-sensitive services, or anything with a strict availability SLA, spot is generally not appropriate as a primary node pool.
The pattern that works at scale is a layered node pool strategy: a baseline on-demand or reserved node pool sized for your guaranteed minimum workload, a spot node pool for burst capacity and cost-tolerant workloads, and clear pod tolerations and node selectors to control which workloads land where.
Azure Reserved VM Instances offer 1-year or 3-year commitments with discounts of roughly 30-50% compared to pay-as-you-go, which makes sense for node pools running at consistent utilization.
The math on reservations is only favorable if you’re reasonably confident the capacity will be used. Buying reserved instances for workloads that are still in active migration or whose demand patterns aren’t stable yet just ends up locking in a different kind of overspend.
The operational requirement for multi-cluster Kubernetes monitoring is a query layer that doesn’t require you to know which cluster a problem is on before you can investigate it.
When an alert fires, the first question shouldn’t be “which cluster?”. You should be able to start from a symptom and narrow down to a specific workload regardless of where it’s running.
This is achievable with a centralized Azure Monitor workspace fed by Container Insights from all clusters, combined with a Thanos or Azure Managed Prometheus layer for workload metrics.
The Grafana dashboards on top of this stack should present a fleet-level view first, showing cluster health, node utilization, and pod scheduling errors across all environments, and then support drill-down to individual clusters and workloads without switching tools.
Cross-cluster monitoring at the fleet level should track a small set of high-signal indicators: API server request latency and error rate, node ready status and pressure conditions, pod scheduling failure rate, persistent volume attachment errors, and cluster autoscaler decision logs. These are the signals that indicate infrastructure-level problems.
Application-level signals like RED metrics for your services, JVM heap, and request queues, belong on separate dashboards that are scoped to individual clusters and namespaces, because they’re team-specific and the fleet operations team doesn’t need to watch them all simultaneously.
Alert fatigue in multi-cluster environments is why your on-call engineer has started to dismiss PagerDuty notifications on reflex because the ratio of actionable alerts to noise is bad enough that triage has become the job.
The fix is a systematic audit of what you’re alerting on and why. This is one of the areas where AI SRE tooling has a concrete impact. Automated alert correlation and noise suppression reduce the manual triage burden without requiring you to rewrite every alert rule by hand.
For each alert in your current ruleset, the question is: “What action does this alert trigger, and how often does it trigger the right action?” Alerts that generate tickets that get closed as “resolved on its own” after 20 minutes are not providing value. They’re providing toil.
For AKS specifically, common noise sources include node memory pressure alerts that fire during GC spikes, pod restart alerts without a CrashLoopBackOff qualifier, and pending pod alerts that fire before the autoscaler has had time to provision capacity.
Tuning these alerts to require sustained conditions over a meaningful window of 5-10 minutes rather than one evaluation cycle eliminates most of the false-positive volume without meaningfully increasing time-to-detection for real issues.
The non-technical part of multi-cluster AKS monitoring that teams consistently underinvest in is ownership clarity.
When an alert fires in a multi-team environment, the worst outcome is not that no one responds. It’s that everyone looks at it for 10 minutes and then assumes someone else is handling it.
Every cluster, every critical namespace, and every high-severity alert path needs a named owner or owning team, and that ownership needs to be reflected in the alert routing, not just in a wiki page that hasn’t been updated since the platform team reorganized.
Runbooks don’t need to be exhaustive to be useful. A runbook that says “this alert fires when the API server p99 latency exceeds 200ms for more than 5 minutes; check etcd leader election status and API server CPU first; escalate to platform-oncall if etcd is healthy” takes 20 minutes to write and saves 45 minutes on every incident where someone unfamiliar with the component gets paged.
For multi-cluster environments, runbooks should include cluster-specific context like regional infrastructure differences, unusual workload profiles, and known quirks, and should be linked directly from alert annotations so the on-call engineer finds them automatically rather than searching during an active incident.
Managing AKS monitoring across multiple clusters brings a coordination problem, and the teams that solve it treat their platform as a product with real ownership, clean cost attribution, and monitoring that supports decisions rather than just recording what happened.
The technical foundation is achievable with the Azure-native stack combined with open-source tooling; the harder work is the process and ownership model that makes the data actionable.
Komodor brings this together for enterprise AKS environments as an AI SRE platform, providing a centralized operational layer across clusters that connects workload health, cluster state, cost signals, and your organizational context, without requiring you to build and maintain the analytical layers yourself.
From autonomous self-healing AI agents to AKS cluster cost analysis and rightsizing recommendations grounded in real workload data with reliability in mind, Komodor reduces the toil involved in running Kubernetes at scale and gives platform teams and application engineers a shared, accurate, and actionable view of what’s happening across their environments.
If your team is spending more time maintaining observability infrastructure than acting on it, contact the Komodor team to see how the platform fits your multi-cluster AKS setup.
The most effective approach to monitor metrics in AKS across multiple clusters is to centralize your telemetry into a shared Azure Monitor workspace and query layer.
Container Insights from each cluster should route into a single Log Analytics workspace or one per environment boundary, giving you cross-cluster query capability without switching contexts.
Start with what Azure already gives you: Azure Cost Management with resource group tags, plus node-level CPU and memory metrics from Container Insights. That combination tells you what infrastructure you’re paying for and roughly how utilized it is.
For workload-level attribution, OpenCost is an open-source CNCF project that calculates per-namespace and per-deployment cost based on actual resource consumption and on-demand node pricing.
It deploys in a few minutes and gives you a cost breakdown by team within hours. The tagging and labeling discipline takes longer to get right, but starting with OpenCost on current labels is better than waiting until your labeling strategy is perfect.
At a minimum, monthly for cost-related settings like node pool sizing, reservation coverage, and spot/on-demand split, and quarterly for architectural choices like VM SKU selection, autoscaler configuration, and node pool structure.
In practice, Kubernetes cost optimization should be triggered by events like a new team onboarding, a workload migration, a significant traffic shift, or a budget review.
The teams that treat optimization as a one-time project tend to find that 18 months later, they’re paying for infrastructure that no longer matches their workload profile.
Container Insights is Azure’s managed, agent-based monitoring layer. It covers node and container resource usage, pod lifecycle events, and logs with minimal configuration and integrates natively with Log Analytics.
Prometheus is a pull-based metrics system that scrapes workload-level and custom application metrics, and it comes with a large ecosystem of exporters, alerting rules, and community tooling that Container Insights doesn’t replicate.
Most mature AKS environments run both, because Container Insights tells you what the infrastructure is doing, and Prometheus tells you what your applications are doing.
The highest-leverage changes are adding duration conditions to alerts that currently trigger on a single evaluation, raising thresholds on resource alerts to match real incident history rather than theoretical limits, and auditing which alerts have generated tickets that were closed as no action needed in the last 90 days.
In multi-cluster environments, also audit for duplicate alerts. If you’re running the same alert rules per-cluster, you’re potentially paging multiple times for the same underlying infrastructure condition.
A deduplicated, routed alerting setup with clear severity levels reduces on-call load more than any dashboard improvement.
Share:
Gain instant visibility into your clusters and resolve issues faster.
May 12 · 9:00EST / 15:00 CET · Live & Online
🎯 8+ Sessions 🎙️ 10+ Speakers ⚡ 100% Free
By registering you agree to our Privacy Policy. No spam. Unsubscribe anytime.
Check your inbox for a confirmation. We'll send session links closer to May 12.
