# Bringing Prometheus Metrics and Grafana Dashboard for Cost Allocation on Kubernetes Clusters This story introduces a Prometheus Exporter along with a Grafana Dashboard intending to provide cost-oriented consolidated resource usage analytics for Kubernetes clusters. Those analytics actually aim at highlighting factual metrics to help organizations easily make cost allocation and capacity planning decisions on short-, mid-, and long terms. > ***Sidenote:*** *Readers may also be interested in this* [*related story*](https://blog.kubeledger.io/introducing-an-analytics-tool-for-kubernetes-cost-allocation-and-capacity-planning) ## **Introduction** A couple of months ago [Kubernetes Opex Analytics](https://github.com/rchakode/kube-opex-analytics) has been introduced as an original open source resource usage analytics tool to make cost sharing and capacity planning decisions easy on Kubernetes clusters. Released under the terms of Apache 2.0 License the tool comes with built-in analytics charts covering various use cases. It received a lot of user feedback, some requesting the ability to expose its metrics to existing Prometheus environments. A Prometheus exporter has been introduced in version 0.3.0, so in this story I’ll be pleased to introduce and describe it. This comes along with an integrated Grafana dashboard to let users easily getting started. All that comes in complement of the native Kubernetes Opex Analytics features and dashboard. ### **What will be covered next** The story will give details on the exposed metrics and provide steps to setup the Exporter and the provided Grafana dashboard. Before that we’ll review the core features of Kubernetes Opex Analytics for readers not familiar with it. > *If you’re already familiar with the concepts of Kubernetes Opex Analytics, you can skip the next section and move forward.* ## **Kubernetes Opex Analytics in a Nutshell** Kubernetes Opex Analytics is designed on the following core concepts: * **Namespace-focused**: Means that consolidated resource usage metrics consider individual namespaces as fundamental units for resource sharing. A special care is taken to also account and highlight `non-allocatable resources` . * **Hourly Usage & Trends**: Like on public clouds, resource use for each namespace is consolidated on a hourly-basic. This actually corresponds to the ratio (`%`) of resource used per namespace during each hour. It’s the foundation for cost calculation and also allows to get over time trends about resources being consuming per namespace and also at the Kubernetes cluster scale. * **Daily and Monthly Usage Costs:** Provide for each period (daily/monthly), namespace, and resource type (CPU/memory), *consolidated cost* computed given one of the following ways: *(i)* accumulated hourly usage over the period; *(ii)* actual costs computed based on resource usage and a given hourly billing rate; *(iii)* normalized ratio of usage per namespace compared against the global cluster usage. * **Efficient Visualization:** For metrics it generates, Kubernetes Opex Analytics provides dashboards with relevant charts covering as well the last couple of hours than the last 12 months (i.e. year) as shown below. ![](https://miro.medium.com/v2/resize:fit:1000/1*cGfFeGTUezdDmo7_zGBEtg.png align="left") Kubernetes Opex Analytics — Screenshot of the Built-in Dashboard ## **Installing Kubernetes Opex Analytics** Next on we assume that the installation will be done on a Kubernetes cluster, as you can also install it on Docker as described [here](https://github.com/rchakode/kube-opex-analytics#start-koa-on-docker). There is a [Helm chart](https://github.com/rchakode/kube-opex-analytics/tree/master/helm/kube-opex-analytics) to ease the deployment on Kubernetes, either by using Helm `Tiller` or `kubectl`. In both cases check the `values.yaml` file to modify the configuration options according to your needs (e.g. to have persistent volume for data storage). Using `Helm Tiller`: ```plaintext helm upgrade \ --install kube-opex-analytics \ helm/kube-opex-analytics/ ``` Using `kubectl`: ```plaintext helm template \ --name kube-opex-analytics \ helm/kube-opex-analytics/ | kubectl apply -f - ``` > *This will enable the built-in dashboard via a HTTP service named* `kube-opex-analytics` on port `80`. ## **Prometheus Exporter** Metrics are exposed for Prometheus via the `/metrics` endpoint. ### **Exposed Metrics** As shown on the sample on Figure below the exposed metrics are: * `koa_namespace_hourly_usage` : exposes for each namespace its current hourly resource usage for both CPU and memory. * `koa_namespace_daily_usage` : exposes for each namespace and for the ongoing day, its current resource usage for both CPU and memory. * `koa_namespace_monthly_usage` : exposes for each namespace and for the ongoing month, its current resource usage for both CPU and memory. Press enter or click to view image in full size ![](https://miro.medium.com/v2/resize:fit:700/1*obCxbKRH3j_i4Wf4zvgD8Q.png align="left") Kubernetes Opex Analytics — Sample of Metrics Exposed for Prometheus ### **Prometheus Scrape Job** The job can be configured like below. An interval less than 5 minutes (i.e. `300s`) is useless as no new metrics will be generated within the meantime. ```plaintext scrape_configs: - job_name: 'kube-opex-analytics' scrape_interval: 300s static_configs: - targets: ['kube-opex-analytics:80'] ``` > *Recall that Kubernetes Opex Analytics works with hourly-consolidated metrics, so you may need to wait at least an hour to have all metrics available.* ## **Grafana Dashboard** Once metrics available in Prometheus, get [this Grafana Dashboard](https://grafana.com/dashboards/10282) and import it into Grafana. The dashboard relies on a variable `KOA_DS_PROMETHEUS` that shall point to your Prometheus data source. Once configured properly the dashboard shall just work out of the box to display charts as described hereafter. ### **Hourly Usage** There are two panels displaying respectively usage charts for CPU (left) and memory (right) over the selected interval (7 days by default). Series for the different namespaces are stacked. This makes usage comparison easy and also helps show how loaded the cluster is. On the below example we can see that during the last 5 days, global CPU and memory usage reached more than 90%. ![](https://miro.medium.com/v2/resize:fit:1000/1*5bKOYCGVnlizBFeFEZMWRg.png align="left") Kubernetes Opex Analytics — Hourly Resource Usage ### **Current Day’s Usage** The two panels display respectively cost charts for CPU (left) and memory (right) over the ongoing day. Values are computed using cost algorithms described early in this story. ![](https://miro.medium.com/v2/resize:fit:1000/1*PSEQP-oQ5MoqGt97RDIQag.png align="left") Kubernetes Opex Analytics — Current’s Day Resource Usage ### **Current Month’s Usage** The two panels below display respectively cost charts for CPU (left) and memory (right) over the ongoing month. Values computed using cost algorithms described early in this story. Press enter or click to view image in full size ![](https://miro.medium.com/v2/resize:fit:1000/1*mU0qKLMq8iIQ1gnl_6jPgg.png align="left") Kubernetes Opex Analytics — Current’s Month Resource Usage ## **Move forward** In short we’ve introduced in this story a Prometheus exporter as well as a Grafana dashboard for [Kubernetes Opex Analytics](https://github.com/rchakode/kube-opex-analytics). As you may have noticed charts of the described Grafana dashboard are less rich than the ones of the built-in dashboard of Kubernetes Opex Analytics. For instance daily and monthly usage are limited to the current day, respectively current month. This makes difficult to compare current usage with previous ones. [These are limitations inherent in how Grafana handles bar charts based on series names](https://community.grafana.com/t/display-stacked-series/3402/15). The current implementation hence leaves room for further improvements, any contributions will be really appreciated. I want to recall that Kubernetes Opex Analytics is open source, it’s an open-to-contribution project. We’re always pleased to receive feedback and contributions on [Github](https://github.com/rchakode/kube-opex-analytics): submit an issue if you encounter problems or have some ideas of improvement; make a pull request; or give a star. Enjoy!