Monitor Resource Quotas

Currently, resource quotas can only be monitored through Prometheus metrics. Prometheus metrics must be manually retrieved using queries that are described in this topic. You must run Prometheus queries (in PromQL) on the Prometheus service of the KubeSlice Controller. You can easily track the usage and violation using the KubeSlice Manager dashboard. However, if you want to track the quota usage or violation through Prometheus queries, then read the following sections.

Access the KubeSlice Controller-installed Prometheus

KubeSlice Controller is shipped with Prometheus installed, which monitors the resource quota and generates metrics. You can access this Prometheus service by port forwarding and accessing it as a local host.

To access the Prometheus service:

1. Do port forwarding to access the Prometheus service of the KubeSlice Controller using the following command:

   kubectl port-forward svc/kubeslice-controller-prometheus-service 9090:9090 -n kubeslice-controller

2. Access the Prometheus user interface from the local host, http://localhost:9090.

Run PromQL Queries on the Prometheus User Interface

To access the Prometheus endpoint:

1. Go to http://localhost:9090 on a web browser.

   

2. On the Prometheus page, in the query text box, enter the query to get the metrics of slice or namespace quotas.

3. (Optional) Under Table, set the date and time for which you want to see the metrics of usage/violation.

   

4. Click Execute.

5. Analyze the output to check the usage or violation of the quotas for a slice or a namespace.

Monitor Limit Usage

Monitor Limit Slice Usage

Use PromQL queries to get the individual limit usage metrics of a resource for a a slice or a namespace. The query output provides the usage value of the corresponding resource.

info

The slice quota usage is reported every 5 minutes. The usage metrics are obtained using the metrics API.

Slice CPU Limit Usage

Run the following query to get the CPU limit usage metrics of a slice:

info

In the query, replace the slice name with your slice name.

kubeslice_controller_slice_quota_cpu_utilization{slice_name="red", quota_type="limit"}

Expected Output

kubeslice_controller_slice_quota_cpu_utilization{control_plane="controller-manager", instance="10.7.2.16:18080", job="kubernetes-pods", kubernetes_namespace="kubeslice-controller", kubernetes_pod_name="kubeslice-controller-manager-bb89cf748-p7jl4", pod_template_hash="bb89cf748", project_name="cisco", quota_type="limit", resource_type="cpu", slice_name="red"}
30

The usage value is in the second line of the output. The unit value of the CPU is millicores.

Slice Ephemeral Storage Limit Usage

Run the following query to get the ephemeral storage limit usage metrics of a slice:

info

In the query, replace the slice name with your slice name.

kubeslice_controller_slice_quota_ephemeralstorage_utilization{slice_name="red", quota_type="limit"}

Expected Output

kubeslice_controller_slice_quota_ephemeralstorage_utilization{control_plane="controller-manager", instance="10.7.2.16:18080", job="kubernetes-pods", kubernetes_namespace="kubeslice-controller", kubernetes_pod_name="kubeslice-controller-manager-bb89cf748-p7jl4", pod_template_hash="bb89cf748", project_name="cisco", quota_type="limit", resource_type="ephemeralStorage", slice_name="red"}
1024000

The usage value is in the second line of the output. The unit value of the CPU is millicores.

Slice Memory Limit Usage

Run the following query to get the memory limit usage metrics of a slice:

info

In the query, replace the slice name with your slice name.

kubeslice_controller_slice_quota_memory_utilization{slice_name="red", quota_type="limit"}

Expected Output

kubeslice_controller_slice_quota_memory_utilization{control_plane="controller-manager", instance="10.7.2.16:18080", job="kubernetes-pods", kubernetes_namespace="kubeslice-controller", kubernetes_pod_name="kubeslice-controller-manager-bb89cf748-p7jl4", pod_template_hash="bb89cf748", project_name="cisco", quota_type="limit", resource_type="memory", slice_name="red"}
97280

The usage value is in the second line of the output. The unit value of the memory is KiB.

Slice Pod Count Limit Usage

Run the following query to get the pod count limit usage metrics:

info

In the query, replace the slice name with your slice name.

kubeslice_controller_slice_quota_podcount_utilization{slice_name="red", quota_type="limit"}

Expected Output

kubeslice_controller_slice_quota_podcount_utilization{control_plane="controller-manager", instance="10.7.2.16:18080", job="kubernetes-pods", kubernetes_namespace="kubeslice-controller", kubernetes_pod_name="kubeslice-controller-manager-bb89cf748-p7jl4", pod_template_hash="bb89cf748", project_name="cisco", quota_type="limit", resource_type="podCount", slice_name="red"}
4

The usage value is in the second line of the output. The unit value of the pod count is number.

Monitor Limit Namespace Usage

Namespace CPU Limit Usage

Run the following query to get the CPU limit usage metrics of a namespace:

info

In the query, replace the name of the slice, worker cluster, and the namespace with the corresponding name of your slice, worker cluster, and namespace.

kubeslice_controller_namespace_quota_cpu_utilization{slice_name="red", cluster_name="cluster-1", namespace="namespace-1", quota_type="limit"}

Expected Output

kubeslice_controller_namespace_quota_cpu_utilization{cluster_name="cluster-1", control_plane="controller-manager", instance="10.7.2.16:18080", job="kubernetes-pods", kubernetes_namespace="kubeslice-controller", kubernetes_pod_name="kubeslice-controller-manager-bb89cf748-p7jl4", namespace="namespace-1", pod_template_hash="bb89cf748", project_name="cisco", quota_type="limit", resource_type="cpu", slice_name="red"}
30

The usage value is in the second line of the output. The unit value of the CPU is millicores.

Namespace Ephemeral Storage Limit Usage

Run the following query to get the ephemeral storage limit usage metrics of a namespace:

info

In the query, replace the name of the slice, worker cluster, and the namespace with the corresponding name of your slice, worker cluster, and namespace.

kubeslice_controller_namespace_quota_ephemeralstorage_utilization{slice_name="red", cluster_name="cluster-1", namespace="namespace-1", quota_type="limit"}

Expected Output

kubeslice_controller_namespace_quota_ephemeralstorage_utilization{cluster_name="cluster-1", control_plane="controller-manager", instance="10.7.2.16:18080", job="kubernetes-pods", kubernetes_namespace="kubeslice-controller", kubernetes_pod_name="kubeslice-controller-manager-bb89cf748-p7jl4", namespace="namespace-1", pod_template_hash="bb89cf748", project_name="cisco", quota_type="limit", resource_type="ephemeralStorage", slice_name="red"}
1024000

The usage value is in the second line of the output. The unit value of the CPU is millicores.

Namespace Memory Limit Usage

Run the following query to get the memory limit usage metrics of a namespace:

info

In the query, replace the name of the slice, worker cluster, and the namespace with the corresponding name of your slice, worker cluster, and namespace.

kubeslice_controller_namespace_quota_memory_utilization{slice_name="red", cluster_name="cluster-1", namespace="namespace-1", quota_type="limit"}

Expected Output

kubeslice_controller_namespace_quota_memory_utilization{cluster_name="cluster-1", control_plane="controller-manager", instance="10.7.2.16:18080", job="kubernetes-pods", kubernetes_namespace="kubeslice-controller", kubernetes_pod_name="kubeslice-controller-manager-bb89cf748-p7jl4", namespace="namespace-1", pod_template_hash="bb89cf748", project_name="cisco", quota_type="limit", resource_type="memory", slice_name="red"}
97280

The usage value is in the second line of the output.The unit value of the memory is KiB.

Namespace Pod Count Limit Usage

Run the following query to get the pod count usage metrics of a namespace:

info

In the query, replace the name of the slice, worker cluster, and the namespace with the corresponding name of your slice, worker cluster, and namespace.

kubeslice_controller_namespace_quota_podcount_utilization{slice_name="red", cluster_name="cluster-1", namespace="namespace-1", quota_type="limit"}

Expected Output

kubeslice_controller_namespace_quota_podcount_utilization{cluster_name="cluster-1", control_plane="controller-manager", instance="10.7.2.16:18080", job="kubernetes-pods", kubernetes_namespace="kubeslice-controller", kubernetes_pod_name="kubeslice-controller-manager-bb89cf748-p7jl4", namespace="namespace-1", pod_template_hash="bb89cf748", project_name="cisco", quota_type="limit", resource_type="podCount", slice_name="red"}
4

The usage value is in the second line of the output. The unit value of the pod count is number.

Monitor Request Usage

Monitor Slice Requests Usage

Use PromQL queries to get the individual request usage metrics of a resource for a a slice or a namespace. The query output provides the usage value of the corresponding resource.

info

The slice quota usage is reported every 5 minutes. The usage metrics are obtained using the metrics API.

Slice CPU Request Usage

Run the following query to get the CPU request usage metrics of a slice:

info

In the query, replace the slice name with your slice name.

kubeslice_controller_slice_quota_cpu_utilization{slice_name="red", quota_type="request"}

Expected Output

kubeslice_controller_slice_quota_cpu_utilization{control_plane="controller-manager", instance="10.7.2.16:18080", job="kubernetes-pods", kubernetes_namespace="kubeslice-controller", kubernetes_pod_name="kubeslice-controller-manager-bb89cf748-p7jl4", pod_template_hash="bb89cf748", project_name="cisco", quota_type="request", resource_type="cpu", slice_name="red"}
30

The usage value is in the second line of the output. The unit value of the CPU is millicores.

Slice Ephemeral Storage Request Usage

Run the following query to get the ephemeral storage request usage metrics of a slice:

info

In the query, replace the slice name with your slice name.

kubeslice_controller_slice_quota_ephemeralstorage_utilization{slice_name="red", quota_type="request"}

Expected Output

kubeslice_controller_slice_quota_ephemeralstorage_utilization{control_plane="controller-manager", instance="10.7.2.16:18080", job="kubernetes-pods", kubernetes_namespace="kubeslice-controller", kubernetes_pod_name="kubeslice-controller-manager-bb89cf748-p7jl4", pod_template_hash="bb89cf748", project_name="cisco", quota_type="request", resource_type="ephemeralStorage", slice_name="red"}
102400

The usage value is in the second line of the output. The unit value of the CPU is millicores.

Slice Memory Request Usage

Run the following query to get the memory request usage metrics of a slice:

info

In the query, replace the slice name with your slice name.

kubeslice_controller_slice_quota_memory_utilization{slice_name="red", quota_type="request"}

Expected Output

kubeslice_controller_slice_quota_memory_utilization{control_plane="controller-manager", instance="10.7.2.16:18080", job="kubernetes-pods", kubernetes_namespace="kubeslice-controller", kubernetes_pod_name="kubeslice-controller-manager-bb89cf748-p7jl4", pod_template_hash="bb89cf748", project_name="cisco", quota_type="request", resource_type="memory", slice_name="red"}
97280

The usage value is in the second line of the output. The unit value of the memory is KiB.

Monitor Request Namespace Usage

Namespace CPU Request Usage

Run the following query to get the CPU request usage metrics of a namespace:

info

In the query, replace the name of the slice, worker cluster, and the namespace with the corresponding name of your slice, worker cluster, and namespace.

kubeslice_controller_namespace_quota_cpu_utilization{slice_name="red", cluster_name="cluster-1", namespace="namespace-1", quota_type="request"}

Expected Output

kubeslice_controller_namespace_quota_cpu_utilization{cluster_name="cluster-1", control_plane="controller-manager", instance="10.7.2.16:18080", job="kubernetes-pods", kubernetes_namespace="kubeslice-controller", kubernetes_pod_name="kubeslice-controller-manager-bb89cf748-p7jl4", namespace="namespace-1", pod_template_hash="bb89cf748", project_name="cisco", quota_type="request", resource_type="cpu", slice_name="red"}
30

The usage value is in the second line of the output. The unit value of the CPU is millicores.

Namespace Ephemeral Storage Request Usage

Run the following query to get the ephemeral storage request usage metrics of a namespace:

info

In the query, replace the name of the slice, worker cluster, and the namespace with the corresponding name of your slice, worker cluster, and namespace.

kubeslice_controller_namespace_quota_ephemeralstorage_utilization{slice_name="red", cluster_name="cluster-1", namespace="namespace-1", quota_type="request"}

Expected Output

kubeslice_controller_namespace_quota_ephemeralstorage_utilization{cluster_name="cluster-1", control_plane="controller-manager", instance="10.7.2.16:18080", job="kubernetes-pods", kubernetes_namespace="kubeslice-controller", kubernetes_pod_name="kubeslice-controller-manager-bb89cf748-p7jl4", namespace="namespace-1", pod_template_hash="bb89cf748", project_name="cisco", quota_type="request", resource_type="ephemeralStorage", slice_name="red"}
102400

The usage value is in the second line of the output. The unit value of the CPU is millicores.

Namespace Memory Request Usage

Run the following query to get the memory request usage metrics of a namespace:

info

In the query, replace the name of the slice, worker cluster, and the namespace with the corresponding name of your slice, worker cluster, and namespace.

kubeslice_controller_namespace_quota_memory_utilization{slice_name="red", cluster_name="cluster-1", namespace="namespace-1", quota_type="request"}

Expected Output

kubeslice_controller_namespace_quota_memory_utilization{cluster_name="cluster-1", control_plane="controller-manager", instance="10.7.2.16:18080", job="kubernetes-pods", kubernetes_namespace="kubeslice-controller", kubernetes_pod_name="kubeslice-controller-manager-bb89cf748-p7jl4", namespace="namespace-1", pod_template_hash="bb89cf748", project_name="cisco", quota_type="request", resource_type="memory", slice_name="red"}
97280

The usage value is in the second line of the output. The unit value of the memory is KiB.

Monitor Limit Violation

Use PromQL queries to get the individual limit violation metrics of a resource for a a slice or a namespace.The query output provides the violation value of the corresponding resource.

The violation value in the query output is indicated by a Boolean value with either of the following options:

• 0 indicates that the resource quota for a slice or a namespace is equal to or within the limits.
• 1 indicates that the resource quota for a slice or a namespace is violated.

Slice Monitor Limit Violations

Slice CPU Limit Violation

Run the following query to get the CPU violation metrics of a slice:

info

In the query, replace the slice name with your slice name.

kubeslice_controller_slice_quota_violation{violated_resource_type="cpu",slice_name="red",quota_type="limit"}

Expected Output

kubeslice_controller_slice_quota_violation{control_plane="controller-manager", instance="10.7.2.16:18080", job="kubernetes-pods", kubernetes_namespace="kubeslice-controller", kubernetes_pod_name="kubeslice-controller-manager-bb89cf748-p7jl4", pod_template_hash="bb89cf748", project_name="cisco", quota_type="limit", slice_name="red", violated_resource_type="cpu"}
0

The violation is in the second line of the output. 1 indicates that the CPU quota of the slice is violated and 0 indicates the CPU quota of the slice is less than or equal to the limits.

Slice Ephemeral Limit Violation

Run the following query to get the CPU violation metrics of a slice:

info

In the query, replace the slice name with your slice name.

kubeslice_controller_slice_quota_violation{violated_resource_type="ephemeralStorage",slice_name="red",quota_type="limit"}

Expected Output

kubeslice_controller_slice_quota_violation{control_plane="controller-manager", instance="10.7.2.16:18080", job="kubernetes-pods", kubernetes_namespace="kubeslice-controller", kubernetes_pod_name="kubeslice-controller-manager-bb89cf748-p7jl4", pod_template_hash="bb89cf748", project_name="cisco", quota_type="limit", slice_name="red", violated_resource_type="ephemeralStorage"}
1

The violation is in the second line of the output. 1 indicates that the CPU quota of the slice is violated and 0 indicates the CPU quota of the slice is less than or equal to the limits.

Slice Memory Limit Violation

Run the following query to get the memory violation metrics of a slice:

info

In the query, replace the slice name with your slice name.

kubeslice_controller_slice_quota_violation{violated_resource_type="memory",slice_name="red",quota_type="limit"}

Expected Output

kubeslice_controller_slice_quota_violation{control_plane="controller-manager", instance="10.7.2.16:18080", job="kubernetes-pods", kubernetes_namespace="kubeslice-controller", kubernetes_pod_name="kubeslice-controller-manager-bb89cf748-p7jl4", pod_template_hash="bb89cf748", project_name="cisco", quota_type="limit", slice_name="red", violated_resource_type="memory"}
0

The violation is in the second line of the output. 1 indicates that the memory quota of the slice is violated and 0 indicates the memory quota of the slice is less than or equal to the limits.

Slice Pod Count Limit Violation

Run the following query to get the pod count violation metrics of a slice:

info

In the query, replace the slice name with your slice name.

kubeslice_controller_slice_quota_violation{violated_resource_type="podCount",slice_name="red",quota_type="limit"}

Expected Output

kubeslice_controller_slice_quota_violation{control_plane="controller-manager", instance="10.7.2.16:18080", job="kubernetes-pods", kubernetes_namespace="kubeslice-controller", kubernetes_pod_name="kubeslice-controller-manager-bb89cf748-p7jl4", pod_template_hash="bb89cf748", project_name="cisco", quota_type="limit", slice_name="red", violated_resource_type="podCount"}
0

The violation is in the second line of the output. 1 indicates that the pod count quota of the slice is violated and 0 indicates the pod count quota of the slice is less than or equal to the limits.

Namespace Limit Violations

CPU Limit Violation

Run the following query to get the CPU limit violation metrics of a namespace:

info

In the query, replace the name of the slice, worker cluster, and the namespace with the corresponding name of your slice, worker cluster, and namespace.

kubeslice_controller_namespace_quota_violation{violated_resource_type="cpu",slice_name="red",cluster_name="cluster-1",namespace="namespace-1",quota_type="limit"}

Expected Output

kubeslice_controller_namespace_quota_violation{cluster_name="cluster-1", control_plane="controller-manager", instance="10.7.2.16:18080", job="kubernetes-pods", kubernetes_namespace="kubeslice-controller", kubernetes_pod_name="kubeslice-controller-manager-bb89cf748-p7jl4", namespace="namespace-1", pod_template_hash="bb89cf748", project_name="cisco", quota_type="limit", slice_name="red", violated_resource_type="cpu"}
0

The violation is in the second line of the output. 1 indicates that the CPU quota of the namespace is violated and 0 indicates the CPU quota of the namespace is less than or equal to the limits.

Namespace Ephemeral Storage Limit Violation

Run the following query to get the ephemeral storage limit violation metrics of a namespace:

info

In the query, replace the name of the slice, worker cluster, and the namespace with the corresponding name of your slice, worker cluster, and namespace.

kubeslice_controller_namespace_quota_violation{violated_resource_type="ephemeralStorage",slice_name="red",cluster_name="cluster-1",namespace="namespace-1",quota_type="limit"}

Expected Output

kubeslice_controller_namespace_quota_violation{cluster_name="cluster-1", control_plane="controller-manager", instance="10.7.2.16:18080", job="kubernetes-pods", kubernetes_namespace="kubeslice-controller", kubernetes_pod_name="kubeslice-controller-manager-bb89cf748-p7jl4", namespace="namespace-1", pod_template_hash="bb89cf748", project_name="cisco", quota_type="limit", slice_name="red", violated_resource_type="ephemeralStorage"}
1

The violation is in the second line of the output. 1 indicates that the CPU quota of the namespace is violated and 0 indicates the CPU quota of the namespace is less than or equal to the limits.

Namespace Memory Limit Violation

Run the following query to get the memory limit violation metrics of a namespace:

info

In the query, replace the name of the slice, worker cluster, and the namespace with the corresponding name of your slice, worker cluster, and namespace.

kubeslice_controller_namespace_quota_violation{violated_resource_type="memory",slice_name="red",cluster_name="cluster-1",namespace="namespace-1",quota_type="limit"}

Expected Output

kubeslice_controller_namespace_quota_violation{cluster_name="cluster-1", control_plane="controller-manager", instance="10.7.2.16:18080", job="kubernetes-pods", kubernetes_namespace="kubeslice-controller", kubernetes_pod_name="kubeslice-controller-manager-bb89cf748-p7jl4", namespace="namespace-1", pod_template_hash="bb89cf748", project_name="cisco", quota_type="limit", slice_name="red", violated_resource_type="memory"}
1

The violation is in the second line of the output. 1 indicates that the memory quota of the namespace is violated and 0 indicates the memory quota of the namespace is less than or equal to the limits.

Namespace Pod Count Limit Violation

Run the following query to get the pod count limit violation metrics of a namespace:

info

In the query, replace the name of the slice, worker cluster, and the namespace with the corresponding name of your slice, worker cluster, and namespace.

kubeslice_controller_namespace_quota_violation{violated_resource_type="podCount",slice_name="red",cluster_name="cluster-1",namespace="namespace-1",quota_type="limit"}

Expected Output

kubeslice_controller_namespace_quota_violation{cluster_name="cluster-1", control_plane="controller-manager", instance="10.7.2.16:18080", job="kubernetes-pods", kubernetes_namespace="kubeslice-controller", kubernetes_pod_name="kubeslice-controller-manager-bb89cf748-p7jl4", namespace="namespace-1", pod_template_hash="bb89cf748", project_name="cisco", quota_type="limit", slice_name="red", violated_resource_type="podCount"}

The violation is in the second line of the output. 1 indicates that the pod count quota of the namespace is violated and 0 indicates the pod count quota of the namespace is less than or equal to the limits.

Monitor Request Violation

Use PromQL queries to get the individual request violation metrics of a resource for a a slice or a namespace.The query output provides the violation value of the corresponding resource.

The request violation value in the query output is indicated by a Boolean value with either of the following options:

• 0 indicates that the resource quota for a slice or a namespace is equal to or within the limits.
• 1 indicates that the resource quota for a slice or a namespace is violated.

Slice Monitor Request Violations

Slice CPU Request Violation

Run the following query to get the CPU request violation metrics of a slice:

info

In the query, replace the slice name with your slice name.

kubeslice_controller_slice_quota_violation{violated_resource_type="cpu",slice_name="red",quota_type="request"}

Expected Output

kubeslice_controller_slice_quota_violation{control_plane="controller-manager", instance="10.7.2.16:18080", job="kubernetes-pods", kubernetes_namespace="kubeslice-controller", kubernetes_pod_name="kubeslice-controller-manager-bb89cf748-p7jl4", pod_template_hash="bb89cf748", project_name="cisco", quota_type="request", slice_name="red", violated_resource_type="cpu"}
0

The violation is in the second line of the output. 1 indicates that the CPU quota of the slice is violated and 0 indicates the CPU quota of the slice is less than or equal to the limits.

Slice Ephemeral Storage Request Violation

Run the following query to get the ephemeral storage request violation metrics of a slice:

info

In the query, replace the slice name with your slice name.

kubeslice_controller_slice_quota_violation{violated_resource_type="ephemeralStorage",slice_name="red",quota_type="request"}

Expected Output

kubeslice_controller_slice_quota_violation{control_plane="controller-manager", instance="10.7.2.16:18080", job="kubernetes-pods", kubernetes_namespace="kubeslice-controller", kubernetes_pod_name="kubeslice-controller-manager-bb89cf748-p7jl4", pod_template_hash="bb89cf748", project_name="cisco", quota_type="request", slice_name="red", violated_resource_type="ephemeralStorage"}
0

The violation is in the second line of the output. 1 indicates that the CPU quota of the slice is violated and 0 indicates the CPU quota of the slice is less than or equal to the limits.

Slice Memory Request Violation

Run the following query to get the memory request violation metrics of a slice:

info

In the query, replace the slice name with your slice name.

kubeslice_controller_slice_quota_violation{violated_resource_type="memory",slice_name="red",quota_type="request"}

Expected Output

kubeslice_controller_slice_quota_violation{control_plane="controller-manager", instance="10.7.2.16:18080", job="kubernetes-pods", kubernetes_namespace="kubeslice-controller", kubernetes_pod_name="kubeslice-controller-manager-bb89cf748-p7jl4", pod_template_hash="bb89cf748", project_name="cisco", quota_type="request", slice_name="red", violated_resource_type="memory"}
0

The violation is in the second line of the output. 1 indicates that the memory quota of the slice is violated and 0 indicates the memory quota of the slice is less than or equal to the limits.

Slice CPU Request Violation

Run the following query to get the CPU request violation metrics of a namespace:

info

In the query, replace the name of the slice, worker cluster, and the namespace with the corresponding name of your slice, worker cluster, and namespace.

kubeslice_controller_namespace_quota_violation{violated_resource_type="cpu",slice_name="red",cluster_name="cluster-1",namespace="namespace-1",quota_type="request"}

Expected Output

kubeslice_controller_namespace_quota_violation{cluster_name="cluster-1", control_plane="controller-manager", instance="10.7.2.16:18080", job="kubernetes-pods", kubernetes_namespace="kubeslice-controller", kubernetes_pod_name="kubeslice-controller-manager-bb89cf748-p7jl4", namespace="namespace-1", pod_template_hash="bb89cf748", project_name="cisco", quota_type="request", slice_name="red", violated_resource_type="cpu"}
0

The violation is in the second line of the output. 1 indicates that the CPU quota of the namespace is violated and 0 indicates the CPU quota of the namespace is less than or equal to the limits.

Namespace Request Violations

Namespace Ephemeral Storage Request Violation

Run the following query to get the ephemeral storage request violation metrics of a namespace:

info

In the query, replace the name of the slice, worker cluster, and the namespace with the corresponding name of your slice, worker cluster, and namespace.

kubeslice_controller_namespace_quota_violation{violated_resource_type="ephemeralStorage",slice_name="red",cluster_name="cluster-1",namespace="namespace-1",quota_type="request"}

Expected Output

kubeslice_controller_namespace_quota_violation{cluster_name="cluster-1", control_plane="controller-manager", instance="10.7.2.16:18080", job="kubernetes-pods", kubernetes_namespace="kubeslice-controller", kubernetes_pod_name="kubeslice-controller-manager-bb89cf748-p7jl4", namespace="namespace-1", pod_template_hash="bb89cf748", project_name="cisco", quota_type="request", slice_name="red", violated_resource_type="ephemeralStorage"}
1

The violation is in the second line of the output. 1 indicates that the CPU quota of the namespace is violated and 0 indicates the CPU quota of the namespace is less than or equal to the limits.

Namespace Memory Request Violation

Run the following query to get the memory request violation metrics of a namespace:

info

In the query, replace the name of the slice, worker cluster, and the namespace with the corresponding name of your slice, worker cluster, and namespace.

kubeslice_controller_namespace_quota_violation{violated_resource_type="memory",slice_name="red",cluster_name="cluster-1",namespace="namespace-1",quota_type="request"}

Expected Output

kubeslice_controller_namespace_quota_violation{cluster_name="cluster-1", control_plane="controller-manager", instance="10.7.2.16:18080", job="kubernetes-pods", kubernetes_namespace="kubeslice-controller", kubernetes_pod_name="kubeslice-controller-manager-bb89cf748-p7jl4", namespace="namespace-1", pod_template_hash="bb89cf748", project_name="cisco", quota_type="request", slice_name="red", violated_resource_type="memory"}
1

The violation is in the second line of the output. 1 indicates that the memory quota of the namespace is violated and 0 indicates the memory quota of the namespace is less than or equal to the limits.

Monitoring Violation through Events

You can also track violation by retrieving the events using the following command:

k get events -n <project name>

Example

k get events -n kubeslice-avesha

Expected Output

LAST SEEN   TYPE      REASON             OBJECT                                    MESSAGE
5m34s       Normal    SUCCESS            workersliceresourcequota/red-worker-2     Updated workersliceresourcequota kubeslice-avesha as delta is more than 5 percent
55m         Normal    SUCCESS            workersliceresourcequota/red-worker-1     Updated workersliceresourcequota kubeslice-avesha as delta is more than 5 percent
41m         Normal    SUCCESS            workersliceresourcequota/red-worker-1     Updated workersliceresourcequota kubeslice-avesha as delta is more than 5 percent
29m         Normal    SUCCESS            workersliceresourcequota/red-worker-1     Updated workersliceresourcequota kubeslice-avesha as delta is more than 5 percent
27m         Normal    SUCCESS            workersliceresourcequota/red-worker-1     Updated workersliceresourcequota kubeslice-avesha as delta is more than 5 percent
22m         Normal    SUCCESS            workersliceresourcequota/red-worker-1     Updated workersliceresourcequota kubeslice-avesha as delta is more than 5 percent
20m         Normal    SUCCESS            workersliceresourcequota/red-worker-1     Updated workersliceresourcequota kubeslice-avesha as delta is more than 5 percent
15m         Normal    SUCCESS            workersliceresourcequota/red-worker-1     Updated workersliceresourcequota kubeslice-avesha as delta is more than 5 percent
4m34s       Warning   VIOLATED           sliceresourcequotaconfig/red              Resource quota of type limit-podCount is violated for namespace iperf, cluster worker-2, slice red
29m         Warning   VIOLATED           sliceresourcequotaconfig/red              Resource quota of type limit-podCount is violated for slice red