{"id":14621,"date":"2025-01-23T02:43:11","date_gmt":"2025-01-23T02:43:11","guid":{"rendered":"https:\/\/dmsretail.com\/RetailNews\/optimizing-ai-workloads-with-nvida-gpus-time-slicing-and-karpenter-part-2\/"},"modified":"2025-01-23T02:43:11","modified_gmt":"2025-01-23T02:43:11","slug":"optimizing-ai-workloads-with-nvida-gpus-time-slicing-and-karpenter-part-2","status":"publish","type":"post","link":"https:\/\/dmsretail.com\/RetailNews\/optimizing-ai-workloads-with-nvida-gpus-time-slicing-and-karpenter-part-2\/","title":{"rendered":"Optimizing AI Workloads with NVIDA GPUs, Time Slicing, and Karpenter (Part 2)"},"content":{"rendered":"<p> <p><a href=\"https:\/\/dmsretail.com\/online-workshops-list\/\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-496\" src=\"https:\/\/dmsretail.com\/RetailNews\/wp-content\/uploads\/2022\/05\/RETAIL-ONLINE-TRAINING-728-X-90.png\" alt=\"Retail Online Training\" width=\"729\" height=\"91\" srcset=\"https:\/\/dmsretail.com\/RetailNews\/wp-content\/uploads\/2022\/05\/RETAIL-ONLINE-TRAINING-728-X-90.png 729w, https:\/\/dmsretail.com\/RetailNews\/wp-content\/uploads\/2022\/05\/RETAIL-ONLINE-TRAINING-728-X-90-300x37.png 300w\" sizes=\"auto, (max-width: 729px) 100vw, 729px\" \/><\/a><\/p><br \/>\n<\/p>\n<div>\n<p><b><span data-contrast=\"none\">Introduction: Overcoming GPU Management Challenges\u00a0<\/span><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">In Part 1 of this blog series, we explored the challenges of hosting large language models (LLMs) on CPU-based workloads within an EKS cluster. We discussed the inefficiencies associated with using CPUs for such tasks, primarily due to the large model sizes and slower inference speeds. The introduction of GPU resources offered a significant performance boost, but it also brought about the need for efficient management of these high-cost resources.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">In this second part, we will delve deeper into how to optimize GPU usage for these workloads. We will cover the following key areas:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<ul>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"1\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">NVIDIA Device Plugin Setup<\/span><\/b><span data-contrast=\"auto\">: This section will explain the importance of the NVIDIA device plugin for Kubernetes, detailing its role in resource discovery, allocation, and isolation.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"1\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Time Slicing<\/span><\/b><span data-contrast=\"auto\">: We\u2019ll discuss how time slicing allows multiple processes to share GPU resources effectively, ensuring maximum utilization.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"1\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Node Autoscaling with Karpenter<\/span><\/b><span data-contrast=\"auto\">: This section will describe how Karpenter dynamically manages node scaling based on real-time demand, optimizing resource utilization and reducing costs.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/li>\n<\/ul>\n<p><b><i><span data-contrast=\"none\">Challenges Addressed<\/span><\/i><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:281,&quot;335559739&quot;:281}\">\u00a0<\/span><\/p>\n<ol>\n<li data-leveltext=\"%1.\" data-font=\"Aptos\" data-listid=\"2\" data-list-defn-props=\"{&quot;335552541&quot;:0,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769242&quot;:[65533,0],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;%1.&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Efficient GPU Management<\/span><\/b><span data-contrast=\"auto\">: Ensuring GPUs are fully utilized to justify their high cost.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/li>\n<li data-leveltext=\"%1.\" data-font=\"Aptos\" data-listid=\"2\" data-list-defn-props=\"{&quot;335552541&quot;:0,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769242&quot;:[65533,0],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;%1.&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Concurrency Handling<\/span><\/b><span data-contrast=\"auto\">: Allowing multiple workloads to share GPU resources effectively.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/li>\n<li data-leveltext=\"%1.\" data-font=\"Aptos\" data-listid=\"2\" data-list-defn-props=\"{&quot;335552541&quot;:0,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769242&quot;:[65533,0],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;%1.&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Dynamic Scaling<\/span><\/b><span data-contrast=\"auto\">: Automatically adjusting the number of nodes based on workload demands.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/li>\n<\/ol>\n<p><span data-contrast=\"auto\">\u00a0<\/span><b><span data-contrast=\"none\">Section 1: Introduction to NVIDIA Device Plugin<\/span><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:281,&quot;335559739&quot;:281}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">\u00a0The NVIDIA device plugin for Kubernetes is a component that simplifies the management and usage of NVIDIA GPUs in Kubernetes clusters. It allows Kubernetes to recognize and allocate GPU resources to pods, enabling GPU-accelerated workloads.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><b><i><span data-contrast=\"none\">Why We Need the NVIDIA Device Plugin<\/span><\/i><\/b><span data-ccp-props=\"{}\">\u00a0<\/span><\/p>\n<ul>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"3\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Resource Discovery<\/span><\/b><span data-contrast=\"auto\">: Automatically detects NVIDIA GPU resources on each node.<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"3\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Resource Allocation<\/span><\/b><span data-contrast=\"auto\">: Manages the distribution of GPU resources to pods based on their requests.<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"3\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Isolation<\/span><\/b><span data-contrast=\"auto\">: Ensures secure and efficient utilization of GPU resources among different pods.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/li>\n<\/ul>\n<p><span data-contrast=\"auto\">\u00a0The NVIDIA device plugin simplifies GPU management in Kubernetes clusters. It automates the installation of the <\/span><b><span data-contrast=\"auto\">NVIDIA driver<\/span><\/b><span data-contrast=\"auto\">, <\/span><b><span data-contrast=\"auto\">container toolkit<\/span><\/b><span data-contrast=\"auto\">, and <\/span><b><span data-contrast=\"auto\">CUDA<\/span><\/b><span data-contrast=\"auto\">, ensuring that GPU resources are available for workloads without requiring manual setup.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<ul>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"4\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">NVIDIA Driver<\/span><\/b><span data-contrast=\"auto\">: Required for nvidia-smi and basic GPU operations. Interfacing with the GPU hardware. The screenshot below displays the output of the nvidia-smi command, which shows key information such as the driver version, CUDA version, and detailed GPU configuration, confirming that the GPU is properly configured and ready for use<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/li>\n<\/ul>\n<p><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-466826 aligncenter\" src=\"https:\/\/storage.googleapis.com\/blogs-images-new\/ciscoblogs\/1\/2025\/01\/Screenshot-2025-01-14-at-12.13.23\u202fPM-300x102.png\" alt=\"\" width=\"968\" height=\"329\" srcset=\"https:\/\/storage.googleapis.com\/blogs-images-new\/ciscoblogs\/1\/2025\/01\/Screenshot-2025-01-14-at-12.13.23\u202fPM-300x102.png 300w, https:\/\/storage.googleapis.com\/blogs-images-new\/ciscoblogs\/1\/2025\/01\/Screenshot-2025-01-14-at-12.13.23\u202fPM-768x262.png 768w\" sizes=\"auto, (max-width: 968px) 100vw, 968px\"\/><\/span><\/p>\n<p><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/p>\n<ul>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"5\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">NVIDIA Container Toolkit<\/span><\/b><span data-contrast=\"auto\">: Required for using GPUs with containerd. Below we can see the version of the container toolkit version and the status of the service running on the instance<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/li>\n<\/ul>\n<pre><span style=\"color: #99cc00;\" data-contrast=\"none\">#Installed Version<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">rpm <\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">qa <\/span><span data-contrast=\"none\">|<\/span><span data-contrast=\"none\"> grep <\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">i nvidia<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">container<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">toolkit<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">nvidia<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">container<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">toolkit<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">base<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">1.15<\/span><span data-contrast=\"none\">.0<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">1.<\/span><span data-contrast=\"none\">x86_64<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">nvidia<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">container<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">toolkit<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">1.15<\/span><span data-contrast=\"none\">.0<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">1.<\/span><span data-contrast=\"none\">x86_64<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span><\/pre>\n<ul>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"5\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"2\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">CUDA<\/span><\/b><span data-contrast=\"auto\">: Required for GPU-accelerated applications and libraries. Below is the output of the nvcc command, showing the version of CUDA installed on the system:<\/span><span data-ccp-props=\"{}\"><br \/><\/span><\/li>\n<\/ul>\n<pre><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">usr<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">local<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">cuda<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">bin<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">nvcc <\/span><span style=\"color: #ff0000;\" data-contrast=\"none\">--<\/span><span data-contrast=\"none\">version<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span style=\"color: #0000ff;\" data-contrast=\"none\">nvcc<\/span><span data-contrast=\"none\">: NVIDIA (R) Cuda compiler driver<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">Copyright (c) <\/span><span style=\"color: #008000;\">2005-2023<\/span><span data-contrast=\"none\"> NVIDIA Corporation<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">Built on <\/span><span style=\"color: #ff0000;\">Tue_Aug_15_22:02:13_PDT_2023<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">Cuda compilation tools, release <\/span><span data-contrast=\"none\">12.2<\/span><span data-contrast=\"none\">, V12.2.140<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">Build cuda_12.2.r12.2<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">compiler.33191640_0<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span><\/pre>\n<p><b><i><span data-contrast=\"none\">Setting Up the NVIDIA Device Plugin<\/span><\/i><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">To ensure the DaemonSet runs exclusively on GPU-based instances, we label the node with the key \u201cnvidia.com\/gpu\u201d and the value \u201ctrue\u201d. This is achieved using <\/span><b><span data-contrast=\"auto\">Node affinity, <\/span><\/b><b><span data-contrast=\"auto\">Node selector<\/span><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\"> and <\/span><b><span data-contrast=\"auto\">Taints and Tolerations.<\/span><\/b><\/p>\n<p><span data-contrast=\"auto\">Let us now delve into each of these components in detail.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<ul>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"7\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Node Affinity:<\/span><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\"> \u00a0<\/span>Node affinity allows to schedule pod on the nodes based on the node labels<span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><span data-contrast=\"auto\">requiredDuringSchedulingIgnoredDuringExecution<\/span><span data-contrast=\"none\">: The scheduler cannot schedule the Pod unless the rule is met, and the key is \u201cnvidia.com\/gpu\u201d and operator is \u201cin,\u201d and the values is \u201ctrue.\u201d<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/li>\n<\/ul>\n<pre><span style=\"color: #0000ff;\">affinity:\u00a0<\/span>&#13;\n<span style=\"color: #0000ff;\"> \u00a0\u00a0 nodeAffinity:\u00a0<\/span>&#13;\n<span style=\"color: #0000ff;\"> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 requiredDuringSchedulingIgnoredDuringExecution:\u00a0<\/span>&#13;\n<span style=\"color: #0000ff;\"> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 nodeSelectorTerms<\/span><span data-contrast=\"none\">:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\"> matchExpressions:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\"> key: feature.node.kubernetes.io<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">pci<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">10de<\/span><span data-contrast=\"none\">.present<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span style=\"color: #0000ff;\" data-contrast=\"none\">operator<\/span><span data-contrast=\"none\">: In<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span style=\"color: #0000ff;\" data-contrast=\"none\">values<\/span><span data-contrast=\"none\">:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">-<\/span> <span style=\"color: #ff0000;\" data-contrast=\"none\">\"true\"<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\"> matchExpressions:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\"> key: feature.node.kubernetes.io<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">cpu<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">model.vendor_id<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span style=\"color: #0000ff;\" data-contrast=\"none\">operator<\/span><span data-contrast=\"none\">: In<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span style=\"color: #0000ff;\" data-contrast=\"none\">values<\/span><span data-contrast=\"none\">:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\"> NVIDIA<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\"> matchExpressions:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\"> key: nvidia.com<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">gpu<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span style=\"color: #0000ff;\" data-contrast=\"none\">operator<\/span><span data-contrast=\"none\">: In<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span style=\"color: #0000ff;\" data-contrast=\"none\">values<\/span><span data-contrast=\"none\">:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">-<\/span> <span style=\"color: #ff0000;\" data-contrast=\"none\">\"true\"<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span><\/pre>\n<ul>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"8\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Node selector:<\/span><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0 \u00a0N<\/span>ode selector is the simplest recommendation form for node selection constraints <span data-contrast=\"none\">nvidia.com\/gpu<\/span><span data-contrast=\"auto\">: \u201ctrue\u201d<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"8\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Taints and Tolerations:<\/span><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\"> Tolerations are added to the Daemon Set to ensure it can be scheduled on the tainted GPU nodes(nvidia.com\/gpu=true:Noschedule).<\/span><\/li>\n<\/ul>\n<pre><span data-contrast=\"none\">kubectl taint node ip<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">10<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">20<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">23<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">199.<\/span><span data-contrast=\"none\">us<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">west<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">1.<\/span><span data-contrast=\"none\">compute.internal nvidia.com<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">gpu<\/span><span data-contrast=\"none\">=<\/span><span data-contrast=\"none\">true:Noschedule<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">kubectl describe node ip<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">10<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">20<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">23<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">199.<\/span><span data-contrast=\"none\">us<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">west<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">1.<\/span><span data-contrast=\"none\">compute.internal <\/span><span data-contrast=\"none\">|<\/span><span data-contrast=\"none\"> grep <\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">i taint<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span style=\"color: #0000ff;\" data-contrast=\"none\">Taints<\/span><span data-contrast=\"none\">: nvidia.com<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">gpu<\/span><span data-contrast=\"none\">=<\/span><span data-contrast=\"none\">true:NoSchedule<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n&#13;\n<span style=\"color: #0000ff;\" data-contrast=\"none\">tolerations<\/span><span data-contrast=\"none\">:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0 <\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\"> effect: NoSchedule<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0 <\/span><span style=\"color: #0000ff;\" data-contrast=\"none\">key<\/span><span data-contrast=\"none\">: nvidia.com<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">gpu<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0 <\/span><span style=\"color: #0000ff;\" data-contrast=\"none\">operator<\/span><span data-contrast=\"none\">: Exists<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span><\/pre>\n<p><span data-contrast=\"auto\">After implementing the node labeling, affinity, node selector, and taints\/tolerations, we can ensure the Daemon Set runs exclusively on GPU-based instances. We can verify the deployment of the NVIDIA device plugin using the following command:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<pre><span data-contrast=\"none\">kubectl get ds <\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">n kube<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">system<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">NAME\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 DESIRED\u00a0\u00a0 CURRENT\u00a0\u00a0 READY\u00a0\u00a0 UP<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">TO<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">DATE\u00a0\u00a0 AVAILABLE\u00a0 NODE SELECTOR\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0                      AGE<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n&#13;\n<span data-contrast=\"none\">nvidia<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">device<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">plugin\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">1<\/span><span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">1<\/span><span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">1<\/span><span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">1<\/span><span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">1<\/span><span data-contrast=\"none\"> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 nvidia.com<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">gpu<\/span><span data-contrast=\"none\">=<\/span><span data-contrast=\"none\">true\u00a0\u00a0\u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0                      <\/span><span data-contrast=\"none\">75d<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">nvidia<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">device<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">plugin<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">mps<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">control<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">daemon\u00a0\u00a0 <\/span><span data-contrast=\"none\">0<\/span><span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">0<\/span><span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">0<\/span><span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">0<\/span><span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">0<\/span><span data-contrast=\"none\"> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 nvidia.com<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">gpu<\/span><span data-contrast=\"none\">=<\/span><span data-contrast=\"none\">true<\/span><span data-contrast=\"none\">,nvidia.com<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">mps.capable<\/span><span data-contrast=\"none\">=<\/span><span data-contrast=\"none\">true\u00a0\u00a0 <\/span><span data-contrast=\"none\">75d<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span><\/pre>\n<p><span data-contrast=\"auto\">But the challenge here is GPUs are so expensive and need to make sure the maximum utilization of GPU\u2019s and let us explore more on GPU Concurrency.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><b><span data-contrast=\"auto\">GPU Concurrency:\u00a0\u00a0<\/span><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><b><span data-contrast=\"auto\">Refers to the ability to execute multiple tasks or threads simultaneously on a GPU<\/span><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<ul>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"10\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><span data-contrast=\"auto\">Single Process: In a single process setup, only one application or container uses the GPU at a time. This approach is straightforward but may lead to underutilization of the GPU resources if the application does not fully load the GPU.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"10\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><span data-contrast=\"auto\">Multi-Process Service (MPS): NVIDIA\u2019s Multi-Process Service (MPS) allows multiple CUDA applications to share a single GPU concurrently, improving GPU utilization and reducing the overhead of context switching.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"10\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><span data-contrast=\"auto\">Time slicing:\u00a0 Time slicing involves dividing the GPU time between different processes in other words multiple process takes turns on GPU\u2019s (Round Robin context Switching)<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"10\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><span data-contrast=\"auto\">Multi Instance GPU(MIG): MIG is a feature available on NVIDIA A100 GPUs that allows a single GPU to be partitioned into multiple smaller, isolated instances, each behaving like a separate GPU.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"10\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><span data-contrast=\"auto\">Virtualization: GPU virtualization allows a single physical GPU to be shared among multiple virtual machines (VMs) or containers, providing each with a virtual GPU.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/li>\n<\/ul>\n<p><b><span data-contrast=\"none\">\u00a0Section 2: Implementing Time Slicing for GPUs<\/span><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;201341983&quot;:0,&quot;335551550&quot;:1,&quot;335551620&quot;:1,&quot;335559685&quot;:0,&quot;335559737&quot;:0,&quot;335559738&quot;:281,&quot;335559739&quot;:281,&quot;335559740&quot;:279}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">Time-slicing in the context of NVIDIA GPUs and Kubernetes refers to sharing a physical GPU among multiple containers or pods in a Kubernetes cluster. The technology involves partitioning the GPU\u2019s processing time into smaller intervals and allocating those intervals to different containers or pods.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<ul>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"11\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><span data-contrast=\"auto\">Time Slice Allocation: The GPU scheduler allocates time slices to each vGPU configured on the physical GPU.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"11\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><span data-contrast=\"auto\">Preemption and Context Switching: At the end of a vGPU\u2019s time slice, the GPU scheduler preempts its execution, saves its context, and switches to the next vGPU\u2019s context.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"11\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><span data-contrast=\"auto\">Context Switching: The GPU scheduler ensures smooth context switching between vGPUs, minimizing overhead, and ensuring efficient use of GPU resources.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"11\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><span data-contrast=\"auto\">Task Completion: Processes within containers complete their GPU-accelerated tasks within their allocated time slices.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"11\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><span data-contrast=\"auto\">Resource Management and Monitoring<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"11\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><span data-contrast=\"auto\">Resource Release: As tasks complete, GPU resources are released back to Kubernetes for reallocation to other pods or containers<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/li>\n<\/ul>\n<p><b><i><span data-contrast=\"none\">Why We Need Time Slicing<\/span><\/i><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:319,&quot;335559739&quot;:319}\">\u00a0<\/span><\/p>\n<ul>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"12\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Cost Efficiency<\/span><\/b><span data-contrast=\"auto\">: Ensures high-cost GPUs are not underutilized.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"12\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Concurrency<\/span><\/b><span data-contrast=\"auto\">: Allows multiple applications to use the GPU simultaneously.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/li>\n<\/ul>\n<p><b><i><span data-contrast=\"none\">\u00a0Configuration Example for Time Slicing\u00a0<\/span><\/i><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">Let us apply the time slicing config using config map as shown below. Here <\/span><b><span data-contrast=\"auto\">replicas: 3<\/span><\/b><span data-contrast=\"auto\"> specifies the number of replicas for GPU resources that means that GPU resource can be sliced into 3 sharing instances<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<pre><span style=\"color: #0000ff;\" data-contrast=\"none\">apiVersion<\/span><span data-contrast=\"none\">: v1<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span style=\"color: #0000ff;\" data-contrast=\"none\">kind<\/span><span data-contrast=\"none\">: ConfigMap<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span style=\"color: #0000ff;\" data-contrast=\"none\">metadata<\/span><span data-contrast=\"none\">:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0 <\/span><span style=\"color: #0000ff;\" data-contrast=\"none\">name<\/span><span data-contrast=\"none\">: nvidia<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">device<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">plugin<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0 <\/span><span style=\"color: #0000ff;\" data-contrast=\"none\">namespace<\/span><span data-contrast=\"none\">: kube<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">system<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span style=\"color: #0000ff;\" data-contrast=\"none\">data<\/span><span data-contrast=\"none\">:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0 <\/span><span style=\"color: #0000ff;\" data-contrast=\"none\">any<\/span><span data-contrast=\"none\">: <\/span><span data-contrast=\"none\">|-<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0 <\/span><span style=\"color: #0000ff;\" data-contrast=\"none\">version<\/span><span data-contrast=\"none\">: v1<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0 <\/span><span style=\"color: #0000ff;\" data-contrast=\"none\">flags<\/span><span data-contrast=\"none\">:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span style=\"color: #0000ff;\" data-contrast=\"none\">migStrategy<\/span><span data-contrast=\"none\">: none<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0 <\/span><span style=\"color: #0000ff;\" data-contrast=\"none\">sharing<\/span><span data-contrast=\"none\">:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span style=\"color: #0000ff;\" data-contrast=\"none\">timeSlicing<\/span><span data-contrast=\"none\">:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span style=\"color: #0000ff;\" data-contrast=\"none\">resources<\/span><span data-contrast=\"none\">:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">-<\/span> <span style=\"color: #0000ff;\" data-contrast=\"none\">name<\/span><span data-contrast=\"none\">: nvidia.com<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">gpu<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span style=\"color: #0000ff;\" data-contrast=\"none\">replicas<\/span><span data-contrast=\"none\">: <\/span><span data-contrast=\"none\">3<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span style=\"color: #339966;\">#We can verify the GPU resources available on your nodes using the following command: <\/span><span data-contrast=\"none\">\u00a0\u00a0\u00a0<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">kubectl get nodes <\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">o json <\/span><span data-contrast=\"none\">|<\/span><span data-contrast=\"none\"> jq <\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">r <\/span><span style=\"color: #ff0000;\">'.items[] | select(.status.capacity.\"nvidia.com\/gpu\" != null) &#13;\n| {name: .metadata.name, capacity: .status.capacity}'<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span style=\"color: #ff0000;\">{\u00a0<\/span>&#13;\n<span style=\"color: #ff0000;\">\u00a0 \"name\": \"ip-10-20-23-199.us-west-1.compute.internal\",\u00a0<\/span>&#13;\n<span style=\"color: #ff0000;\">\u00a0 \"capacity\": {\u00a0<\/span>&#13;\n<span style=\"color: #ff0000;\">\u00a0\u00a0\u00a0 \"cpu\": \"4\",\u00a0<\/span>&#13;\n<span style=\"color: #ff0000;\">\u00a0\u00a0\u00a0 \"ephemeral-storage\": \"104845292Ki\",\u00a0<\/span>&#13;\n<span style=\"color: #ff0000;\">\u00a0\u00a0\u00a0 \"hugepages-1Gi\": \"0\",\u00a0<\/span>&#13;\n<span style=\"color: #ff0000;\">\u00a0\u00a0\u00a0 \"hugepages-2Mi\": \"0\",\u00a0<\/span>&#13;\n<span style=\"color: #ff0000;\">\u00a0\u00a0\u00a0 \"memory\": \"16069060Ki\",\u00a0<\/span>&#13;\n<span style=\"color: #ff0000;\">\u00a0\u00a0\u00a0 \"nvidia.com\/gpu\": \"3\",\u00a0<\/span>&#13;\n<span style=\"color: #ff0000;\">\u00a0\u00a0\u00a0 \"pods\": \"110\"\u00a0<\/span>&#13;\n<span style=\"color: #ff0000;\">\u00a0 }\u00a0<\/span>&#13;\n<span style=\"color: #ff0000;\">}\u00a0<\/span>&#13;\n<span style=\"color: #339966;\">#The above output shows that the node ip-10-20-23-199.us-west-1. compute.internal has 3 virtual GPUs available. <\/span>&#13;\n<span style=\"color: #339966;\">#We can request GPU resources in their pod specifications by setting resource limits <\/span>&#13;\n<span style=\"color: #0000ff;\" data-contrast=\"none\">resources<\/span><span data-contrast=\"none\">:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span style=\"color: #0000ff;\" data-contrast=\"none\">limits<\/span><span data-contrast=\"none\">:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span style=\"color: #0000ff;\" data-contrast=\"none\">cpu<\/span><span data-contrast=\"none\">: <\/span><span style=\"color: #ff0000;\" data-contrast=\"none\">\"1\"<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span style=\"color: #0000ff;\" data-contrast=\"none\">memory<\/span><span data-contrast=\"none\">: <\/span><span style=\"color: #ff0000;\" data-contrast=\"none\">2G<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 nvidia.com<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">gpu: <\/span><span style=\"color: #ff0000;\" data-contrast=\"none\">\"1\"<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span style=\"color: #0000ff;\" data-contrast=\"none\">requests<\/span><span data-contrast=\"none\">:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span style=\"color: #0000ff;\" data-contrast=\"none\">cpu<\/span><span data-contrast=\"none\">: <\/span><span style=\"color: #ff0000;\" data-contrast=\"none\">\"1\"<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span style=\"color: #0000ff;\" data-contrast=\"none\">memory<\/span><span data-contrast=\"none\">: <\/span><span style=\"color: #ff0000;\">2G\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 nvidia.com<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">gpu: <\/span><span style=\"color: #ff0000;\" data-contrast=\"none\">\"1\"<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span><\/pre>\n<p><span data-contrast=\"auto\">In our case we can be able to host 3 pods in a single node ip-10-20-23-199.us-west-1. compute. Internal and because of time slicing these 3 pods can use 3 virtual GPU\u2019s as below<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/p>\n<p><span data-ccp-props=\"{}\"> <img decoding=\"async\" loading=\"lazy\" class=\"wp-image-466838 aligncenter\" src=\"https:\/\/storage.googleapis.com\/blogs-images-new\/ciscoblogs\/1\/2025\/01\/Picture1-300x38.png\" alt=\"\" width=\"1097\" height=\"139\" srcset=\"https:\/\/storage.googleapis.com\/blogs-images-new\/ciscoblogs\/1\/2025\/01\/Picture1-300x38.png 300w, https:\/\/storage.googleapis.com\/blogs-images-new\/ciscoblogs\/1\/2025\/01\/Picture1-768x98.png 768w, https:\/\/storage.googleapis.com\/blogs-images-new\/ciscoblogs\/1\/2025\/01\/Picture1.png 936w\" sizes=\"auto, (max-width: 1097px) 100vw, 1097px\"\/><\/span><\/p>\n<p><span data-contrast=\"auto\">GPUs have been shared virtually among the pods, and we can see the PIDS assigned for each of the processes below.<\/span><span data-ccp-props=\"{}\">\u00a0<\/span><\/p>\n<p><span data-ccp-props=\"{}\"> <img decoding=\"async\" loading=\"lazy\" class=\"wp-image-466840 aligncenter\" src=\"https:\/\/storage.googleapis.com\/blogs-images-new\/ciscoblogs\/1\/2025\/01\/Picture2-300x143.png\" alt=\"\" width=\"831\" height=\"396\" srcset=\"https:\/\/storage.googleapis.com\/blogs-images-new\/ciscoblogs\/1\/2025\/01\/Picture2-300x143.png 300w, https:\/\/storage.googleapis.com\/blogs-images-new\/ciscoblogs\/1\/2025\/01\/Picture2-768x366.png 768w, https:\/\/storage.googleapis.com\/blogs-images-new\/ciscoblogs\/1\/2025\/01\/Picture2.png 874w\" sizes=\"auto, (max-width: 831px) 100vw, 831px\"\/><\/span><\/p>\n<p><span data-contrast=\"auto\">Now we optimized GPU at the pod level, let us now focus on optimizing GPU resources at the node level. We can achieve this by using a cluster autoscaling solution called <\/span><b><span data-contrast=\"auto\">Karpenter<\/span><\/b><span data-contrast=\"auto\">. This is particularly important as the learning labs may not always have a constant load or user activity, and GPUs are extremely expensive. By leveraging <\/span><b><span data-contrast=\"auto\">Karpenter<\/span><\/b><span data-contrast=\"auto\">, we can dynamically scale GPU nodes up or down based on demand, ensuring cost-efficiency and optimal resource utilization.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><b><span data-contrast=\"none\">Section 3: Node Autoscaling with Karpenter<\/span><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:281,&quot;335559739&quot;:281}\">\u00a0<\/span><\/p>\n<p><b><span data-contrast=\"auto\">Karpenter <\/span><\/b><span data-contrast=\"auto\">is an open-source node lifecycle management for Kubernetes. It automates provisioning and deprovisioning of nodes based on the scheduling needs of pods, allowing efficient scaling and cost optimization<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<ul>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"13\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><span data-contrast=\"auto\">Dynamic Node Provisioning: Automatically scales nodes based on demand.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"13\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><span data-contrast=\"auto\">Optimizes Resource Utilization: Matches node capacity with workload needs.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"13\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><span data-contrast=\"auto\">Reduces Operational Costs: Minimizes unnecessary resource expenses.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"13\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><span data-contrast=\"auto\">Improves Cluster Efficiency: Enhances overall performance and responsiveness.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/li>\n<\/ul>\n<p><b><i><span data-contrast=\"none\">Why Use Karpenter for Dynamic Scaling<\/span><\/i><\/b><span data-ccp-props=\"{}\">\u00a0<\/span><\/p>\n<ul>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"14\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Dynamic Scaling<\/span><\/b><span data-contrast=\"auto\">: Automatically adjusts node count based on workload demands.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"14\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Cost Optimization<\/span><\/b><span data-contrast=\"auto\">: Ensures resources are only provisioned when needed, reducing expenses.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/li>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"14\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Efficient Resource Management:<\/span><\/b><span data-contrast=\"auto\"> Tracks pods unable to be scheduled due to lack of resources, reviews their requirements, provisions nodes to accommodate them, schedules the pods, and decommissions nodes when redundant.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/li>\n<\/ul>\n<p><b><i><span data-contrast=\"none\">Installing Karpenter:<\/span><\/i><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:281,&quot;335559739&quot;:281}\">\u00a0<\/span><\/p>\n<pre><strong>\u00a0<\/strong><span style=\"color: #339966;\"><b>#Install Karpenter using HELM:&#13;\n<\/b><\/span><span data-contrast=\"none\">helm upgrade <\/span><span data-contrast=\"none\">--<\/span><span data-contrast=\"none\">install karpenter oci:<\/span><span data-contrast=\"none\">\/\/<\/span><span data-contrast=\"none\">public.ecr.aws<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">karpenter<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">karpenter <\/span><span data-contrast=\"none\">--<\/span><span data-contrast=\"none\">version <\/span><span data-contrast=\"none\">\"<span style=\"color: #ff0000;\">$<\/span><\/span><span style=\"color: #00ccff;\" data-contrast=\"none\">{KARPENTER_VERSION}<\/span><span data-contrast=\"none\">\"<\/span> &#13;\n<span data-contrast=\"none\">--<\/span><span data-contrast=\"none\">namespace <\/span><span data-contrast=\"none\">\"<span style=\"color: #ff0000;\">$<\/span><\/span><span style=\"color: #00ccff;\" data-contrast=\"none\">{KARPENTER_NAMESPACE}<\/span><span data-contrast=\"none\">\"<\/span> <span data-contrast=\"none\">--<\/span><span data-contrast=\"none\">create<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">namespace\u00a0\u00a0 <\/span><span data-contrast=\"none\">--<\/span><span data-contrast=\"none\">set<\/span> <span data-contrast=\"none\">\"<span style=\"color: #993300;\">settings.clusterName=<\/span><span style=\"color: #ff0000;\">$<\/span><\/span><span style=\"color: #00ccff;\" data-contrast=\"none\">{CLUSTER_NAME}<\/span><span data-contrast=\"none\">\"<\/span><span data-contrast=\"none\">\u00a0\u00a0\u00a0 &#13;\n<\/span><span data-contrast=\"none\">--<\/span><span data-contrast=\"none\">set<\/span> <span data-contrast=\"none\">\"<span style=\"color: #993300;\">settings.interruptionQueue=<\/span><span style=\"color: #ff0000;\">$<\/span><\/span><span style=\"color: #00ccff;\" data-contrast=\"none\">{CLUSTER_NAME}<\/span><span data-contrast=\"none\">\"<\/span><span data-contrast=\"none\">\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">--<\/span><span data-contrast=\"none\">set<\/span><span data-contrast=\"none\"> controller.resources.requests.cpu<\/span><span data-contrast=\"none\">=<\/span><span data-contrast=\"none\">1<\/span><span data-contrast=\"none\">\u00a0\u00a0\u00a0 &#13;\n<\/span><span data-contrast=\"none\">--<\/span><span data-contrast=\"none\">set<\/span><span data-contrast=\"none\"> controller.resources.requests.memory<\/span><span data-contrast=\"none\">=<\/span><span data-contrast=\"none\">1Gi<\/span><span data-contrast=\"none\">\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">--<\/span><span data-contrast=\"none\">set<\/span><span data-contrast=\"none\"> controller.resources.limits.cpu<\/span><span data-contrast=\"none\">=<\/span><span data-contrast=\"none\">1<\/span><span data-contrast=\"none\">\u00a0\u00a0\u00a0 &#13;\n<\/span><span data-contrast=\"none\">--<\/span><span data-contrast=\"none\">set<\/span><span data-contrast=\"none\"> controller.resources.limits.memory<\/span><span data-contrast=\"none\">=<\/span><span data-contrast=\"none\">1Gi<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n&#13;\n<span style=\"color: #339966;\">#Verify Karpenter Installation:\u00a0<\/span>&#13;\n<span data-contrast=\"none\">kubectl get pod <\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">n kube<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">system <\/span><span data-contrast=\"none\">|<\/span><span data-contrast=\"none\"> grep <\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">i karpenter<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">karpenter<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">7df6c54cc<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">rsv8s\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">1<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">1<\/span><span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0 Running\u00a0\u00a0 <\/span><span data-contrast=\"none\">2<\/span><span data-contrast=\"none\"> (<\/span><span data-contrast=\"none\">10d<\/span><span data-contrast=\"none\"> ago)\u00a0\u00a0 <\/span><span data-contrast=\"none\">53d<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">karpenter<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">7df6c54cc<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">zrl9n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">1<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">1<\/span><span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0 Running\u00a0\u00a0 <\/span><span data-contrast=\"none\">0<\/span><span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">53d<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span><\/pre>\n<p><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><b><i><span data-contrast=\"none\">Configuring Karpenter with NodePools and NodeClasses:\u00a0<\/span><\/i><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;201341983&quot;:0,&quot;335551550&quot;:1,&quot;335551620&quot;:1,&quot;335559685&quot;:0,&quot;335559737&quot;:0,&quot;335559738&quot;:281,&quot;335559739&quot;:281,&quot;335559740&quot;:279}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">Karpenter can be configured with <\/span><b><span data-contrast=\"auto\">NodePools <\/span><\/b><span data-contrast=\"auto\">and <\/span><b><span data-contrast=\"auto\">NodeClasses <\/span><\/b><span data-contrast=\"auto\">to automate the provisioning and scaling of nodes based on the specific needs of your workloads<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<ul>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"15\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Karpenter NodePool<\/span><\/b><span data-contrast=\"auto\">: Nodepool is a custom resource that defines a set of nodes with shared specifications and constraints in a Kubernetes cluster. Karpenter uses NodePools to dynamically manage and scale node resources based on the requirements of running workloads<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/li>\n<\/ul>\n<pre><span data-contrast=\"none\">apiVersion<\/span><span data-contrast=\"none\">: karpenter.sh<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">v1beta1<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">kind<\/span><span data-contrast=\"none\">: NodePool<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">metadata<\/span><span data-contrast=\"none\">:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0 <\/span><span data-contrast=\"none\">name<\/span><span data-contrast=\"none\">: g4<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">nodepool<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">spec<\/span><span data-contrast=\"none\">:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0 <\/span><span data-contrast=\"none\">template<\/span><span data-contrast=\"none\">:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">metadata<\/span><span data-contrast=\"none\">:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">labels<\/span><span data-contrast=\"none\">:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 nvidia.com<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">gpu: <\/span><span data-contrast=\"none\">\"<\/span><span data-contrast=\"none\">true<\/span><span data-contrast=\"none\">\"<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0 spec:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">taints<\/span><span data-contrast=\"none\">:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\"> effect: NoSchedule<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">key<\/span><span data-contrast=\"none\">: nvidia.com<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">gpu<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">value<\/span><span data-contrast=\"none\">: <\/span><span data-contrast=\"none\">\"<\/span><span data-contrast=\"none\">true<\/span><span data-contrast=\"none\">\"<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0 requirements:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">-<\/span> <span data-contrast=\"none\">key<\/span><span data-contrast=\"none\">: kubernetes.io<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">arch<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">operator<\/span><span data-contrast=\"none\">: In<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">values<\/span><span data-contrast=\"none\">: [<\/span><span data-contrast=\"none\">\"amd64\"<\/span><span data-contrast=\"none\">]<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">-<\/span> <span data-contrast=\"none\">key<\/span><span data-contrast=\"none\">: kubernetes.io<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">os<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">operator<\/span><span data-contrast=\"none\">: In<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">values<\/span><span data-contrast=\"none\">: [<\/span><span data-contrast=\"none\">\"linux\"<\/span><span data-contrast=\"none\">]<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">-<\/span> <span data-contrast=\"none\">key<\/span><span data-contrast=\"none\">: karpenter.sh<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">capacity<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">type<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">operator<\/span><span data-contrast=\"none\">: In<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">values<\/span><span data-contrast=\"none\">: [<\/span><span data-contrast=\"none\">\"on-demand\"<\/span><span data-contrast=\"none\">]<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">-<\/span> <span data-contrast=\"none\">key<\/span><span data-contrast=\"none\">: node.kubernetes.io<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">instance<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">type<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">operator<\/span><span data-contrast=\"none\">: In<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">values<\/span><span data-contrast=\"none\">: [<\/span><span data-contrast=\"none\">\"g4dn.xlarge\"<\/span><span data-contrast=\"none\"> ]<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0 nodeClassRef:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">apiVersion<\/span><span data-contrast=\"none\">: karpenter.k8s.aws<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">v1beta1<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">kind<\/span><span data-contrast=\"none\">: EC2NodeClass<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">name<\/span><span data-contrast=\"none\">: g4<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">nodeclass<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0 limits:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">cpu<\/span><span data-contrast=\"none\">: <\/span><span data-contrast=\"none\">1000<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0 <\/span><span data-contrast=\"none\">disruption<\/span><span data-contrast=\"none\">:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">expireAfter<\/span><span data-contrast=\"none\">: <\/span><span data-contrast=\"none\">120m<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">consolidationPolicy<\/span><span data-contrast=\"none\">: WhenUnderutilized<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span><\/pre>\n<ul>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"15\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"2\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">NodeClasses<\/span><\/b><span data-contrast=\"auto\"> are configurations that define the characteristics and parameters for the nodes that Karpenter can provision in a Kubernetes cluster. A NodeClass specifies the underlying infrastructure details for nodes, such as instance types, launch template configurations and specific cloud provider settings.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/li>\n<\/ul>\n<p><b><i><span data-contrast=\"none\">Note: The userData section contains scripts to bootstrap the EC2 instance, including pulling a TensorFlow GPU Docker image and configuring the instance to join the Kubernetes cluster.<\/span><\/i><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559685&quot;:0,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/p>\n<pre><span data-contrast=\"none\">apiVersion: karpenter.k8s.aws<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">v1beta1<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">kind<\/span><span data-contrast=\"none\">: EC2NodeClass<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">metadata<\/span><span data-contrast=\"none\">:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0 <\/span><span data-contrast=\"none\">name<\/span><span data-contrast=\"none\">: g4<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">nodeclass<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">spec<\/span><span data-contrast=\"none\">:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0 <\/span><span data-contrast=\"none\">amiFamily<\/span><span data-contrast=\"none\">: AL2<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0 <\/span><span data-contrast=\"none\">launchTemplate<\/span><span data-contrast=\"none\">:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">name<\/span><span data-contrast=\"none\">: <\/span><span data-contrast=\"none\">\"ack_nodegroup_template_new\"<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">version<\/span><span data-contrast=\"none\">: <\/span><span data-contrast=\"none\">\"7\"<\/span><span data-contrast=\"none\">\u00a0<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0 <\/span><span data-contrast=\"none\">role<\/span><span data-contrast=\"none\">: <\/span><span data-contrast=\"none\">\"KarpenterNodeRole\"<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0 <\/span><span data-contrast=\"none\">subnetSelectorTerms<\/span><span data-contrast=\"none\">:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\"> tags:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 karpenter.sh<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">discovery: <\/span><span data-contrast=\"none\">\"nextgen-learninglab\"<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0 securityGroupSelectorTerms:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\"> tags:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 karpenter.sh<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">discovery: <\/span><span data-contrast=\"none\">\"nextgen-learninglab\"<\/span><span data-contrast=\"none\">\u00a0\u00a0\u00a0 <\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0 <\/span><span data-contrast=\"none\">blockDeviceMappings<\/span><span data-contrast=\"none\">:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\"> deviceName: <\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">dev<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">xvda<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">ebs<\/span><span data-contrast=\"none\">:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">volumeSize<\/span><span data-contrast=\"none\">: <\/span><span data-contrast=\"none\">100Gi<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">volumeType<\/span><span data-contrast=\"none\">: gp3<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">iops<\/span><span data-contrast=\"none\">: <\/span><span data-contrast=\"none\">10000<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">encrypted<\/span><span data-contrast=\"none\">: <\/span><span data-contrast=\"none\">true<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">deleteOnTermination<\/span><span data-contrast=\"none\">: <\/span><span data-contrast=\"none\">true<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">throughput<\/span><span data-contrast=\"none\">: <\/span><span data-contrast=\"none\">125<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0 tags:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">Name<\/span><span data-contrast=\"none\">: Learninglab<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">Staging<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">Auto<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">GPU<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">Node<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0 <\/span><span data-contrast=\"none\">userData<\/span><span data-contrast=\"none\">: <\/span><span data-contrast=\"none\">|<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 MIME<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">Version: <\/span><span data-contrast=\"none\">1.0<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Content<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">Type: multipart<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">mixed; <\/span><span data-contrast=\"none\">boundary<\/span><span data-contrast=\"none\">=<\/span><span data-contrast=\"none\">\"\/\/\"<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">--<\/span><span data-contrast=\"none\">\/\/<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Content<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">Type: text<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">x<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">shellscript; <\/span><span data-contrast=\"none\">charset<\/span><span data-contrast=\"none\">=<\/span><span data-contrast=\"none\">\"us-ascii\"<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">set<\/span> <span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">ex<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 sudo ctr <\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">n<\/span><span data-contrast=\"none\">=<\/span><span data-contrast=\"none\">k8s.io image pull docker.io<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">tensorflow<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">tensorflow:<\/span><span data-contrast=\"none\">2.12<\/span><span data-contrast=\"none\">.0<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">gpu<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">--<\/span><span data-contrast=\"none\">\/\/<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Content<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">Type: text<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">x<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">shellscript; <\/span><span data-contrast=\"none\">charset<\/span><span data-contrast=\"none\">=<\/span><span data-contrast=\"none\">\"us-ascii\"<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">B64_CLUSTER_CA<\/span><span data-contrast=\"none\">=<\/span><span data-contrast=\"none\">\" \"<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">API_SERVER_URL<\/span><span data-contrast=\"none\">=<\/span><span data-contrast=\"none\">\"\"<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">etc<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">eks<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">bootstrap.sh nextgen<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">learninglab-eks <\/span><span data-contrast=\"none\">--<\/span><span data-contrast=\"none\">kubelet<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">extra<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">args <\/span><span data-contrast=\"none\">'--node-labels=eks.amazonaws.com\/capacityType=ON_DEMAND &#13;\n--pod-max-pids=32768 --max-pods=110'<\/span> <span data-contrast=\"none\">--<\/span><span data-contrast=\"none\"> b64<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">cluster<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">ca <\/span><span data-contrast=\"none\">$<\/span><span data-contrast=\"none\">B64_CLUSTER_CA <\/span><span data-contrast=\"none\">--<\/span><span data-contrast=\"none\">apiserver<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">endpoint <\/span><span data-contrast=\"none\">$<\/span><span data-contrast=\"none\">API_SERVER_URL <\/span><span data-contrast=\"none\">--<\/span><span data-contrast=\"none\">use<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">max<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">pods false<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">--<\/span><span data-contrast=\"none\">\/\/<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Content<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">Type: text<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">x<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">shellscript; <\/span><span data-contrast=\"none\">charset<\/span><span data-contrast=\"none\">=<\/span><span data-contrast=\"none\">\"us-ascii\"<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">KUBELET_CONFIG<\/span><span data-contrast=\"none\">=\/<\/span><span data-contrast=\"none\">etc<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">kubernetes<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">kubelet<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">kubelet<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">config.json<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 echo <\/span><span data-contrast=\"none\">\"$(jq \"<\/span><span data-contrast=\"none\">.podPidsLimit<\/span><span data-contrast=\"none\">=<\/span><span data-contrast=\"none\">32768<\/span><span data-contrast=\"none\">\" $KUBELET_CONFIG)\"<\/span> <span data-contrast=\"none\">&gt;<\/span> <span data-contrast=\"none\">$<\/span><span data-contrast=\"none\">KUBELET_CONFIG<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span><span data-contrast=\"none\">--<\/span><span data-contrast=\"none\">\/\/<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Content<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">Type: text<\/span><span data-contrast=\"none\">\/<\/span><span data-contrast=\"none\">x<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">shellscript; <\/span><span data-contrast=\"none\">charset<\/span><span data-contrast=\"none\">=<\/span><span data-contrast=\"none\">\"us-ascii\"<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 systemctl stop kubelet<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\">\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 systemctl daemon<\/span><span data-contrast=\"none\">-<\/span><span data-contrast=\"none\">reload<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335557856&quot;:2039583,&quot;335559738&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:285}\">\u00a0<\/span>&#13;\n<span data-contrast=\"none\"> \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 systemctl start kubelet&#13;\n        --\/\/--<\/span><\/pre>\n<p><span data-contrast=\"auto\">In this scenario, each node (e.g., <\/span><span data-contrast=\"auto\">ip-10-20-23-199.us-west-1.compute.internal<\/span><span data-contrast=\"auto\">) can accommodate up to three pods. If the deployment is scaled to add another pod, the resources will be insufficient, causing the new pod to remain in a pending state.\u00a0<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/p>\n<p><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\"> <img decoding=\"async\" loading=\"lazy\" class=\"wp-image-466842 aligncenter\" src=\"https:\/\/storage.googleapis.com\/blogs-images-new\/ciscoblogs\/1\/2025\/01\/Picture3-300x69.png\" alt=\"\" width=\"943\" height=\"217\" srcset=\"https:\/\/storage.googleapis.com\/blogs-images-new\/ciscoblogs\/1\/2025\/01\/Picture3-300x69.png 300w, https:\/\/storage.googleapis.com\/blogs-images-new\/ciscoblogs\/1\/2025\/01\/Picture3-768x177.png 768w, https:\/\/storage.googleapis.com\/blogs-images-new\/ciscoblogs\/1\/2025\/01\/Picture3.png 936w\" sizes=\"auto, (max-width: 943px) 100vw, 943px\"\/>\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">Karpenter monitors these Un schedulable pods and assesses their resource requirements to act accordingly. There will be nodeclaim which claims the node from the nodepool and Karpenter thus provision a node based on the requirement.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/p>\n<p><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\"><img decoding=\"async\" loading=\"lazy\" class=\"wp-image-466843 aligncenter\" src=\"https:\/\/storage.googleapis.com\/blogs-images-new\/ciscoblogs\/1\/2025\/01\/Picture4-300x40.png\" alt=\"\" width=\"991\" height=\"132\" srcset=\"https:\/\/storage.googleapis.com\/blogs-images-new\/ciscoblogs\/1\/2025\/01\/Picture4-300x40.png 300w, https:\/\/storage.googleapis.com\/blogs-images-new\/ciscoblogs\/1\/2025\/01\/Picture4-768x103.png 768w, https:\/\/storage.googleapis.com\/blogs-images-new\/ciscoblogs\/1\/2025\/01\/Picture4.png 936w\" sizes=\"auto, (max-width: 991px) 100vw, 991px\"\/><\/span><\/p>\n<p><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><b><i><span data-contrast=\"none\">Conclusion: Efficient GPU Resource Management in Kubernetes<\/span><\/i><\/b><span data-ccp-props=\"{}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">With the growing demand for GPU-accelerated workloads in Kubernetes, managing GPU resources effectively is essential. The combination of <\/span><b><span data-contrast=\"auto\">NVIDIA Device Plugin<\/span><\/b><span data-contrast=\"auto\">, <\/span><b><span data-contrast=\"auto\">time slicing<\/span><\/b><span data-contrast=\"auto\">, and <\/span><b><span data-contrast=\"auto\">Karpenter<\/span><\/b><span data-contrast=\"auto\"> provides a powerful approach to manage, optimize, and scale GPU resources in a Kubernetes cluster, delivering high performance with efficient resource utilization. This solution has been implemented to host pilot GPU-enabled Learning Labs on <\/span><span data-contrast=\"none\">developer.cisco.com\/learning<\/span><span data-contrast=\"auto\">, providing GPU-powered learning experiences.<\/span><\/p>\n<p>Share:<\/p>\n<p>\n  \t<\/div>\n<p><p><a href=\"https:\/\/dmsretail.com\/online-workshops-list\/\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-496\" src=\"https:\/\/dmsretail.com\/RetailNews\/wp-content\/uploads\/2022\/05\/RETAIL-ONLINE-TRAINING-728-X-90.png\" alt=\"Retail Online Training\" width=\"729\" height=\"91\" srcset=\"https:\/\/dmsretail.com\/RetailNews\/wp-content\/uploads\/2022\/05\/RETAIL-ONLINE-TRAINING-728-X-90.png 729w, https:\/\/dmsretail.com\/RetailNews\/wp-content\/uploads\/2022\/05\/RETAIL-ONLINE-TRAINING-728-X-90-300x37.png 300w\" sizes=\"auto, (max-width: 729px) 100vw, 729px\" \/><\/a><\/p><br \/><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Introduction: Overcoming GPU Management Challenges\u00a0\u00a0 In Part 1 of this blog series, we explored the challenges of hosting large language models (LLMs) on CPU-based workloads [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":14622,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5],"tags":[],"class_list":["post-14621","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology"],"_links":{"self":[{"href":"https:\/\/dmsretail.com\/RetailNews\/wp-json\/wp\/v2\/posts\/14621","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dmsretail.com\/RetailNews\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dmsretail.com\/RetailNews\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dmsretail.com\/RetailNews\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dmsretail.com\/RetailNews\/wp-json\/wp\/v2\/comments?post=14621"}],"version-history":[{"count":0,"href":"https:\/\/dmsretail.com\/RetailNews\/wp-json\/wp\/v2\/posts\/14621\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/dmsretail.com\/RetailNews\/wp-json\/wp\/v2\/media\/14622"}],"wp:attachment":[{"href":"https:\/\/dmsretail.com\/RetailNews\/wp-json\/wp\/v2\/media?parent=14621"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dmsretail.com\/RetailNews\/wp-json\/wp\/v2\/categories?post=14621"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dmsretail.com\/RetailNews\/wp-json\/wp\/v2\/tags?post=14621"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}