AWS adds NVIDIA GPU partitioning to SageMaker – Unite.AI inference endpoints

Amazon Web Services (AWS) announced NVIDIA multi-instance GPU support for Amazon SageMaker Inference on November 6, 2026, allowing multiple hardware-isolated models to share an accelerator across real-time endpoints instead of assigning each deployment an entire GPU.

The feature solves two GPU-wide allocation constraints. Accelerator counting limits the number of models an instance can host, regardless of how much memory or compute each model uses, and customers considering a newer GPU must allocate the entire device. Multi-instance GPU (MIG) instead assigns each partition a fixed quota of compute units and a dedicated memory block, with isolation applied in hardware. Because each model remains resident, the arrangement also avoids the delay that occurs when a multi-model endpoint loads a model onto its shared device for first use.

Partitioning is limited to real-time endpoints that use inference components, SageMaker’s construct to deploy several models to an endpoint and set the computation received by each. Single-model endpoints, serverless endpoints, and heterogeneous instance pools cannot be partitioned. Asynchronous workloads are also out of scope of the feature because the asynchronous call cannot address an inference component.

Salesforce helped define the requirements and validate the initial implementation. Richard Chang, a software engineering architect at Salesforce, said the company expects MIG to securely consolidate workloads, improve GPU utilization and reduce inference costs while preserving predictable performance and isolation. Salesforce is preparing to validate the software with representative production workloads, so such benefits are expectations rather than reported production results.

Two API fields control partitioning

The SageMaker AI developer documentation defines two optional fields: AcceleratorPartitionLayouts declares the carving on a production variant, e AcceleratorPartitionType selects the profile used by an inference component. There are no new API operations, resource types, or AWS Identity and Access Management operations, and omitting both fields preserves the behavior of the existing endpoint.

The selected layout applies uniformly to each accelerator on each instance in the variant. The AWS example divides each of the eight A100s onto a ml.p4d.24xlarge in mig-4g.20gb, mig-2g.10gb AND mig-1g.5gb partitions, producing 24 partitions on the endpoint. Each copy of the inference component occupies a required partition, while InvokeEndpoint continues to address models by component name without a partition identifier. In its testing, AWS said carving the accelerators added about 17 seconds to provisioning.

The isolation covers the compute and memory of the GPU, not the entire host. Models on an instance still share host CPU, host memory, storage, networking, and endpoint routing. A component requesting a profile absent from the layout is rejected rather than receiving an entire GPU. Changing the layout of a running endpoint requires a new endpoint and traffic migration; moving a component to another profile requires a new component.

AWS tests an H100 partition against a full A100 partition

AWS compared one mig-3g.40gb H100 partition with a full A100-40GB, using Llama-3.1-8B-Instruct on production SageMaker real-time endpoints. Both parties used identical model replication, containers, tokenizers, and client configurations, with genai-bench driving five forms of traffic at concurrency levels 2 to 512. NVIDIA’s profile guide assigns the tested H100 profile three of seven streaming multiprocessor shares and four of eight memory drives. The two configurations have comparable memory capacities but different compute allocations.

In testing, AWS reported 53% to 89% more output tokens per second from the H100 partition to the load where the cost benefit of each scenario was greatest. Latency between tokens was 8.9% to 42.1% lower, and time to first token was 5.6% to 47.5% lower. At maximum concurrency on 6,000 and 8,000 token workloads, the A100 missed its service deadline in one-third to one-half of requests while the partition failed none.

AWS’s cost analysis, presented as a case-by-case comparison rather than a universal verdict, found an 18% to 34% lower cost per million output tokens once traffic exceeded a workload-specific break-even point. This point occurred at concurrency 64 for the 100-token prompts and at concurrency 4 for the 8,000-token prompts. Below that, the entire A100 remained cheaper, leaving short-input or low-concurrency workloads potentially better suited to the older entire GPU. The unit cost comparison divided On-Demand Instance prices by the number of units tested on each instance.

Supported hardware and operational limits

The launch covers four instance families, with partition counts and profiles determined by the accelerator:

GPU partitioning is available in US East (N. Virginia), US East (Ohio), US West (Oregon), Asia Pacific (Singapore), Asia Pacific (Tokyo), Europe (Frankfurt), and Europe (Ireland), subject to regional instance availability and endpoint quota. No partitions span accelerators, so models that require a full GPU or multiple GPUs still need full devices.

Customers continue to pay for the full GPU instance; partitioning can increase usable capacity but does not reduce the hourly price. Additionally, partitioned endpoints and their inference components do not publish GPU, CPU, or memory usage metrics to CloudWatch, although invocation and latency metrics remain available. On P4d, the mig-1g.10gb The profile requires the NVIDIA R525 or newer driver, and AWS directs customers to a newer inference AMI because the default image does not provide it.

As for capacity planning, the launch turns each supported accelerator into different fixed GPU sizes, but a denser, hardware-isolated deployment features full instance billing, layouts that can’t be changed in-place, and reduced usage telemetry. The AWS benchmark shows the value of measuring actual traffic: in these tests the H100 partition provided performance and cost advantages under heavier and longer context conditions, while break-even results left an entire A100 cheaper for lighter workloads.