When the GPU work is on a client's data, this is where I go on AWS. I trained deep neural networks on it for an Australian agriculture-technology company: weather forecasts built from years of its own sensor data, GPU training on demand, and nothing running between experiments (the project). AWS now calls the service SageMaker AI.
The value isn't the GPU, which costs about 40% more than the same EC2 instance. It's everything around it, inside the client's account: the data never leaves S3, a training job starts, bills by the second and stops on its own, an endpoint autoscales or scales to zero, and IAM, the VPC and the compliance scope (IRAP PROTECTED, HIPAA) come with it. That's what a client's production system needs, and what a GPU marketplace can't give it.
For my own research and experiments I don't use it: Runpod rents the same card for far less, rents cards AWS doesn't, and has no quota to wait for. Studio is also a poor home for an agent that installs things all day: its Docker can't run privileged containers or map ports. The usual path is both: prototype on Runpod, then train and serve on SageMaker once the method works. The numbers are in Runpod or AWS for a GPU?
Use it when
- Training or fine-tuning on a client's data: the data stays in S3 in the client's account, a training job bills per second while it runs, and nothing runs between experiments.
- The work must stay in Australia: in Sydney, SageMaker has T4, A10G and L4 instances, and A100 and H100 machines.
- Inference in a client's production system: real-time endpoints that autoscale, and asynchronous endpoints for batch work (up to 1 GB a request and an hour of processing) that scale to zero when idle.
- Australian government or health work: SageMaker is in scope for IRAP PROTECTED in Sydney and Melbourne, and HIPAA-eligible.
Don't use it when
- Research and experiments with an agent working in the box: Studio's Docker can't run privileged containers, use Docker networks or map ports, and it costs about 40% more than the same EC2 instance. Runpod, or EC2 when the data must stay in the account.
- A model you can call through an API: Bedrock or OpenRouter, with no GPU to manage.
- Serverless inference on a GPU: SageMaker Serverless Inference has none.
Pricing
us-east-1, on demand: ml.g6.xlarge (L4, 24 GB) $1.127 an hour, ml.g6e.xlarge (L40S, 48 GB) $2.61, ml.p5.4xlarge (H100) $7.91; Sydney ml.g6.xlarge $1.465. That's 40% over the same EC2 instance for the 24 and 48 GB cards, and 15% for the H100. Billed per second. SageMaker Savings Plans take up to 64% off. Checked 1 November 2026.
Gotchas, with fixes
Gotcha: Every SageMaker GPU quota starts at 0 in a new account, per instance type and per use: Studio, notebooks, training, endpoints.
Fix: Request the exact instance and use on day one (ml.g6.xlarge for training job usage, for example). Approval takes from a day to over a week.
Gotcha: A Studio space or a notebook instance left running bills by the hour, like any instance.
Fix: Set an idle shutdown on the domain (JupyterLab and Code Editor support it), and run experiments as training jobs, which stop by themselves.
Gotcha: An endpoint scaled to zero answers the first request with a capacity error while an instance starts (NoCapacityInvocationFailures).
Fix: Use an asynchronous endpoint for batch work, which queues the request, and keep one instance running for traffic that can't wait.
Gotcha: Serverless Inference has no GPUs, and caps memory at 6 GB.
Fix: For GPU inference that scales to zero, use an asynchronous endpoint, or a real-time endpoint with inference components.
Provisioning
Official Pulumi package · Official Terraform provider
aws.sagemaker.Domain, Space, Model, EndpointConfiguration and Endpoint in @pulumi/aws, and the same resources in the hashicorp/aws provider. Training jobs aren't infrastructure: the code starts them.
Instead, consider
- Runpod Research and experiments on data that isn't the client's: the same card for far less, and no quota to wait for.
- EC2 A full GPU VM in the account, or inference running 24/7 on an EC2 Savings Plan.
- Amazon Bedrock A model behind an API: no GPU to run.
Decision guides
The concept behind it
In our client work
Where it fits
- A GPU for models and experiments: Pick 02.
When the data is the client's: training jobs and endpoints inside its account, Sydney included.
Sources
- SageMaker AI pricing
- SageMaker AI quotas
- SageMaker asynchronous inference
- SageMaker inference endpoints scale down to zero
- SageMaker Serverless Inference: feature exclusions
- SageMaker Studio: idle shutdown
- AWS services in scope for IRAP
Last reviewed by Nicolas Dao