Run GPU based docker on AWS EC2

 

In this post we will review the steps to run a GPU based docker container on AWS EC2.

A. Launch EC2 Instance

The first step is to launch a new EC2 instance, however, there some issues to notice.

First we should select a suitable AMI that includes the drivers to enable the GPU usage. The best match I found is to select Ubuntu with AMI: Deep Learning Base OSS Nvidia GPU AMI.

I wanted the cheapest instance type that includes GPU, and selected the g4dn.xlarge

One more thing is to configure a larger storage than the default 75G, since most LLM models and python libraries require a lot of disk space.

We will need a way to connect to the instance, so we probably need to configure security group that allows our IPs to connect to the EC2 instance using SSH, and allocate a public IPv4 to the instance.

B. Run The Docker Container

Once the EC2 instance is up, connect to instance using SSH, and install docker support for GPU.

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo systemctl restart docker

Now we can build our docker image, and run it with a flag enabling it to use the GPU. 

We would probably want to expose the relevant ports as well.

docker run --rm --name models -it -p 0.0.0.0:9090:9090 --gpus all  my-image:latest

That's all, our GPU base container is up and serving requests.