Docker and Virtualization
Docker is a tool we use extensively at Triton UAS, and in general it and similar container tools are very useful for making a consistent development or production environment in which a program can run. If you have ever tried to run a program that worked on your friend's computer but not yours, then you have felt the pain that Docker can solve.
Setup
To get setup with Docker we will download Docker and run a simple "Hello World" container.
Install Docker
If you don't already have Docker installed, follow the steps on this page for your operating system.
To check if you have it installed, open a terminal and run docker --version. On Windows you can open Command Prompt, Powershell or the Windows Terminal. On macOS you can use the built in Terminal application. On Linux you can use whatever terminal app your distribution ships with.
Once you have Docker installed, you should see something like this:
Note for Mac users
Make sure to download the correct version of Docker Desktop for your CPU architecture. If you have an Apple Sillicon processor (such as the M1 or M2 chips), download the Apple Sillicon version. If you have an older Mac with an Intel chip, download the Intel version. You can check this by clicking on the Apple logo on the top left and clicking on "About this Mac".
Note for Linux users
If you previously had Docker Engine installed, you dont need to install Docker Desktop for Linux. Read this page if you're interested in learning the differences. You can continue to use the same commands mentioned in this tutorial. If you would like to use one over the other you can switch using the following commands:
Run docker context ls to see your installed docker contexts:
NAME DESCRIPTION DOCKER ENDPOINT ERROR
default * Current DOCKER_HOST based configuration unix:///var/run/docker.sock
desktop-linux Docker Desktop unix:///home/atarbinian/.docker/desktop/docker.sock
Then do docker context use <context-name> to switch between them.
For example, if you wanted to switch to Docker Desktop:
Hello World
To test that our Docker installation works, we're going to pull a simple image and run it as a container.
First, open a terminal window on your computer.
- Mac: Open the built-in Terminal application
- Windows: Open either Command Prompt, Powershell or Windows Terminal.
- Linux: Open any terminal emulator that you have installed. Most distributions ship with one.
Next, run the following command:
You should see Docker pull the image and print out the following message:
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
(amd64)
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/
For more examples and ideas, visit:
https://docs.docker.com/get-started/
If you see this message, you're good to move onto the next steps. If not, ask a software lead for help.
Deeper Dive
What is Docker?
As mentioned at the start, Docker is a containerization technology that creates an environment for isolated applications to run in. Think of if you started up a program on your computer that was isolated to it's own sandbox. The application would have it's own file system and have no knowledge of the host system. As far as it's concerned, it's running in this isolated environment.
Why would we want this? Doesn't this sound like it just increases the complexity of running our applciation?
To name a few benefits, Docker provides us isolation, reproducibility, and security.
-
Isolation
- When you are running an application within a Docker container, it exists within the isolated environment of the container. This means that the application can only modify the files within the container. If it modifies some system path or needs some extra configuration, those changes are isolated to the sandbox of the container and will not pollute your host system.
-
Reproducibility
- With Docker images we can share the same environemnt between multiple hosts, regardless of the host's operating system and configuration. This means that you don't need to run any setup on the host system. The Docker container should execute the same way regardless of where it is running.
-
Security
- As mentioned, the containers are running inside their own isolated environment. This means that if there was a malicuous application, it would only be able to compromise the container itself and not your host system.
-
Performance
- Since each container is not running as a virtual machine, the performance is generally better than a virtual machine.
Image vs Container?
So far we've been talking about images and containers without explaining them. You might think these are interchangable terms, however they are completely different entities.
Image
An image is where we can define what should be included in the sandboxed environment. It includes an operating system, any system packages, other configuration, and any application code/binaries. You can think of them as a filesystem that includes everything you need to run your application.
Let's say I want to create an image for my web server. Well, the image must include the binary or code of the server itself. The web server can't run on it's own. It needs to run within a operating system. We might also have some configuration file that details how the server should work. One can encapsulate all these things into a Docker image and we will explore exactly how to do that in the section on building a Custom Image.
Container
A container is the runtime environemnt where an application runs within. It takes the specification described by the image and actually starts up the required processes and sets up the specified filesystem.
Ubuntu Container
Let's try to run a more complicated container than the hello-world one.
Ubuntu is a popular distribution of Linux that publishes Docker images that we can run as containers.
Let's pull the Ubuntu image and run it as a container.
To pull the Ubuntu image we can use the docker pull command:
Let's figure out what's happening here.
docker pull will pull any given image from a registry. A registry is just a place where images are hosted on the Internet. By default, Docker will pull images from an image registry called Docker Hub.
Next we specify the argument ubuntu:latest. This means that we want to pull the Ubuntu iamge that exists on Docker Hub. Here is the page for the Ubuntu image. On Docker Hub you can find many types of images that will have various pieces of software available.
The next part of this argument after the colon, is the tag. A tag is a way of speciying what type of image we want. In this case we want the latest tag. If you visit the Docker Hub page for the Ubuntu image, you'll see many other tags that are available to use.
Now that we've pulled our image, let's verify that Docker has it stored on our computer. You can see all Docker images that are availalbe locally with the docker images command.
You might see other images on your computer if you've used Docker before. If you pulled hello-world from earlier, it should show up here as well.
Now that we've pulled this image, we can start up a container.
To run a container, we can use the docker run command.
What's happening in this command? Let's dissect every part of the command.
The first flag --rm is to tell Docker to clean up the container after we are finished running it. This isn't strictly necessary but it just makes our lives easier once we're done with the container.
The next two flags --interactive and --tty gives us a terminal interface to interact with the container. You could abbreviate this to just -it and your command would be:
Next, we specify the image name and the image tag. This should be familiar since it's the same syntax we used when running docker pull to download the image earlier.
Finally we have the argument /bin/bash. This just specifies what program the container should run. In this case, we specify it to run the bash shell. /bin/bash is just an absolute file path to the bash executable program. If you need a reminder, a shell is a way for humans to interface with the underlying system. We can issue text-based commands to have programs be executed. All the Docker commands we've been running have been input to a shell.
If all went well, you should see something like this:
This is our shell prompt where we can issue commands inside the container. Feel free to poke around the container to see what's available. If you forgot how to use some of the command line programs, feel free to check back at our Unix Basics workshop.
Let's run a few commands to see what's inside the container.
bin boot dev etc home lib lib32 lib64 libx32 media mnt opt proc root run sbin srv sys tmp usr var
ls at the root directory should show us that we have what looks like a standard Linux file system.
Running ls in the home directory should show us nothing in an unmodified container.
Once you're done playing around, you can close the container by just hitting Ctrl-C and close it like any other program. If for some reason you didn't run the command with the same flags provided you might end up having the container not shut down even after hitting Ctrl-C. You can also kill the container by exiting the bash process. To do so, you can type the exit command or hit Ctrl-D. You can also kill it with the keyboard combination Ctrl-P and Ctrl-Q. To force kill the container, run docker ps to get the container ID and then run docker stop <container-id> to kill it.
Ok so, it seems we have succesfully started up an Ubuntu container. As mentioned previously, this is useful for isolating our application from our host system. It has it's own Linux sandbox where it can do whatever it wants and not endanger our host system.
Custom Image
Let's do something more interesting than just a base Ubuntu image. Let's define our own image to ping the TUAS website.
Find a directory on your host system where you're ok with creating a new file. In this directory, create a new file called Dockerfile.
Open this Dockerfile in your favorite text editor. This can be VS Code, Nano, Vim, TextEdit or even Notepad.
This Dockerfile will be the definition of the custom image that we'll be creating. If you remember what an image is, it's a predefined representation of what's included in our sandboxed environment. We will define the image's underlying operating system, any extra packages and other configuration.
On the first line of the Dockerfile write the following line:
ThisFROM line defines what image our custom image should be based on. In this case, we are basing our image on top of Ubuntu. This means that we will inherit all the files and resources associated with the Ubuntu docker image. We didn't have to base our image on top of Ubuntu, we could have chosen any other image. One popular choice is Alpine Linux which is a more minimal Linux distribution than Ubuntu. There are other popular base images that provide tools for working with certain programming languages such as the Python and Golang images.
Now that we have our base image, let's install any packages we might need.
These RUN lines will run any commands we provide and modify the contents of the image that's being built. In this case we run the two commands apt-get update and apt-get install -y iputils-ping. What these commands do is not crucial to understanding Docker. These are just the way that we obtain packages on Ubuntu and other Debian-based Linux distributions. The first line update's the system's list of packages by checking the Ubuntu servers for any updates. The second command will install the package that we specify, in this case iputils-ping which will give us the ping command we care about.
Finally, let's finish our image definition by specifying what action the image should perform when it's run as a container. As we mentioned, we want this container to ping the TUAS website. We can specify what application should be run in the container with the CMD keyword. Here we run ping with the arguments -c 10 which will ping the TUAS website 10 times and exit.
The final Dockerfile should look like this:
FROM ubuntu:latest
RUN apt-get update
RUN apt-get install -y iputils-ping
CMD ping -c 10 tuas.ucsd.edu
Now that we've defined our image in a Dockerfile, we must build it to create the image itself. This can be done with the docker build command.
Open your terminal and ensure that you're in the same directory as the Dockerfile that you just created.
The --tag custom-ping is how we specify the name of the image and any additional tags. Remember how when we were pulling ubuntu, we pulled it as ubuntu:latest. ubuntu was the image name and latest was the tag. Here we are tagging our image to have the name custom-ping. We don't explicitly specify a tag but it should have a default tag of latest.
The next part is the . which specifies the build context directory. The build context is used by the build process to know where to refer to paths on the host system from. We didn't go over them, but there are parts of the Dockerfile which can interact with files on the host system. Those lines would use paths that are relative from the build context.
Now that the image is build, you should see it on your system with the docker images command.
Next, let's run our container and see how it works.
The syntax will be similar to how we ran the Ubuntu container.
You should see output to your terminal that indicates that the TUAS website is being pinged.
64 bytes from acsweb.ucsd.edu (128.54.65.138): icmp_seq=20 ttl=47 time=102 ms
64 bytes from acsweb.ucsd.edu (128.54.65.138): icmp_seq=21 ttl=47 time=125 ms
64 bytes from acsweb.ucsd.edu (128.54.65.138): icmp_seq=22 ttl=47 time=44.5 ms
64 bytes from acsweb.ucsd.edu (128.54.65.138): icmp_seq=23 ttl=47 time=66.6 ms
64 bytes from acsweb.ucsd.edu (128.54.65.138): icmp_seq=24 ttl=47 time=89.5 ms
64 bytes from acsweb.ucsd.edu (128.54.65.138): icmp_seq=25 ttl=47 time=112 ms
64 bytes from acsweb.ucsd.edu (128.54.65.138): icmp_seq=26 ttl=47 time=32.0 ms
64 bytes from acsweb.ucsd.edu (128.54.65.138): icmp_seq=27 ttl=47 time=54.4 ms
64 bytes from acsweb.ucsd.edu (128.54.65.138): icmp_seq=28 ttl=47 time=76.5 ms
64 bytes from acsweb.ucsd.edu (128.54.65.138): icmp_seq=29 ttl=47 time=98.8 ms
If you see this output then the image we defined was correct and works fine when run as a container.
To see what else you can do inside a Dockerfile, you can check the reference. You can also see one of the Dockerfiles we use on the GCS repository.
Docker Compose
Docker Compose is a tool used to make it easy to have multiple Docker container be run simultaneously.
You might be wondering why we'd want that. There are many types of software where this is necessary. Let's say I am creating a web server. The web server doesn't exist in isolation. It might need to talk to a database to save/query data. We can have the web server run in one container and the database in another. Docker compose makes it easy to do so and have the two containers communicate to each other.
Even on our GCS, we use Docker Compose to run the GCS server, a database, a simulated plane, and more. You can see this Docker Compose definition here.
To define how containers should run with Docker Compose, the configuration is stored in a YAML file usually called docker-compose.yml
Let's look at a simplified version of our GCS Docker Compose.
version: "3"
services:
gcs:
image: tritonuas/gcs
ports:
- 5000:5000
environment:
- INFLUXDB_URI=http://influxdb:8086
influxdb:
image: influxdb:2.0-alpine
ports:
- 8086:8086
volumes:
- influxdb_data:/var/lib/influxdb
Here we define two services, gcs and influxdb. gcs is our backend web server and influxdb is our database.
There's a few things happening in this compose specification, but let's focus on one thing. How do these containers communicate with each other? As mentioned, previously, each container runs inside it's own isolated environment. How would they be able to talk to other containers they're running besides.
To get around this, Docker has a network that they all share and Docker Compose will allow each container to easily access each other. To access another container across the network, a container can use the container's service name from the YAML file. This means that if the gcs wants to send a network request to influxdb, it can use the string influxdb as the domain name. We see this in the example where the environment variable for the GCS called INFLUXDB_URI is defined as http://influxdb:8086. When the GCS makes network requests to that address, Docker Compose will know that it's trying to contact the influxdb container.
To try out using compose you can follow along this official guide from Docker.
Here is the complete specification for how you can configure a compose YAML file.
Dev Containers
Dev Containers are not a feature build into Docker itself, but is a feature that can be added to VS Code that uses Docker under the hood.
The idea is to have the same development environment between multiple people working on the same project. Each developer can take a shared Dockerfile and spawn up the same development environment as their peers.
Dev Containers are a useful resource for us at Triton UAS since we have so many members with wildly different host systems. It also makes it easier to get set up since you don't need to worry about downloading many dependencies to your host system. You just need to start up the container and everything should start up.
To use Dev Containers you'll need two things:
Once you have these, you can open any repository with a Dev Container and open the project in the container environment (assuming the project has a Dockerfile defined).
If you aren't prompted to open the project in a Dev Container, you can manually do so via the VS Code command pallette. Hit the keyboard combination Ctrl-Shift-P or Cmd-Shift-P on macOS. This should open a little text box with options. Select the option that says Dev Containers: Open Folder in Container... and follow the prompts. It may ask you to locate a Dockerfile where you'll have to point to the project's Dockerfile.
Summary
At this point, you should be familiar with what Docker is, why it's useful, how to pull images, write our own images, run containers, define docker compose configurations, and use VS Code Dev Containers. Docker will be an indespensible tool that you are likely to run into with any large scale software you ever come across.