Basics of Unix and the Command Line
Here are the slides that accompany this workshop.
It is imperative for anyone interested in Software Development to become familiar with the command line and Unix programs. They power many development workflows and allow software engineers to interact with their systems in a simple and efficient manner.
Here's a brief history lesson before we dive into the main content. The original Unix operating system was developed by AT&T Bell Labs in the 1960s and 70s by some of the pioneers of computing. While the original Unix operating system isn't used much today, many modern systems are inspired by it. Even if you don't personally own a Unix-like system, you will find yourself interacting with one through remote servers or virtual machines.
Setup
If you're using macOS or Linux, then you already have a Unix-like system. If you're on Windows, follow the instructions here. For the purposes of this workshop, it may be quicker to get setup with WSL.
Terminal and Shell
The terminal is an application on your system that allows you to interact with your shell. We'll get to what a shell is in just a second.
The following are a few methods for accessing a terminal:
- macOS has a built-in application called "Terminal". You can access it by searching for it in Spotlight.
- To access WSL on Windows, search for "wsl" in your start menu. You can also access WSL through the Windows Terminal application.
- Your Linux distribution probably comes packaged with a terminal emulator, so you can use that one.
The shell is a program that runs inside the terminal and allows the user to interact with the operating system using text based commands. The shell reads in text input and runs the corresponding programs. When you open the terminal, you are greeted by a blank window with a cursor. This is the shell, and you can type commands that you want the shell to run.
Besides the cursor, you'll also see the shell prompt. The shell prompt can greatly vary depending on your shell and configuration, however it usually has information about the current user or current directory.
Here's an example of what your shell prompt might look like:
Let's break down what each part means:
-
user- The text before the @ sign is the username of the current user.
-
laptop- This is the hostname of the computer. A hostname is a label given to a computer for when it communicates to other computers over the network.
~/Desktop- This is the current working directory of the shell. As we'll talk about later, you can use your shell to navigate through directories on your system. This is similar to how you would do so with a traditional file explorer (ex: Finder or Windows Explorer). Think of the current working directory as meaning that we currently have a certain folder open. In this case, we have the Desktop folder open.
The user inputted text goes after the dollar sign symbol. If you start typing some characters then you should see them show up in the terminal.
Let's start running some programs.
echo
The echo command will print out any text that you provide to it. Let's say we want to print out the text "Hello world!" to the screen.
You can do so with the following command:
If you type this into your shell and hit the "Enter" or "Return" key, the command will be executed. You should see "Hello world" printed out in the terminal.
Let's try a deviation of the previous command.
You might notice that we aren't using quotes around the text we want to print. This is totally fine since echo will look at all of its arguments to determine what to print. Arguments are just additional data that we pass into commands. These come after the command name and are separated by spaces. In the fist scenario (echo "Hello world"), we wrapped Hello world in quotes. This meant that the shell treated "Hello world" as a single argument. In our new example of echo Hello world, we don't have quotes, so the shell treats Hello and world as two different arguments. The number of spaces between the arguments doesn't matter, so they only count as two arguments. Since these are seen as two separate arguments, echo will print them out as two strings with a single space between them.
Demonstration:
Unix Filesystem
Before we explore the cd command, let's explore the basics of a Unix filesystem.
Root Directory /
The root directory is the top level of the filesystem. Every file and directory of the filesystem is a sub-child of the root directory.
The symbol for the root directory is a forward slash /. When you want to describe a path to a file or directory that starts at the root directory, you start with a / symbol. This is called an absolute path.
The following represents an absolute path that points to the bin directory. This directory is located one level below the root.
The following is another absolute path that points to a fonts directory that is within the etc directory. Note that only the first forward slash represents the root directory. Subsequent slashes are simply path separators between directories and files.
Home Directory ~
On Unix-like systems, every user of a computer is assigned a home directory. Every user's home directory contains user-specific files.
The symbol for the home directory is a tilde ~. Similarly to the root directory, we can describe paths that start from the home directory (or are relative to the home directory).
The following represents a relative path from the home directory that points to my user's Downloads directory.
This path points to a Python source code file within my home directory.
Since all directories on a filesystem must be under the root directory, the same applies for every home directory. Usually they follow the following pattern.
To see your current user's home directory location you can useecho to display the HOME environment variable. We'll get to what an environment variables are later in this workshop.
Current Directory .
The current directory is the directory which the shell is currently working in.
The current working directory is denoted by a single period . symbol.
The following path represents a directory named build that's in the current directory.
Parent Directory ..
The parent directory is the directory above the current directory in the filesystem hierarchy.
It's denoted by two periods ..
cd
The cd command will change the current working directory. As mentioned above, you can use the shell to move around the directories of your computer's filesystem just as you would with a graphical file explorer.
cd will change to the directory that you provide the path of. There are a few ways that you can describe paths on your system using the symbols that we just discussed.
We can change directory to the root directory or describe an absolute path that starts from the root directory.
We can also cd into directories that are relative to the home directory. The paths begin with a ~ character.
cd ~ or simply cd on its own.
We can also change to directories that are relative to our current working directory. It's important to note that these paths mean different things depending on your current working directory. This is in contrast to the previous commands which would work no matter what your current directory is.
Let's say there's a directory named build in our current directory. We can cd into it with the following command.
We can also remove the ./ which will also evaluate the path relative to our current working directory.
To move to the parent directory we can use the following command.
Demonstration:
ls
The ls command will list files or directories. If you just type the ls command on its own then it will list files and directories in the current directory.
You can also specify a path to a directory to list files/directories in. The path can be either absolute or relative.
Demonstration:
There are many other options that you can specify to change the behavior of ls. You can change ls to also list out "hidden files/directories". These hidden files/directories aren't normally listed by ls. You can tell a file/directory is hidden if it's prepended with a period (example: .bashrc).
You can specify a command line flag for ls to list all files, including hidden ones. A command line flag is an option that you can specify to change the behavior of a program. Every program has its own flags and way of handling them, so you'll need to check its respective documentation.
To list all files with ls you can use the -a flag.
There are many other flags you can provide to ls. The -l flag will have every result take up its own line.
You can combine these flags so that both hidden files are shown and everything is displayed on separate lines.
Demonstration:
Now that we've gone through the basics of cd and ls we can efficiently browse our filesystem and find the files that we want to interact with.
pwd
pwd will print the current working directory. This is helpful if you get lost and forget what directory you're in.
Demonstration:
cat
cat will output the contents of a specified file.
Demonstration:
Demonstration:
mkdir
mkdir will create a new directory that matches the name you provide.
This command makes a new, empty directory named tuas.
Demonstration:
cp
cp will copy a file from its current location to somewhere else on your filesystem.
You can specify a path to a file to copy and a path to a destination directory.
If you want to change the name to something else, you can specify a filename in the destination path. You can also copy multiple files to a destination. The following command will copy bothindex.html and new.html to the Documents directory.
If you try the previous commands to try to copy a directory, they will not work. This is because you need to specify a command line flag to recursively copy a directory's contents. This basically means that the directory and all the content it contains will be copied to the destination.
The following command uses the -r flag to recursively copy the tuas directory and everything inside it to the Documents directory. After running this command we should see a tuas directory inside Documents. We can check by running ls ~/Documents.
Demonstration:
mv
mv will move a file from one location to another.
Let's say I want to move the index.html file to the Documents directory.
mv without the need of an extra flag like with cp. The following command will move the tuas directory to be inside the Documents directory.
You can also use mv to rename files or directories.
Be careful, if you try to rename a file/directory to something that already exists, the existing file/directory will be overwritten by the one you are moving.
rm
rm will remove a file or directory. This is a potentially dangerous command since it will delete the file/directory forever. There is no Trash or Recycle Bin that files go to. They are essentially unrecoverable after running rm. So use wisely ...
To remove a file you can specify a path to a file.
To delete a directory you need to pass in the -r flag to recursively delete a directory and its contents.
whoami
whoami will display the user that's currently logged in. This command isn't as useful as some of the others, however it might come in handy from time to time.
sudo
sudo allows you to elevate your permissions and run a command as the root user. Typically, Unix-like systems have a set of permissions on files/directories of the system. Only certain users may be able to interact with some system resources. The root user is the most powerful user on a system. They have permissions to everything on the system. Sometimes it makes sense to temporarily grant yourself root user permissions. There may be a file or directory that you need to modify but usually don't have permission to do so.
You might be wondering why we don't just run every command as the root user to ensure that we always have sufficient permissions. Always being root can potentially be dangerous because it becomes very easy to tamper with system files that are essential for a working system.
This command will run a cp command to copy a file to a path that our normal user doesn't have permissions to write to. A regular user only has write permissions to the files in its own user directory. Files above this user directory and in other user directories will not have write permissions. With sudo, we can circumvent this by temporarily having superuser permissions. Running a command with sudo will prompt you for a sudo password. This is usually the same as your user's login password unless you explicitly changed it.
Demonstration:
Stopping programs early
Sometimes there might be a program running that you want to quit before it finishes.
You can accomplish this by pressing Ctrl-C while a program is running.
As a quick demo, we can try to kill the sleep program. If you run sleep 60, there will be 60 second delay where the terminal will be unusable. We can kill this program with Ctrl-C and gain back control over our terminal.
Demonstration:
make
make is a useful development utility that helps with running shell tasks that are commonly used.
Let's say you have a project with a complicated command to compile the code. Or maybe there is a set of commands you need to execute for an application to run. make can help in these scenarios by providing a convenient way to run these commonly used workflows.
make can get pretty complicated and could be a whole workshop on its own. For this workshop, we'll show you how to run a make target and how to write a simple one.
When it comes to make, the most important thing is the Makefile. This is where everything associated with make will live. For our case, we're going to be talking about the targets that are defined in this file.
A target can be thought of an "action" we define in the Makefile. We can have a sequence of commands that will be executed when a target is run.
The following is an example make target. The part at the beginning is the name of the target. The lines that are indented represent the commands run when the target is called.
build:
cp resources/* .
ln -sf /usr/lib/libGL.so ./libGL.so
gcc main.c -Wall -g -Og -o library.o app
mv app bin/
@echo "Build completed successfully"
Don't pay attention to the content of the commands in the target. They don't really make sense when put together. But the point is that you might have some more complicated build workflow that is longer than a single command. This example shows how there could be some setup and cleanup that you want to do.
Even if your target is only a single command, it may be useful to avoid writing out long commands. Here's one place where we do this in our GCS Makefile:
To actually run a make target you can run make <target_name>. It's important to note that you're current working directory must have a Makefile inside of it for make to work. For the two target examples shown here are the commands:
Here's the Makefile for our gcs project. You can glance at it and see all the make targets that are defined for the project.
For a great, more comprehensive tutorial on make, check out this page.
If you want a deep dive on make, check out the GNU make book.
Terminal Text Editors
You'll find that it may be more convenient to quickly edit a file in the terminal rather than open up an entire graphical text editor.
Here are two of the most commonly used terminal text editors:
Nano
Open file in Nano:
To navigate the file you can use the arrow keys to move left/right across a line and up/down the lines. You can type as you normally would to make your edits.
Once you're satisfied with your changes you can hit Ctrl-O to save your changes to disk. Then you can press Ctrl-X to quit Nano.
Demonstration:
Vim
Vim is another terminal text editor. However, Vim works a bit differently compared to most text editors. Vim is another topic that could be its own workshop. We're just going to go over the basics to get you up and running with Vim.
Open a file in Vim:
When you enter Vim, you'll notice that you can't type as you normally would. This is because Vim is a modal editor. This means that there are different modes that change how you interact with the document. In this workshop, we're only going to talk about NORMAL, INSERT and COMMAND modes.
When you first launch Vim, you'll be in NORMAL mode. Here, you can't modify the document, but you can navigate around it.
Navigation keys:
jto move down a linekto move up a linehto move one character to the leftlto move one character to the right
You can also prepend each of these motions with a number to represent how many times the motion should be repeated. For example, if you wanted to move 5 lines down, you would type 5j while in NORMAL mode.
Here are a few more handy navigation keystrokes:
0to move to the start of a line$to move to the end of a linewto move to the start of the next wordbto move to the start of the previous wordggto move to the start of a fileGto move to the end of a file
To enter INSERT mode, press the i key. You should see text on the bottom left of your terminal that indicates that you're in INSERT mode.
Once you're in INSERT mode, you can proceed to edit the file and modify text as you normally would.
To exit INSERT and return to NORMAL mode, press the escape key.
Once you're done with your edits you can save and quit Vim. To save the changes to disk, press the colon key to enter COMMAND mode, type w and hit enter. To quit, do the same thing but type q instead. You can do both at the same time with the combination of :wq.
Demonstration:
If you want to become more proficient with Vim, vimtutor is a fantastic guide that should already be installed on your system. Just type vimtutor in your terminal and follow the lessons.
Here's a handy Vim cheatsheet
You can also get used to Vim binds by using them in other applications. For example, there is a VS Code extension that provides Vim binds for editing.
Shell Configuration
There are many ways which we can customize the behavior of our shells. One approach is to modify the shell configuration file. The configuration file is simply a list of commands that are executed when the shell is initialized.
Most shells will have a configuration file located in a user's home directory. If you're using Bash, your configuration file is called .bashrc (Note that it is a hidden file. See the ls section for more information about those). If you're using Zsh, it is called .zshrc. If you're not sure what shell you're using, run either echo $SHELL or echo $0. If you're on a recent version of macOS, Zsh is the default shell. Most Linux distributions use Bash by default. The same goes for WSL.
Let's modify our shell configuration file. You're going to need a text editor to do this. You can use the terminal text editors, such as Nano or Vim, which were discussed in the previous section. A graphical text editor, such as TextEdit, VS Code, also will work for this. Note that since .bashrc and .zshrc are hidden files, they may not show up in your editor's file picker by default.
Let's open the shell configuration file in our editor of choice. For this workshop, I'm using vim to edit my .bashrc file.
As mentioned, the shell configuration file holds a list of commands that are executed when the shell starts up. This means that we can set options that will persist even after we close the shell.
Let's add some commands that will greet us when our shell starts up.
If we add the following line to our configuration file and save the file, the next time we open our shell we'll see "Hello there" being printed to the screen. Try adding this line and open a new terminal window.
We can run any command in this file, so let's try the date command. This will simply print out the date when the shell is started.
Let's combine the two and print out a greeting that has the date embedded in it. We can use backticks to execute a command and place it's output inside another string.
These examples might not seem too useful, however the following sections will show a few use cases that you will encounter in development. There are many other ways you can customize the look and feel of your shell (customize the prompt, change colorscheme, etc.).
Environment variables
Environment variables are system-wide pieces of data that can be used to convey information about your current system. Earlier we saw the HOME environment variable and how it stores the username of the current user.
As demonstrated previously, we can print out an environment using echo and a dollar sign $ symbol prepended to its name.
There are many more environment variables that are commonly used. One of the most important is PATH.
PATH
The PATH environment variable is used by the shell to determine where binary files and executables on the system are located.
Executables are programs and scripts that you can run at the shell. Every command that we've gone over in this workshop are executables.
PATH points to multiple directories that contain executable files. Directories are separated from each other using the : character.
When you type in a program (such as ls), the shell looks for the executable in one of the directories described by PATH. Sometimes, programs are installed to directories that are not in your PATH, so you must add a directory to your PATH.
The following command prepends a directory to the PATH environment variable. The directory being added is $HOME/.local/bin/. The command sets path to that directory and appends the old value of PATH to the end. Note the colon that separates them.
For this change to PATH to be persistent, you need to add it to your shell configuration file (ex: .bashrc, .zshrc).
alias
The alias command can be used to register a command with a different string.
Let's say I don't want to keep typing out ls -al whenever I want to list all files in a list format. If I want to alias the command to something shorter like l, I can run the following alias command:
SSH
You will often find yourself wanting to connect to remote servers and computers during development. A server might have a preferred development environment that is difficult to set up or a computer might be in a production environment.
The ssh program is a simple way to do so via the command line. SSH stands for "Secure Shell" and allows us to connect to a remote computer and use its shell.
When connecting via SSH, the command will generally follow this format:
username is just the username of the user you want to connect as on the remote server.
host is the network address of the remote server. This can be an IP address (ex: 100.23.24.121) or a domain name (ex: ieng6.ucsd.edu).
During this workshop, the software leads should have a computer setup that you can SSH into. You can also try SSH-ing into UCSD's ieng6 server.
Demonstration:
scp
scp is a command line program that uses SSH to copy files from a local computer to a remote computer and vice versa.
Copying from local to remote computer.
Copying from remote to local computer.
Getting Help
It's easy to feel lost when it comes to working with command line applications. There's no intuitive user interface to point you where you need to be. If you forget how to use a command line program or are learning how to use a new one, there are several ways you can quickly get up to speed.
Many commands have manual pages that explain how to use a program and all its options. You can use the man command to view this information. You can scroll around the manual with the arrow keys. To quit, just press q.
You can also use the --help command line flag which will also give usage information. Usually, --help is a bit more brief than a man page. However, that might be all you need.
There's also this handy program called tldr which has short descriptions of common use cases of command line programs. It's great if you want to quickly get launched into using a program. There are several clients you can find here and this web application that has the same information.
You can also feel free to search online to find what you need. In most cases, this might be faster than digging through a man page or help options.
Also feel free to ask other people on the team! We're all learning from each other and most people are more than happy to help out.
Demo
Ok enough explaining, let's do something with the knowledge we've gained.
We're going to make some folders/files and create a Makefile that does a basic task.
First, let's navigate to a directory where we can mess around in. If you remember, you can use a combination of cd and ls commands to navigate around your system.
You can run pwd to get your bearings and see what directory you're in. Then run ls to see what files/directories are in the current working directory. Then cd into the directory you want to.
Let's say I want to put the files for this demo in a ~/tuas/unix_workshop folder.
I'm going to assume you're in your home directory by default. You can run cd on its own to return to your home directory.
As mentioned, you can verify you're in your home directory by running the pwd command.
That looks correct (for my user at least). Next we need to navigate to the folder we want (which is ~/tuas/unix_workshop). The first part of the path after the home directory is the tuas directory. You might not have a tuas directory in your home directory, so we may need to create it. We can check if you have a tuas directory with the ls command. If you run ls and don't see a directory entry that says tuas, we have to create it. If you remember, we can create a directory with the mkdir command.
Now that the tuas directory exists, we can cd into it.
ls, you shouldn't see any files (assuming you just ran mkdir). This makes sense since mkdir will create an empty directory. We also wanted a unix_workshop directory so let's also make that one.
Now if you run ls, you should see unix_workshop as an entry.
Let's cd into unix_workshop.
pwd to verify this is true.
Note that you could have done this with two commands:
When you pass the -p flag to mkdir, it will take a path and create any directories that don't exist within the path. The cd command will just place us directly in the directory we want to end up in.
Now that we're in the unix_workshop folder, let's make some files. Let's open a new file in a terminal text editor. You can use either Nano or Vim. See their respective sections above for usage information.
Let's say we want to edit a new file called date.txt. To open a new file in either editor, use the following commands:
For Vim:
For Nano:Feel free to add any text you want (remember that you need to enter INSERT mode for Vim). Once you're done, save the file and quit from the editor.
To save exit Vim type in :wq.
For Nano, it's Ctrl-O then Ctrl-X.
Now let's edit a new file. This time let's make a Makefile. As mentioned above, this is what controls the behavior of make and allows you to define custom make targets.
So, open the editor of your choice and create a new file called Makefile.
Inside the Makefile, write this:
Here we're defining a new make target called run. As mentioned in the make section, the commands inside a target are indented from the target name. This run target has two commands that are executed when the target is run.
It's not essential to know what this target is doing, but we'll quickly explain it. The echo command is writing some text to the date.txt file. The two angle brackets (>>) signs mean that whatever text echo outputs will be added onto the end of a file (in this case date.txt). The echo command also has these backticks around the word date. This means that the output of the command will be placed in its spot. date is a command that will output the current date and time. So the echo will output "Hello it is: " with the current date at the end. The cat command will just output the contents of the date.txt file to the screen.
Once you've written this text, we can save and close the file.
Now let's execute the command. If you remember, to run a make target the command looks like make <target_name>.
You should see the contents of the file be outputted to your terminal.
Run the make target again and see how the contents of the file have changed.
Great! We've created a make target and successfully ran it. Now let's do some cleanup.
Let's remove the unix_workshop directory that we created.
First, let's cd out of the unix_workshop directory to the parent directory.
Run ls to verify that you see unix_workshop listed. If so, then we can remove it. Recall that we can use the rm command to remove files and directories. We need to pass in the -r flag to recursively delete directories and what's inside them.
Awesome! We've completed the demo. Hopefully you have some more familiarity when it comes to working with the command line.