Basic Knowledge for Shell

Why is it important

combine programs in creative ways to automate nearly any task.
navigate the world of open-source software.

Navigating in the shell

missing:~$

missing: machine name.
~: short for “home”.
$: you are not the root user.

missing:~$ date
Fri 10 Jan 2020 11:49:31 AM EST
missing:~$

missing:~$ echo hello
hello

missing:~$ echo 'hello world'
hello world
missing:~$ echo hello\ world
hello world
missing:~$ echo "hello world"
hello world

execute a program (with arguments).
shell parses the command by splitting it by whitespace.
provide an argument that contains spaces or other special characters: quote the argument with ' or ", or escape just the relevant characters with \ .

missing:~$ man date
missing:~$ date --help

man(manual): look up more information about any command on your system.
or a short version using --help .

missing:~$ pwd                                     #print current working directory
missing:~$ echo $PWD                                 #or using environment variable


missing:~$ cd /
missing:/$ cd bin
missing:/bin$

missing:~$ cd /
missing:/$ cd bin/../bin/../bin/././../bin/..
missing:/$

absolute path: start with / and give the full set of directories.
relative paths: relative to the current working directory.
. is “this directory”, and .. is “the parent directory”.

What is available in the shell

How does the shell know how to find programs like date or echo?

missing:~$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
missing:~$ which echo
/bin/echo
missing:~$ /bin/echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

missing:~$ ls /bin            #listing the contents of all the directories on $PATH

$PATH is an environment variable that lists which directories the shell should search for programs.
When executing a program, the shell consults $PATH, searching through the :-separated list of directories by that name.
find out which file is executed using which.
bypass $PATH by giving the path.

some of the most important programs

cat file, which prints the contents of file.
sort file, which prints out the lines of file in sorted order.
uniq file, which eliminates consecutive duplicate lines from file.
head file and tail file, which respectively print the first and last few lines of file.
grep pattern file, which finds lines matching pattern in file. pattern is a regular expression which can express very complex patterns.

missing:~$ grep "error" log.txt
#pass -r to recursively search all the files in a directory.  
missing:~$ grep -r "TODO" .

sed, which is a programmatic file editor.

missing:~$ sed -i 's/pattern/replacement/g' file

# 1. 基础替换（只打印不修改原文件）
$ sed 's/foo/bar/' file.txt
# 2. 就地修改（-i），并替换每行所有匹配项（/g）
$ sed -i 's/foo/bar/g' file.txt
# 3. 使用正则表达式：把每行开头的 "http://" 换成 "https://"
$ sed 's/^http:/https:/' file.txt
# 4. 反向引用（Capture Group）示例：
# 假设你想把 "date: 2023-10-27" 变成 "2023/10/27"
# 这里用 \(...\) 定义捕获组，用 \1 \2 \3 来引用它们
$ echo "date: 2023-10-27" | sed 's/date: \([0-9]*\)-\([0-9]*\)-\([0-9]*\)/\1\/\2\/\3/'
# 输出结果：2023/10/27

find, which lets you find files (recursively) that match certain conditions.

#finds ZIP files in the download directory that are older than 30 days.
missing:~$ find ~/Downloads -type f -name "*.zip" -mtime +30

# 找到当前目录下所有 .tmp 结尾的文件并删除它们
# -exec takes a command terminated with a stand-alone ;(which we need to escape much like a space)
# {} is replaced with each matching file path by `find`.
$ find . -name "*.tmp" -exec rm {} \;

awk, which is built for parsing files.

#prints the second whitespace-separated column of every line of file
missing:~$ awk '{print $2}' file

compose shell programs pipes | : string together the output of one program with the input of another.

missing:~$ ssh myserver 'journalctl -u sshd -b-1 | grep "Disconnected from"' \
  | sed -E 's/.*Disconnected from .* user (.*) [^ ]+ port.*/\1/' \
  | sort | uniq -c \
  | sort -nk1,1 | tail -n10 \
  | awk '{print $2}' | paste -sd,
postgres,mysql,oracle,dell,ubuntu,inspur,test,admin,user,root

The shell language (bash)

Most shells implement a full programming language (like bash), just like Python or Ruby. It has variables, conditionals, loops, and functions.

redirects: >file, which takes the standard output of a program and write it to file. >>file will append to file rather than overwrite it. <file , which tells the shell to read from file.
conditionals: if command1; then command2; command3; fi.
loops: while command1; do command2; command3; done functions. for varname in a b c d; do command; done. executes command four times, each time with $varname set to one of a, b, c, and d.