Advanced Shell Commands
Table of Contents
Introduction and Document Notation
This is a brief set of notes that are helpful when using a Unix system and processing data! Much of the documentation here was taken from the Man Pages that are available on all Unix machines. To access the Man Pages on any command, simply type `man <command>`. Examples are provides and encouraged to be tested.
Keywords written in all CAPS are meant to be replaced with option and are self documenting, i.e. FILE means a file of some sort, paths to files work as well. PATTERN or expression are interchangeable, could be simply a keyword or in some commands could be a regular expression (regex). SCRIPT is dependent on the command, but usually a set of logical rules that dictate the operations for the command. All options in square braces, [OPTION], are optional arguments. All options followed by ellipsis, …, means that the option can be repeated as many times, i.e. FILE… means one or more files can be given.
Basic shortcut notation seen in various applications
Bash/Shell/POSIX notation
[] optional argument
If you see '[FILE]' or something similar wrapped in square braces, this indicates an optional argument to the program. Similarly, '[FILE | DIR]' indicates one or the other but not both.
- single letter argument
Often single argument flags are used because it is faster than typing the entire flag, most of the time '-' is used for single letter arguments (not always) and these letters can be grouped together, i.e. `tar -xzvf`
– string argument
Often '–' is used to avoid ambiguity that an argument is multiple flags or one string as a flag, i.e. `command [–str] [-s] [-t] [-r]` could be used as `command –str -str`. Most programs have -h or –help option, which indicates the correct use of arguments to that program.
Nano notation
^X
the '' indicates Ctrl and 'X' indicates lowercase x. together 'X' is read or executed as Ctrl+x.
M-X
'M-' indicates Meta (a modifier like Control or Shift), but on modern systems Meta is mapped to Alt. 'M-X' would be read or executed as Alt-x.
Vim notation
<Up>
When reading documentation on vim, '<>' indicate that this is a single stroke, but are optional for documentation (it is just more verbose). So <Up> is simply the up arrow on the keyboard. Could be <Left>, <Right>, or <Down> for arrow keys.
<S-Up>
Similar to the <Up> example, 'S-' usually indicates Shift, so this is Shift-Up Arrow.
<C-Up>
Similar to the above example, 'C-' usually indicates Control, so this is Ctrl-Up Arrow.
Emacs notation
Emacs notion is a combination of the vim and nano notations, except <> are used for specific keys like enter is <RET> or f8 is <F8>.
Shell shortcuts
Ctrl-r – Reverse history search
Ctrl-p – Previous command in history
Ctrl-n – Next command in history
Commands on all *Nix systems
Most text here is sourced from the man pages
grep
Grep is very useful in searching for keywords in files or command output
grep [OPTIONS] PATTERN [FILE...] grep [OPTIONS] [-e PATTERN]... [-f FILE]... [FILE...]
examples of grep use cases:
grep PATTERN FILE
Searches the file for the keyword, then prints the lines containing the keywork
<command w/ output> | grep PATTERN
Searches the output of the command for keyword, then prints the lines containing the keyword
<command w/ output> | grep -v PATTERN
Searches the output of the command for keyword, then prints the lines EXCLUDING the keyword
Other options that maybe useful:
- -E, –extend-regexp
Interpret PATTERN as an extended regular expression.
- -n, –line-number
Prefix each line of output with the 1-based line number within its input file.
- –line-buffered
Use line buffering on output. This can cause a performance penalty. This means that if grep gets input, then it waits until the input ends before outputting. Using this options changes the described behavior for continuous input and continuous output for long running processes such as watching logs.
pushd/popd
These are very useful for hopping between directories, that way you never have to type out full paths over and over.
`pushd [dir]` saves the current directory to a stack, then changes current directory to 'dir', which becomes the new top of stack.
`popd` changes current directory to whatever is top of the stack after removing the current directory from the top of the stack.
See man pages about rotating the stack.
find
Search for files in a directory hierarchy
find [start-point...] [expression] find [-H] [-L] [-P] [-D debugopts] [-Olevel] [starting-point...] [expression]
find PATTERN
Search for PATTERN in current directory.
find DIR PATTERN
Search for PATTERN in DIR
find -P [DIR] [PATTERN]
Same as above, do not follow symbolic links
find -L [DIR] [PATTERN]
Same as above, follow symbolic links
find -P [DIR] [PATTERN]
Same as above, do not follow symbolic links, except in arguments.
See man pages
For more details on more find options, see man pages.
cut
Remove sections from each lines of input. Really useful for formatting output of a program that organizes the data in columns.
cut OPTION... [FILE]...
cut -d DELIM -f LIST [FILE]
LIST could be a range or comma separated list of numbers indicating which columns to keep. Columns are determined to be anything between DELIM, usually a single character like space or comma or semi-colon. Range is inclusive on both ends, and 1 indexing.
<command w/ output> | cut -d DELIM -f LIST
Same as above, except input is from another command instead of a file.
sed
Sed is a stream editor for filtering and transforming. Most would not write a program or text file in sed, that would be a lot more work! Sed is useful for making specific changes to text files that you may need to automatically correct or edit. Very useful for bash scripts to install or update files.
for more information see linuxconfig.org
sed [OPTIONS] {SCRIPT} [FILE]
Examples of sed use cases:
sed 's/QUERY/REPLACEMENT/' file
Finds the first instance of QUERY of each line, and replaces it with REPLACEMENT. Note that QUERY could be a regex, and REPLACEMENT could include regex group matching, i.e. \1, etc. REPLACEMENT could be left empty to remove the instance of QUERY from the output.
sed 's/QUERY/REPLACEMENT/g' file
Same as above, but 'g' is for global, so all instances in a line will be changed now. Note that QUERY could be a regex, and REPLACEMENT could include regex group matching, i.e. \1, etc.
<command w/ output> | sed SCRIPT
Sed can take input from another command with the same effect as providing a file. This is useful for manipulating output to match a specific format.
Other useful options
- -i[SUFFIX], –in-place[=SUFFIX]
edit files in place (makes backup if extension supplied). The default operation mode is to break symbolic and hard links. This can be changed with –follow-symlinks and –copy.
- -e script, –expression=script
add the script to the commands to be executed
- -f script-file, –file=script-file
add the contents of script-file to the commands to be executed
awk
(gawk) Pattern scanning and processing language. Awk is a full programming language implemented as a commandline tool for processing data. There are lots of ways to use this tool, even as a way to compute results very quickly. Here I will only use it for simple formatting similar to cut or sed
awk [OPTIONS] -f SCRIPT [--] FILE ... awk [OPTIONS] [--] TEXT FILE
awk '{print $1}' FILE
Prints first column in FILE, SCRIPT is indicated between {} and the single quotes are needed two reasons. 1) the inline script needs to be read as a single argument, which single or double quotes allow. 2) Bash uses $VAL to reference environment variables inside double quotes, not the case in single quotes.
awk -F DELIM '{print $1}' FILE
Same as above except we want the column delimiter to be DELIM, maybe comma or semi-colon, not white space.
<command w/ output> | awk '{print $1}'
Same as above, input is coming from another command.
echo "Hi,Hello,Bye" | awk -F ',' '{$3="Hey"; print $0}'
Takes the input and replaces the 3rd column with the specified string, then prints whole line space separated
output:
Hi Hello Hey
Try the examples!
Example 1
Take data.log, copy and paste into text file, and use the above commands (Not the editors) to create an identical output.
The output tells us the number of attempts with a user name from an IP address. The first line reads "2 attempts using 'admin' from '10.18.0.7'".
output should be identical to:
2 admin 10.18.0.7 6 admin 218.49.183.17 3 guest 10.18.0.7 1 guest 218.49.183.17 3 root 10.18.0.7 9 root 218.49.183.17 4 test 10.18.0.7 4 test 218.49.183.17 1 user 10.18.0.7 3 user 218.49.183.17
Example 2
Similar to example 1, but a solution is not provided. Match the output below
2 10.18.0.7 admin 3 10.18.0.7 guest 3 10.18.0.7 root 4 10.18.0.7 test 1 10.18.0.7 user 6 218.49.183.17 admin 1 218.49.183.17 guest 9 218.49.183.17 root 4 218.49.183.17 test 3 218.49.183.17 user
Data.log
# Source: https://ossec-docs.readthedocs.io/en/latest/log_samples/auth/sshd.html # Slightly modified IP Addresses Aug 1 18:27:45 knight sshd[20325]: Illegal user test from 10.18.0.7 Aug 1 18:27:46 knight sshd[20325]: Failed password for illegal user test from 10.18.0.7 port 48849 ssh2 Aug 1 18:27:46 knight sshd[20325]: error: Could not get shadow information for NOUSER Aug 1 18:27:48 knight sshd[20327]: Illegal user guest from 10.18.0.7 Aug 1 18:27:49 knight sshd[20327]: Failed password for illegal user guest from 10.18.0.7 port 49090 ssh2 Aug 1 18:27:49 knight sshd[20327]: error: Could not get shadow information for NOUSER Aug 1 18:27:52 knight sshd[20329]: Failed password for admin from 10.18.0.7 port 49266 ssh2 Aug 1 18:27:56 knight sshd[20331]: Failed password for admin from 10.18.0.7 port 49468 ssh2 Aug 1 18:27:58 knight sshd[20334]: Illegal user user from 10.18.0.7 Aug 1 18:27:59 knight sshd[20334]: Failed password for illegal user user from 10.18.0.7 port 49680 ssh2 Aug 1 18:27:59 knight sshd[20334]: error: Could not get shadow information for NOUSER Aug 1 18:28:02 knight sshd[20336]: Failed password for root from 10.18.0.7 port 49869 ssh2 Aug 1 18:28:05 knight sshd[20347]: Failed password for root from 10.18.0.7 port 50063 ssh2 Aug 1 18:28:12 knight sshd[20349]: Failed password for root from 10.18.0.7 port 50245 ssh2 Aug 1 18:28:14 knight sshd[20352]: Illegal user test from 10.18.0.7 Aug 1 18:28:19 knight sshd[20352]: Failed password for illegal user test from 10.18.0.7 port 50671 ssh2 Aug 1 18:28:19 knight sshd[20352]: error: Could not get shadow information for NOUSER Aug 1 18:29:55 knight sshd[20402]: Illegal user test from 10.18.0.7 Aug 1 18:29:56 knight sshd[20402]: Failed password for illegal user test from 10.18.0.7 port 52244 ssh2 Aug 1 18:29:56 knight sshd[20402]: error: Could not get shadow information for NOUSER Aug 1 18:29:58 knight sshd[20404]: Illegal user guest from 10.18.0.7 Aug 1 18:30:02 knight sshd[20406]: Illegal user test from 10.18.0.7 Aug 1 18:30:03 knight sshd[20404]: Failed password for illegal user guest from 10.18.0.7 port 52416 ssh2 Aug 1 18:30:03 knight sshd[20404]: error: Could not get shadow information for NOUSER Aug 1 18:30:03 knight sshd[20406]: Failed password for illegal user test from 10.18.0.7 port 52558 ssh2 Aug 1 18:30:03 knight sshd[20406]: error: Could not get shadow information for NOUSER Aug 1 18:30:05 knight sshd[20439]: Failed password for illegal user guest from 10.18.0.7 port 52818 ssh2 Aug 1 18:30:05 knight sshd[20439]: Illegal user guest from 10.18.0.7 Aug 1 18:30:05 knight sshd[20439]: error: Could not get shadow information for NOUSER Aug 1 18:30:06 knight sshd[20441]: Failed password for admin from 218.49.183.17 port 52851 ssh2 Aug 1 18:30:08 knight sshd[20443]: Failed password for admin from 218.49.183.17 port 53014 ssh2 Aug 1 18:30:09 knight sshd[20445]: Failed password for admin from 218.49.183.17 port 53040 ssh2 Aug 1 18:30:11 knight sshd[20447]: Failed password for admin from 218.49.183.17 port 53192 ssh2 Aug 1 18:30:11 knight sshd[20449]: Illegal user user from 218.49.183.17 Aug 1 18:30:12 knight sshd[20449]: Failed password for illegal user user from 218.49.183.17 port 53230 ssh2 Aug 1 18:30:12 knight sshd[20449]: error: Could not get shadow information for NOUSER Aug 1 18:30:13 knight sshd[20451]: Illegal user user from 218.49.183.17 Aug 1 18:30:14 knight sshd[20451]: Failed password for illegal user user from 218.49.183.17 port 53404 ssh2 Aug 1 18:30:14 knight sshd[20451]: error: Could not get shadow information for NOUSER Aug 1 18:30:14 knight sshd[20453]: Failed password for root from 218.49.183.17 port 53425 ssh2 Aug 1 18:30:21 knight sshd[20455]: Failed password for root from 218.49.183.17 port 53571 ssh2 Aug 1 18:30:22 knight sshd[20457]: Failed password for root from 218.49.183.17 port 53615 ssh2 Aug 1 18:30:24 knight sshd[20476]: Failed password for root from 218.49.183.17 port 54033 ssh2 Aug 1 18:30:24 knight sshd[20484]: Failed password for root from 218.49.183.17 port 54078 ssh2 Aug 1 18:30:26 knight sshd[20488]: Illegal user test from 218.49.183.17 Aug 1 18:30:27 knight sshd[20486]: Failed password for root from 218.49.183.17 port 54243 ssh2 Aug 1 18:30:27 knight sshd[20488]: Failed password for illegal user test from 218.49.183.17 port 54285 ssh2 Aug 1 18:30:27 knight sshd[20488]: error: Could not get shadow information for NOUSER Aug 1 18:30:29 knight sshd[20490]: Illegal user test from 218.49.183.17 Aug 1 18:30:34 knight sshd[20490]: Failed password for illegal user test from 218.49.183.17 port 54423 ssh2 Aug 1 18:30:34 knight sshd[20490]: error: Could not get shadow information for NOUSER Aug 1 18:35:53 knight sshd[20658]: Illegal user test from 218.49.183.17 Aug 1 18:35:54 knight sshd[20658]: Failed password for illegal user test from 218.49.183.17 port 39604 ssh2 Aug 1 18:35:54 knight sshd[20658]: error: Could not get shadow information for NOUSER Aug 1 18:35:56 knight sshd[20660]: Illegal user guest from 218.49.183.17 Aug 1 18:35:57 knight sshd[20660]: Failed password for illegal user guest from 218.49.183.17 port 39811 ssh2 Aug 1 18:35:57 knight sshd[20660]: error: Could not get shadow information for NOUSER Aug 1 18:36:00 knight sshd[20664]: Failed password for admin from 218.49.183.17 port 40009 ssh2 Aug 1 18:36:04 knight sshd[20666]: Failed password for admin from 218.49.183.17 port 40217 ssh2 Aug 1 18:36:06 knight sshd[20675]: Illegal user user from 218.49.183.17 Aug 1 18:36:11 knight sshd[20675]: Failed password for illegal user user from 218.49.183.17 port 40470 ssh2 Aug 1 18:36:11 knight sshd[20675]: error: Could not get shadow information for NOUSER Aug 1 18:36:14 knight sshd[20677]: Failed password for root from 218.49.183.17 port 40973 ssh2 Aug 1 18:36:21 knight sshd[20679]: Failed password for root from 218.49.183.17 port 41159 ssh2 Aug 1 18:36:24 knight sshd[20681]: Failed password for root from 218.49.183.17 port 41541 ssh2 Aug 1 18:36:27 knight sshd[20683]: Illegal user test from 217.49.183.17 Aug 1 18:36:28 knight sshd[20683]: Failed password for illegal user test from 218.49.183.17 port 41630 ssh Aug 1 18:36:28 knight sshd[20683]: error: Could not get shadow information for NOUSER