Special Commands: sed, awk, grep, sort
The four tools that do most of the work in most shell scripts, and the small subset of each that is worth memorising.
These four have enormous manual pages and you need perhaps ten per cent of each. This page is that ten per cent, plus the portability traps that make scripts work on Linux and fail on macOS.
grep — find lines#
grep -E 'ERROR|FATAL' app.log # -E: extended regex. use it always.
grep -i warning app.log # case insensitive
grep -v DEBUG app.log # invert: lines NOT matching
grep -c ERROR app.log # count matching lines
grep -n ERROR app.log # with line numbers
grep -l ERROR *.log # just the filenames
grep -q ERROR app.log # silent; exit status only — for `if`
grep -o -E '[0-9]+ms' app.log # print each MATCH, not the line
grep -A3 -B1 ERROR app.log # 3 lines after, 1 before
grep -r TODO src/ # recursive
grep -F 'a.b.c' file # fixed string: no regex, much fasterThe three worth internalising:
-qfor conditions.if grep -q pattern file; then— no output, just the exit status.-oto extract. Prints the matched text rather than the line. With-Ethis is a general-purpose extractor.-Ffor literals. When you are searching for a string that contains dots or brackets,-Fis both correct and faster.
# every unique IP in a log
grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}' access.log | sort -used — edit a stream#
Ninety per cent of sed use is one command: substitute.
sed 's/old/new/' # first occurrence on each line
sed 's/old/new/g' # every occurrence
sed 's/old/new/2' # the second occurrence only
sed 's|/usr/local|/opt|g' # any delimiter — use | for paths
sed -E 's/([0-9]{4})-([0-9]{2})/\2\/\1/' # -E for ERE, \1 \2 for groupsOther things worth knowing:
sed -n '5,10p' file # print lines 5-10 (-n suppresses default printing)
sed '/^#/d' file # delete comment lines
sed '/^$/d' file # delete blank lines
sed -n '/START/,/END/p' # print a range between two patterns
sed '2i\inserted' # insert before line 2awk — work with columns#
awk is a whole language, but the useful core is: for each line, split into fields, run a rule.
awk '{ print $1, $3 }' file # first and third field
awk '{ print $NF }' file # last field
awk 'NF' file # drop blank lines
awk -F, '{ print $2 }' data.csv # comma separated
awk -F'\t' '{ print $2 }' data.tsv$0 is the whole line, $1..$n the fields, NF the field count, NR the line number.
Patterns and actions#
awk '$3 > 100' access.log # print lines where field 3 > 100
awk '/ERROR/ { print $1 }' app.log # match then act
awk 'NR > 1' data.csv # skip a header row
awk '$2 == "ERROR" { errors++ } END { print errors+0 }' app.logThe three one-liners worth memorising#
# 1. sum a column
awk '{ sum += $3 } END { print sum }' data.txt
# 2. group and count
awk '{ count[$1]++ } END { for (k in count) print count[k], k }' access.log \
| sort -rn | head
# 3. average
awk '{ s += $2; n++ } END { if (n) printf "%.2f\n", s/n }' data.txtThat second one — an associative array plus END — is the pattern that replaces a surprising number of scripts. "Top ten IPs by request count" is one line.
awk '{ print $1 }' access.log | sort | uniq -c | sort -rn | headsort and friends#
sort file # lexical
sort -n file # numeric — "9" before "10"
sort -rn file # numeric, descending
sort -u file # unique
sort -k2 file # by the second field
sort -t, -k3 -n data.csv # comma-separated, third field, numeric
sort -h # human-readable sizes: 2K < 1M < 1Guniq only collapses adjacent duplicates, so it is almost always preceded by sort:
sort file | uniq # unique lines
sort file | uniq -c # with counts
sort file | uniq -d # only the duplicated ones
sort file | uniq -c | sort -rn # ranked by frequencyPutting them together#
# top 10 slowest endpoints from an access log
awk '{ print $7, $NF }' access.log \
| sort -k2 -rn \
| head -10 \
| awk '{ printf "%-40s %sms\n", $1, $2 }'
# how many errors per hour today
grep ERROR app.log \
| grep -oE '^[0-9-]+T[0-9]{2}' \
| sort | uniq -cEach stage does one thing. That is the shell working as intended.
When to stop#
These tools are excellent at line-oriented text. They are the wrong tool for:
- JSON →
jq. Always. A regex over JSON works until it does not. - YAML →
yq. - CSV with quoted fields containing commas →
awk -F,will split inside the quotes and give you wrong data silently. Usecsvkit, or Python. - Anything needing more than three pipeline stages of logic → Python. If your
awkprogram has functions in it, you have written a program in the wrong language.
The rule
grep, sed, awk and sort are for filtering and reshaping lines of text between two other programs. The moment the data has structure — nesting, quoting, types — use a tool that understands that structure.
Portability summary#
| Thing | Note |
|---|---|
sed -i | needs '' on macOS, nothing on Linux. Avoid; write to a temp file. |
grep -P | GNU only. Use -E and POSIX classes. |
\d \w \s | not in ERE. Use [[:digit:]] etc. |
sort order | locale-dependent. LC_ALL=C when it matters. |
awk | macOS ships BSD awk; gawk has extensions BSD lacks. Stick to POSIX awk. |
readlink -f | GNU only. Use a cd/pwd subshell instead. |
Exercise#
#!/bin/bash
# From the log lines below, print the count of each level, most frequent first,
# in the form "<count> <LEVEL>".
set -euo pipefail
printf '%s\n' \
"2026-09-04 ERROR db timeout" \
"2026-09-04 INFO started" \
"2026-09-04 ERROR db timeout" \
"2026-09-04 WARN slow query" \
"2026-09-04 ERROR connection refused" \
"2026-09-04 INFO ready" > /tmp/app.log
# write your code hereCommon questions#
awk or a Python script?#
awk for one-liners over columns in a pipeline — it is faster to write and faster to run. Python the moment you need data structures, error handling, or anything you would want to unit test. A multi-line awk program with functions is a signal you crossed the line two edits ago.
Why does my sed -i work locally and fail in CI?#
macOS ships BSD sed, which requires a backup-suffix argument to -i; GNU sed does not accept one. Write to a temporary file and mv it into place — that works on both and is one extra token.
grep, egrep or grep -E?#
grep -E. egrep is deprecated and prints a warning on newer GNU grep. Using -E everywhere also means one regex dialect across grep, sed -E and [[ =~ ]].
Get the Bash agent pack
A battle-tested AGENTS.md, the review checklist, and the failure-mode cheat sheet for Bash. One email, then occasional updates when the tooling shifts. No course pitch.
AGENTS.md now — no email needed.