# Special Commands: sed, awk, grep, sort

> Source: https://learn-bash.net/special-commands-sed-awk-grep-sort/
> Part of Learn Bash, free to read.

These four have enormous manual pages and you need perhaps ten per cent of each. This page is that ten per cent, plus the portability traps that make scripts work on Linux and fail on macOS.

## grep — find lines

```bash
grep -E 'ERROR|FATAL' app.log      # -E: extended regex. use it always.
grep -i warning app.log            # case insensitive
grep -v DEBUG app.log              # invert: lines NOT matching
grep -c ERROR app.log              # count matching lines
grep -n ERROR app.log              # with line numbers
grep -l ERROR *.log                # just the filenames
grep -q ERROR app.log              # silent; exit status only — for `if`
grep -o -E '[0-9]+ms' app.log      # print each MATCH, not the line
grep -A3 -B1 ERROR app.log         # 3 lines after, 1 before
grep -r TODO src/                  # recursive
grep -F 'a.b.c' file               # fixed string: no regex, much faster
```

The three worth internalising:

- **`-q` for conditions.** `if grep -q pattern file; then` — no output, just the exit status.
- **`-o` to extract.** Prints the matched text rather than the line. With `-E` this is a general-purpose extractor.
- **`-F` for literals.** When you are searching for a string that contains dots or brackets, `-F` is both correct and faster.

```bash
# every unique IP in a log
grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}' access.log | sort -u
```

:::tip `ripgrep` is a drop-in upgrade
`rg` is much faster, respects `.gitignore`, and uses the same regex syntax you already know. `grep` is what is installed everywhere, so scripts should use `grep`; interactively, use `rg`.
:::

## sed — edit a stream

Ninety per cent of `sed` use is one command: substitute.

```bash
sed 's/old/new/'      # first occurrence on each line
sed 's/old/new/g'     # every occurrence
sed 's/old/new/2'     # the second occurrence only
sed 's|/usr/local|/opt|g'          # any delimiter — use | for paths
sed -E 's/([0-9]{4})-([0-9]{2})/\2\/\1/'   # -E for ERE, \1 \2 for groups
```

Other things worth knowing:

```bash
sed -n '5,10p' file       # print lines 5-10 (-n suppresses default printing)
sed '/^#/d' file          # delete comment lines
sed '/^$/d' file          # delete blank lines
sed -n '/START/,/END/p'   # print a range between two patterns
sed '2i\inserted'         # insert before line 2
```

:::danger `sed -i` is not portable
This is the number one shell portability trap.

```bash
sed -i    's/a/b/' f      # GNU (Linux): edits in place
sed -i '' 's/a/b/' f      # BSD (macOS): requires an empty backup suffix
```

A script written on Linux with `sed -i` fails on macOS with a confusing error; written on macOS with `sed -i ''` it fails on Linux. Portable options:

```bash
sed 's/a/b/' f > f.tmp && mv f.tmp f       # works everywhere
perl -i -pe 's/a/b/' f                      # perl is everywhere too
```

Generated scripts get this wrong constantly, because most examples online assume GNU.
:::

## awk — work with columns

`awk` is a whole language, but the useful core is: for each line, split into fields, run a rule.

```bash
awk '{ print $1, $3 }' file        # first and third field
awk '{ print $NF }' file           # last field
awk 'NF' file                      # drop blank lines
awk -F, '{ print $2 }' data.csv    # comma separated
awk -F'\t' '{ print $2 }' data.tsv
```

`$0` is the whole line, `$1`..`$n` the fields, `NF` the field count, `NR` the line number.

### Patterns and actions

```bash
awk '$3 > 100' access.log                     # print lines where field 3 > 100
awk '/ERROR/ { print $1 }' app.log            # match then act
awk 'NR > 1' data.csv                         # skip a header row
awk '$2 == "ERROR" { errors++ } END { print errors+0 }' app.log
```

### The three one-liners worth memorising

```bash
# 1. sum a column
awk '{ sum += $3 } END { print sum }' data.txt

# 2. group and count
awk '{ count[$1]++ } END { for (k in count) print count[k], k }' access.log \
  | sort -rn | head

# 3. average
awk '{ s += $2; n++ } END { if (n) printf "%.2f\n", s/n }' data.txt
```

That second one — an associative array plus `END` — is the pattern that replaces a surprising number of scripts. "Top ten IPs by request count" is one line.

```bash
awk '{ print $1 }' access.log | sort | uniq -c | sort -rn | head
```

## sort and friends

```bash
sort file                  # lexical
sort -n file               # numeric — "9" before "10"
sort -rn file              # numeric, descending
sort -u file               # unique
sort -k2 file              # by the second field
sort -t, -k3 -n data.csv   # comma-separated, third field, numeric
sort -h                    # human-readable sizes: 2K < 1M < 1G
```

`uniq` only collapses **adjacent** duplicates, so it is almost always preceded by `sort`:

```bash
sort file | uniq            # unique lines
sort file | uniq -c         # with counts
sort file | uniq -d         # only the duplicated ones
sort file | uniq -c | sort -rn   # ranked by frequency
```

:::warn `sort` is locale-dependent
Sort order changes with `LC_ALL`, which means a script can produce different output on two machines. If the order matters — you are diffing, checksumming, or generating a file — force it:

```bash
LC_ALL=C sort file
```
:::

## Putting them together

```bash
# top 10 slowest endpoints from an access log
awk '{ print $7, $NF }' access.log \
  | sort -k2 -rn \
  | head -10 \
  | awk '{ printf "%-40s %sms\n", $1, $2 }'

# how many errors per hour today
grep ERROR app.log \
  | grep -oE '^[0-9-]+T[0-9]{2}' \
  | sort | uniq -c
```

Each stage does one thing. That is the shell working as intended.

## When to stop

These tools are excellent at line-oriented text. They are the wrong tool for:

- **JSON** → `jq`. Always. A regex over JSON works until it does not.
- **YAML** → `yq`.
- **CSV with quoted fields containing commas** → `awk -F,` will split inside the quotes and give you wrong data silently. Use `csvkit`, or Python.
- **Anything needing more than three pipeline stages of logic** → Python. If your `awk` program has functions in it, you have written a program in the wrong language.

:::verdict The rule
`grep`, `sed`, `awk` and `sort` are for filtering and reshaping lines of text between two other programs. The moment the data has structure — nesting, quoting, types — use a tool that understands that structure.
:::

## Portability summary

| Thing | Note |
|---|---|
| `sed -i` | needs `''` on macOS, nothing on Linux. Avoid; write to a temp file. |
| `grep -P` | GNU only. Use `-E` and POSIX classes. |
| `\d` `\w` `\s` | not in ERE. Use `[[:digit:]]` etc. |
| `sort` order | locale-dependent. `LC_ALL=C` when it matters. |
| `awk` | macOS ships BSD awk; `gawk` has extensions BSD lacks. Stick to POSIX awk. |
| `readlink -f` | GNU only. Use a `cd`/`pwd` subshell instead. |

## Exercise

```bash
#!/bin/bash
# From the log lines below, print the count of each level, most frequent first,
# in the form "<count> <LEVEL>".

set -euo pipefail
printf '%s\n' \
  "2026-09-04 ERROR db timeout" \
  "2026-09-04 INFO  started" \
  "2026-09-04 ERROR db timeout" \
  "2026-09-04 WARN  slow query" \
  "2026-09-04 ERROR connection refused" \
  "2026-09-04 INFO  ready" > /tmp/app.log

# write your code here
```

## Common questions

### awk or a Python script?

`awk` for one-liners over columns in a pipeline — it is faster to write and faster to run. Python the moment you need data structures, error handling, or anything you would want to unit test. A multi-line `awk` program with functions is a signal you crossed the line two edits ago.

### Why does my `sed -i` work locally and fail in CI?

macOS ships BSD sed, which requires a backup-suffix argument to `-i`; GNU sed does not accept one. Write to a temporary file and `mv` it into place — that works on both and is one extra token.

### grep, egrep or `grep -E`?

`grep -E`. `egrep` is deprecated and prints a warning on newer GNU grep. Using `-E` everywhere also means one regex dialect across `grep`, `sed -E` and `[[ =~ ]]`.