# Redirection and Here Documents

> Source: https://learn-bash.net/here-documents-and-redirection/
> Part of Learn Bash, free to read.

Every process starts with three open file descriptors:

```text
0  stdin    input
1  stdout   normal output
2  stderr   diagnostics
```

Redirection changes where they point.

```bash
cmd > out.txt         # stdout to a file, TRUNCATING it
cmd >> out.txt        # stdout appended
cmd 2> err.txt        # stderr to a file
cmd < in.txt          # stdin from a file
cmd > /dev/null       # discard stdout
```

## Combining streams

```bash
cmd > all.txt 2>&1        # stdout to the file, then stderr to wherever stdout goes
cmd &> all.txt            # bash shorthand for the same
cmd >> all.txt 2>&1       # appending version
cmd 2>&1 | tee log.txt    # both streams into a pipe AND a file
```

:::warn Order matters, and it is counter-intuitive
```bash
cmd > file 2>&1     # correct: stdout to file, then stderr follows it there
cmd 2>&1 > file     # WRONG: stderr goes to the TERMINAL, stdout to the file
```
`2>&1` means "make fd 2 point wherever fd 1 points **right now**". In the second form fd 1 is still the terminal at that moment, so stderr is bound to the terminal before stdout is moved. Reading it right to left helps.
:::

## Sending diagnostics to stderr

```bash
echo "processing $file"            # to stdout — part of the output
echo "warning: skipping" >&2       # to stderr — a diagnostic
```

Getting this right is what lets a caller do `./script.sh > results.txt` and still see the warnings. Anything that is not the script's actual output belongs on stderr — usage text, progress, errors.

```bash
die() { printf '%s\n' "$*" >&2; exit 1; }
```

## Here documents

A block of text fed to a command's stdin.

```bash
cat <<EOF
Deploying $APP_NAME
Target: $TARGET
User:   $(whoami)
EOF
```

Variables and command substitutions are expanded. **Quote the delimiter to switch that off:**

```bash
cat <<'EOF'
This $VARIABLE is printed literally.
So is $(this) and `this`.
EOF
```

The quoted form is what you want whenever the text contains shell syntax you do not want evaluated — a config file, a generated script, an awk program.

```bash
generate_service_file() {
  cat > /etc/systemd/system/myapp.service <<EOF
[Unit]
Description=$APP_NAME

[Service]
ExecStart=$INSTALL_DIR/bin/myapp
Restart=always
EOF
}
```

Note the mixed use: the delimiter is unquoted here **because** we want `$APP_NAME` and `$INSTALL_DIR` expanded.

### Indented here documents

`<<-` strips leading **tabs** (not spaces), so the block can be indented with the surrounding code:

```bash
if [[ -n "$verbose" ]]; then
	cat <<-'EOF'
	This text is indented in the source
	but printed flush left.
	EOF
fi
```

The indentation must be actual tab characters. That requirement catches people out often enough that many scripts just leave here-docs unindented.

## Here strings

For a single line, `<<<` is shorter:

```bash
grep -q "error" <<< "$log_line"
jq -r '.name' <<< "$json"
read -r first second <<< "$line"
```

`<<<` adds a trailing newline, which matters when hashing or comparing exact bytes — use `printf '%s'` piped instead if that is a problem.

## Process substitution

`<(cmd)` makes a command's output look like a file:

```bash
diff <(sort a.txt) <(sort b.txt)          # compare without temp files
comm -13 <(sort old.txt) <(sort new.txt)  # lines only in new
```

And the crucial use — feeding a `while read` loop without a subshell:

```bash
count=0
while read -r line; do
  count=$((count + 1))
done < <(find . -name '*.log')
echo "$count"           # correct

count=0
find . -name '*.log' | while read -r line; do
  count=$((count + 1))  # runs in a SUBSHELL
done
echo "$count"           # 0 — the variable never escaped
```

That second form is one of the most common shell bugs, and process substitution is the fix. Note the space in `< <(` — `<<(` is a syntax error.

`>(cmd)` works the other way, treating a command's input as a file:

```bash
tar czf >(ssh backup-host 'cat > backup.tgz') /data
```

## Reading files safely

```bash
while IFS= read -r line; do
  printf '%s\n' "$line"
done < input.txt
```

Three deliberate parts: `IFS=` preserves leading and trailing whitespace, `-r` stops backslash interpretation, and redirecting the file avoids the subshell. Generated scripts routinely omit all three.

```bash
# handle a final line with no trailing newline
while IFS= read -r line || [[ -n "$line" ]]; do
  printf '%s\n' "$line"
done < input.txt
```

## Exercise

```bash
#!/bin/bash
set -euo pipefail

# 1. Write a `usage` function printing a multi-line here-doc to STDERR
#    with the delimiter quoted so nothing is expanded.
# 2. Write `render_config` that produces a config block WITH $APP and $PORT
#    expanded, redirected to a file.
# 3. Count the lines of `ls` output in a while loop such that the count is
#    still correct after the loop.
APP=myapp PORT=8080
# write your code here
```

## Common questions

### Why did `cmd 2>&1 > file` not capture my errors?

Because redirections apply left to right. `2>&1` binds stderr to whatever stdout points at *at that moment* — still the terminal. Move stdout first: `cmd > file 2>&1`.

### Quoted or unquoted here-doc delimiter?

Quote it (`<<'EOF'`) unless you specifically want expansion. Unquoted is right for templating in variables; quoted is right for embedding config, scripts, or anything containing `$` you want preserved literally.

### Why does my variable lose its value after a `while` loop?

The right-hand side of a pipe runs in a subshell, so assignments inside it do not survive. Use process substitution — `done < <(command)` — and the loop runs in the current shell.