Shell Course 6

Chapter 6: Advanced Tools and Practical Tips

## Shell Customization

Make the shell truly yours by customizing its behavior and appearance.

Configuration Files

The main files that control shell behavior:

1
2
3
4
5
6
# For Bash
~/.bashrc        # Settings for interactive shells
~/.bash_profile  # Settings for login shells

# For Zsh (macOS default)
~/.zshrc         # Settings for interactive shells

Customizing Your Prompt

Change your prompt to show useful information:

1
2
3
4
5
# Basic prompt customization in ~/.bashrc
PS1="\u@\h:\w\$ "  # username@hostname:current_directory$

# Colorful prompt
PS1="\[\033[32m\]\u@\h\[\033[00m\]:\[\033[34m\]\w\[\033[00m\]\$ "

Creating Aliases

Create shortcuts for common commands:

1
2
3
4
# Add to ~/.bashrc
alias ll='ls -la'
alias ..='cd ..'
alias update='sudo apt update && sudo apt upgrade'

Useful Functions

For more complex operations, use functions:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# Create and enter directory in one step
mkcd() {
    mkdir -p "$1" && cd "$1"
}

# Extract various archive formats
extract() {
    if [ -f "$1" ]; then
        case "$1" in
            *.tar.bz2) tar xjf "$1" ;;
            *.tar.gz)  tar xzf "$1" ;;
            *.zip)     unzip "$1"   ;;
            *)         echo "Unknown format" ;;
        esac
    else
        echo "'$1' is not a valid file"
    fi
}

Remember to run source ~/.bashrc after making changes to apply them to your current session.

## Text Processing Tools

Master these powerful text manipulation utilities to process data efficiently.

Advanced grep Techniques

1
2
3
4
5
6
7
8
# Search recursively with context
grep -r -A2 -B1 "error" /var/log/

# Count matches
grep -c "authentication failure" /var/log/auth.log

# Show only matching filenames
grep -l "TODO" *.txt

Text Processing with sed

The stream editor for transforming text:

1
2
3
4
5
6
7
8
9
10
11
# Replace text
sed 's/old/new/' file.txt

# Global replacement (all occurrences in each line)
sed 's/old/new/g' file.txt

# Delete lines matching pattern
sed '/pattern/d' file.txt

# In-place editing with backup
sed -i.bak 's/old/new/g' file.txt

Data Processing with awk

Perfect for column-based data:

1
2
3
4
5
6
7
8
9
10
11
# Print specific columns
awk '{print $1, $3}' file.txt

# Process CSV files
awk -F, '{print $2}' data.csv

# Sum values in a column
awk '{sum += $3} END {print sum}' data.txt

# Calculate average
awk '{sum += $1; count++} END {print sum/count}' data.txt

Combining these tools creates powerful data processing pipelines, like grep "ERROR" logs.txt | awk '{print $1, $2}' | sort | uniq to find unique timestamps of errors.

## Advanced File Operations

Go beyond basic file commands with these advanced techniques.

Find and Act on Files

The versatile find command:

1
2
3
4
5
6
7
8
# Find and delete old files
find /tmp -type f -mtime +30 -delete

# Find and change permissions
find . -type f -name "*.sh" -exec chmod +x {} \;

# Find large files
find . -type f -size +10M

Efficient Remote File Operations

Copy files between systems:

1
2
3
4
5
6
7
8
# Copy to remote system
scp file.txt user@remote:/path/

# Copy from remote to local
scp user@remote:/path/file.txt .

# Synchronize directories with rsync
rsync -avz --progress local_dir/ user@remote:/path/

The trailing slash in rsync source paths matters! source/ means “contents of source” while source (no slash) means “the source directory itself.”

## Package Management

Install and manage software efficiently.

Advanced apt (Debian/Ubuntu)

1
2
3
4
5
6
7
8
9
10
11
# Search for packages
apt search keyword

# Show package details
apt show package_name

# List installed packages
apt list --installed

# Remove unused dependencies
apt autoremove

Advanced brew (macOS)

1
2
3
4
5
6
7
8
9
10
11
# Search for formulae
brew search keyword

# Show formula information
brew info formula

# List outdated formulae
brew outdated

# Services management
brew services start mysql

## Network Tools

Essential tools for network operations and diagnostics.

Downloading and API Requests

1
2
3
4
5
6
7
8
# Download with curl
curl -O https://example.com/file.zip

# Resume interrupted download
curl -C - -O https://example.com/large-file.iso

# Make API request and format JSON
curl https://api.example.com/data | jq

SSH Advanced Techniques

1
2
3
4
5
6
7
8
9
10
11
# Run a command on remote server
ssh user@server "df -h"

# Local port forwarding (access remote service locally)
ssh -L 8080:localhost:80 user@server

# Configure SSH with ~/.ssh/config
Host myserver
    HostName server.example.com
    User username
    Port 2222

Network Diagnostics

1
2
3
4
5
6
7
8
# Trace network route
traceroute example.com

# DNS lookup
dig example.com

# Check open ports
netstat -tuln

## Shell Productivity Boosters

These shortcuts and techniques will make you much more efficient.

Command History Navigation

1
2
3
4
5
history           # Show command history
!42               # Run command #42 from history
!!                # Repeat last command
!string           # Run most recent command starting with "string"
^old^new          # Replace "old" with "new" in previous command

Essential Keyboard Shortcuts

1
2
3
4
5
6
Ctrl+a            # Move to beginning of line
Ctrl+e            # Move to end of line
Ctrl+u            # Delete from cursor to start of line
Ctrl+k            # Delete from cursor to end of line
Ctrl+r            # Reverse search history
Alt+.             # Insert last argument of previous command

Brace Expansion

Generate strings with patterns:

1
2
3
4
echo file{1,2,3}.txt     # file1.txt file2.txt file3.txt
mkdir -p project/{src,doc,test}
cp file.txt{,.bak}       # Copies file.txt to file.txt.bak
echo {1..5}              # 1 2 3 4 5

Process Substitution

Treat command output as a file:

1
2
diff <(ls dir1) <(ls dir2)
cat <(grep "error" log1) <(grep "error" log2)

Learning these shortcuts might seem like a small investment, but the time savings add up dramatically over your career.

## Terminal Multiplexers

Manage multiple terminal sessions within a single window.

tmux Basics

1
2
3
tmux                  # Start new session
tmux ls               # List sessions
tmux attach -t 0      # Attach to session 0

Inside tmux, use the prefix key (Ctrl+b) followed by:

  • c: Create new window
  • %: Split vertically
  • ": Split horizontally
  • arrow keys: Move between panes
  • d: Detach from session

tmux keeps your sessions running even if your connection drops, making it essential for remote work.

## Shell Security Best Practices

Keep your scripts and systems secure with these practices.

Secure Temporary Files

1
2
3
4
5
6
# Bad: Predictable filename
TEMP_FILE="/tmp/myapp_data.txt"

# Better: Random filename with cleanup
TEMP_FILE=$(mktemp)
trap 'rm -f "$TEMP_FILE"' EXIT

Robust Error Handling

1
2
3
4
5
6
7
8
9
#!/bin/bash
set -euo pipefail  # Exit on error, undefined var, pipeline failure

error_handler() {
    echo "Error on line $1" >&2
    exit 1
}

trap 'error_handler $LINENO' ERR

Validating Input

1
2
3
4
5
# Check if value is numeric
if [[ ! "$input" =~ ^[0-9]+$ ]]; then
    echo "Error: Input must be numeric" >&2
    exit 1
fi

## Quick Practical Examples

Small, useful scripts you can adapt to your needs.

System Status Checker

1
2
3
4
5
6
7
8
9
10
11
12
13
#!/bin/bash
# quick_status.sh - Basic system status check

{
    echo "=== System Status at $(date) ==="
    echo "Load average: $(uptime | awk -F'[a-z]:' '{ print $2}')"
    echo "Memory usage:"
    free -h | grep "Mem"
    echo "Disk usage:"
    df -h / | tail -n 1
    echo "Most CPU-intensive processes:"
    ps aux --sort=-%cpu | head -n 6
} | tee ~/status_report.txt

Log Monitor

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
#!/bin/bash
# quick_log_monitor.sh - Check logs for errors

LOG_FILE="/var/log/syslog"
KEYWORDS=("error" "failed" "critical")

echo "Checking $LOG_FILE for issues..."
for keyword in "${KEYWORDS[@]}"; do
    count=$(grep -i "$keyword" "$LOG_FILE" | wc -l)
    if [ "$count" -gt 0 ]; then
        echo "Found $count lines with '$keyword'"
        grep -i "$keyword" "$LOG_FILE" | tail -n 3
        echo
    fi
done

Backup Script

1
2
3
4
5
6
7
8
9
10
11
#!/bin/bash
# quick_backup.sh - Simple backup tool

SOURCE="$HOME/Documents"
DEST="$HOME/Backups"
ARCHIVE="backup_$(date +%Y%m%d).tar.gz"

mkdir -p "$DEST"
echo "Backing up $SOURCE to $DEST/$ARCHIVE..."
tar -czf "$DEST/$ARCHIVE" -C "$(dirname "$SOURCE")" "$(basename "$SOURCE")"
echo "Done! Archive size: $(du -h "$DEST/$ARCHIVE" | cut -f1)"

These simple scripts demonstrate how powerful the shell can be for automating routine tasks. Adapt them to your specific needs and build your toolkit over time.

## Conclusion

Throughout this course, we’ve journeyed from basic shell commands to advanced techniques and tools. You now have the knowledge to navigate the filesystem, manipulate files, process text, automate tasks with scripts, and enhance your productivity with various shell features.

The shell is more than just a command prompt—it’s a powerful environment for solving problems and extending your capabilities as a developer or system administrator.

Final tips for shell mastery:

  1. Learn incrementally: Focus on understanding concepts rather than memorizing commands
  2. Practice regularly: Use the shell for your daily tasks
  3. Read others’ scripts: Learn from experienced users
  4. Customize thoughtfully: Build your configuration files over time
  5. Embrace the philosophy: Simple tools combined in powerful ways

The shell may seem arcane at times, with its terse commands and cryptic options, but it’s a tool that rewards investment. The time you spend learning the shell will be repaid many times over in increased productivity throughout your computing career.

Happy shell scripting!