adminGrep Awk Sort Uniq Log Filtering Checklist
grep, awk, sort and uniq turn large logs into useful evidence. They help system admins find attack patterns, top IPs, status code spikes, repeated URLs, failed login attempts and noisy bots.
Core principle
Logs are raw data. Filtering and counting turn raw data into operational intelligence.
Checklist
- Identify the log file to inspect.
- Filter by keyword or endpoint.
- Extract IP addresses.
- Count top IPs.
- Count status codes.
- Find repeated URLs.
- Filter by time range if possible.
- Separate normal traffic from suspicious traffic.
- Save useful commands for reuse.
- Turn repeated findings into monitoring rules.
Reusable lesson
This applies to Nginx access logs, auth logs, WordPress attacks, API errors, webhook abuse, crawler detection and security monitoring.
When to Use This Checklist
Use this checklist when analyzing logs to find repeated errors, abusive traffic, attack patterns, bot spikes or high-volume endpoints.
Required Tools
SSH access, log files, grep, awk, sort, uniq, incident time, access log format
Before You Start
Do not block traffic based on one line. Look for repeated patterns and confirm whether the traffic is truly abusive.
Structured Checklist Steps
- Pick log file.
- Filter keyword.
- Extract IPs.
- Count top IPs.
- Count status codes.
- Find repeated URLs.
- Filter by time if possible.
- Classify traffic.
- Save command.
- Create monitoring rule.
Verification Steps
- Top IPs are known.
- Status code distribution is visible.
- Repeated URLs are identified.
- Suspicious pattern is evidence-based.
- Reusable command is saved.
Rollback Plan
If a blocking decision based on log analysis causes false positives, remove the block and refine the pattern with narrower evidence.
Common Mistakes
- Blocking after one request.
- Not counting frequency.
- Ignoring status codes.
- Not checking user agents or URLs.
- No saved investigation commands.
Related Commands
sudo awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -nr | head
sudo awk '{print $9}' /var/log/nginx/access.log | sort | uniq -c | sort -nr
sudo grep "wp-login.php" /var/log/nginx/access.log | tail -n 50
sudo awk '{print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -nr | head