adminNginx Access Log Analysis Checklist for Bot Traffic
Nginx access logs show who requested your website, which URLs were hit, what status codes were returned and how often requests happened. This is useful for finding aggressive bots, crawl waste, suspicious traffic and overloaded URLs.
What to look for
- Too many requests from one IP.
- Repeated hits to wp-login.php.
- Repeated 404 URLs.
- Suspicious query strings.
- Heavy bot traffic from unknown user agents.
- Requests that trigger 502 or 504 errors.
Checklist
- Locate the access log file.
- View recent requests.
- Find IPs with high request counts.
- Find URLs requested most often.
- Check status codes like 404, 500, 502 and 504.
- Check requests to wp-login.php and xmlrpc.php.
- Look for suspicious query strings.
- Compare access log with error log.
- Block abusive IPs only after confirming behavior.
- Document recurring bot patterns.
Final check
Access log analysis helps you stop guessing. It shows traffic patterns that may explain slow websites, server overload and security problems.
When to Use This Checklist
Use this checklist when your website is slow, overloaded or receiving suspicious bot traffic.
Required Tools
SSH access, Nginx access log, Nginx error log, awk, sort, uniq, firewall access
Before You Start
Do not block IPs blindly. First confirm the traffic pattern and make sure the IP is not a legitimate service or search engine.
Structured Checklist Steps
- Locate access log.
- View recent requests.
- Count top IPs.
- Find top requested URLs.
- Check status codes.
- Check wp-login.php traffic.
- Check xmlrpc.php traffic.
- Look for suspicious query strings.
- Compare with error log.
- Document and block only confirmed abuse.
Verification Steps
- Top abusive IPs are identified.
- Heavy URLs are known.
- Status code patterns are understood.
- Any firewall block is documented.
- Server load is checked after mitigation.
Rollback Plan
If a firewall rule blocks legitimate traffic, remove the rule and use more specific conditions such as path-based protection or rate limiting.
Common Mistakes
- Blocking Googlebot by mistake.
- Only reading error log and ignoring access log.
- Not checking status codes.
- Confusing high traffic with attack.
- No documentation of blocked IPs.
Related Commands
sudo tail -n 100 /var/log/nginx/access.log
sudo awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -nr | head
sudo awk '{print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -nr | head
sudo tail -n 100 /var/log/nginx/error.log