Nginx Access Log Analysis Checklist for Bot Traffic
Nginx access logs show who requested your website, which URLs were hit, what status codes were returned and how often requests happened. This is useful for finding aggressive bots, crawl waste, suspicious traffic and overloaded URLs.
What to look for
- Too many requests from one IP.
- Repeated hits to wp-login.php.
- Repeated 404 URLs.
- Suspicious query strings.
- Heavy bot traffic from unknown user agents.
- Requests that trigger 502 or 504 errors.
Checklist
- Locate the access log file.
- View recent requests.
- Find IPs with high request counts.
- Find URLs requested most often.
- Check status codes like 404, 500, 502 and 504.
- Check requests to wp-login.php and xmlrpc.php.
- Look for suspicious query strings.
- Compare access log with error log.
- Block abusive IPs only after confirming behavior.
- Document recurring bot patterns.
Final check
Access log analysis helps you stop guessing. It shows traffic patterns that may explain slow websites, server overload and security problems.
Estimated Time
30–60 minutes
Use Case
Use this checklist when your website is slow, overloaded or receiving suspicious bot traffic.
Common Mistakes
- Blocking Googlebot by mistake.
- Only reading error log and ignoring access log.
- Not checking status codes.
- Confusing high traffic with attack.
- No documentation of blocked IPs.
Related Commands
sudo tail -n 100 /var/log/nginx/access.log
sudo awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -nr | head
sudo awk '{print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -nr | head
sudo tail -n 100 /var/log/nginx/error.log