Samurai Penguin's Lair
Bash one-line script to find common referrers 
Monday, May 5, 2008, 02:55 PM
Recently, I had a need to find common referring URLS for a site NOW (which meant no time to install webalizer, which I highly recommend.) This is a fairly standard bash one-liner, but it might be useful to someone in a pinch:

grep "/page.html" access.* | awk '{print $11}' | sort | uniq -c | sort -n

Basically, this checks for all requests to page.html in all access logs. It then spits out only the 11th column(referrer), then counts and sorts.

Other useful columns (assuming that you're using the standard "combined" log format:
- $1 - IP address
- $12+ - user agent
- $6 - request type(GET, POST, PUT, etc.)

This can be useful in a number of different ways. For example, trying to find abusive behaviors can be done by looking for a lot of POST and PUT requests from a specific IP:

grep "POST\|PUT" access.* | awk '{print $1}' | sort | uniq -c | sort -n

Hopefully this is of use to someone!

Comments 

Add Comment

Fill out the form below to add your own comments.









Insert Special: