Log File Parsing for SEO’s

What is a log file?

Every time a request is made to a server this request is logged in the site’s log files. This means that log files record every single request whether it is made by a bot or by a real visitor to your site.

These logs record the following useful information:

  • IP Address of the visitor
  • The Date & Time of the request
  • The resource requested
  • The status code of this resource
  • The number of bytes downloaded

Why does this matter to SEO’s?

SEO’s need to know how search engine spiders are interacting with their sites. Google Search console provides some limited data including the number of pages visited and the amount of information downloaded. But this limited information doesn’t give us any real insight into how Googlebot is interacting with different templates, and different directories.

Looking at this log file data often turns up some interesting insights. When looking at one client’s site, we discovered that Googlebot was spending a large portion of it’s time crawling pages which drove only 3% of overall organic traffic. These pages where essentially eating massively into the site’s overall crawl budget, leading to a decrease in overall organic traffic. This data point would simply not be available to someone who just looked at the top level crawl data contained in Google’s Search Console.

What Can We Discover in Log Files

Log files allow us to discover what resources are being requested by Google. This can help us achieve the following:

  • See how frequently important URLs are crawled
  • See the full list of errors discovered by Googlebot during crawls
  • See whether Google is fully rendering pages using front-end frameworks such as React and AngularJS
  • See whether site structure can be improved to ensure commercially valuable URLs are crawled
  • Check implementation of Robots.txt rules, ensuring blocked pages are not being crawled and that certain page types are not being unexpectedly blocked.

Options Available 

There are a number of different options available for SEO’s who want to dive into their log file data.

Analysing Data in Excel

As log files are essentially space separated text files, it’s possible to open these log files up in Excel and then analyse them. This can be fine when you are working with a very small site, but with bigger sites it isn’t a viable option. Just verifying whether traffic is really from Googlebot can be quite a pain and Excel is not really designed for this type of analysis.

Commercial Options

There are a number of commercial options available for those who want to undertake analysis of their log files. Botify offer a log file analysis service to users depending on their subscription package. Of course the Botify subscription is very pricey, and is not going to be a viable option for many SEO’s.

Screaming Frog also offer a log file parsing tool, which is available for £99 a year. As with Screaming Frog’s SEO Spider the mileage you will get with this tool really depends on the size of your available RAM and how big the site you are dealing with is.

Parsing Log Files with Python

Log files are plain text files and can easily be parse by Python. The valuable information contained on each line of a log file is delimited by a space. This means that we are able to run through the file line by line, and pull out the relevant data.

There are a couple of challenges when parsing a log file for SEO purposes:

  • We need to pull out Googlebot results and verify that they are actually from Google IP address.
  • We need to normalise date formats to make our analysis of our log files easier.

I have written a Python script which deals with both of these issues and outputs the underlying data to a SQLite database. Once we have the data into SQL format, we can either export this into a CSV file for further analysis or query the database to pull out specific information.

Our Script

The above script can be used without any Python knowledge, by simply moving the script to the directory where you have downloaded your log files. On the last line of the script, the user changes out the ‘example-extension’ to the extension that their log files have been saved with and updates ‘exampledomain.com’ with the domain in question.

As the above script is not reliant on libraries outside of Python’s standard library, anyone with Python should be able to save the script and simply run it.

Leave a Reply

Your email address will not be published. Required fields are marked *