if you are not seeing this from your website developer or making statements like below about your web developer contact us today!

Working with Michael has been a dream. As the owner of a web design company, getting it right the first time to meet deadlines is super important. Not only is Michael on time, but everything always works flawlessly and above my expectations. He’s also a great problem solver – bringing unique and fresh ideas to the table in order to achieve a look or function we’re going for. Now, I can focus on client relations and design because I know I can can rely on him to implement the project on time and on budget. If you’re looking for a reliable web developer who knows his stuff, you have to use Michael. He’s fantastic!

view more of what our customers say >

Google Analytics – Blocking SPAM Bots and Crawlers - December 31, 2014

You log into Google Analytics and are looking over your analytics information. Digging through the data you want to understand who is referring traffic to your website and move down to Acquisition >> All referrals in the main menu. Your excitement fades as you realize that a large portion of your traffic comes from sites that you have never heard of, nor ones that you are aware of. These domain names will look SPAMMY and like websites you would not normally visit or expect to visit your site.

You may not be aware of this but you have just entered the world of Google Analytics SPAM bot / crawler traffic!

What is a Google Analytics SPAM bot / crawler?

SPAM bots / crawlers are automated programs that collect information about websites. They are constantly running, visiting sites, and operate on a random basis with no discernable pattern.

Most of these bots / crawlers will have a 100% bounce rate. That means they visit your website, gather a quick amount of information and then move onto the next site in their list. 

What does this mean for your Google Analytics?

Now you may be curious if this is important or affects you at all.

We will keep this simply and use some round numbers to explain what this means for you and your Google Analytics.

Over a 30 day period you have 100 visits to your website, 50 are New Sessions / New Users, and your bounce rate is 50%. – Easy numbers to work with right?

You notice that a site listed in your Acquisition >> All Referrals has visited your site 25 times during that period with a 100% bounce rate.

How has this bot / crawler program affected your Google Analytics Numbers?

  1. First: You actually had 75 legitimate visits once you remove the 25 bot / crawler visits
  2. Second: You actually had 25 New Sessions as these bot / crawler visits will usually be categorized as New Sessions / New Users
  3. Third: Your actual bounce rate is not 50% it is 30%
    1. Instead of you have 100 visits with a 50% bounce rate (equals 50 users that bounced from your site) you have 75 visits with 25 users that bounced. This equals 75/25 = 30%.
  4. Fourth: You have other numbers that are affected such as Pages / Session, Avg. Session Duration, Goals and Conversions, etc. 

You can see with the information above how this can affects your actual Google Analytics numbers and how one program impacts your analytics. Since there are numerous bots / crawlers running all the time, you can three or more Source entries that are skewing your numbers.

So the next question is: What can I do to filter out these bots / crawlers from my Google Analytics numbers?

Some of these bots, like semalt.semalt.com, have a page that explains what their bot / crawler does and include a “Remove a site from our seed list” form you can fill out. You can access the page here

There is an easier method though and you can do this right from Google Analytics.

IMPORTANT: Once you follow the steps below you WILL NOT be able to have data returned to your Google Analytics account. Use this technique only if you are sure you want to filter out these bot / crawler programs.

  1. Log into your Google Analytics account
  2. Click on Home
    1. If you have more than one Account listed you will need to scroll down to find the correct account. Click on the View that loads the information you want to filter.
  3. You should now be in the Reporting Tab for the account.
  4. Scroll down and click on Acquisition >> All Traffic in the left hand menu
    1. This has the list of sites you want to filter out referral traffic for
    2. Ensure that you are removing sites that are indeed SPAM bots / crawlers.
      1. Typically they have all of the following traits:
        1. Fair amount of traffic to your site
        2. Very high New Sessions
        3. Very high New Users
        4. Very high Bounce Rate or the Bounce Rate is at 100%
        5. Pages / Session is at 1 or very close to 1
        6. Average Session Duration
      2. IMPORTANT: If in doubt, go to that domain and see if it is one that looks like it should be linking to your site.
  5. Now to make it easier to copy and paste in the top menu Right Click on the Admin tab in the gray menu bar and open in a new tab or window
  6. Find the heading of Property and click on Tracking Info
  7. Click on Referral Exclusion List
  8. You will see an Orange / Red button that says “+ Add Referral Exclusion” on it. Click on that button
  9. The page will refresh and you will see a heading of Exclude this domain from your referral traffic, the word domain, and a text box. If you see this you are in the right spot.
  10. Go back to your All Referrals browser window that has the list of referral sites you want to exclude.
    1. Copy and paste the domain names you want to exclude
    2. Click the Blue Create button
    3. Repeat for each domain you want to exclude

It is important to remember that once you apply this filter you will no longer see that information in Google Analytics account starting on the day and time you added the exclusion. New traffic will not include these crawlers / bots, however, that information is still show for any reports that include days before you added the exclusion. You can remove any of these domains at any time and tracking will start once they are no longer included in your exclusion list.

If you have questions or are not confident you can make the changes above, you can talk with your web developer and have them implement them for you. If you are without a web developer, contact Spotlight Web Design today and we will work with you to implement and make the changes.