How To Block Web Crawlers & Reduce Google Analytics Bounce Rates

Contents

We speak about conversions on this blog all the time and one of the biggest questions we received from clients is why doesn’t our traffic convert into business more often. Business owners often tell me that their website receives 100 visitors a day yet rarely receive any enquiries or sales. 100 visitors a day for most business would be a solid amount of traffic and unless you had the worse web design in the world I would expect some of it to convert.

This scenario is all to common however and something that’s well worth investigating. Often upon review of Google Analytics it’s easy to pick out some clear warning signs that the visitors might not be of the human kind and instead bots and crawlers scanning your website for whatever reason. There are few different categories of bots and crawlers and some of them you really want and others not so much.

For example Google will crawl your website on regular basis and index your pages into their search engine, I think we all agree this is definitely wanted. However there are also a number of others such as semalt, simple-share-buttons and more that don’t add any value in terms of search referrals and can actually increase statistics such as bounce rate and average session duration. From a client perspective I want to present them with realistic traffic rather than explaining that it was nothing of interest.

Here is an example of a new website that was launched recently upon a day where it recorded a modest 75 sessions for the day. Not too bad in the scheme of things right? Most small business owners would be happy but most would quickly begin wondering why no enquiries were sent. Is the contact form broken? Let’s take a look at some of the key statistics in the below screenshot and see if we can understand what might be occurring.

bounce-rate-bots-crawlers

Channel traffic

Out of the 75 sessions for the day 90% of the visitor traffic was sent via a referral website. Out of those 68 sessions there was a 100% bounce rate!?!?! 68 users that have visited your website and not browsed anything after landing on the initial page. This is a warning sign that maybe this traffic isn’t exactly what you expected it to be. Finally we can see that the average session duration is 00:00:00 i.e. nothing. So 90% of the visitors that have come to our website haven’t browsed any additional pages or interacted on our website in any way at all.

Doesn’t sound like a human kind of interaction to me.

Referral traffic

Let’s hop over to the referral tab and have a look at the websites that are sending traffic to our website.

referrals-bounce-rate-crawlers-bots

Immediately I notice that for this particular day the top three referrals are a website called simple-share-buttons.com.  After quick review online it seems that this website might be some kind of scanner imitating the legitimate site which is actually called simplesharebuttons.com. Once again we can see that out of all the visitors from this domain there was a 100% bounce rate and session duration of zero.

As you can see the greater the amount of traffic we receive from these types of sites the larger our average bounce rate percentage becomes. It also decreases the average duration of visitors on our website and individual pages. Now not everyone agrees but bounce rate is a potential ranking factor that Google can use to determine whether a websites content is engaging with your visitors. Although there is nothing written to say Google uses this information it would make sense to for them to use it as factor.

If the traffic is of no value then it makes sense then we should look at preserving our statistics as best we can

How to Block Crawlers & Bots in .htacess

Fortunately for us we are able to block this kind of traffic to our website using our .htaccess file. We spoke briefly about how to access the .htaccess on our guide on how to launch a new website and maintain your SEO so please read that for ideas on how to access it. In your .htaccess add a section such as the following for the domains you wish to block. You should be careful blocking websites that aren’t affecting your statistics and only block them when you feel it’s necessary. Each block should be tested to ensure that legitimate traffic is not affected.

## STOP BOTS ## 
SetEnvIfNoCase Referer evil-spam-domain.com spammer=yes 
SetEnvIfNoCase Referer evil-spam-keyword spammer=yes 
SetEnvIfNoCase Referer semalt.com spammer=yes 
SetEnvIfNoCase Referer poker spammer=yes 
SetEnvIfNoCase Referer buttons-for-website.com spammer=yes 
SetEnvIfNoCase Referer simple-share-buttons.com spammer=yes
SetEnvIfNoCase Referer 4webmasters.org spammer=yes
SetEnvIfNoCase Referer anticrawler.org spammer=yes
SetEnvIfNoCase Referer bestwebsitesawards.com spammer=yes
SetEnvIfNoCase Referer best-seo-solution.com spammer=yes
SetEnvIfNoCase Referer best-seo-offer.com spammer=yes
SetEnvIfNoCase Referer blackhatworth.com spammer=yes
SetEnvIfNoCase Referer buttons-for-website.com spammer=yes
SetEnvIfNoCase Referer buttons-for-your-website.com spammer=yes
SetEnvIfNoCase Referer buy-cheap-online.info spammer=yes
SetEnvIfNoCase Referer econom.co spammer=yes
SetEnvIfNoCase Referer darodar.com spammer=yes
SetEnvIfNoCase Referer hulfingtonpost.com spammer=yes
SetEnvIfNoCase Referer Get-Free-Traffic-Now.com spammer=yes
SetEnvIfNoCase Referer googlsucks.com spammer=yes
SetEnvIfNoCase Referer free-share-buttons.com spammer=yes
SetEnvIfNoCase Referer humanorightswatch.org spammer=yes
SetEnvIfNoCase Referer ilovevitaly.com spammer=yes
SetEnvIfNoCase Referer 7makemoneyonline.com spammer=yes
SetEnvIfNoCase Referer o-o-6-o-o.com spammer=yes
SetEnvIfNoCase Referer priceg.com spammer=yes
SetEnvIfNoCase Referer social-buttons.com spammer=yes
SetEnvIfNoCase Referer semalt.com spammer=yes
SetEnvIfNoCase Referer smailik.org spammer=yes
SetEnvIfNoCase Referer theguardlan.com spammer=yes
SetEnvIfNoCase Referer simple-share-buttons.com spammer=yes
SetEnvIfNoCase Referer 100dollars-seo.com spammer=yes
SetEnvIfNoCase Referer 100dollars-seo.com spammer=yes
SetEnvIfNoCase Referer success-seo.com spammer=yes
SetEnvIfNoCase Referer videos-for-your-business.com spammer=yes

Order allow,deny 
Allow from all 
Deny from env=spammer 
## STOP BOTS ##

Now that you've got that in place continue to check your statistics over the coming weeks and confirm that you're no longer seeing referrals for unwanted domains.

Frequently Asked Questions

Do Google ads work for lawyers?

Yes, Google Ads can be an effective way for lawyers to reach potential clients. It allows them to target specific keywords and phrases related to their legal services and geographic locations. However, it can be difficult for beginners and quickly become expensive as a lawyer is one of the most expensive PPC keywords.

How can I optimise my law firm’s Google My Business profile?

Optimise your law firm’s Google My Business profile by providing accurate and consistent information, selecting relevant categories, verifying your listing, utilizing Google’s post feature, uploading photos and videos, and optimizing for SEO keywords.

Leave a Reply

Your email address will not be published. Required fields are marked *

About the Author

In 2013 Byron took the bold move to transition into his own agency and his hard work paid off, eventually turning PixelRush into a 14-strong digital marketing team that has helped over 150-200 businesses to date in countless which including spending over 10 million in ad spend and optimisation over 400+ landing pages. His personal motto is to lead with this value and this blog is here to provide you with successful strategies so you can learn faster, more efficiently and without the countless hours and hard lessons he’s had to learn along the way.
Share This Article:

You might also like

The legal industry is not immune to the wave of digital transformation sweeping across various sectors. The advent of artificial intelligence (AI) is revolutionising the way law firms market their services, streamline operations, and enhance client experiences.
In today’s competitive legal landscape, establishing a strong online presence is crucial for attracting potential clients. Using Google My Business for lawyers is a powerful tool to help law firms stand out in local search results…
Are you striving to raise your law firm’s web visibility and search engine rankings? Link building is the answer! This piece will provide valuable insights into how link-building strategies for law firms can give you an edge over competition…

Ready to take your Business
to the next level?

Book your free video audit of your website and let’s talk about how we can help you grow your business. 

30 mins free video analysis + 30 mins discovery session