What the hell is a robots.txt file and how do we use it for SEO?
Note: If you're a client of mine this is already done.
Ok, so it may sounds like I'm going a little sci-fi on you. To understand how we use a robots.txt file for SEO, we need to get clear on a couple of things.
- What is a robot in this context?
- What then is a robots.txt file?
- Finally, how do we use it for SEO?
We should say that if you're an existing client of ours, you can assume that it is already taken care.
What is a robot in this context?
When you perform a search using the popular search engines like Google, you are only seeing results from their copy of the web. Many people who we speak to think that it is a live copy and are surprised to learn that..
The search engines each use robots to copy the web into their database. Google call theirs the Googlebot and Bing the Bingbot. Their journey around the web is referred to as crawling.
Once they have crawled a page it is then saved in their database and this is known as indexing.
To see what pages on your website have been indexed in Google you can perform a search for "site:yourwebsite.com". Just replace yourwebsite.com with your actual website address ie "site:onlinebusinessbuilders.co.uk".
What then is a robots.txt file?
A robots.txt file is a set of instructions for the robots when they reach your website.
We could instruct them to leave immediately. We can tell them to index everything. We can tell them about our Sitemap, so they know all our pages, learn more about sitemaps here..
To see if your website has one already go to yourwebsite.com/robots.txt. Just replace yourwebsite.com with your actual website address ie http://onlinebusinessbuilders.co.uk/robots.txt
How do we use it for SEO?
You've no doubt read the "SEO content is king" posts on the web. Although that is true, quality is of the greatest importance.
There is no point having lots of low quality content in Google as it will not help your ranking in the slightest. So We tend to consider the robots.txt file as a way of keeping low quality and/or private pages out of Google.
Take for example this website. It's only internal people who ever need to sign in. So using the robots.txt file, We tell the robots to stay away from the sign in / forgot your password pages. Why would we want them in Google?
Personally We like to run the "site:yourwebsite.com" search on Google. We'll then look through all the results and use the robots.txt to remove the low quality links.
Please contact us regarding any web design / development / SEO services that you may require.
