Robots.txt – An Introduction Part I
Robots.txt file is a text file (yea, the ones on notepad) that resides on your server, and controls a whole lot of features on your website (whatever platform it is built on). It's a simple text file in which there are a few lines of text, but it's very powerful that it can even decide whether your website should be shown on Google or not, what part of your website should be shown to the search engines (like Google, Yahoo and MSN).
There are several questions which have been as under answered
- What is a Robots.txt file technically ?
- Where do the Robots come from ?
- What do Robots do ?
- What does a Robots.txt file do ?
- Where to spot the Robots.txt file ?
- Why is Robots.txt file and Robots important to a webmaster ?
- How does a Robots.txt file look like ?
- How and what can you control using Robots.txt ?
- How to set up a Robots.txt file ?
Essentially, Robots.txt is an excellent tool to control how search engines scan your website, and gather up information from them. The more complex and careful you plan your website design, the better your search engine positions would be. But many websites simply ignore this and leave everything to the search engines to decide. Is that a good thing to do? I'd say it all depends on how you want it, if you think "not showing" a folders content to Google will avoid un necessary information being passed to it, and you know exactly how you can accomplish it, then why not use the options?
Be The First One To Comment