Robots.txt – An Introduction Part II
What is a Robots.txt file technically?
Ans :: In order to understand what robots.txt files are, you have to first understand what a Robot (the web one) is. A robot - is technically a program from search engines like Google, Yahoo and MSN that are set out on the internet to do the job of finding out new websites, indexing them and gathering the right information about the website. They are sometime called "spiders", "crawlers" and even "bots".
Where do the Robots come from?
Robots are commonly set out by search engines like Google, Yahoo, MSN, Altavista, Ask.com and others. Mainly, these are web servers of the search engines, that are on the constant look out of information on the internet. And they gather information (which ultimately goes to the search engines index) by visiting new websites, gathering up new information from them, following links and calculating and analyzing a whole lot of information from them.
Also Read ::
Be The First One To Comment