Robots.txt convention permits you to direct bugs on your site so they just slither the pages you believe they should creep. Albeit this could appear to be in opposition to the objectives of Web optimization, keeping the bugs from investigating a portion of your substance can really be gainful. For instance, this could apply when you are building another site, or on the other hand on the off chance that you have various variants of a page on your site and you would rather not get punished for copy content.
The starting points of the convention can be followed back to the mid-1990s when early website admins started to have worries about what bugs visited their sites. The thought has developed to address the issues of present day website specialists and the ebb and flow adaptation will be acknowledged by the insects sent from the significant web indexes.
Since robots.txt tells search engine spiders not to creep specific pages, some site proprietors could erroneously imagine that this is an effective method for keeping specific data hidden. Sadly, albeit legit spiders, for example, those from Google or Bing, will regard the convention, there are a lot of noxious spiders that will not, and consequently your data can in any case be taken.
It is additionally feasible for the data to wind up ordered in alternate ways, for example, another site connecting to the substance. In the event that you have individual data, you ought to have a lot more grounded security, like through a firewall.