Robots.txt can be a file that exists on the basis directory of each Site and can be used to instruct search engines like google on which directories/information of the web site they're able to crawl and consist of in their index. You may additionally look at developing individual pillar pages https://technik.forum