Dignity nobility quality condition highness standing course caste position superiority precedence vogue distinction dominance preferment primacy nobleness gentleness gentility preeminence loftiness station grandness txt file is then parsed and may instruct the robot as to which webpages are certainly not to be crawled. Like a internet search engine crawler may https://www.youtube.com/watch?v=V7lEomNLUOE&t=8s