In order to avoid undesirable material during the look for indexes, webmasters can instruct spiders never to crawl specific documents or directories with the conventional robots.txt file in the basis directory of your domain. Also, a web page is often explicitly excluded from a online search engine's database by using https://news.rafeeg.ae