10. Gzip Compression: Enable Gzip compression in your server to reduce the sizing of data files transferred amongst the server along with the user’s browser. txt file is then parsed and can instruct the robotic regarding which pages are usually not to become crawled. For a internet search engine crawler https://www.youtube.com/watch?v=DhNbKpYq6Ro