Following, you'll want to choose how you need the crawler to handle different Web technologies. There is certainly an ongoing discussion regarding the intelligence of search engine crawlers. It isn't totally crystal clear if they are full-blown headless browsers or simply glorified curl scripts (or something between). By default, https://www.youtube.com/watch?v=EVeTP0aV8gY