Future, you must decide how you wish the crawler to take care of different Web systems. You can find an ongoing debate with regard to the intelligence of internet search engine crawlers. It isn't really entirely obvious If they're full-blown headless browsers or just glorified curl scripts (or a https://youtu.be/EVeTP0aV8gY?si=5FHbTy4kbLwUVoJD