Crawlers (or bots) are used to gather information out there on the web. By utilizing website navigation menus, and reading inside and external links, the bots begin to grasp the context of a web page. Of course, the words, photographs, and other knowledge on pages additionally help search engines like google and yahoo perceive what they’re about.