In an effort to push for an official web crawler standard, Google has made its robots.txt parsing and matching library open source with the hope that web developers will soon be able to agree on a standard for how web crawlers operate online. The C++ library is responsible for powering the company’s own web crawler …
