Build a robots.txt with crawler groups, allow and disallow rules and sitemaps, choose which AI crawlers to block, and test whether any URL is allowed for any crawler.