Generate a robots.txt file for your website. Configure user-agent rules, disallow/allow paths, crawl delay, and sitemap.
robots.txt