Robots.txt Generator
Build a clean, standards-compliant robots.txt with allow/disallow rules, sitemap, host, and crawl delay — validated as you type.
Crawl Rules
Define user-agent groups and their allow/disallow paths.
robots.txt
Copy or download the file and place it at your site root.
User-agent: *
Disallow: /admin
Disallow: /private
Allow: /public
Sitemap: https://vizotool.com/sitemap.xml
By Vizo ToolLast updated August 20264 min read
Robots.txt Generator — Control How Search Engines Crawl Your Site
Create a clean, standards-compliant robots.txt in seconds. Add allow and disallow rules per user-agent, point crawlers at your sitemap, set a host and crawl delay, and download a file ready to drop into your site root.
The generator validates the syntax as you build, so you never accidentally block your entire site. It's the fastest way to control what Google and other bots can crawl — right in your browser.
- Allow & Disallow rules
- Sitemap + Host directives
- Crawl-delay support
- Syntax validation
Why Use Robots.txt Generator?
Visual Rule Builder
Add and reorder allow/disallow rules without remembering robots.txt syntax.
Sitemap & Host Directives
Automatically add Sitemap: and Host: lines to help crawlers discover your content.
Crawl-Delay Support
Set a crawl delay for bots that respect it, like Bing and Yandex.
Syntax Validation
Real-time checks warn you about malformed rules before you publish.
Features
Multiple User-Agents
Separate rule sets for Googlebot, Bingbot, and * (all bots) with independent allow/disallow rules.
Allow & Disallow
Add precise path rules — disallow /admin/, allow /public/, and everything in between.
Sitemap Directive
One click adds your sitemap URL so search engines find it from the root file.
Host & Crawl Delay
Optional Host: and Crawl-delay: lines for complete control over crawling behavior.
Live Syntax Check
Invalid paths or duplicate groups are flagged immediately with a validation summary.
Copy & Download
Copy the file with one click or download robots.txt ready to upload to your site root.
How to Generate a Robots.txt
Build a crawler-friendly robots.txt in three quick steps.
Add user agents
Create rule groups for Googlebot, Bingbot, or * — or just use the default group for all bots.
Set allow & disallow rules
Add paths you want to block (like /admin or /cart) and paths you want to explicitly allow.
Add sitemap & download
Enter your sitemap URL, review the live preview, and copy or download the file to your site root.
Frequently Asked Questions
Got questions? We've got answers.
What is a robots.txt file?▼
robots.txt is a plain-text file in your site root that tells search engine crawlers which URLs they may or may not crawl.
Where do I put my robots.txt file?▼
Upload it to the root of your domain, e.g. https://yoursite.com/robots.txt. Next.js serves it automatically from app/robots.ts.
Does robots.txt block indexing?▼
No — robots.txt only controls crawling, not indexing. To remove a page from search results entirely, use a noindex meta tag or header.
What does Disallow: / mean?▼
Disallow: / tells all bots not to crawl any part of the site. Only use it if you truly want to block crawling site-wide.
Do I need to list Googlebot separately?▼
The * group covers all bots. Add a Googlebot group only when you need behavior specific to Google, like different crawl rules or no crawl delay.
What is a Sitemap: line?▼
A Sitemap: line points crawlers to your XML sitemap location. It's the standard way to submit sitemaps in robots.txt.
What is Crawl-delay?▼
Crawl-delay tells a bot how many seconds to wait between requests. Googlebot ignores it, but Bing and Yandex respect it.
Is the robots.txt generator free?▼
Yes, it's completely free with no sign-ups, no watermarks, and no usage limits.
Can I have rules for multiple bots?▼
Yes — create separate user-agent groups (Googlebot, Bingbot, etc.) each with their own allow and disallow rules.
What happens if I make a syntax error?▼
Crawlers ignore malformed rules, which can leave pages accidentally exposed or blocked. The generator validates syntax as you type to prevent this.
Does robots.txt affect SEO?▼
Indirectly — blocking important pages wastes crawl budget, while accidentally exposing private pages can hurt you. A correct file keeps crawling efficient.
Should I disallow /wp-admin or /admin?▼
Many sites disallow admin and backend paths to keep crawlers out. Just make sure the paths you block don't include resources your pages need.
Can robots.txt have comments?▼
Yes — lines starting with # are comments. Use them to document why each rule exists.
How do I test my robots.txt?▼
Use Google Search Console's robots.txt tester or the URL Inspection tool to see which rules apply to a specific URL.
Does the tool support a Host directive?▼
Yes — the optional Host: line (used mainly by Yandex) declares your preferred domain, and the generator adds it if you enable it.
Related Tools
Try more free browser-based image tools.
Compress
Reduce image file size instantly
Resize
Perfect dimensions every time
Flip
Mirror & rotate instantly
Convert
PNG, JPEG, WEBP — instantly
Watermark
Protect photos with text or logo
Remove Background
AI background remover in your browser
Passport Photo
Passport-size photos for 25+ countries
Image to PDF
Merge JPG, PNG & HEIC into one PDF
PDF to Image
Convert PDF pages to JPG or PNG
Signature Resizer
Resize signatures to any size & KB limit
Social Media Resizer
Exact sizes for every social platform
JSON Formatter
Beautify, minify & validate JSON instantly
JSON Validator
Validate JSON with error line numbers
Base64 Encoder
Encode text or files to Base64
Base64 Decoder
Decode Base64 back to text or files
Password Generator
Strong random passwords with a strength meter
UUID Generator
Generate UUID v1, v4 & v7 in bulk
QR Code Generator
QR codes for URLs, WiFi, WhatsApp & more
Gradient Generator
Linear, radial & conic CSS gradients
Box Shadow Generator
Design CSS box shadows visually
JWT Decoder
Decode JWT tokens & check expiry
SQL Formatter
Beautify & minify SQL queries
HTML/CSS/JS Playground
CodePen-style editor with live preview & console
SQL Playground
Run SQLite queries in your browser with sql.js
Meta Tag Generator
Create perfect SEO meta tags & SERP preview
Schema Generator
Generate JSON-LD structured data for 15+ types
Open Graph Generator
Design social share cards with live previews
Sitemap Generator
Create XML sitemaps with image & video entries
UTM Builder
Build trackable campaign URLs with QR code
SERP Preview
See how your listing looks in Google search
Slug Generator
Create clean SEO-friendly URL slugs
Meta Tag Analyzer
Audit any page's meta tags & SEO score
Heading Checker
Check H1–H6 hierarchy & duplicate headings
Traffic Checker
Estimate website traffic from public SEO signals
SIP Calculator
Estimate the future value of your monthly SIP investments
Compound Interest
See how your principal grows with compound interest
EMI Calculator
Monthly EMI, interest & full amortization schedule
GST Calculator
Add or remove GST with CGST/SGST split
FD Calculator
Fixed deposit maturity value & interest earned
CAGR Calculator
Annualized growth rate of your investment
ROI Calculator
Return on investment & annualized ROI
Income Tax
Estimate Indian income tax — old & new regimes
Retirement Calculator
Project your retirement corpus & monthly income
Discount Calculator
Final price and savings after a discount
Profit Margin
Profit, margin & markup from cost and price
Stock Average
Average buy price across multiple purchases
Salary Calculator
Estimate take-home salary from CTC
Inflation Calculator
How inflation erodes purchasing power
Thumbnail Downloader
Download YouTube thumbnails in all available resolutions
Tags Extractor
Analyze video tags & get smart tag suggestions
Transcript Extractor
Get the transcript of any YouTube video with timestamps
Title Generator
Generate catchy, SEO-scored YouTube titles instantly
Description Generator
Write SEO-ready video descriptions with hashtags
Video Downloader
Save videos & audio — MP4, MP3, M4A with quality options