Plugin / WordPress Robots.txt optimization (+ Sitemap) – Website traffic, ranking & SEO Booster + Woocommerce
Pagup
Frequently Asked Questions (FAQ)
Better Robots.txt creates a WordPress virtual robots.txt file. Please make sure that your permalinks are enabled from Settings > Permalinks. If permalinks are working then make sure that there is no physical robots.txt file on your server. Since it can’t write over physical file, so you must connect to FTP and rename or delete robots.txt from your domain root directory. It usually in /public_html/ folder on cPanel hostings. If you can’t find your domain root directory, please ask your hosting provider for help. If issue persists after taking these measures, please post it in support section or send a message to [email protected]
If you have a pshysical robots.txt on your web hosting server, then this plugin will not work. As mentioned, it creates a WordPress virtual robots.txt file. Please follow steps in above answer if you want to use robots.txt file with this plugin.
This feature is allowed in Better Robots.txt Pro version, which automatically add sitemap in robots.txt file. It detects sitemap from Yoast SEO plugin. In-case you’re using a different sitemap plugin or a manually generated sitemap then you can simply add sitemap URL in sitemap input field. If Yoast XML sitemaps are also enabled then you need to disable it first by simply going to Yoast General Settings > Features and disable XML Sitemaps feature.
Why not? Considering that the robots.txt is the very first file read when your website is loaded by a browser, why not enable crawlers to continuously index your content? The simple fact of adding your Sitemap in the Robots.txt is simply common sense. Why? Did you list your website on Google Search Console, did your webmaster do it? How to tell the crawlers that you have new content available for indexation on your website? If you want this content to be found on search engines (Google, Bing, …), you have to have it indexed. That’s exacly what this instruction (adding the sitemap) aim to. One last point. The main reason why this plugin exists is due to the fact that 95% of the time (based of thousands of SEO analysis), the robots.txt is either missing, empty or misued, simply because it is either misunderstood or forgotten. Imagine now if it was activated and fully functional.
Actually, this plugin will increase your website indexation capacity which leads to improve your ranking on Google. How ? Well, the idea of creating this plugin was taken after making hundreds of SEO optimization on professional and corporative websites. As mentioned before, 95% of analysed websites did not have what we could call an “optimized” robots.txt file and, while we were optimizing these websites, we realized that simply modifying the content of this file was actually “unlocking” these websites (based on daily SEMrush analysises). As we were used to working in 2 steps (periods of time), starting with this simple modification was already generating a significant impact on Google Ranking, and this, even before we started deeply modifying either the content, the site arborescence or META Data. The more you help search engines at understanding your website, the better you help your capacity of getting better results on SERPs.
While you can view the contents of your robots.txt by navigating to the robots.txt URL, the best way to test and validate it, is through the robots.txt Tester option of Google Search Console.
Login to your Google Search Console Account. Click on robots.txt Tester, found under Crawl options. Click the Test button.
If everything is ok, the Test button will turn green and the label will change to ALLOWED. If there is a problem, the line that causes a disallow will be highlighted.
WordPress by default is using a virtual robots.txt file. This means that you cannot directly edit the file or find it in the root of your directory.
The only way to view the contents of the file, is to type https://www.yourdomain.com/robots.txt in your browser.
The default values of WordPress robots.txt are:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
When you enable the “Discourage search engines from indexing this site” option under Search Engine Visibility Settings, the robots.txt becomes:
User-agent: *
Disallow: /
Which basically blocks all crawlers from accessing the website.
There are 3 main reasons that you’d want to use a robots.txt file.
Block Non-Public Pages: Sometimes you have pages on your site that you don’t want indexed. For example, you might have a staging version of a page. Or a login page. These pages need to exist. But you don’t want random people landing on them. This is a case where you’d use robots.txt to block these pages from search engine crawlers and bots.
Maximize Crawl Budget: If you’re having a tough time getting all of your pages indexed, you might have a crawl budget problem. By blocking unimportant pages with robots.txt, Googlebot can spend more of your crawl budget on the pages that actually matter.
Prevent Indexing of Resources: Using meta directives can work just as well as Robots.txt for preventing pages from getting indexed. However, meta directives don’t work well for multimedia resources, like PDFs and images. That’s where robots.txt comes into play.
You can check how many pages you have indexed in the Google Search Console.
If the number matches the number of pages that you want indexed, you don’t need to bother with a Robots.txt file.
But if that number of higher than you expected (and you notice indexed URLs that shouldn’t be indexed), then it’s time to create a robots.txt file for your website.
Why would you use robots.txt when you can block pages at the page-level with the “noindex” meta tag?
As mentioned before, the noindex tag is tricky to implement on multimedia resources, like videos and PDFs.
Also, if you have thousands of pages that you want to block, it’s sometimes easier to block the entire section of that site with robots.txt instead of manually adding a noindex tag to every single page.
There are also edge cases where you don’t want to waste any crawl budget on Google landing on pages with the noindex tag.
Robots.txt must be in the main folder, i.e., domain.com/robots.txt.
Each subdomain needs its own robots.txt (sub1.domain.com, sub2.domain.com, … ) while multisites require only ONE robots.txt (domain.com/multi1, domain.com/multi2, …).
Some crawlers can ignore robots.txt.
URLs and the robots.txt file are case-sensitive.
Crawl-delay is not honored by Google (as it has its own crawl-budget), but you can manage crawl settings in Google Search Console.
Validate your robots.txt file in Google Search Console and Bing Webmaster Tools.
Don’t block crawling to avoid duplicate content. Don’t disallow pages which are redirected. Crawlers won’t be able to follow the redirect.
The max size for a robots.txt file is 500 KB.
$300 worth free guest posting on WPSocket.com!
Get Started With Free Trial
Get Started With Free Trial
Explore the offer details: Discover it now
Ratings
4.7
48 reviews
Rating breakdown
Details Information
Version
1.3.0.1
First Released
15 May, 2018
Total Downloads
55,621
Wordpress Version
4.1 or higher
Tested up to:
5.3
Require PHP Version:
5.2.4 or higher
Tags
Contributors
Languages
DIRECTORY DISCLAIMER
The information provided in this THEME/PLUGIN DIRECTORY is made available for information purposes only, and intended to serve as a resource to enable visitors to select a relevant theme or plugin. wpSocket gives no warranty of any kind, express or implied with regard to the information, including without limitation any warranty that the particular theme or plugin that you select is qualified on your situation.
The information in the individual theme or plugin displayed in the Directory is provided by the owners and contributors themselves. wpSocket gives no warranty as to the accuracy of the information and will not be liable to you for any loss or damage suffered by you as a consequence of your reliance on the information.
Links to respective sites are offered to assist in accessing additional information. The links may be outdated or broken. Connect to outside sites at your own risk. The Theme/Plugin Directory does not endorse the content or accuracy of any listing or external website.
While information is made available, no guarantee is given that the details provided are correct, complete or up-to-date.
wpSocket is not related to the theme or plugin, and also not responsible and expressly disclaims all liability for, damages of any kind, arising out of the use, reference to, or reliance on, any information or business listed throughout our site.