Decided to stop including robots.txt in my WordPress sites for the time being. With Google presenting fully rendered pages in search results I prefer not blocking those webcrawlers and preventing them from finding required css and js files. The smart people at Yoast wrote up a good article explaining in more detail.
About the Author: Garrett Hunter
Founder and Principal Consultant of BLACKTOWER, Inc., Garrett leads organizations through the complex process of selecting and implementing software systems and realizing tangible benefits from their investment. In his spare time he writes about leadership, personal growth, and technology trends. Contact Garret at [email protected]
This site uses Akismet to reduce spam. Learn how your comment data is processed.