Shopify Robots.txt – Everything you need to Know

If you’ve recently received an email from Google about your Shopify Robots txt file being blocked, you are probably wondering what this is and how to fix it.

In this article I will explain everything you need to know about your Shopify Robots txt file, how it works, and what all those warnings mean.

Make sure to read until the end of the article so you can get a full understanding of what is normal, and what things to look out for.

Watch our Quick Video Tutorial

Update 2021: Shopify Robots.txt file is now Editable

What is a robots.txt file?

A robots.txt file is a file that tells these search engine bots which pages they can access and which ones they cannot.

A  search engine crawler (aka: Googlebot) is a robot that crawls the web and visits many websites to better understand what they are all about.

If you have a Shopify store, a search engine crawler has most likely your site in the past and reported the information they found back to Google.

The purpose of the robots.txt file is to prevent your site from being overwhelmed with “requests” which can slow it down.

Index though blocked by Robots.txt?

Robots.txt blocked page Example

For example, a search engine like Google can crawl a product page on your site in search of more data, but they can’t crawl your checkout page… why is this?

This has to do both with security measures as well as SEO. A checkout page does not need to rank for SEO, so it’s best that it is blocked from crawlers.

You don’t want bots crawling your checkout page, in fact you want these pages to be blocked by Shopify robots.txt.

When goes says they’ve “indexed” these pages, that’s a good thing. When a page is “indexed” it means Google  visited the page, knows its content and it’s eligible to be shown on search results.

What pages shouldn’t be blocked by robots.txt?

You should definitely be aware of each page is that is being blocked by robots.txt.

One good way to monitor this is by signing up for a free Google Search Console account. This will send you automatic emails and alert you about any potential issues.

Just remember Google sends out an email notice about ALL pages blocked by robots.txt, even the pages that are totally normal, so no need to freak out. See section below.

When should you actually worry? For example if you happen to see a product page being blocked by your robots.txt file this could be a potential issue and worth looking into.

What pages should be blocked by robots.txt?

Some common pages that you will see blocked are: register, login, checkouts and checkout to name a few.

Keep in mind this is totally normal and nothing to worry about. This is working as expected.

Examples:

amysdogtreats.com/account/register

amysdogtreats.com/account/login

amysdogtreats.com/checkout

amysdogtreats.com/checkouts

Where is my robots.txt file located?

You can find your robots.txt file at the root directory of your Shopify store’s primary domain name.

So it will look like your websites name slash robots.txt

For example: amysdogtreats.com/robots.txt

Can I edit my Shopify robots.txt file?

Nope.

The simple answer is you cannot edit your robots.txt file if you are using a hosting service such as Shopify. 

Your robots txt file  is handled automatically for you by Shopify, and you can rest assured they are doing the job correctly.

For example, your shopping cart page will be blocked from search engines. This is helpful for your SEO because it allows your product pages to be found more easily by customers on search results.

What use is it if a customer find your shopping cart page on your website? The idea is to help your product pages rank higher than your checkout page.

How to hide a page from search engines?

As Google states in their help docs, “You should not use robots.txt as a means to hide your web pages from Google Search results.”

If you want to hide a page from search engines, Google recommends you use noindex.

With a shopify store, this will require you to add some code to your theme’s liquid.file to noindex particular pages.

How to test your Robots.txt file?

Google has a tester you can use to test out your robots txt file which can be found here.

This is a helpful tool if you want to see which specific URLs are being blocked from a Google web crawler.

Best Shopify SEO App for Google monitoring

The best Shopify SEO app available for monitoring your Google activity is Get Clicked SEO. This app takes the guess work out of updating your site’s SEO.

A key feature of Get Clicked SEO is that it offers a robust automatic “Rich snippet Code” which can help you gain visibility on Google search console.

Rich snippets are those customer reviews and ratings that pop out on search results with yellow stars, pricing, and in-stock availability.

Some other key features of Get Clicked SEO is it helps you update your page titles, meta descriptions, and alt text images. It also has a keyword monitoring tool as well as a keyword research tool (it’s the only SEO app on Shopify to have both!)

Get Clicked SEO is one of the only USA made Shopify apps available providing VIP customer service, along with video tutorials to help you through your SEO journey. 

If you don’t have Get Clicked SEO installed today, you can try out their free 7 day trial ( money back 100% guaranteed if you don’t like the app).

Conclusion

Thanks for reading this article all about what is a robots txt file and how you can monitor yours to make sure nothing fishy is happening. In summary, we covered that your robots.txt file is being automatically generated and maintained by Shopify.

If your eCommerce website is not being hosted by Shopify, please check out Google’s help documents for how to edit this file.

We also discussed which pages are okay to be blocked by Google and which pages shouldn’t be blocked. And lastly, we wrapped up this tutorial by showing you where you can find your file and how to test it with Google’s tools.

If you have any more questions about your Shopify robots txt file, feel free to get in touch with our team at Adolab here or leave a comment down below.