How To Edit The Robots.txt File On Your Shopify Store

Introduction

The Robots.txt file is used by search engines so that they can crawl your site and index its contents. Without proper crawling, your site will be inaccessible to people who are looking for information on products you sell. The main purpose of the Robots.txt file is to tell GoogleBot (and other search crawlers) what pages to not index when crawling your site. In this blog post we will explain what the Robots.txt file is and how you can use it on your Shopify store to improve visibility of product listings in search results.

What is the Robots.txt File?

The Robots.txt file is a list of instructions for Google Bot that shows which pages of your website should be crawled and indexed by Google. This is helpful for SEO purposes, so that Google is able to index your pages and return search results. The file is usually placed in the root of your website and is formatted in plain text. If you haven’t set up Google Search Console yet, you can see your site’s status and any errors.

How to edit the Robots.txt file on your Shopify store?

There are a couple of ways you can edit the Robots.txt file on your Shopify store. You can either use the built-in Shopify theme editor (see how). There are some caveats to this, each time you publish a new theme you will need to copy your changes across, and you might forget! Or if you don’t feel comfortable doing that you can use our handy app, using our easy-to-use dashboard and our app will sync your robots.txt file with any theme you publish.

View our app here: Shopify Robots.txt Editor.

When does the Robots.txt file come into play?

You’ve probably heard a lot about the importance of the Robots.txt file. Well, Google has a set of guidelines that many search engines like Google+, Bing, and Yahoo! follow. If a website doesn’t have a Robots.txt file, then it’s considered to be “open” and available for bots to crawl. If you want to prevent bots from making your site crawlable, then you need to add a Robots.txt file.

How to use the Robots.txt file in SEO?

Googlebot crawls all pages of your store and looks for the instructions in the Robots.txt file. If it finds instructions, then it ignores the page. If the Googlebot finds your page, it will skip it. This means that your page won’t be indexed and your visitors won’t be able to find it. Googlebot can’t crawl all pages, so some of the pages it might be able to crawl, but will ignore them. However, the pages it doesn’t crawl can’t be found by anyone. GoogleBot is the most popular and widely used bot, so it’s important to make your pages not crawlable to Googlebot.

Bottom line

The Robots.txt file is used by search engines so that they can crawl your site and index its contents. Without proper crawling, your site will be inaccessible to people who are looking for information on products you sell. The file is used to tell GoogleBot (and other search crawlers) what pages to not index when crawling your site. You can use the file to disallow certain pages from being crawled or to instruct them to not crawl certain pages. The file is usually placed in the root of your website and is formatted in plain text.

Conclusion

Now that you know what the Robots.txt file is, how to edit the file on your Shopify store, and when to use it, it’s time for action! Install our app on your Shopify store: Shopify Robots.txt Editor.