SEO For Small Businesses: Crawling and Indexing

June 15, 2021

By Taryn Hefner

Taryn runs Marketing at Join It and is a lover of all things data! You can frequently find her experimenting in the kitchen, absorbed in a book, or brushing up on her Python coding skills.

View other posts by Taryn

On this blog, we’ve started an informal SEO series all about search engine optimization tips for small businesses! One of the first things any website owner needs to do is make sure that their website is both crawlable and indexable. 

Let's go over some definitions! 

What Is A Web Spider? A web spider is a robot that consistently and methodically collects information on every page on the internet; a bot that follows links to other links to discover new pages and new websites across the internet. 

What Does Crawlable Mean? A page that is crawlable is a page that a spider can find and access, and that content on it can be understood by the spider including written content, images, videos, and links to other pages or websites 

What Does Indexable Mean?  If a website is indexable, it means search engines can catalog and store the information on your website in order to serve it to users searching for information. In order for a page to be indexable, it must be crawlable. 

  • Some examples of pages that are indexable are recipe pages, news articles, and product pages in online shops. 
  • Some pages that aren’t indexable are personal banking pages, medical information for patients, and anything that requires a user to log in to view. 

If you want users to be able to find your website via a Google or Bing search, your site has to have pages that are both crawlable and indexable. 

To ensure your site is known by major search engines, you can start by submitting your sitemap to Google through Google Search Console, and for Bing in Bing Webmaster Tools. You can also link to different relevant pages within your website to improve the network of links between pages. 

The great thing about these tools is that they’ll tell you whether or not your pages have been indexed… and what you can do about it if they’re not! 

Not sure if spiders can read your web content properly? Check out some of the best paid and free SEO tools to help you view your website the way spiders would. 

For more of our SEO guides, check out all of our SEO for Small Businesses posts here: