In this article I’m going to cover a lot of things but the major takeaways will be:
- How to submit your website to the major search engines like Google, Bing, and Yahoo
- How to check and make sure your website (or new website pages) have been indexed
Is this still even necessary in 2021 though?
Why Do I Need to Submit My Website to Search Engines?
If search engines (like Google and Bing) don’t know your website exists then your website will never appear in any searches.
That means you’re throwing away a lot of opportunities for people to find your site and visit it.
If your website is a part of your business and you depend upon people visiting it to generate revenue, that means you’re throwing away a lot more than just opportunities: you’re throwing away money.
Your website doesn’t just automatically appear on Google and the other top search engines by magic either.
In fact, several weeks can pass by without the search engines ever knowing your entire website exists.
Search engines have programs called bots that crawl the web and create massive indices of the world wide web.
These bots primarily rely on links from one page to another to discover them and index them. They’re pretty damn good at this but as you can imagine, the world wide web is huge.
If links to your site don’t exist on other pages already in that search engine’s index then your site won’t get automatically indexed (link building should be a part of your SEO campaigns).
What Does Crawling a Website Mean?
Imagine you’re on Wikipedia reading an article about lawyers. While reading the article you see a link to a topic about Juris Doctorates and click on it because you want to read more about it.
That same concept is how bots crawl the web. They visit web pages and follow the links on those pages.
As a page gets crawled by a bot it gets indexed in the search engine.
Now, there’s a lot of nuances that we haven’t covered but if you want to learn more about it, then you can watch this video by Matt Cutts, the former head of Google’s webspam team.
Reasons Why You Should Still Submit Your Website to Search Engines Manually
- Low effort, high reward. There’s no excuse not to. It’s just too easy to submit your website for indexing.
- Search engines aren’t perfect. They can’t find everything on their own. Submitting your website manually makes it easier for the major players to begin ranking your website for keywords. By submitting your website manually you can also give the search engines information they couldn’t figure out on their own such as the importance of each of your website’s pages.
- You’ll get access to some pretty helpful dashboards of data. Both Google and Bing each have dashboards you can use to gain insight into how you’re performing in the world of search. You’ll get access to some pretty useful tools also.
Time for the bad news. Don’t be fooled into thinking that you’ll automatically start ranking for keywords after getting your site indexed.
In fact, most sites don’t rank on the first page for major search terms for a very long time. For example, in some niches like SEO for attorneys, things are so competitive that you’ll spend months just climbing to the first page.
So if you thought this was a magic bullet. Sorry. It’s just the first step.
How to Submit Your Website to Google
Performing a search engine submission with Google is pretty simple. Just follow the instructions below.
To submit a website URL you need to use a website/tool called Google Search Console (formerly Google webmaster tools). You’ll need a Gmail account to use Search Console (and basically all of their tools) so either sign in or create one.
Once signed into Search Console you have two options to submit your site for indexing.
- Submitting an XML Sitemap
- Typing the URL into the search bar at the top of the page.
NoteWhile Google did discontinue their URL submission tool via Search Console in July 2018, the currently version of Search Console has this functionality back (as of April 2019).
What is an XML Sitemap?
An XML sitemap is a file that lists all of the page URLs of a website in XML (eXtensible Markup Language) format. XML is just a file format, like PDF or JPEG.
If the sitemap has been formatted for easy readability it will look something like this:
Note:If you have a WordPress website and use a plugin like Yoast then it will automatically generate a sitemap like this for you.
How Can I Check if my Website has a Sitemap?
1. Check yourdomain.com/sitemap.xml
Your sitemap xml file is usually located at yourdomain.com/sitemap.xml. Example: Let’s say your website address is https://www.rundoyen.com/ Then you can check if your sitemap exists by visiting: https://www.rundoyen.com/sitemap.xml If the page isn’t found that doesn’t mean you don’t have a sitemap. It’s possible that it has a different URL.
2. Check Your robots.txt File
You can also check if your sitemap has been listed in your robots.txt file. To check that try typing in yourdomain.com/robots.txt and you should see something like the following:
As you can see, this site has two sitemaps listed and neither are using the traditional /sitemap.xml path.
Use Advanced Search Operators
If your sitemap isn’t traditionally named AND isn’t in your robots.txt file, then you can also check Google’s search engine directly by using advanced search operators on your domain name.
Go to Google and use the following advanced search operator sequence to see if your sitemap indexed:site:yourdomain.com inurl:sitemap
If Your Website Doesn't Have a SitemapYou need to make one then. If you are using WordPress, there are a variety of plugins available that will do this for you. My recommendation is SEOPress. I've been using it ever since the Yoast SEO fiasco from early 2018.
If you have located your sitemap then you can go ahead and submit it to Search Console.
Just go to Search Console → Select Your Website → Sitemaps → Paste Sitemap URL → Click Submit
Website Not Verified?If your website hasn't been verified you'll need to do so first. Follow this tutorial to do so and then return to this guide.
How to Submit Multiple Sitemaps
If you have multiple sitemaps you can just repeat this process for each sitemap URL.
It’s as easy as that.
How to Manually Submit a Web Page with Google
If you already have your website’s sitemap submitted, then it will crawl your site at scheduled intervals.
What if you just published a new page or blog post you’re really excited about though? How can you get that submitted faster? With a manual submission.
The submission process is easy, don’t worry.
Just visit Search Console and enter your URL in the search bar at the very top. If the page hasn’t been indexed yet Search Console will let you know and you can click the “Request Indexing” button to index the page to do so.
If your page is indexed already, you’ll see results like this:
How to Submit Your Website to Bing
You’ll need to use Bing’s Webmaster Tools (similar to Search Console) to submit your full sitemap to Bing for indexing.
If you haven’t set up your Bing Webmaster Tools account yet you can authenticate it using Google. Let us know if you have any issues.
How to Submit Your Website to Yahoo
This one is the easiest. There’s nothing to do! Bing powers Yahoo so by indexing your site with Bing you’ll automatically submit your site to yahoo to be indexed as well!
How to Know if Your Website has been Indexed
You can use the advanced search operators we talked about earlier to check if your page has been indexed.
Follow this syntax: site:yourdomain.com/page-subfolder-to-check/
Example: Let’s say we just published a new page on rundoyen.com that’s located at https://www.rundoyen.com/coach-match/ and need to check it.
We would type in the following:
site:rundoyen.com/coach-match/
If you do that and don’t get any results for that advanced search then your page likely has not been indexed by the search engine yet (the advanced search operator site: works for both Google and Bing).
It can sometimes take days to get your page indexed so check back in a few days if it doesn’t appear when with the site: search operator after you’ve submitted the page to be indexed.
I’ve had pages get indexed within 30 seconds before (but not always). So it doesn’t hurt to check right afterwards.
Using Google Search Console to Check if your Site has been Indexed
Go to Google’s Search Console → Index → Coverage → Click on the Valid tab and then click submitted and indexed and Search Console will return a list of pages that have been indexed.
What if my Website isn't Indexed?
There are a few common culprits that we can check if you have determined your website/web page isn’t indexed:
- Flagged with a noindex tag
- Blocked by robots.txt
- Blocked by .htaccess
Web Page Flagged noindex
The noindex meta tag directs the search engine crawlers not to index a certain page. This tag exists at the page level. Search through your page’s HTML to figure out if it has been flagged for no index.
The easiest way to do this is to open up the DevTools for the page and look for the term “noindex” using the search function (Windows: Control + F, MacOS: ⌘ + F ). Follow this link to learn how to use DevTools in Chrome.
Blocked by robots.txt File
If your site’s robots.txt file is blocking Google’s bot (Googlebot) from crawling your website, then you will want to fix that.
You can either review your robots.txt file directly by visiting yourdomain.com/robots.txt or using Google’s Robots Testing Tool.
Note that this tool is a part of the old Search Console and isn’t present in the current version of Search Console. You can still access the older Robots Testing Tool here.
Blocked by .htaccess File
If your site is running on an Apache-based server then you’ll have an .htaccess file in your root directory. Locate this file and look to see if the following is present:
Header set X-Robots-Tag "noindex, nofollow"
If you find that, then you’ll need to remove it to grant Googlebot access to crawl and index your website.
If this is above your technical skills just reach out to whoever maintains your website or feel free to ask us for help with technical SEO as well.
My Website is Indexed but Not Ranking
There are a lot of sites indexed in the search engines. Just go to Google and type in any term and you’ll see how many pages are indexed for that term.
The query “indexing with Google” has over 46 million results indexed.
But only 10 show up on the first page.
When’s the last time you searched for something and checked the second page?
Exactly.
Pages that aren’t ranking on the first page receive very little traffic. You really need to be in the top 3 results to capture significant organic traffic for a search term.
Getting to the first page is one battle. Climbing the ranks from position #10 to the top 3 positions is another battle.
Ranking higher is, in my experience, primarily correlated with how many backlinks your website is receiving from other sites and how well you satisfy searcher’s intent.
Backlinks will still play a role in climbing up the ranks from 10 to the top 3 but it is at this point that on-page SEO factors really begin to matter.
Be sure you’re tackling both objectives: backlink building and on-page SEO.
Final Thoughts
If you want to take things casually and aren’t all that concerned with ensuring that the search engines index your site, then you can simply do nothing. Odds are they will find your site eventually…
If you’ve spent money on creating a website, hosting it, and so on, then it really doesn’t make any sense to not manually index it with the search engines though. You’re not taking your ROI seriously if you don’t.
This is especially true if you run a business and depend upon your website for lead generation or sales.
As I said before though, indexing is just the first step. In most cases, you’ll need to invest in SEO to get the results you want.