Where Are My URLs?
The other day I got an email from one of our clients regarding a community site they created for their customers.
Google searching on things that should be returning matches to your community are coming up with no links to site. I wonder why the Google crawlers are not picking up those pages?
Apparently the pages on this community site weren’t showing up in Google, even when they used incredibly specific keywords like author name + brand. Obviously most site owners know that they can’t and won’t rank well for every single search term they can think of, but to not see your website at all (especially for branded searches) can be very disconcerting. I took a quick look at their Webmaster Tools account and the code of their website to see if I could figure it out.
Here are 2 reasons why some URLs on your website aren’t appearing in the SERPs:
Google hasn’t crawled them yet.
Even though a URL is live and shareable and trackable the moment you hit publish, Google might not find that URL for several days (or even weeks!) depending on how frequently the search bots call your site. Take a look at your Webmaster Tools account; when was the last time Google crawled your sitemap? If your blog went live AFTER that date chances are Google doesn’t know your new blog post even exists yet. Google doesn’t crawl the entire web every second of every day, which is why we always suggest clients wait a solid month before they start wondering why Google hasn’t indexed any major chances on their site. You have to give Google time to find those changes/new content AND give them time to take effect.
And keep in mind that even though every new URL gets to piggyback off the value of your domain, each new URL also has to earn its own stripes and place in the SERPs. A new blog post might be buried in the SERPs (even after Google finds it) for several weeks or months as it earns it’s way to the top of the SERPs. Even though you THINK it should rank really well for XYZ keywords, your post won’t appear there instantly. There is more content being created in a day than one person could consume in a life time, but there are still only ten spots on page 1.
This is what happened to one of our other clients. Their new blog posts took about 2-3 weeks to be indexed, so even though he was looking for the title of the post they were nowhere to be found. However, when we searched for older URLs they were easily uncovered. He was also searching using keywords that had nothing to do with the blog post (he was using terms the site was optimized for), which also made it harder to find his blog posts.
Google will never crawl them.
If this code appears on your website, then Google will NEVER crawl and index those URLs:
<meta name=’robots’ content=’noindex,follow’ />
The noindex tag tells Google to essentially skip those URLs and NOT submit them to the SERPs. A site owner might choose to do this if they have multiple URLs with similar content (to prevent a duplicate content issue), or if they want to keep some content for customers only and not give it up for free to searchers. This is what had happened to the client I mentioned in the first paragraph. Once you left the homepage of their community website, all the pages had the noindex tag right in the top of the code, ensuring that Google would never crawl those pages.
Any content that you have locked behind a login or download form is also uncrawlabe by the Google search bots and will never appear in the SERPs. That login form stops the search spiders just like it stops a human from accessing your content. If there is content you only want customers or prospects to have access to then gating it would be your best bet.
Categorized in: Search Engines
Like what you've read? Please share this article