Where Are My URLs?

Written by Brick Marketing

The other day I got an email from one of our clients regarding a community site they created for their customers.

Google searching on things that should be returning matches to your community are coming up with no links to site.  I wonder why the Google crawlers are not picking up those pages?

Apparently the pages on this community site weren’t showing up in Google, even when they used incredibly specific keywords like author name + brand. Obviously most site owners know that they can’t and won’t rank well for every single search term they can think of, but to not see your website at all (especially for branded searches) can be very disconcerting. I took a quick look at their Webmaster Tools account and the code of their website to see if I could figure it out.

Here are 2 reasons why some URLs on your website aren’t appearing in the SERPs:

Google hasn’t crawled them yet.

Even though a URL is live and shareable and trackable the moment you hit publish, Google might not find that URL for several days (or even weeks!) depending on how frequently the search bots call your site. Take a look at your Webmaster Lost URLsTools account; when was the last time Google crawled your sitemap? If your blog went live AFTER that date chances are Google doesn’t know your new blog post even exists yet. Google doesn’t crawl the entire web every second of every day, which is why we always suggest clients wait a solid month before they start wondering why Google hasn’t indexed any major chances on their site. You have to give Google time to find those changes/new content AND give them time to take effect.

And keep in mind that even though every new URL gets to piggyback off the value of your domain, each new URL also has to earn its own stripes and place in the SERPs. A new blog post might be buried in the SERPs (even after Google finds it) for several weeks or months as it earns it’s way to the top of the SERPs. Even though you THINK it should rank really well for XYZ keywords, your post won’t appear there instantly. There is more content being created in a day than one person could consume in a life time, but there are still only ten spots on page 1.

This is what happened to one of our other clients. Their new blog posts took about 2-3 weeks to be indexed, so even though he was looking for the title of the post they were nowhere to be found. However, when we searched for older URLs they were easily uncovered. He was also searching using keywords that had nothing to do with the blog post (he was using terms the site was optimized for), which also made it harder to find his blog posts.

Google will never crawl them.

If this code appears on your website, then Google will NEVER crawl and index those URLs:

<meta name=’robots’ content=’noindex,follow’ />

The noindex tag tells Google to essentially skip those URLs and NOT submit them to the SERPs. A site owner might choose to do this if they have multiple URLs with similar content (to prevent a duplicate content issue), or if they want to keep some content for customers only and not give it up for free to searchers. This is what had happened to the client I mentioned in the first paragraph. Once you left the homepage of their community website, all the pages had the noindex tag right in the top of the code, ensuring that Google would never crawl those pages.

Any content that you have locked behind a login or download form is also uncrawlabe by the Google search bots and will never appear in the SERPs. That login form stops the search spiders just like it stops a human from accessing your content. If there is content you only want customers or prospects to have access to then gating it would be your best bet.

Categorized in:

Like what you've read? Please share this article

Read What Full Service SEO Clients Say About SEO Firm Brick Marketing

"highly recommend Brick Marketing to all of my clients"

“After working with many other SEO firms and not being satisfied I finally was introduced to the Brick Marketing President and Founder, Nick Stamoulis. Nick Stamoulis has educated me about SEO and has provided me with a well rounded SEO package, not only does he offer top quality services he also educates his clients and spends the time to explain everything and their SEO pricing is competitive. I will highly recommend Brick Marketing to all of my clients. Brick Marketing is an A+ for SEO services.”

– Mike Cardella, Integrated Computer Services

"went above and beyond to help us with any SEO issues"

“We hired Brick Marketing to manage our SEO, but they ended up also managing our company blog, social media marketing, helped us launch a pay per click advertising campaign, migrated our website to a new domain and so much more! Our SEO Specialist is always quick to respond whenever we had a question and went above and beyond to help us with any SEO issues.”

– Mory Creighton, CEO, Pinpoint Laser Systems

"Brick Marketing is rock solid and essential in building a strong marketing foundation."

“Brick Marketing is extremely valuable for feedback and advice in many aspects of marketing, not just SEO and I look forward to continuing to work with them over the long term to continually improve our business and increase sales. That’s what it’s all about. Just like their name implies, Brick Marketing is rock solid and essential in building a strong marketing foundation.”

– Jeff Nicholson, President Freely Creative, Inc. / Websticker.com

"superior knowledge of marketing on the internet"

“I have had the good fortune to work with SEO expert Nick Stamoulis and his team at Brick Marketing and it was clear form the get go that Nick had superior knowledge of marketing on the internet. The SEO campaign and technical writing that Brick Marketing provides has been impressive, where in the past we struggled to find the high levels of competence in these two skill sets. Were only just beginning with Brick Marketing and look forward to growing with their expert guidance. Thank you Nick Stamoulis and the Brick Marketing team!”

– Craig Raubenheimer, President, Roan Solutions, Inc.

"dependable, professional SEO company"

“Brick Marketing has been a dependable, professional SEO company that has helped us get results. In the last 6 months of using their services, visits to our website have increased by almost 30%. Our dedicated SEO Specialist was pleasant to deal with. Her suggestions for articles and press releases were industry specific. Brick Marketing always answered our phone calls and emails within an hour which made us feel valued as a client. I would recommend Brick Marketing to all businesses to handle their SEO needs.”

– JoAnne Loftus, Archival Designs, Inc.

"Our keyword rankings have improved and traffic has increased"

“The Brick Marketing team, was critical to ensuring that our new Web site is optimized to drive maximum traffic. Our keyword rankings have improved and traffic has increased thanks to her assistance and advice. In addition, they executed a strategic link building program that helped further increase Web traffic.”

– Lauren LaFronz, Director of Marketing, enChoice, Inc.

"traffic has increased over 80% year-over-year"

“When we started working with Brick Marketing, we had just been subjected to a Google algorithm update and our websites both took substantial hits to traffic. Within a few months, Brick has taken both of our websites to pre-algorithm traffic and conversion levels, and for the CallFinder site, the traffic has increased over 80% year-over-year. Their expertise in SEO, responsive design, and content marketing helped us turn the situation around, and we could not be happier with the results we’ve seen in just our first four months of working with them. The team we work with at Brick Marketing is always available for off-the-cuff consultations and are at the ready to provide recommendations and suggestions to improve our site’s appearance and performance. They are truly a hands-on partner, which is immensely valuable to our business.​”

– Jeanne Landau, Director of Marketing www.800response.com | www.mycallfinder.com

"Brick Marketing is "solid" when it comes to support for SEO marketing advice"

“I had been impressed for a long time with the content that Brick Marketing was sharing in their informative blog posts and articles. I chatted with Nick Stamoulis a couple times and decided that he was the expert I wanted to work with. I have worked with Brick Marketing for about six months and they have helped us resolve several SEO related issues pertaining to our website. Our account rep is always just an email away with answers to any questions I have and suggestions for how we can improve what we’re doing. Brick Marketing is “solid” when it comes to support for SEO marketing advice. I definitely recommend them if you want to feel more secure about how your website is performing in searches and have the confidence that everything being done to improve your rank is white hat and legit.”

– Paul Rarick, Debbiegrattan.com

"Brick Marketing has been a tremendous resource for our business"

“Brick Marketing has been a tremendous resource for our business. Through their expertise with the ever changing world of SEO, our web presence is as strong as ever. Our working relationship with Nick Stamoulis and Danielle Bachini has been outstanding. In collaboration with web designer Chris Roberts, we were also able to develop the perfect responsive website that truly reflects our business. Thank you Brick Marketing!​”

– Ellen Parlee, Parlee Farms

"Brick Marketing is levels above all others...an integral part of my team"

“I have formed an invaluable marketing partnership with Brick Marketing. Nick Stamoulis and the rest of the Brick Marketing team are professional, timely, thorough and take time to, not only succeed at the tasks, but also educate myself and my team on the strategies in the process. Since my first encounter working with Brick, I’ve changed organizations and have taken them along with me…they are that good! In my experience in working with many outside agencies who over-promise and under-communicate, I can truly state that Brick Marketing is levels above all others and vested in our relationship. They are not just an SEO consultant, but an integral part of my team. I highly recommend Brick Marketing for any company looking to significantly increase search engine competitiveness and internet presence.​”

– Jodi Murphy, Marketing Director EUXmedia

Contact Us about brick marketing Industries We Serve Case Studies

Need Assistance? Please Contact Us Below: