My site is not being regularly crawled?
-
My site used to be crawled regularly, but not anymore. My pages aren't showing up in the index months after they've been up. I've added them to the sitemap and everything. I now have to submit them through webmaster tools to get them to index. And then they don't really rank?
Before you go spouting off the standard SEO resolutions...
- Yes, I checked for crawl errors on Google Webmaster and no, there aren't any issues
- No, the pages are not noindex. These pages are index,follow
- No, the pages are not canonical
- No, the robots.txt does not block any of these pages
- No, there is nothing funky going on in my .htaccess. The pages load fine
- No, I don't have any URL parameters set
What else would be interfereing?
Here is one of the URLs that wasn't crawled for over a month: http://www.howlatthemoon.com/locations/location-st-louis
-
Google may have reduced their crawl budget for your site if they found enough pages to be low quality, thin, duplicate, etc... Here are some examples that you can probably apply a noindex tag to in order to reduce crawl budget waste on them.
Google these:
site:howlatthemoon.com/ inurl:tag
1,360 indexed pages that don't need to be in the SERPssite:howlatthemoon.com/ inurl:tag page
Even if you want the tag pages indexed, you don't need their paginated pages indexed toosite:howlatthemoon.com/ inurl:page inurl:category
A few dozen category pagination pages in the SERPs, many more on the site that have been booted from the index
Example:
http://www.howlatthemoon.com/dueling_piano_bar/category/nightlife/page/2/I would install the Yoast Wordpress SEO plugin, which should fix these in the following ways:
Use rel canonical tags
Use rel Next/Prev tagsNoindex tag pages
etc... -
That page is indexed. Can you give me an example of a page that hasn't been indexed yet? Your onsite SEO seems fine except you should get rid of your meta keywords tag and your Meta Description tag is a bit long.
-
oh sorry! Here is one of the pages that wasn't crawled since it was posted in early November.
-
You forgot one...what is the URL of your site
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why is this site ranking higher?
We've put a fair bit of effort into delivering value here: https://lewescountycars.co.uk/ But a search for "Lewes taxis" or "taxis Lewes" puts this site above us: https://www.lewestowntaxis.co.uk/ As you can you see, this is a tiny site that we outperform in most ways.... what can we do to rank above it that we haven't already done? Thanks in advance - Gerard.
Technical SEO | | Paul7301 -
Site Navigation
Hello, I have some questions about best practices with site navigation & internal linking. I'm currently assisting aplossoftware.com with its navigation. The site has about 200 pages total. They currently have a very sparse header with a lot of links in the footer. The three most important keywords they want to rank for are nonprofit accounting software, church accounting software and file 990 online. 1. What are your thoughts about including a drop down menu in the header for the different products? (they have 3 main products). This would allow us to include a few more links in the header and give more real estate to include full keywords in anchor text. 2. They have a good blog with content that gets regularly updated. Currently it's linked in the footer and gets a tiny amount of visits. What are your thoughts about including it as a link in the header instead? 3. What are best practices with using (or not using) no follow with site navigation and footer links? How about with links to social media pages like Facebook/Twitter? Any other thoughts/ideas about the site navigation for this site (www.aplossoftware.com) would be much appreciated. Thanks!
Technical SEO | | stageagent0 -
Re-using site code.
Hi, I'm looking at launching a new website, and am keen to understand whether re-using the basic code behind one of my other sites will cause me an issue. I'll be changing the directory structure/ file names, etc - but it will basically leave me with a very similar-looking site to another in my portfolio - using code thats all ready out there, etc. Thanks, David
Technical SEO | | newstd1000 -
Site Redesign - Regaining Rankings
We just finished designing a whole new site that will hopefully convert better than our previous site and we are currently coding it. We are hoping to get the site out in the next month or two (or three!). We want to know what to expect in regard to our sales from SEO. If you successfully launched a site redesign and your conversion rate improved, can you answer this question? How long will it take for my rankings to regain their initial ranking and then hopefully rank even higher?
Technical SEO | | EcomLkwd0 -
Should I show archives on site?
Should I show my archives on my site? I have WordPress and have dragged the archive widget to the bottom. Would this be considered duplicate content?This is what it looks like. October 2012 September 2012 August 2012 July 2012 June 2012 May 2012 April 2012 February 2012
Technical SEO | | MyAllenMedia0 -
Have my site been penalised ?
Hi, I recently hired a link builder (who works for a digital marketing agency and wanted to earn some extra cash ) to do some link building for me on keyword Carpet Cleaner Hire Basically , May my ranking for this work has moved from 29 up to 13 , then down to 500 + and now up to 67 ?. It's been all over the place and I am worried that maybe google thinks its unnatural or to many links etc. I looked at some of the work this chap did and there was a log of bookmarks done , apparently to increase social awareness and some articles on random forums I never heard of but the articles didn't look to good and in my eyes were a little spammy.. I don't have any messages in WMT saying I have been penalised or anything but I am naturally worried that this chap may have caused damaged and that I was manually penalised. I have since asked him to stop immediately.. does this sound normal for the ranking to be all over the place in this manner when link building ? How can I tell if I have been manually penalized in any way ? What;s the general consensus ?. Should I contact google or leave it and see what happens ? Other keywords seems to be okay though. Any advice much appreciated. thanks Sarah.
Technical SEO | | SarahCollins1 -
Site not indexing correctly
I am trying to figure out what is going on with my site listings. Google is only displaying my title and url - no description. You can see it when you search for Franchises for Sale. The site is www.franchisesolutions.com. Why could this happen? Also I saw a big drop off in a handful of keyword rankings today. Could this be related?
Technical SEO | | franchisesolutions0