I've had a sudden a increase in crawl issues as of yesterday (like 300 from a steady 10, does anyone else have this issue?
-
the main issue is that it's now indexing both www and http:// - anyone else got this issue or had any changes suddenly on their crawl results?
-
If it makes things easier, based on your settings in that campaign you've always been crawling "all sites," including http://domain.com and http://www.domain.com. If this is due to a redirect, I'd guess that it's due to a redirect having been removed.
Have there been any changes made on that site recently?
-
Thank you both. I was wondering why it had done this now (the domain is at least 4 years old) but i'm going to do what you say and hope it fixes so thanks again!
-
My first response was going to be just what Ryan suggested!
I would look into why both are being indexed, which could be as simple as needing to redirect one version to the other, and then change your preferences in GWT & Bing. The issue should sort itself out after that has been taken care of.
-
Hi Rebecca. Do you have a 301 redirect in place to set your preferred domain? i.e. If someone goes to http:// are they redirected to the www, vice versa, or do both work? If both work you could get crawling on both versions. Ideally you'll pick your preferred setup, add the 301, and also set that same preference within GWT and Bing.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Looks like keyword stuffing, but it isn't
(There was a similar older question on the forum, but it wasn't really answered so please forgive me if this looks like a repeated question) Looks like keyword stuffing, but it isn't We have a massive web store with 80k "commodity products" (and this amount will only increase) which aren't sold the same way normal products are sold (i.e. by brand and model). Commodity products are sold by specification, therefore their product names are actually descriptions of the product. In our case, industrial fasteners (nuts, bolts, washers, screws, etc) sold in bulk. If you click on the link below, you will see that our catalog involves a tremendous amount of repetition, where the products all appear the same, but are varying by dimensions and/or package quantities. The solutions the web store software offers to solve this problem cause issues for us (i.e. displaying the dimensions and quantities ONLY under a common header) but more importantly, we are concerned that search engines are seeing this as keyword stuffing and penalizing the pages. http://www.aspenfasteners.com/Step-Bolts-Inch-Standard-s/407.htm If we can't change the presentation of the page, should we be concerned and if so, how do we let a search engine know that the repetition is legitimate?
Technical SEO | | AspenFasteners0 -
Structure Data Issue
HiI found few errors in Google webmaster tools under structure data. The error shows "Missing: name' but when I click 'Test Live Data' it shows 'All good'. Currently we are using Drupal CMS and please find attached error screenshot.Please advice on this issue.Thanks,SatlaqlGEyp7
Technical SEO | | TrulyTravel0 -
Anyone a Bing SEO Expert?
I have a website that is getting great long tail search positions in Google for a website.The site is not getting any Bing or Yahoo positions for these long tail terms. I'm trying to find someone has experience or can give advice on Bing rankings. Is their anything specific I should watch out for when trying to rank in Bing?
Technical SEO | | EngNet0 -
Absurdly High Crawl Stats
Over the past month and a half, our crawl stats have been rising violently. A few weeks ago, our crawl stats rose, such that the pages crawled per day worked out to the entire site being crawled 6 times a day, with a corresponding rise in KB downloaded per day. Last week, the crawl rate jumped again, such that the site is being crawled roughly 30x a day. I'm not seeing any chatter at there about an algorithm change, and I've checked and double-checked the site for signs of duplicate content, changes in our backlink profile, or anything else. We haven't seen appreciable changes in our search volume, either impressions or clicks. Any ideas what could be going on?
Technical SEO | | Tyler-Brown0 -
'External nofollow' in a robots meta tag? (advertorial links)
I believe this has never worked? It'd be an easy way of preventing any penalties from Google's recent crackdown on paid links via advertorials. When it's not possible to nofollow each external link individually, what are people doing? Nofollowing and/or noindexing the whole page?
Technical SEO | | Alex-Harford0 -
Facebook Like button issue
In looking through my top pages in Google Analytics, my #2 page (oddly enough) looked like this "/?fb_xd_fragment=". Apparently, this is because we added the Facebook Like button to many of our pages. But I'm worried these show very skewed PageView data and lower Time Spent on each page. The average time on this page is 5 seconds whereas the average sitewide time is much higher. Further, it shows 9,000 pageviews coming from only 250 Unique Visitors. I'm sure this is messing with our SEO. Is there a fix for this? Should I even be worried about it? I heard that I can remove it from my GA stat reporting, but I don't want it to be causing problems in the background. Please advise..my boss wants to keep the Facebook Like button the pages as it has brought us some good response. The page that this is on is: www.accupos.com Maybe there's an alternate version of the Facebook Like that we don't know about... I would appreciate any help on this DM
Technical SEO | | DerekM880 -
301 redirect Issues
my clients site is www.greenbayharvest.co.uk When you enter that URL it redirects to www.greenbayharvest.co.uk/shop, dont ask why, thats the way they set it up and thats what im stuck with. So, how do i resolve the 301 issue here. we want all things to point to www.greenbayharvest.co.uk, in terms of SEO but does the fact that there is a redirect going to /shop make this an issue? we appear to have: www.greenbayharvest.co.uk/shop www.greenbayharvest.co.uk greenbayharvest.co.uk greenbayharvest.co.uk/shop all these URL's go to the same same page so what is the best way to correct this? thanks for any help on this Lee
Technical SEO | | IPIM0 -
Crawling image folders / crawl allowance
We recently removed /img and /imgp from our robots.txt file thus allowing googlebot to crawl our image folders. Not sure why we had these blocked in the first place, but we opened them up in response to an email from Google Product Search about not being able to crawl images - which can/has hurt our traffic from Google Shopping. My question is: will allowing Google to crawl our image files eat up our 'crawl allowance'? We wouldn't want Google to not crawl/index certain pages, and ding our organic traffic, because more of our allotted crawl bandwidth is getting chewed up crawling image files. Outside of the non-detailed crawl stat graphs from Webmaster Tools, what's the best way to check how frequently/ deeply our site is getting crawled? Thanks all!
Technical SEO | | evoNick0