Why do we have so many pages scanned by bots (over 250,000) and our biggest competitors have about 70,000? Seems like something is very wrong.
-
We are trying to figure out why last year we had a huge (80%) and sudden (within two days) drop in our google searches. The only "outlier" in our site that we can find is a huge number of pages reported in MOZ as scanned by search engines. Is this a problem? How did we get so many pages reported? What can we do to bring the number of searched pages back to a "normal" level?
BT
-
Hi. A mystery indeed! Have you recently upgraded or changed Web platforms or changed or upgraded what you are using for your site navigation?
-
Stewart_SEO
Thanks for your quick response. We did review the robots.txt of the competitors. Not line by line - they took surprisingly different approaches to the robots.txt. But there were the usual exclusions for wish lists, etc. We've gone back and tightened up our robots.txt and haven't yet seen any changes. Several months ago we were at about 600,000 pages and it is dropping. Very mysterious.
-
Have you looked at your competitors robots.txt file? they are probably blocking the very same searches you are talking about. if there is a particular bot like a Chinese crawler for example baidu that you don't want to come to your site you can block them via the command: User-agent: Baiduspider
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Multiple links from same domain (different pages) considered in credibility of backlinks?
Hi, Let's say there are multiple backlinks from different pages of same domain to different pages of other domain like below: Website A: Page 1 -----------> Website B: Page 1 Website A: Page 2 -----------> Website B: Page 2 Do the pages of Website B pages will get backlinks authority equally or they don't get much backlinks impact as they have multiple backlinks from same domain? There were old school stories that Google ignores second link from same domain.....etc... So, please suggest on this. Thank you. Note: The question is NOT about content relevancy or domain authority score of the backlinks.
Algorithm Updates | | vtmoz1 -
What date tags are required/best to use for articles or other landing pages?
Does anyone have suggestions on which date tag(s) are most important to use and how to use them on the frontend? (i.e. dateModified, dateCreated, and datePublished). The Structured Data Testing Tool is coming up with errors for my article pages, but I'm a bit confused which ones should be in the code vs. showing on the frontend.
Algorithm Updates | | ElsaT0 -
Category Containing a Product searched shows up higher in google then the product page itself?
Hello Moz Wizards, We have recently launched a new eCommerce website www.memoky.com and think we did a pretty good job with the markup structure for feeding the hungry google bot all information available about a the products. However google doesn't like us very much : ( It seems every time you google a product that we carry; the category pages that contain that product will show up, but the product page itself does not. Below are two examples, however this seems to be site-wide which makes me feel like there is an underlying issue that we are missing. Examples
Algorithm Updates | | Memoky
when searched for "Eduardo floor lamp - matt black/matt yellow shade"
Shows ups - http://www.memoky.com/lighting/floor-lamps.html
Does not - http://www.memoky.com/eduardo-floor-lamp-matt-black-matt-yellow-shade.html when searched for "Derrick arm chair - white leather/ walnut"
Shows ups - http://www.memoky.com/living/lounge-chairs.html_
Does not - http://www.memoky.com/derrick-arm-chair-white-leather.html_ that is the pattern for almost all the products on this site. Any thoughts on why this could be the case?0 -
How much is Page Rank really worth?
We are in a position to purchase a domain, made of relevant keywords to our company with a current page ranking of 4 for their home page. However in looking at their analytics and other information they do not do well on significant keywords and have very low site traffic. In fact they do very, very poorly. With their high page ranking would it be relatively easy to conduct a successful SEO campaign on the domain if we were to take it over as our own and attempt to climb in the SERP's? I know Page Rank doesn't mean everything when it comes to your ranking, but 4 is relatively high in our field, so I don't really understand why they do so poorly when it comes to their actual rankings on key words.
Algorithm Updates | | absoauto0 -
Will too many [img no alt-text] links harm a link profile?
Hi everyone. I have a client who has a lot of sponsorships etc and therefore a lot of inbound image links (many of them sitewide). Unfortunately most of these don't have alt text, and [img no alt-text] links now make up over 50% of their link profile. Should I be trying to correct this and requesting updates from the people who are linking? Obviously I wouldn't want loads of keyword stuffed alt texts, but maybe I should request alt texts on the brand name or URL instead. Do you think this would make a significant difference and be worth the time it will take to contact all these webmasters? Thanks in advance.
Algorithm Updates | | QubaSEO0 -
Should you include Website Title in all page title tags?
We recently spent analyzing some of the best SEO software companies on the U.S. market fishing for the best practices in SEO and I saw one thing in common : They all had website titles in all the page title tags separated by " | " Is that the best practice for SEO or is it just for Branding? Interestingly enough, the website titles were completely unrelated to the pages' content or keywords. (Here's my personal opinion on what it looked like: "riding on a bicycle" | Ferrari ) But when I looked up the keywords ... ranked #1 or #2 spots, in some serious competition. (So in the example above, "bicycle" would be in the top spot)
Algorithm Updates | | HMCOE0 -
Google +1 link on Domain or Page?
Since its release, I've seen Google +1 being used across an entire domain but only reference the root href in the code snippet. At the same time, you see other sites use +1 more naturally with the button being specific to the page you're on. What's your take on this? To clarfiy, do you add: or .. on each page.
Algorithm Updates | | noeltock0 -
Does Google index Wordpress pages with frames
Does Google or other search engines index Wordpress pages that use frames? Here is the site in question: http://www.source-nutrition.com/son/
Algorithm Updates | | BradBorst0