SEOMOZ crawl all my pages
-
SEOMOZ crawl all my pages including ".do" (all web pages after sign up )
. Coz of this it finishes all my 10.000 crawl page quota and be exposed to dublicate pages.
Google is not crawling pages that user reach after sign up. Because these are private pages for customers I guess
The main question is how we can limit SEOMOZ crawl bot. If the bot can stay out of ".do" java extensions it'll perfect to starting SEO analysis.
Do you know think about it?
Cheers
Example;
.do java extension (after sign up page) (Google can't crawl)
Normal Page (Google can crawl)
http://magaza.turkcell.com.tr/telefon/Apple-iPhone-3GS-8GB/1001694/.html
-
Hi There,
Thanks for writing in and sorry for the confusion.
It actually isn't possible for the SEOmoz crawler to access pages that require a user login. I went to those URLs and I was able to access the pages with out be logged in as a user, so they don't require user sign up to access them. Since these pages are linked to by other pages on your site and our crawler is not being blocked from these pages and the pages don't actually require a user to be signed in to access them, we will crawl them. I can't say why Google wouldn't be crawling those pages, but there is definitely nothing in place that would stop our crawler from accessing them.
If you would like to stop our crawler from accessing those pages in the future, you may consider adding a disallow directive in your robots.txt file using the user-agent rogerbot.
I hope this helps. Let me know if you have any other questions.
Chiaryn
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What is the proper way to execute 'page to page redirection'
I need to redirection every page of my website to a new url of another site I've made. I intend to add:"Redirect 301 /oldpage.html http://www.example.com/newpage.html"I will use the 301 per page to redirect every page of my site, but I'm confused that if I add:"Redirect 301 / http://mt-example.com/" it will redirect all of my pages to the homepage and ignore the URLs i have separately mentioned for redirection.Please guide me.
Intermediate & Advanced SEO | | NABSID0 -
Does it make sense to create new pages with friendlier URLs then redirect old pages to new?
Hi Moz! My client has messy URLs. does it make sense to write new clean URLs, then 301 redirect all old URLs to the new ones? Thanks for reading!
Intermediate & Advanced SEO | | DA20130 -
Indexing Dynamic Pages
http://www.oreillyauto.com/site/c/search/Wiper+Blade/03300/C0047.oap?make=Honda&model=Accord&year=2005&vi=1430764 How is O'Reilly getting this page indexed? It shows up in organic results for [2005 honda accord windshield wiper size].
Intermediate & Advanced SEO | | Kingof50 -
I've seen and heard alot about city-specific landing pages for businesses with multiple locations, but what about city-specific landing pages for cities nearby that you aren't actually located in? Is it ok to create landing pages for nearby cities?
I asked here https://www.google.com/moderator/#7/e=adbf4 but figured out ask the Moz Community also! Is it actually best practice to create landing pages for nearby cities if you don't have an actual address there? Even if your target customers are there? For example, If I am in Miami, but have a lot of customers who come from nearby cities like Fort Lauderdale is it okay to create those LP's? I've heard this described as best practice, but I'm beginning to question whether Google sees it that way.
Intermediate & Advanced SEO | | RickyShockley2 -
What do you do with the page of a product that has been deleted?
As anyone know with an ecommerce website, products are constantly being added and removed. Once products are removed, the corresponding product pages are not reachable. Currently, I am redirecting to the Search page, if a product page is reached, whose corresponding product has been deleted. I am not sure if that is the correct, recommended technique from a SEO perspective. Should I try to show related products on the redirected page? Does anyone here know what is the best thing to do with this product page?
Intermediate & Advanced SEO | | amitramani0 -
Cleaning bad pages
We have 10,000 of bad pages, which panda could track and penalize us for that. If we delete them we will get 404 error, and after that we could again get penality from G algo. How can i delete them to follow google rules and avoid penalities? If we make redirect of 10k pages with 301 to index, can 10k old pages be treated as duplicate?
Intermediate & Advanced SEO | | bele0 -
Could you use a robots.txt file to disalow a duplicate content page from being crawled?
A website has duplicate content pages to make it easier for users to find the information from a couple spots in the site navigation. Site owner would like to keep it this way without hurting SEO. I've thought of using the robots.txt file to disallow search engines from crawling one of the pages. Would you think this is a workable/acceptable solution?
Intermediate & Advanced SEO | | gregelwell0 -
How to Preserve PageRank for Disappearing Pages?
Pretend that USA Today has a section of their site where the sell electronics, which they locate at http://electronics.usatoday.com. The subdomain is powered by an online electronics store called NewCo via a white label. Many of the pages on this subdomain have relatively high PageRank. But few, if any, external sites link to the subdomain--the PageRank of the subdomain is largely due to internal links from the usatoday.com root domain. USA Today's deal with NewCo expires and they decide to partner with my startup instead. But, unlike NewCo, we won't be providing a white-label solution; rather, USA Today will be redirecting all of the electronics-related links on their root domain to my site instead of the electronics.usatoday.com subdomain. They also agree to direct all of the pages on electronics.usatoday.com to me. Ideally USA Today would add 301's to all of their pages on electronics.usatoday.com that direct to the corresponding pages on my site, but they don't have the engineering wherewithal or resources to do this. Therefore, what is the best way to pass the PageRank from the electronics.usatoday.com pages to my site? Would it work to have USA Today change the CNAME for electronics.usatoday.com to my site and then create pages on my site that mimic the USA today URL structure? For example, let's say there was a page located at electronics.usatoday.com/ipods. Could we give electronics.usatoday.com a CNAME form my site and then create a page on my site located at mysite.com/ipods that 301'ed to the ipod page on my site? Would that preserve the PageRank?
Intermediate & Advanced SEO | | jack789078900