Is there a way to control a crawl?
-
I need to decrease our campaigns to 1,000 crawls we are currently at 10000 per
-
Hey Josh!
With the latest increase of crawl limits for all campaign, unfortunately the lowest is set for 10,000. If you have less than 10,000 pages then we will only crawl the total amount of pages you have. Takeshi has great tips on excluding certain pages from being crawled in robots.txt
Our user-agent is rogerbot so an example would be
User-agent: rogerbot
Disallow: /categories/Hope this helps!
-
Yes I am talking about pages crawled.
-
Kind of what I was wondering. I think this is a product deal, since they referenced 10000.
-
Hi Josh!
Thanks for posting in Q&A! Just to clarify, are you referring to your _pages _crawled in Moz Analytics?
-
Are you asking about crawls or indexation?
I can't think of any reason why you would want to decrease crawls, but you could probably do so by getting backlinks removed, blocking sections of your site from being crawled through robots.txt or nofollow links, and decreasing the change frequency in your xml sitemaps.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can i manipulate a picture in a such way that it is usable. if yes how?
Lets suppose i downloaded a photo from a XYZ website and want to use it on my own website, and also i want to rank for same keyword, and would like to rank just below XYZ site, i know there could be copyright issue. what can be done to avoid this issue. Can i manipulate the picture in a such way that it is usable. if yes how? How can i use that official websites picture for my website, i mean, can i cite that website as a source? what is the best practice in this case? i dont want to use stock photo,i really like xyz sites pics.
Image & Video Optimization | | Sam09schulz0 -
Can Google crawl this type of content?
Dear all, I have already asked few questions like that but its some different. it is videos portal. All videos are on server side like on JWS server. when page is loaded after that my video is loaded on loading time dynamically. But Google is unable to read all content of my videos. I have done all the things in sitemap as well. Can Google fetch the video content which is pure dynamic? Please have a look my issue. than only static few pages are coming in Google and two three videos. Please help me anyone regard this. Please guys...........
Image & Video Optimization | | docbeans0 -
Local Search Clean up Done. Now Best Way to Index?
I have spent hours doing some local search clean up and have a list of all the URLs to index, about 75. Is there a place I can dump these to get them all indexed quickly or do I have to wait for the hand of Google to come down and bless me with indexing. I don't want to dump all these to Google +, Facebook, or Twitter; that would just be wrong. Any ideas for fast indexing. I want them to all get indexed before the business address changes in a couple years 🙂
Image & Video Optimization | | photoseo10 -
Best way to publish same content on 2 blogs?
I currently have a couple of blogs and I want to put a post on one of them and then copy it onto my other blog, but give the credit and the link back to the original (lose you yet?) I want to write a blog and publish it on my site (http://goo.gl/WRwR8) After it's on the blog on my main e-commerce site, i want to copy it onto my wordpress blog (http://goo.gl/0SSti) What is the best way to accomplish this without creating duplicate content? Is there a way to state on the second wordpress blog that this post originated from the ecommerce site? I could really use some guidance... Thanks
Image & Video Optimization | | Prime850 -
What is the best way to embed a link back to our site in our video embed codes?
Our site has over 100 great videos. We are currently streaming them all from our YouTube channel. I noticed SEOMoz is using http://www.wistia.com and that when I copy and paste a video from SEOMoz that code has a link back to the SEOMoz site in it, which of course is awesome for SEO. My question has several parts: Is this possible to do with an embed code from YouTube? Specifically directed towards SEOMoz: Which of the 3 types of player did you choose, HTML5, Flash or javascript? Why? Is Wistia just creating a video feed, or are they actually hosting your video content too? Our goal is two-fold: 1. We want people who embed our videos to be also embeddding a link back to our site and 2. We want our videos to be found in SERPs and have those direct people to our site instead of our YouTUbe channel. I'd welcome any comments regarding video SEO in this thread. Thanks!
Image & Video Optimization | | danatanseo0 -
Australian Service that Allows you to Control all your Local Listings
Hey, After a few stumbles upon a variety of services that allows you to manage all your local listings from the one profile (the latest one was Yext.com). It got me thinking - is there an Australian equivalent? My research revealed nothing. Thanks.
Image & Video Optimization | | LukeyJamo1 -
Quick ways of finding out of date / inconsistent citations
Hi fellow mozzers! We are currently relaunching our website and are in the process of ensuring all our local citations are up to date and consistent. In doing so, I've noticed that there are still countless listings under our old address, which probably isn't doing our local rankings a whole world of good! I have been hunting these down by searching "company name" "old address" to bring up any references, but I wondered if there was a tool to find these quicker, in the same way Whitespark finds citation sources to begin with? Similarly, is there a tool that flags up listings that have the correct NAP info, but that has maybe been formatted differently enough so as to cause consistency probs? Thanks all!
Image & Video Optimization | | themegroup0 -
Is there a very quick way to geenrate the google top 100 for a any given keyword
I ma looking to produce a list of the top 100 websites in a list format from any given keyword. Bascially a process or ftool that will generate a list so i dont have to copy and paste all of them.....any ideas guys and gals
Image & Video Optimization | | raytax0