Is there a way to control a crawl?
-
I need to decrease our campaigns to 1,000 crawls we are currently at 10000 per
-
Hey Josh!
With the latest increase of crawl limits for all campaign, unfortunately the lowest is set for 10,000. If you have less than 10,000 pages then we will only crawl the total amount of pages you have. Takeshi has great tips on excluding certain pages from being crawled in robots.txt
Our user-agent is rogerbot so an example would be
User-agent: rogerbot
Disallow: /categories/Hope this helps!
-
Yes I am talking about pages crawled.
-
Kind of what I was wondering. I think this is a product deal, since they referenced 10000.
-
Hi Josh!
Thanks for posting in Q&A! Just to clarify, are you referring to your _pages _crawled in Moz Analytics?
-
Are you asking about crawls or indexation?
I can't think of any reason why you would want to decrease crawls, but you could probably do so by getting backlinks removed, blocking sections of your site from being crawled through robots.txt or nofollow links, and decreasing the change frequency in your xml sitemaps.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can i manipulate a picture in a such way that it is usable. if yes how?
Lets suppose i downloaded a photo from a XYZ website and want to use it on my own website, and also i want to rank for same keyword, and would like to rank just below XYZ site, i know there could be copyright issue. what can be done to avoid this issue. Can i manipulate the picture in a such way that it is usable. if yes how? How can i use that official websites picture for my website, i mean, can i cite that website as a source? what is the best practice in this case? i dont want to use stock photo,i really like xyz sites pics.
Image & Video Optimization | | Sam09schulz0 -
How to see SERPs the way that moz sees SERPs
I'm getting to grips with the New Keyword Explorer. I've found that one of my top keyword phrases has a 'Featured Snippet' result. However I'm struggling to see it in my own browser. Is there a clean way of viewing SERPs un enhanced by history, location etc? I'd also like to see the other SERPs features - is that possible? I'd like to be able to see the results as Moz clearly can view them! Thank you Mozers C09Fb
Image & Video Optimization | | Catherine_Selectaglaze0 -
Can Google crawl this type of content?
Dear all, I have already asked few questions like that but its some different. it is videos portal. All videos are on server side like on JWS server. when page is loaded after that my video is loaded on loading time dynamically. But Google is unable to read all content of my videos. I have done all the things in sitemap as well. Can Google fetch the video content which is pure dynamic? Please have a look my issue. than only static few pages are coming in Google and two three videos. Please help me anyone regard this. Please guys...........
Image & Video Optimization | | docbeans0 -
Does duplicate content damage ranking in YouTube the same way it does on websites?
I'm asking this question due to the following scenario: When we have videos produced by Third Party video companies they upload the video to their YouTube account before us (the customer), has chance to upload it to ours. If this was an article that was uploaded to the web then the website uploading it first would be classed as the original piece and all other website uploading the article would be considered duplicates and not rank (As I believe). Hence my question, does YouTube work in the same way and hence our videos when searched in YouTube be secondary to our Third Party video companies as they are posting them first? Regards, Tom
Image & Video Optimization | | CoGri0 -
Best way to use popular YouTube channel for SEO purposes
Hi all, We have a YouTube channel containing about 25 videos we created professionally a few years ago. They are all product features and are hosted on our site for applicable products (a very small percentage of our overall number of products). This channel was largely forgotten about for a few years. It has now reached over 160k views. What's the best way to utilize these videos for SEO benefits? Any techniques anyone has to maximize our ever increasing number of views? Thanks
Image & Video Optimization | | bradkrussell0 -
What's the best way to host video for SEO purposes?
Hello SEOmoz folks, I'm curious as to whether it will be more beneficial for a website to host a video on the site itself or to host it on YouTube and then embed the code on your site. The options so far, as it seems, are: 1.) Host the video on YouTube or Vimeo or wherever else, then embed the YouTube code into your own website's HTML. 2.) Host the video on your own website by itself. 3.) Host the video on your own website AND YouTube AND Vimeo. I'm wondering which option is best. And if hosting your video on your own website is a good way to go, whether to go that route solely or in addition to hosting on YouTube, what is the best way to go about that? Do I have to embed a player AND the video info into the HTML of my website? What player is best for this? Flash, or something else? And once I get the video on my site one way or the other, how do I write the code to make sure the video thumbnail shows up next to my search results in Google to maximize the SEO potential for the video clip? Sorry for the long question, but I've been searching everywhere for this and I'd love just a direct answer on it. Thanks for your help! Cheers, Dani
Image & Video Optimization | | MountainMedia0 -
Quick ways of finding out of date / inconsistent citations
Hi fellow mozzers! We are currently relaunching our website and are in the process of ensuring all our local citations are up to date and consistent. In doing so, I've noticed that there are still countless listings under our old address, which probably isn't doing our local rankings a whole world of good! I have been hunting these down by searching "company name" "old address" to bring up any references, but I wondered if there was a tool to find these quicker, in the same way Whitespark finds citation sources to begin with? Similarly, is there a tool that flags up listings that have the correct NAP info, but that has maybe been formatted differently enough so as to cause consistency probs? Thanks all!
Image & Video Optimization | | themegroup0 -
What are some effective ways to take up SERP real estate?
Hey Guys, I was wondering if you've seen or practiced some effective ways to take up real estate in the SERPS. Ideally I'd like to see more results from my own domain although I realize that G seems to only display 3 for any single domain. I'm looking for creative ways to take up more real estate other than subdomains. I recently saw a municipal website with 4 results (1 being a PDF) which suprised me. I have a couple in my head but what's worked best for you?
Image & Video Optimization | | reegs0