Is there a way to control a crawl?
-
I need to decrease our campaigns to 1,000 crawls we are currently at 10000 per
-
Hey Josh!
With the latest increase of crawl limits for all campaign, unfortunately the lowest is set for 10,000. If you have less than 10,000 pages then we will only crawl the total amount of pages you have. Takeshi has great tips on excluding certain pages from being crawled in robots.txt
Our user-agent is rogerbot so an example would be
User-agent: rogerbot
Disallow: /categories/Hope this helps!
-
Yes I am talking about pages crawled.
-
Kind of what I was wondering. I think this is a product deal, since they referenced 10000.
-
Hi Josh!
Thanks for posting in Q&A! Just to clarify, are you referring to your _pages _crawled in Moz Analytics?
-
Are you asking about crawls or indexation?
I can't think of any reason why you would want to decrease crawls, but you could probably do so by getting backlinks removed, blocking sections of your site from being crawled through robots.txt or nofollow links, and decreasing the change frequency in your xml sitemaps.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to see SERPs the way that moz sees SERPs
I'm getting to grips with the New Keyword Explorer. I've found that one of my top keyword phrases has a 'Featured Snippet' result. However I'm struggling to see it in my own browser. Is there a clean way of viewing SERPs un enhanced by history, location etc? I'd also like to see the other SERPs features - is that possible? I'd like to be able to see the results as Moz clearly can view them! Thank you Mozers C09Fb
Image & Video Optimization | | Catherine_Selectaglaze0 -
Best Way to Replace Lost Links
We found a particular client that was using a 3rd party backlinker to build links (they were outsourced). The backlinks appeared to be low quality, just basic text and no substantial content surrounding it, usually like a forum profile. The backlinker was swapped out. However, we've been suffering from a big loss of links. We anticipated that quite a few of the links the previous crew were managing would be lost, however, we've lost over 1500. Some of that is due to the former SEO company using sitesays.com as part of their strategy. That site has since removed all listings/links because they were hit by recent algorithm changes. Every week GWT has fewer and fewer links coming from that site. We expect this to continue until GWT has removed all links from that site. We are also losing ground (quickly) on local rankings too. We believe this is due to the high number of links lost. What is best practice when building links to counter this issue? Is there anything that can be done to stem the link purge and, if possible, gain ground on rankings again?
Image & Video Optimization | | AaronHenry0 -
What is the best way to include video transcripts on your pages?
I just posted a question as a comment on this blog post (http://www.audiotranscription.org/6-reasons-you-need-to-have-your-sites-videos-transcribed/comment-page-1/#comment-4311) and realized I probably should have asked the question here because it is really SEO-related. We are in the process of transcribing all of our videos. Can you someone recommend how best to post the transcript file on our product pages? For example, here's a product page that has a video review on it: http://www.ccisolutions.com/StoreFront/product/behringer-x32-digital-mixing-system. Would it be best to post a link to the transcript file or insert the iframe transcript file from DotSub.com (in that case wouldn't DotSub.com be getting credit for our original content?) or, should we just let our captions on the YouTube stream from there (which is what's happening right now)? Will Google still crawl the transcript if the only place it's available is on YouTube? If so, doesn't YouTube get credited for that content, and not our site? Even if we do post a trascript on the page, if it appeared at YouTube first, wouldn't YouTube get credit for the original version? We want credit for that content...just not sure how to get it? I understand that moving to Wistia.com would probably solve most of these issues. What I'm trying to figure out is how best to handle new transcripts for existing videos over at YouTube?
Image & Video Optimization | | danatanseo0 -
What is the best way to internally link via images?
What is the best way to internally link via images? I have a website about shoes with all different brands. Let's say I'm trying to rank for "ABC sandals", and on the ABC page I have a "related brands" section with 4 logos of related sandal brands, linking to their respective page. Currently the alt text of the logos is "XYZ Logo" with the link title text "XYZ." Would it be beneficial to have change either the image alt text, or the link title text, or both to "XYZ sandals"?
Image & Video Optimization | | Hakkasan0 -
What is the best way to host a video on my site from a SEO perspective?
Hi, we are in the process of creating a video about a topic that is highly relevant for our customers. Once it is completed we want to offer the video to our customers so that they can embed it into their own websites for free. What is the best way to do this and still receive maximum SEO juice? If we use a a video service like youtube or vimeo, then the link will go to vimeo or youtube and not our site. One thing we are thinking about is asking our customers to link to our site in return for embedding the video, but if there is an automatic solution then this would be a lot better. Any tips will be highly appreciated! If you have anyother tips regarding video content then the same of course! Many thanks! Daan
Image & Video Optimization | | daan.loening0 -
Quick ways of finding out of date / inconsistent citations
Hi fellow mozzers! We are currently relaunching our website and are in the process of ensuring all our local citations are up to date and consistent. In doing so, I've noticed that there are still countless listings under our old address, which probably isn't doing our local rankings a whole world of good! I have been hunting these down by searching "company name" "old address" to bring up any references, but I wondered if there was a tool to find these quicker, in the same way Whitespark finds citation sources to begin with? Similarly, is there a tool that flags up listings that have the correct NAP info, but that has maybe been formatted differently enough so as to cause consistency probs? Thanks all!
Image & Video Optimization | | themegroup0 -
What are some effective ways to take up SERP real estate?
Hey Guys, I was wondering if you've seen or practiced some effective ways to take up real estate in the SERPS. Ideally I'd like to see more results from my own domain although I realize that G seems to only display 3 for any single domain. I'm looking for creative ways to take up more real estate other than subdomains. I recently saw a municipal website with 4 results (1 being a PDF) which suprised me. I have a couple in my head but what's worked best for you?
Image & Video Optimization | | reegs0 -
Is there an easy way to see how competitive a local search term is?
Is there a quick and easy way to see how competitive local search terms are? I am looking at helping my clients show up on the local search results. Some times all I have to do is claim a listing and they move right to the top. Sometimes I claim a listing and nothing happens so I get links and I get reviews and it takes awhile. I want to be able to put an accurate price point on the service I am offering. I have looked at the search volume and it hasn't been the best indicator because some industries are more competitve than others. Any help or suggestions would be appreciated.
Image & Video Optimization | | jimmyseo0