Crawl Test produced only 1 page
-
Hi, I recently submitted a crawl for www.cirrato.com using SEOMoz Crawl Test Tool. I have a lot of pages, but the crawl result shows only 1 page, which is the front page and nothing else...
Does anyone know what this could mean or what the problem is?
-
Ok Yusuf, if you blocked all robots than seomoz wouldn't be able to crawl your site.
if you are still working on the website, there is a way to block search engine but let Seomoz robot to cawl the website
let us know if you have a problem with that )
-
Thanks for your comment. I checked my robots.txt as you said and it was set for:
Disallow: *
That was probably the problem. I'm going to do a new crawl and see if the problem remains. I'll let you know here afterwards.
Again, thanks for the comment.
-
what is the url? is everything ok with robots.txt ? i've never experienced that on my campaigns, i'd be curious to take a look
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Filter Pages
Howdy Moz Forum!! I have a headache of a job over here in the UK and I'd welcome any advice! - It's sunny today, only 1 of 5 days in a year and i'm stuck on this! I have a client that currently has 22,000 pages indexed to Google with almost 4000 showing as duplicate content. The site has a "jobs" and "candidates" list. This can cause all sorts of variations such as job title, language, location etc. The filter pages all seem to be indexed. Plus the static pages are indexed. For example if there were 100 jobs at Moz being advertised, it is displaying the jobs on the following URL structure - /moz
Moz Pro | | Slumberjac
/moz/moz-jobs
/moz/moz-jobs/page/2
/moz/moz-jobs/page/3
/moz/moz-jobs/page/4
/moz/moz-jobs/page/5 ETC ETC Imagine this with some going up to page/250 I have checked GA data and can see that although there are tons of pages indexed this way, non of them past the "/moz/moz-jobs" URL get any sort of organic traffic. So, my first question! - Should I use rel-canonical tags on all the /page/2 & /page/3 etc results and point them all at the /moz/moz-jobs parent page?? The reason for this is these pages have the same title and content and fall very close to "duplicate" content even though it does pull in different jobs... I hope i'm making sense? There is also a lot of pages indexed in a way such as- https://www.examplesite.co.uk/moz-jobs/search/page/9/?candidate_search_type=seo-consulant&candidate_search_language=blank-language These are filter pages... and as far as I'm concerned shouldn't really be indexed? Second question! - Should I "no follow" everything after /page in this instance? To keep things tidy? I don't want all the variations indexed! Any help or general thoughts would be much appreciated! Thanks.0 -
Problem crawling a website with age verification page.
Hy every1, Need your help very urgent. I need to crawl a website that first has a page where you need to put your age for verification and after that you are redirected to the website. My problem is that SEOmoz, crawls only that first page, not the whole website. How can I crawl the whole website?, do you need me to upload a link to the website? Thank you very much Catalin
Moz Pro | | catalinmoraru0 -
Page Authority is the same on every page of my site
I'm analyzing a site and the page authority is the exact same for every page in the site. How can this be since the page authority is supposed to be unique to each page?
Moz Pro | | azjayhawk0 -
On-Page Optimization Report: How Are Keywords Chosen?
Apologies if this has already been covered 100 times! Last month I set up a new campaign, and so far the On-Page Optimization tool has only crawled and graded three of my pages so far. I assume it takes time for more pages to be covered? But, here's my real question: I see that the tool is giving my pages grades based on certain keywords, but the tool itself seems to be deciding which keyword to use in grading each page. To use a made-up example, my example has a page about leather gloves, a page about wool mittens, and a page about cotton mittens. The last one is supposed to be optimized for the keyword "cotton mittens," but the tool is grading it based on how well it's optimized for "wool mittens." I can go into the drop-down at the top of the page and change the keyword that the page is graded on, and that gives me a new grade, but only for that instance. The next week, the tool is back to giving the page an F for "wool mittens." Is that because the tool decides that "wool mittens" is the keyword for which the page has the best chance of ranking, no matter what my intentions are? is there any way to permanently tell the tool that I want the page to target "wool mittens" as its main keyword? Thanks in advance for your help!
Moz Pro | | ScottShrum0 -
Third crawl of my sites back to 250 pages
Hi all, I've been waiting some days for the third crawl of my sites, but SEOMOZ only crawled 277 pages. The next phrase appeared on my crawl report: Pages Crawled: 277 | Limit: 250 My last 2 crawls were of about 10K limit. Any idea? Kind regards, Simon.
Moz Pro | | Aureka0 -
Issue: Duplicate Page Content
For pro members: Is there a way to see exactly what "it" is that is duplicate? Also, what % of duplicate content is required to be labeled as dupped. Much thanks, Chenzo
Moz Pro | | Chenzo0 -
Amount of Pages Crawled Dropped Significantly
I am just wondering if something changed with the SEOMoz crawler. I was always getting 10,000 or near 10,000 pages crawled. After the last two crawls I am ending up around 2500 pages. Has anything changed that I would need to look at it see if I am blocking the crawler or something else?
Moz Pro | | jeffmace0 -
On Page Analysis and Grading
I received an email that my on page analysis for my campaigns were completed. But when I click on the link there are no grades there. What does that mean? Another question on this topic....when your campaign is graded are pages graded on all the keywords in the campaign or is each keyword graded invidividually? Thanks!
Moz Pro | | Confections0