Problem crawling a website with age verification page.
-
Hy every1,
Need your help very urgent. I need to crawl a website that first has a page where you need to put your age for verification and after that you are redirected to the website. My problem is that SEOmoz, crawls only that first page, not the whole website. How can I crawl the whole website?, do you need me to upload a link to the website?
Thank you very much
Catalin
-
Hello Catalin,
Our crawler will not be able to get past an age verification page. You will need to find or unlock a subfolder or subdomain to bypass this if you would like our crawlers to be able to get through. Luckily, Google's crawlers are a bit more thorough a will be able to index your site properly. We are hoping to add this ability soon and I hope you can find a way for us to get through in the meantime.
-
the problem is that the pages are not in a subfolder. I have to pass the verification page every time :(. SEOMoz is crawling only the first page.
-
Well that's a small side note to your problem ;-), are you able to just set up a crawl for a sub folder? Or do you have to pass the verification at all times?
-
OK, thank you for your short answer, but the thing is I didn't understand anything from what you wrote :).
I want to add that I do not own the website. I dont have acces to back-end, cms, etc. The client just wants me to crawl the whole website to see if something is wrong. I can see with my own eyes that the website has duplicate content, but seomoz doesnt crawls the website, because of that first page with verification.
-
Hi Catalin,
The best way do to this is of course to include a link to the rest of the Web site (you could remove the link of course when Roger came by). But what you also could is redirect the user based on the user agent when linking wouldn't be an option.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Crawl Diagnostics saids a page is linking but I can't find the link on the page.
Hi I have just got my first Crawl Diagnostics report and I have a questions. It saids that this page: http://goo.gl/8py9wj links to http://goo.gl/Uc7qKq which is a 404. I can't recognize the URL on the page which is a 404 and when searching in the code I can't find the %7Blink%7D in the URL which gives the problems. I hope you can help me to understand what triggers it 🙂
Moz Pro | | SebastianThode0 -
Page with "Missing Title Tag" isn't a page
Hello, I am going through the various errors that the Moz Pro Crawl report and some non-existent pages keep coming up in the report. For example, one error category is "Missing Title Tag" with one page identified. But this page http://www.immigroup.com/news/“http%3A/crs.yorku.ca”?page=2 isn't real. It would have been a 404 were there not a redirect for everything that is /news/gobbledygook to /news. So my question is: when moz (or GA for that matter) identifies these pages as "real" and having errors, do I need to take this seriously? And what do I do about it? Thanks! George
Moz Pro | | canadageorge0 -
Website Issues - Duplicate Content
Hello, I'm fairly new to using Moz and I logged on this morning to find Issues have been found in one of the websites - 22 High Priority and 44 Medium. I know it's due to duplicate content in the blog, but i can't figure out what is duplicated? I've only recently come on board this website so I don't know if the content has been plagiarised or what? The link to the site is here: delacyspa.co.uk Any help would be appreciated. Thanks zFxQmmd
Moz Pro | | Cowbang0 -
How to make a Crawl Report readable?
Hi! I am trying to find out how I make my CSV report neat, so I can interprete the report. I know have a CSV report with just numbers and text all in one column. I tried the button text to columns but that doesn't work because when I do that at column A it overwrites column B in which I have the same problem! Thanks
Moz Pro | | HetCommunicatielokaal0 -
Sitewide links causing problem
Hello, I've had a couple of people look at the backlink profile for nlpca(dot)com and tell me that there were a lot of sitewide links with non-branded anchor text. The spreadsheet is [here](http://thewealthymind.com/wp-content/uploads/2013/03/anchor-text.xls). Actually, because of our company name, all sitewide links have branded anchor text except for http://www.iepdoc.nl
Moz Pro | | BobGW
http://www.ionut-ciurea.com The term "NLP" is not ranking for us, we should be on page 1 or 2. Also, we used to be sitelinked for "NLP California" and aren't anymore. Notice that "nlpca" splits into "nlp" and "ca" for California. Could you look and see if it is just those 2 sites and if all we need to do is contact them to change to branded anchor text. Or if it's something else to get back our rankings for "NLP" and our sitelinks for "NLP California" Thank you0 -
Adjusting SEOmoz Crawling Speed
How do you adjust the SEOmoz crawling speed? SEOmoz tried to crawl 10,000 pages in 3 hours and crashed our MySQL server.
Moz Pro | | cappuccino891 -
Reports for page titles
Is there a report I can run on SEOmoz that shows me the page titles for all pages on my website, along with the link to each page?
Moz Pro | | TalarMade0