Having issues crawling a website
-
We looked to use the Screaming Frog Tool to crawl this website and get a list of all meta-titles from the site, however, it only resulted with the one result - the homepage.
We then sought to obtain a list of the URLs of the site by creating a sitemap using https://www.xml-sitemaps.com/. Once again however, we just go the one result - the homepage.
There is something that seems to be restricting these tools from crawling all pages. If you anyone can shed some light as to what this could be, we'd be most appreciative.
-
That robots.txt should be fine.. its not blocking anything.
The reason the crawl is stopping on the homepage is this code:
<meta name="<a class="attribute-value">robots</a>" content="<a class="attribute-value">nofollow</a>">
Which tells bots to not follow any links on the page. Remove that and you should be good.
-
Hi,
I think it is your robots.txt file that is causing the issue. At the moment you have the following:
**User-agent: ***
Disallow:
I would recommend updating it to the following:
**User-agent: ***
Allow: /
Moz also has a good post about what else you can include in your robots.txt file for best practices etc. :
https://moz.com/learn/seo/robotstxt
Hope that helps
Thanks
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Please Help me! I need advice for my website
I have 2 Domains with the same name vps nine and vpsn ine with same content. How to solve that problem? Do I need to change the content from my main website. My Hosting is having different plans, but with the same features. So many pages were having the same content, and it is not possible to change the content, what is the solution for that? Please let me know how to solve that issue?
Intermediate & Advanced SEO | | Alexa.Hill0 -
How to rank my website in Google UK?
Hi guys, I own a London based rubbish removal company, but don't have enough jobs. I know for sure that some of my competitors get most of their jobs trough Google searches. I also have a website, but don't receive calls from it at all. Can you please tell me how to rank my website on keywords like: "rubbish removal london", "waste clearance london", "junk collection london" and other similar keywords? I know that for person like me (without much experience in online marketing) will be difficult task to optimize the website, but at least - I need some advices from where to start. I'm also thinking to hire an SEO but not sure where to find a trusted company. Most importantly I have no idea how much should pay to expect good results? What is too much and what is too low? I will appreciate all advices.
Intermediate & Advanced SEO | | gorubbishgo0 -
Best Sitemap for Large Website
i have more than 3500 pages on my website. Please let me know the best sitemap plugin for my website.
Intermediate & Advanced SEO | | Michael.Leonard1 -
SEO issues with removing a forum from a site
I'm thinking of adding a Discourse discussion forum to one of my websites. I'm not sure if it's going to be something that works well or not for the site. So I'm thinking ahead and wondering what Google issues I could have if after a few months of having the forum, I decide to remove it. What would Google think about all the then non-existent pages it might have indexed? Would there be a simple wildcard redirect I could do in htaccess that would satisfy that? Or some other thing I should do?
Intermediate & Advanced SEO | | bizzer0 -
First Website
Hi Everyone, I have just published my first website and was wondering if anybody would like to help me with some hints and tips. This is my first time branching into SEO and could really do with some help. Any feedback would be greatly appreciated. The site address is www.theremovalistsguide.com.au which targets the furniture removal industry in Australia. Thanks for your help.
Intermediate & Advanced SEO | | RobSchofield0 -
Yahoo directory listing issue
Hello all, We submit our site http://tinyurl.com/5v9hrql to Yahoo's directory (Standard Listing) on 08/26/2012 at this time the order remains as pending and the site is not listed on our suggested category http://tinyurl.com/d4a5lyf Furthermore, we haven't gotten any email from Yahoo Team; something is wrong with our submission or need to contact to? Thank you for your help
Intermediate & Advanced SEO | | SharewarePros0 -
Production and Priority Issue for SEO and Website Usability
I am a NOVICE .........My website is about 4 months old. My developer/programmer only has 4-6 hours of work a week so it is going to take 4 months to finish two weeks of work. So I have to prioritize the things that are best for SEO (Our architecture is PHP,Apache and Zend) .** If you are interested I would be curious to how you would prioritize some or all of these. Or at least as many as you can until you get bored.** 1. Optimizing Cart/Conversion - 7 hrs - (Extremely low conversion rates)
Intermediate & Advanced SEO | | Boodreaux
2. Optimizing Speed for usability -10+ hrs (Very slow on initial load time) 10-14 sec
3. Filling in all Titles and Metadata - 2 hrs
4. Contact persistence with cookie...enter data only once. - 2 hrs
5. Social panels for sharing content - 3 hrs
6. Custom notifications for those who opt in. for updates - 5 hrs
7. Shorten 12 key URL's and optimize with key words - 3 hrs (I rank this very high)
8. Install Wordpress Blog - 5-10 hrs
9. RSS Feed - 5 hrs ( Run a feed real time on side of page)
10. Create Content Management System for me - 20 hrs (So I can make changes)
11. Keywords for H-1 Tags - 1 hr
12. At tag for images - 1 hr
13. Use of bold /italics - 2 hrs
14. Canonical tag in head - 3 hrs Any expert advice will be greatly appreciated. Boodreaux PS After studying SEO for 1 month I think the priorities should be #7,#3, #2, #1, #5 (on landing pages) #11, #12,#6, #4, #13, #14, #8, #9, #100 -
New Website.. From where to start with a SEO Plan ?
Hi.. We have got two SEO and link builders working here in the office, putting in efforts is not a problem. And we have been doing SEO for our websites since a long time and they all rank well. Since, I have signed up for SEOMOZ, which is the one stop with all the heavy duty SEO guys, so we would like to get some suggestions from the experts. Ok here is the question. We have launched a new website this week and we have got a new SEO client, whose website is 6 months old. 1. What should be a proper SEO plan for a new website ? Got any Plan PDF or Buleprint, which you can send me ? 2. For the client website , he is ranking on 4-7th pages for his 4 keywords. Whats should be the correct way or plan to start doing SEO for them and get them to the first page ? 3. I feel like, doing competitive analysis is lacking with us and we are not good at evaluating that step. Whats the bet way to do competitive analysis ? 4. We are looking to build more and more links to the website . Whats the best way to make links ? Got any SEOMoz Blog post link, which explain this better or any forum thread which has got the tricks for this ?
Intermediate & Advanced SEO | | qubesys0