Website Indexing Issues - Search Bots will only crawl Homepage of Website, Help!
-
Hello Moz World,
I am stuck on a problem, and wanted to get some insight. When I attempt to use Screaming Spider or SEO Powersuite, the software is only crawling the homepage of my website. I have 17 pages associated with the main domain i.e. example.com/home, example.com/sevices, etc. I've done a bit of investigating, and I have found that my client's website does not have Robot.txt file or a site map. However, under Google Search Console, all of my client's website pages have been indexed.
My questions, Why is my software not crawling all of the pages associated with the website? If I integrate a Robot.txt file & sitemap will that resolve the issue?
Thanks ahead of time for all of the great responses.
B/R
Will H.
-
Hi Will,
It'd be impossible to find a solution to the problem without having the domain. If you want to message me the URL, I can take a quick look for ya.
A lack of either of those files shouldn't create any crawl issues. Setting up those files won't fix your problem, but you should add them both anyway.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Weird Indexing Issues with the Pages and Rankings
When I found the my page was non-existent on the search results page, I requested Google to index my page via the Search Console. And then just a few minutes after I did that, that page rose to top 3 ranking on the search page (with the same keyword and browser search). It happens to most of the pages on my website. Maybe a week later the rankings sank again, and I had to do the process again to make my pages to the top. Any reasons to explain this phenomenon, and how I can fix this issue? Thank you in advance.
Intermediate & Advanced SEO | | mrmrsteven0 -
Can I define that one area of my website is a regualr news (no subscription) and the other part of the website is news that only subscribers can read?
Hi I have a client that have a news website, he asked me if he can define one area of his website to be a regular news that google can show on google news search results (no subscription) and the other part of the website is news that only subscribers can read? Thanks Roy
Intermediate & Advanced SEO | | kadut1 -
Google WMT/search console: Thousands of "Links to your site" even only one back-link from a website.
Hi, I can see in my search console that a website giving thousands of links to my site where hardly only one back-link from one of their page to our page. Why this is happening? Here is screenshot: http://imgur.com/a/VleUf
Intermediate & Advanced SEO | | vtmoz0 -
Does Google still don't index Hashtag Links ? No chance to get a Search Result that leads directly to a section of a page? or to one of numeras Hashtag Pages in a single HTML page?
Does Google still don't index Hashtag Links ? No chance to get a Search Result that leads directly to a section of a page? or to one of numeras Hashtag Pages in a single HTML page? If I have 4 or 5 different hashtag link section pages , consolidated into one HTML Page, no chance to get one of the Hashtag Pages to appear as a search result? like, if under one Single Page Travel Guide I have two essential sections: #Attractions #Visa no chance to direct search queries for Visa directly to the Hashtag Link Section of #Visa? Thanks for any help
Intermediate & Advanced SEO | | Muhammad_Jabali0 -
If other websites implement our RSS feed sidewide on there website, can that hurt our own website?
Think about the switching anchors from the backlinks and the 100s of sidewide inlinks... I gues Google will understand that it's just a RSS feed right?
Intermediate & Advanced SEO | | Zanox0 -
Do search results differ greatly when you search on mobile?
If you have a site with responsive design, is Google likely to look upon you more favourably and dramatically change rankings?
Intermediate & Advanced SEO | | BlueLinkERP0 -
Retailers Issue
Hi there, We have 20 retailers who are about to launch websites and are going to be selling our products on their websites, however with they have no content for these products they are wanting to take our content we have for our product pages on place the content on their websites, is this going to cause an issue for me? We are ranking well for competitive keywords in this niche and do not want to do anything to harm it. What I would say is the retailers in question of no intention short term anyway of doing anything with SEO. Thanks for any help
Intermediate & Advanced SEO | | Paul780 -
Duplication Issue?
One of our copywriters has just written a blog to be posted on our own company blog to be reviewed by myself, however I had noticed that the blog post has some duplication issues with one of our own product pages, about 60% duplication, is it still worth posting? Will search engines still index the blog post? Kind Regards,
Intermediate & Advanced SEO | | Paul780