My website pages are not crawled, what to do?
-
Hi all.
I have made some changes on the website so i like to crawled them by the search engines Google especially.
I have made these changes around 2 weeks ago. I have submitted my website on good bookmarking websites.
Also i used a tool available in Google webmasters "Fetch as Google", Resubmitted a sitemap.xml.
Still my pages are not crawled your opinion please.
Thanks
-
this is somenting I would love to know more!-)
I think news and updates, maybe a blog in your site can make google "feel" it has to come back more times to refresh your site again ( "hum dude, they are doing something there)... Links to your site and to new articles in social media could help a lot.
If you find anything let us know!-)
-
SEO Martin,
all these 93 results are index successfully also shows in our Google webmaster account.
But i am worried that why crawler is not visit our website pages on regular basis as we submit sitemap.xml in webmaster account after the changes have made.
what is the reason behind this.
also can you please explain me that what the other methods to force the search engine crawler to crawl our page.
Regards
Lucid
-
Hi Sha,
I just add this meta tag (as below) to help to crawl the page but not sure how it is.
I am just worried that if we make changes in our website pages and submit the site map in webmaster (that's mean that we are put request to search engine crawler to crawl the page ) but why they do not.
what is the actual reason behind this.
also can you please explain me that what the other methods to force the search engine crawler to crawl our page.
Regards
Lucid
-
HI Martin,
Exactly, all url are indexed in search engines but they are not crawled after making a changes that is the main concern for me now.
For example:
Take a look on Magento Development page that showing last crawled 17 Mar 2013. I have made some changes and that should be crawled on a regular basis same as previous crawling.
Url :http://www.lucidsoftech.com/magento-development.html
your thoughts?
Thanks
-
Lucid,
I´ve just made a search on google and I found 93 results for your site, http://www.lucidsoftech.com/ , is that url right?
How many results do you actually own?
-
HI Sha,
yes site is indexed and crawled on regular basis before making these changes. don't know why not this time and not crawled around 2 weeks. yes i checked for thart all have including tag.
Normally when u submitted your pages on few good social bookmarking, workout in webmasters, after generate few good back links all these thing are very helpful for page crawling.
Still waiting for the page crawling.Thanks Sha
-
HI Martin,
Before making these changes they crawled regular basis and I just changed a meta title keywords and description but don't know the reason why after making these changes on few pages my web pages are not crawled in past 2 weeks and and still they are not crawled.
Martin i have my robots.txt in public.html and i allow the page that should be crawled by the bots.
waiting fro the page crawling.Thanks
-
Hi lucidsoftech,
The site is definitely indexed, but without knowing the specific URLs you are waiting on, it is a little difficult.
I ran a quick crawl with Screaming Frog and there are actually quite a few pages on the site which have the noindex meta tag in place - have you checked that the pages you are wanting indexed are not excluded with a tag?
Just a thought,
Sha
-
make sure you have a robots.txt on public folder. You must allow bots and disallow some folders.
Example of robots.txt of wordpress site
User-agent: *
Allow: /Disallow: /*?
Disallow: /feed/
Disallow: /trackback/
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /xmlrpc.php
Disallow: /wp-
Disallow: /tmpAllow: /wp-content/uploads/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Paginated pages are being indexed?
I have lots of paginated pages which are being indexed. Should I add the noindex tag to page 2 onwards? The pages currently have previous and next tags in place. Page one also has a self-referencing canonical.
Technical SEO | | WTH0 -
A crawl revealed two home pages
After doing a site crawl using the moz tool, I have found two home pages-www.domain.com/ and www.domain.com. Both URLS have the exact same metrics and I have set a preferred domain name in google, will this hurt seo? Should I claim the www.domain.com/ as well as www.domain.com and domain.com in the search console? Thanks
Technical SEO | | Tom3_150 -
Site Crawl -> Duplicate Page Content -> Same pages showing up with duplicates that are not
These, for example: | https://im.tapclicks.com/signup.php/?utm_campaign=july15&utm_medium=organic&utm_source=blog | 1 | 2 | 29 | 2 | 200 |
Technical SEO | | writezach
| https://im.tapclicks.com/signup.php?_ga=1.145821812.1573134750.1440742418 | 1 | 1 | 25 | 2 | 200 |
| https://im.tapclicks.com/signup.php?utm_source=tapclicks&utm_medium=blog&utm_campaign=brightpod-article | 1 | 119 | 40 | 4 | 200 |
| https://im.tapclicks.com/signup.php?utm_source=tapclicks&utm_medium=marketplace&utm_campaign=homepage | 1 | 119 | 40 | 4 | 200 |
| https://im.tapclicks.com/signup.php?utm_source=blog&utm_campaign=first-3-must-watch-videos | 1 | 119 | 40 | 4 | 200 |
| https://im.tapclicks.com/signup.php?_ga=1.159789566.2132270851.1418408142 | 1 | 5 | 31 | 2 | 200 |
| https://im.tapclicks.com/signup.php/?utm_source=vocus&utm_medium=PR&utm_campaign=52release | Any suggestions/directions for fixing or should I just disregard this "High Priority" moz issue? Thank you!0 -
I can't crawl the archive of this website with Screaming Frog
Hi I'm trying to crawl this website (http://zeri.info/) with Screaming Frog but because of some technical issue with their site (i can't find what is causing it) i'm able to crawl only the first page of each category (ex. http://zeri.info/sport/) and then it will go to crawl each page of their archive (hundreds of thousands of pages) but it won't crawl the links inside these pages. Thanks a lot!
Technical SEO | | gjergjshala0 -
Contact Page
I'm currently designing a new website for my wife, who just started her own wedding/engagement photography business. I'm trying to build it as SEO friendly as possible, but she brought up an idea that she likes that I've never tried before. Typically on all the websites I've ever built, I've had a dedicated contact page that has the typical contact form. Because that contact form on a wedding photographers website is almost as important as selling a product on an e-commerce site, she brought up the possibility of putting the contact form in the footer site-wide (minus maybe the homepage) rather than having a dedicated contact page. And in the navigation, where you have links such as "Home", "Portfolio", "About", "Prices", "Contact", etc. the "Contact" navigation item would transfer the user to the bottom of the page they are on rather than a new page. Any thoughts on which way would be better for a case like this, and any positives/negatives for doing it each way? One thought I had is that if it's in the footer rather than it's own page, it would lose it's search-ability as it's technically duplicate content on each page. But then again, that's what a footer is. Thanks, Mickey
Technical SEO | | shannmg10 -
My beta site (beta.website.com) has been inadvertently indexed. Its cached pages are taking traffic away from our real website (website.com). Should I just "NO INDEX" the entire beta site and if so, what's the best way to do this? Please advise.
My beta site (beta.website.com) has been inadvertently indexed. Its cached pages are taking traffic away from our real website (website.com). Should I just "NO INDEX" the entire beta site and if so, what's the best way to do this? Are there any other precautions I should be taking? Please advise.
Technical SEO | | BVREID0 -
Find where the not selected pages are from
Hi all Can anyone suggest how I can find where gtoogle is finding approx. 1000 pages not to select? In round numbers I have 110 pages on the site site: searech shows all pages index status shows 110 slected and 1000 not selected. For the life of me I cannot fingure where these pages are coming from. I have set my prefered domain to www., setup 301 's to www. as per below RewriteCond %{HTTP_HOST} ^growingyourownveg.com$
Technical SEO | | spes123
RewriteRule ^(.*)$ "http://www.growingyourownveg.com/$1" [R=301,L] site is www.growingyourownveg.com any suggestions much appreciated Simon0 -
Website Page Structuring and URL re writing - need helpful resources
Hello, I am not technically very sound and I need some good articles that teach me how to think about and go about website pages structuring and url rewriting that is seo friendly. I will be most obliged if some of you great seomoz-ers can pitch in with help. Regards, Talha ZigZag Solutions
Technical SEO | | TopGearMedia0