Google Crawling Issues! How Can I Get Google to Crawl My Website Regularly?
-
Hi Everyone!
My website is not being crawled regularly by Google - there are weeks when it's regular but for the past month or so it does not get crawled for seven to eight days. There are some specific pages, that I want to get ranked but they of late are not being crawled AT ALL unless I use the 'Fetch As Google' tool! That's not normal, right? I have checked and re-checked the on-page metrics for these pages (and the website as a whole, backlinking is a regular and ongoing process as well! Sitemap is in place too! Resubmitted it once too! This issue is detrimental to website traffic and rankings! Would really appreciate insights from you guys! Thanks a lot!
-
Hi Farhan
Yes sure, we've definitely found this helps bring the crawlers back to the site more regular!
Hope it helps
Dan
-
Hi Dan! Thanks a lot for your views! This might be it! Content on the website is not updated very regularly owing to the nature of the site. Perhaps updating the blog more regularly or making tweaks in the meta script could help as per your suggestions?
-
Hi Anders! They are all available to the Googlebot! I’m afraid, it won’t be possible for me to share the URL on a public forum! Please check your inbox! Thank you so much for your insights and concern!
-
Hi Umar! There’s been no change in web hosting nor a penalty! We regularly monitor the Search Console and the speed is good too! Thanks a lot for your time though!
-
Hi
How regular are you updating content on the site?
We've found that regular updating and tweaks to content on pages to improve the pages content & user's experience helps the page indexing & frequency.
Looking at analytics and seeing what customers are searching for on your site can help you then improve the content on pages, by going back amending and improving page content we've found Google returns more often to re-index.
Dan
-
Hi!
Are all pages available for GoogleBot, or are they perhaps blocked by robots.txt or a meta tag?
Do you have a URL for the website so we can take a look?
Anders
-
Hello Farhan,
Yes, it's not normal. Did you change your web hosting in this period? or have you faced any kind of penalty in recent past? Usually this happens when you just recovered.
You also need to look out your Search Console for crawling problems, are there any crawling or server issues you're facing? Sometimes, your .htaccess creates crawling problems, make sure to check again. Also, see how fasts your pages are loading, if the speed is low it also cause the crawling delayed.
Hope this helps!
Umar
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can a page that's 301 redirected get indexed / show in search results?
Hey folks, have searched around and haven't been able to find an answer to this question. I've got a client who has very different search results when including his middle initial. His bio page on his company's website has the slug /people/john-smith; I'm wondering if we set up a duplicate bio page with his middle initial (e.g. /people/john-b-smith) and then 301 redirect it to the existent bio page, whether the latter page would get indexed by google and show in search results for queries that use the middle initial (e.g. "john b smith"). I've already got the metadata based on the middle initial version but I know the slug is a ranking signal and since it's a direct match to one of his higher volume branded queries I thought it might help to get his bio page ranking more highly. Would that work or does the 301'd page effectively cease to exist in Google's eyes?
Technical SEO | | Greentarget0 -
Google Appending Blog URL inbetween my homepage and product page is it issue with base url?
Hi All, Google Appending Blog URL inbetween my homepage and product page. Is it issue or base url or relative url? Can you pls guide me? Looking to both tiny url you will get my point what i am saying. Please help Thanks!
Technical SEO | | amu1230 -
What server issues might cause temporary and repeated Soft 404/500 Errors that appear to be functioning correctly when checked later from Google Webmaster Tools?
We are experiencing unknown server issues (we think) which are causing Soft 404/500 errors at unpredictable times on 2 websites. When we check on the pages, they’re fine but still show errors in Moz/Search Console. What are some measures we can take to protect from this or figure out what is causing this? Example URL for Soft 404 Error: https://www.advancedtraveltherapy.com/jobs/any/occupational-therapist/any/ Example URL for 500 Error: https://www.advancedtraveltherapy.com/job-detail/ms/physical-therapist/87529740/ Example URL for Soft 404 Error: https://www.advancedtravelnursing.com/search/searchresults.php?jobState=CA&tempType=g&specialties= Example URL for 500 Error: https://www.advancedtravelnursing.com/job/ma/registered-nurse/emergency-room/87108662/
Technical SEO | | StaffingRobot0 -
How to get google to forget my old but still working page and list my new fully optimized page for a keyword?
Hi There! (i am beginner in seo) I have dynamic and static pages on our site. I created a static page for a specific keyword. Fully optimized it, (h1, alt, metas, etc.....maybe too optimized). My problem is that this page is alive for weeks, checked it in GWT and it is in robots.txt, google sees it, and indexed it. BUT whenewer i do a search for that keyword, we still appear with the dynamically created link in the google listings. How could i "redirect" google, if sy make a search for that keyword than shows our optimized page? Is there a tool for that? I cant delete the dynamic page... Any ideas? Thx Andrew
Technical SEO | | Neckermann0 -
What I doing wrong when trying to search for links from external websites to my website
This is just the little frustrating question nothing important but I’m sure somebody will know the answer. In the white board Friday this week Rand suggested at one point that when you’re searching for results links to your website if you put a - followed by site followed by your url like –site:yourwebsite.com you get the results of pages with links on other websites but excluding your own webpages but it just doesn’t work I get no results just an error message, any idea why? If I remove the - I get tons of results but there on my own webpages……….
Technical SEO | | whitbycottages0 -
How can i get google adsense to work properly to earn income
Hi i am trying to get google adsense to work properly but i am not winning. What i am trying to do is, to get the adverts to reflect on the content. So for example this page here http://www.in2town.co.uk/news/mark-feehily/westlife-mark-feehily-announces-split-from-long-term-boyfriend I would like google adsense to have celebrity adverts such as celebrity news sites, celebrity fashion, concert tickets etc. I want the adverts to be related to celebrity but it is not happening. Can anyone please let me know how to do this and also if i have the google adsense in the right place as since rebuilding the site we have not earned anything with google adsense many thanks
Technical SEO | | ClaireH-1848860 -
How can i increase my website traffic
Hello, my boss has decide a build website we have more than 12500 products in ourwebsite its mtscellular.com, im new as seo but im confused and need help i want to know how somebody help me to increase my website traffic
Technical SEO | | jimmylora0 -
If two links from one page link to another, how can I get the second link's anchor text to count?
I am working on an e-commerce site and on the category pages each of the product listings link to the product page twice. The first is an image link and then the second is the product name. I want to get the anchor text of the second link to count. If I no-follow the image link will that help at all? If not is there a way to do this?
Technical SEO | | JordanJudson0