Please help us undertsand the things we need to improve so that google crawler visit us more often to reindex pages from our domain
-
we are currently in the process of a massive project which involves us migrating our domain, we realised that Google crawlwer has not been crawling our pages Quiet often. i have observed some cases where google crawled these pages about 6 months back and then never visited the pages again
and we had to manually submit these pages for reindexing in some geographies. can you please help us undertsand the things we need to improve so that google crawler visit us more often to reindex pages from our domain -
Hi Bhaskaran,
Someone with more knowledge than me may be around soon to give you a more in depth answer, but this should get you going.
Google has a limited amount of resources to crawl each webpage, so to be efficient it schedules pages to be crawled based on a few different things. The one i suspect is affecting your crawl times is how often your content is updated. If google does not believe that your site will be updated frequently it sees no reason to crawl it that often.
If you want google to crawl more often you need to give it a reason to do so, that means ensure your site is easily crawled by bots, good internal linking and make sure your content is fresh and up to date.
On a side note, as you are carrying out a domain migration it might be easier to redirect the pages to the new page on the new domain (301 redirect) and ensure your new domain has the points i raised above.
I hope this helps?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Migrated Domain, 90% Drop in Organic Traffic, HELP!!!!
One week ago we migrated our old domain www.nyc-officespace-leader.com to https://www.metro-manhattan.com/. Our organic search traffic from Google has dropped about 90%. Is this normal? If so, how long should it take to recover? We filed a submitted a domain change request on Webmaster tools one week ago. We are noticing that many of the www.nyc-officespace-leader.com pages are still indexed which seems strange after a week. To complicate things, we filed a disavow file on April 9th for spammy links that pointed the NYC site. We filed the identical disavow of those links to the new Metro domain to ensure low quality links don't point to the new domain. Prior to making the domain change request, we migrated 30-40 non critical pages from NYC to Metro domains. Webmaster Tools indicated that the traffic was normal on the migrated pages. We then migrated remaining pages and filed the domain change request on April 4th. It is after April 4th that traffic and ranking declined. I would like to mention that there was no change in content; identical content was migrated from Metro to NYC This does not seem normal. Research prior to the migration indicated that if proper steps were taken it should proceed with limited disruption in traffic and ranking. Any ideas on how to remedy this situation? Thanks, Alan
Intermediate & Advanced SEO | | Kingalan10 -
Sudden Rankings Drop for Good Keywords.. Did I Do This? Please Help :(
Hello, I noticed a gradual rankings drop for 3 important keywords over the last month, with a pretty big plummet the last two weeks. Overall in the last month+ we dropped from position 9 to 41.I noticed this when I dug further after noticing traffic dropping since February (not a drastic traffic drop). I should note that the keywords took people to my client's homepage. Their branded keywords have no suffered and I looked at a couple others that haven't either. Now, there is a link in the site footer (we have site wide header and footer) that takes you to a static page that contains links for the 2 digital flipbook catalogs the customer has (one for US and one for Canada). My concern is that at the end of January I had a developer implement a noindex/nofollow meta robot & robots.txt disallow specifically on the HTML pages/URL of the Canadian catalog ONLY. It specifically pointed to that flipbook URL. This catalog is nearly identical to the US catalog and I thought I'd be eliminating duplicate content and helping with crawl budget. After looking further into it last week (reading up about internal nofollows not necessarily being detrimental, but not recommended) and noticing the drop in search visibility traffic (starting gradually in March), I had the disallow/nofollow removed. This was last week, and over this last week the traffic took an even bigger drop (not amazingly drastic but enough to be concerned) and I noticed the keywords that we did ok for dropped even more this last week (down to 41). I'm concerned this has to do with the change I made at the end of January and reversed back. I should note that I don't think these catalogs or the static page that links to them brought any traffic. The keywords I am concerned about fell on our homepage (where the link to the static page that contains the links to both catalogs is in the sitewide footer) The catalogs are a couple hundred pages. I honestly don't see how this could do it, unless it has something to do with the footer being sitewide? There have been site upgrades/dev changes over the last couple months too (although I am not sure if that affected other clients who received the same upgrade), so this is hard to pinpoint. Sorry this is so long but I'd appreciate someone offering some insight to help ease my mind a bit!
Intermediate & Advanced SEO | | AliMac260 -
HTTP HTTPS Migration Gone Wrong - Please Help!
We have a large (25,000 Products) ecommerce website, and we did an HTTP=>HTTPS migration on 3/14/17, and our rankings went in the tank, but they are slowly coming back. We initially lost 80% of our organic traffic. We are currently down about 50%. Here are some of the issues. In retrospect, we may have been too aggressive in the move. We didn't post our old sitemaps on the new site until about 5 days into the move. We created a new HTTPS property in search console. Our redirects were 302, not 301 We also had some other redirect issues We changed our URL taxonomy from http://www.oursite.com/category-name.html to https://www.oursite.com/category-name (removed the .html) We changed our filters plugin. Proper canonicals were used, but the filters can generate N! canonical pages. I added some parameters (and posted to Search Console) and noindex for pages with multiple filter choices to cut down on our crawl budget yesterday. Here are some observations: Google is crawling like crazy. Since the move, 120,000+ pages per day. These are clearly the filtered pages, but they do have canonicals. Our old sitemaps got error messages "Roboted Out". When we test URLs in Google's robots.txt tester, they test fine. Very Odd. At this point, in search console
Intermediate & Advanced SEO | | GWMSEO
a. HTTPS Property has 23,000 pages indexed
b. HTTP Property has 7800 pages indexed
c. The crawl of our old category sitemap (852 categories) is still pending, and it was posted and submitted on Friday 3/17 Our average daily organic traffic in search console before the move was +/-5,800 clicks. The most recent Search Console had HTTP: 645 Clicks HTTPS: 2000 clicks. Our rank tracker shows a massive drop over 2 days, bottoming out, and then some recovery over the next 3 days. HTTP site is showing 500,000 backlinks. HTTPS is showing 23,000 backilinks. I am planning on resubmitting the old sitemaps today in an attempt to remap our redirects to 301s. Is this typical? Any ideas?0 -
How Do You Remove Video Thumbnails From Google Search Result Pages?
This is going to be a long question, but, in a nutshell, I am asking if anyone knows how to remove video thumbnails from Google's search result pages? We have had video thumbnails show up next to many of our organic listings in Google's search result pages for several months. To be clear, these are organic listings for our site, not results from performing a video search. When you click on the thumbnail or our listing title, you go to the same page on our site - a list of products or the product page. Although it was initially believed that these thumbnails drew the eye to our listings and that we would receive more traffic, we are actually seeing severe year over year declines in traffic to our category pages with thumbnails vs. category pages without thumbnails (where average rank remained relatively constant). We believe this decline is due to several things: An old date stamp that makes our listing look outdated (despite the fact that we can prove Google has spidered and updated their cache of these pages as recent as 2 days ago). We have no idea where Google is getting this datestamp from. An unrelated thumbnail to the page title, etc. - sometimes a picture of a man's face when the category is for women's handbags A difference in intent - user intends to shop or browse, not watch a video. They skip our listing because it looks like a video even though both the thumbnail and our listing click through to a category page of products. So we want to remove these video thumbnails from Google's search results without removing our pages from the index. Does anyone know how to do this? We believed that this connection between category page and video was happening in our video sitemap. We have removed all reference to video and category pages in the sitemap. After making this change and resubmitting the sitemap in Webmaster Tools, we have not seen any changes in the search results (it's been over 2 weeks). I've been reading and it appears many believe that Google can identify video embedded in pages. That makes sense. We can certainly remove videos from our category pages to truly remove the connection between category page URL and video thumbnail. However, I don't believe this is enough because in some cases you can find video thumbnails next to listings where the page has not had a video thumbnail in months (example: search for "leather handbags" and find www.ebags.com/category/handbags/m/leather - that video does not exist on that page and has not for months. Similarly, do a search for "handbags" and find www.ebags.com/department/handbags. That video has not been on that page since 2010. Any ideas?
Intermediate & Advanced SEO | | SharieBags0 -
Google is displaying my pages path instead of URLS (Pages name)
Does anyone knows why Google is displaying my pages path instead of the URL in the search results, i discoverd that while am searching using a keyword of mine then i copied the link http://www.smarttouch.me/services-saudi/web-services/web-design and found all related results are the same, could anyone one tell me why is that and is it really differs? or the URL display is more important than the Path display for SEO!
Intermediate & Advanced SEO | | ali8810 -
Redirecting One Page of Content on Domain A to Domain B
Let's say I have a nice page of content on Domain A, which is a strong domain. That page has a nice number of links from other websites and ranks on the first page of the SERPs for some good keywords. However, I would like to move that single page of content to Domain B using a 301 redirect. Domain B is a slightly weaker domain, however, it has better assets to monetize the traffic that visits this page of content. I expect that the rankings might slip down a few places but I am hoping that I will at least keep some of the credit for the inbound links from other websites. Has anyone ever done this? Did it work as you expected? Did the content hold its rankings after being moved? Any advice or philosophical opinions on this? Thank you!
Intermediate & Advanced SEO | | EGOL2 -
Multiple domains, which does Google like better?
From a business perspective It seems a better idea to have one site and multiple domains pointing to that one site. Is there any benefit as far as ranking well in Google to having a site set up on each domain with links to the main site? Does you have any success from a business point of view in using this strategy? P.S. This is my 4th question in Beta testing, just want to make sure I'm not taking advantage of your generosity and if it is fine to keep asking questions here.
Intermediate & Advanced SEO | | iSenseWebSolutions0