How to activate Page Fetching and Rendering
-
We have a coupons and deal website. Coupons are added and removed from the website on a daily basis. But crawler isn't crawling it that often. Lately we started fetching and rendering the page, but that is a time taking task as we have more than 500 stores with coupons.
So, I was looking for some API or some method using which the crawler would crawl the website as defined. Suppose "x" store page should be crawled every alternate day as we daily update the coupons there, whereas "y" store coupons are update fortnightly so , they can be crawled weekly.
Can somebody suggest me something..
-
Hi Anukul
As far as I am aware, it is not possible to fetch and render via an API - however you can double-check the documentation here: https://developers.google.com/webmaster-tools/
It sounds like what you really need to work on perhaps is Crawl Optimization. If Google thinks the site is important enough, they will crawl it fast enough to catch all of the changes.
AJ Kohn wrote and excellent post on it here which I highly encourage you check out. Here are a few other resources on getting Google to crawl your site faster:
- https://blog.kissmetrics.com/googlebot-optimization/
- http://moz.com/blog/seo-log-file-analysis-part-2
- http://googlewebmastercentral.blogspot.com/2009/08/optimize-your-crawling-indexing.html
Hope those help!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Use Internal Search pages as Landing Pages?
Hi all Just a general discussion question about Internal Search pages and using them for SEO. I've been looking to "noindexing / follow" them, but a lot of the Search pages are actually driving significant traffic & revenue. I've over 9,000 search pages indexed that I was going to remove, but after reading this article (https://www.oncrawl.com/technical-seo/seo-internal-search-results/) I was wondering if any of you guys have had success using these pages for SEO, like with using auto-generated content. Or any success stories about using the "noindexing / follow"" too. Thanks!
Technical SEO | | Frankie-BTDublin0 -
Duplicate Home Page
Hi everyone! So, I;m using the crawl diagnostics in Moz and it's telling that I've got duplicate content for these two pages: http://www.bridgelanguages.com/
Technical SEO | | Bridge_Education_Group
http://www.bridgelanguages.com/index.php?p=3233&source=3 Would a redirect from the 2nd page to the 1st one be a solution? I'm not even sure where that 2nd link is on the site? Any suggestions or has anyone experienced the same? Thanks! Kelly0 -
Page for Link Building
Hello Guys, My question is about a link building process. We all know that some directories/sites do require a reciprocal link. Does it make any sense to creat a page in website exclusively to reciprocal links? And what we do with this webpage in terms of indexing, do folow, crawling...etc. Any sugestions are more then welcome 🙂 Tks in advance! PP
Technical SEO | | PedroM0 -
Splitting Page Authority with two URLs for the same page.
Hello guys, My website is currently holding two different URLs for the same page and I am under the impression such set up is dividing my Page Authority and Link Juice. We currently have the following page with both URLs below: www.wbresearch.com/soldiertechnologyusa/home.aspx
Technical SEO | | JoaoPdaCosta-WBR
www.wbresearch.com/soldiertechnologyusa/ Analysing the page authority and backlinks I identified that we are splitting the amount of backlinks (links from sites, social media and therefore authority). "/home.aspx"
PA: 67
Linking Root Domains: 52
Total Links: 272 "/"
PA: 64
Linking Root Domains: 29
Total Links: 128 I am under the impression that if the URLs were the same we would maximise our backlinks and therefore page authority. My Question: How can I fix this? Should I have a 301 redirect from the page "/" to the "/home.aspx" therefore passing the authority and link juice of “/” directly to “/homes.aspx”? Trying to gather thoughts and ideas on this, suggestions are much appreciated? Thanks!0 -
If a permanent redirect is supposed to transfer SEO from the old page to the new page, why has my domain authority been impacted?
For example, we redirected our old domain to a new one (leaving no duplicate content on the old domain) and saw a 40% decrease in domain authority. Isn't a permanent redirect supposed to transfer link authority to the place it is redirecting to? Did I do something wrong?
Technical SEO | | BlueLinkERP0 -
Do web pages have to be linked to a menu?
I have a situation where people search for terms like, say 1978 one dollar bill. Even though there never was a 1978 one dollar bill. I want to make a page to capture these searches but since there wasn't such a thing as a one dollar bill I don't want it connected to the rest of my content which is reality based. Does that make sense? Anyway, my question is, can I publish pages that aren't linked to my menu structure but that will be searchable or, am I going to have to figure out a way to make these oddball pages accessible through my menu?
Technical SEO | | Banknotes0 -
Search Result Page, Index or Not?
I believe Google doesn't want to index and show other search result pages in there SERP.
Technical SEO | | DigitalJungle
So instead of adding "noindex, follow" tag i have changed the url in my search result page like this: Original
http://www.mysite.com/kb-search.aspx?=travelguide&type=wiki&s=3 To
http://www.mysite.com/travelguide/attraction-guide.html And the search result page contains the title of the articles, a short descriptions (300 chars.) and a link to the articles. Does it help? Or should i add noindex, follow tag? Helps Please?0 -
What Are The Page Linking Options?
Okay, so I'm working with a site that has pretty good domain authority, but the interior pages are not well linked or optimized. So, it ranks for some terms, but everything goes to the home page. So, I'd like to increase the authority of the interior pages. The client is not wild about spreading targeted juice via something like a footer. They also don't like a "Popular Searches" style link list. The objection is that it's not attractive. They currently use cute euphemisms as the linking text, so like "cool stuff" instead of "sheepskin seat covers." In that made up example, they'd like to rank for the latter, but insist on using the former. What about a slide show with alt text/links? Would that increase the authority of those pages in a term-targeted kinda way? Are there other options? Does it matter how prominent those links are, like footers not as good as something higher up the page? They currently use a pull-down kind of thing that still results in some pages having no authority. Do bots use that equally well? Thanks!
Technical SEO | | 945010