Will using http ping, lastmod increase our indexation with Google?
-
If Google knows about our sitemaps and they’re being crawled on a daily basis, why should we use the http ping and /or list the index files in our robots.txt?
- Is there a benefit (i.e. improving indexability) to using both ping and listing index files in robots?
- Is there any benefit to listing the index sitemaps in robots if we’re pinging?
- If we provide a decent <lastmod>date is there going to be any difference in indexing rates between ping and the normal crawl that they do today?</lastmod>
- Do we need to all to cover our bases?
thanks
Marika
-
Will using http ping, lastmod increase our indexation with Google?
No. You can submit a perfect sitemap and ping Google with changes every hour, but that will not increase the number of pages which are indexed.
A few good sources discussing sitemaps and indexing:
http://followmattcutts.com/2010/03/23/matt-cutts-on-sitemap-indexing/
http://faq.bloggertipsandtricks.com/2010/08/html-xml-sitemap-what-difference-matt.html
If you have a site with solid navigation, good architecture and links, then there is no need to use a sitemap. Search engines will determine how often your site should be crawled based on your site's authority. They can also determine which pages have been modified by comparing the header dates with their database.
I still use a sitemap, but it's mostly because the process is fully automated. I know of other sites that are well indexed which do not use site maps at all.
With the above understood, I'll try to offer a bit more information directly related to your questions. When you ask about pinging, I presume you are referring to mainly Google and Bing. For those cases, the answers to all four of your questions is NO.
Listing your sitemap location in robots.txt will help other search engines whom you did not ping to locate your sitemap. This can include the SEOmoz crawler, for example.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can you use no-index to counter duplicate content across separate domains?
Hi Moz Community, I have a client who is splitting out a sub brand from a company website to its own domain. They have lots of content around the theme and they want to migrate most of the content out to the new domain, but they also wanted to keep that content on the main site as the main site gets lots of traffic. My question is, as they want search traffic to go to the new site, but want to keep the best content on the original site too, so it can be found in the nav, if they no-index identical content on main site and index content on the new site will they still be penalised for duplicate content? Our advice has been to keep the thematic content on both sites but make them different enough so they are not considered duplicate - we routinely write the same blog post in 50 different ways for them but their Head of Web asked if the no-index is a route, which means they don't need to pay for and wait for brand new content? They are comfortable in losing traffic until the new domain gets traction. In theory, if they are telling Google not to index or rank the main site content, the new site shouldn't be penalised but I'm not confident giving that advice as I've never been asked to do this before. Thoughts?
Technical SEO | | Algorhythm_jT0 -
Google is indexing our old domain
We changed our primary domain from vivitecsolutions.com to vivitec.net. Google is indexing our new domain, but still has our old domain indexed too. The problem is that the old site is timing out because of the https: Thought on how to make the old indexing go away or properly forward the https?
Technical SEO | | AdsposureDev0 -
Accidental No Index
Hi everyone, We control several client sites at my company. The developers accidentally had a no index robot implemented in the site code when we did the HTTPS upgrade without knowing it (yes it's true). Ten days later we noticed traffic was falling. After a couple days we found the no index tags and removed them and resumbitted the sitemaps. The sites started ranking for their own keywords again within a day or two. The organic traffic is still down considerably and other keywords they are not ranking for in the same spot as they were before or at all. If I look in Google Search console, it says we submitted for example 4,000 URLs and only 160 have been indexed. I feel like maybe Google is taking a long time to re-index to remainder of the sites?? Has anyone has this issue?? We're starting to get very concerned so any input would be appreciate. I read an article on here from 2011 about a company that did the same and they were ranking for their keywords within a week. It's been 8 days since our fix.
Technical SEO | | AliMac260 -
How to know how much pages are indexed on Google?
I have a big site, there are a way to know what page are not indexed? I know that you can use site: but with a big site is a mess to check page by page. This is a tool or a system to check a entire site and automatically find non-indexed pages?
Technical SEO | | markovald0 -
Is it needed to use http:// or not?
Hi, When doing link building and getting my URL to other websites, is it necessary that the other websites include http:// before the domain? Example: I want to increase the number of links to my site www.example.com . When I ask other websites to add a link to my site, should I ask them to use http://www.example.com or is www.example.com enough (without http://)? Or it really does not matter? Thanks in advance, Sam
Technical SEO | | Awaraman0 -
Is Google caching date same as crawling/indexing date?
If a site is cached on say 9 oct 2012 doesn't that also mean that Google crawled it on same date ? And indexed it on same date?
Technical SEO | | Personnel_Concept0 -
Google not using <title>for SERP?</title>
Today I noticed that Google is not using my title tag for one of my pages. Search for "covered call search" Look at organic result 6: Search - Covered Calls Covered call screener filters 150000 options instantly to find the best high yield covered calls that meet your custom criteria. Free newsletter.<cite>https://www.borntosell.com/search</cite> - CachedNow, if you click through to that page you see the meta title tag is:Covered Call ScreenerEven the cached version shows the title tag as Covered Call ScreenerI am not logged in, so I don't believe personalization has anything to do with it.Have others seen this before?It is possible that "search - covered calls" was the title tag 9 months ago (before I understood SEO); I honestly don't remember. I cleaned all my titles up at least 6 months ago.Can I force Google to re-index the page? Its content has changed a few times in the last few months, and Google crawls my site frequently according to webmaster tools.
Technical SEO | | scanlin0