Why is my site not getting crawled by google?
-
Hi Moz Community,
I have an escort directory website that is built out of ajax. We basically followed all the recommendations like implementing the escaped fragment code so Google would be able to see the content. Problem is whenever I submit my sitemap on Google webmastertool it always 700 had been submitted and only 12 static pages had been indexed. I did the site query and only a number of pages where indexed.
Does it have anything to do with my site being on HTTPS and not on HTTP?
My site is under HTTPS and all my content is ajax based.
Thanks
-
Sean, What is the URL for your site?
-
Hi There
As Alan mentioned HTML is going to be a much more guaranteed way to get indexed. The HTTPS alone shouldn't be affecting anything. But do you have a different robots.txt for https and http? Is the https one blocking crawlers? Do you have the https version of the site registered in webmaster tools? When you go to crawl stats, how many pages does it show that they are crawling?
-Dan
-
Why build it with Ajax, ajax is good for functionally that needs to load seamlessly but not good for content.
Using escaped fragment seems to not work well as many are having problems getting indexed.Does your content have to be loaded via ajax? why not load it on the page? would be much simpler
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moving a lot of pdfs to main site. Worth trying to get them indexed?
On my main site we link to pdfs that are located on another one of our domains. The only thing that is on this other domain is the pdfs. It was setup really poorly so I am going to redesign everything and probably move it. Is it worthwhile trying to add these pdfs to our sitemap and to try and get them indexed? They are all connected to a current item, but the content is original.
Intermediate & Advanced SEO | | EcommerceSite0 -
Would you rate-control Googlebot? How much crawling is too much crawling?
One of our sites is very large - over 500M pages. Google has indexed 1/8th of the site - and they tend to crawl between 800k and 1M pages per day. A few times a year, Google will significantly increase their crawl rate - overnight hitting 2M pages per day or more. This creates big problems for us, because at 1M pages per day Google is consuming 70% of our API capacity, and the API overall is at 90% capacity. At 2M pages per day, 20% of our page requests are 500 errors. I've lobbied for an investment / overhaul of the API configuration to allow for more Google bandwidth without compromising user experience. My tech team counters that it's a wasted investment - as Google will crawl to our capacity whatever that capacity is. Questions to Enterprise SEOs: *Is there any validity to the tech team's claim? I thought Google's crawl rate was based on a combination of PageRank and the frequency of page updates. This indicates there is some upper limit - which we perhaps haven't reached - but which would stabilize once reached. *We've asked Google to rate-limit our crawl rate in the past. Is that harmful? I've always looked at a robust crawl rate as a good problem to have. Is 1.5M Googlebot API calls a day desirable, or something any reasonable Enterprise SEO would seek to throttle back? *What about setting a longer refresh rate in the sitemaps? Would that reduce the daily crawl demand? We could set increase it to a month, but at 500M pages Google could still have a ball at the 2M pages/day rate. Thanks
Intermediate & Advanced SEO | | lzhao0 -
Why are these sites outranking me?
I am trying to rank for the phrase "a link between worlds walkthrough" I am on page 1 but there are several results that just outranks me and I cannot see any reason that they would be doing so. My site is hiddentriforce.com/a-link-between-worlds/walkthrough/ For that page I have 5 linking domains, varied anchor text that spans from things like "here" to a variety of related phrases. All of the links come from really good sites My page has 1400 likes, 90 shares, and about 20 each in tweets and +'s DA of 44 PA of 37 The 4 and 5 ranked sites both have WAY less social interactions, lower PA and DA, less links, etc Yet they outrank me why?
Intermediate & Advanced SEO | | Atomicx0 -
Site rankings dropped from ~15 to 500+ but Google says we were not penalized
2 months ago my site was ranking about 15 for my main KWs in the UK (my main market). On July 28 we dropped suddenly to ~500 or not in the top 500 at all in the UK for the 3 main keywords. However, these same terms are still ranking in other markets and we are doing okay in the UK for other terms targeted to the same pages. I cleaned up my links/content and sent a reconsideration request to Google. I then looked closer at my links and realized Google may be counting my affiliate links as if they were mine, so thought maybe that was the problem. I sent Google another reconsideration request and they wrote back the letter pasted below Any ideas about what happened or how I can get my rankings back? This definitely doesn't seem like it was just a simple algorithm change. There were no major changes done to our site, and we are still ranking in other markets. Dear site owner or webmaster
Intermediate & Advanced SEO | | theLotter
We received a request from a site owner to reconsider your site for compliance with Google's Webmaster Guidelines.
We reviewed your site and found no manual actions by the webspam team that might affect your site's ranking in Google. There's no need to file a reconsideration request for your site, because any ranking issues you may be experiencing are not related to a manual action taken by the webspam team.
Of course, there may be other issues with your site that affect your site's ranking. Google's computers determine the order of our search results using a series of formulas known as algorithms. We make hundreds of changes to our search algorithms each year, and we employ more than 200 different signals when ranking pages. As our algorithms change and as the web (including your site) changes, some fluctuation in ranking can happen as we make updates to present the best results to our users.
If you've experienced a change in ranking which you suspect may be more than a simple algorithm change, there are other things you may want to investigate as possible causes, such as a major change to your site's content, content management system, or server architecture. For example, a site may not rank well if your server stops serving pages to Googlebot, or if you've changed the URLs for a large portion of your site's pages. This article has a list of other potential reasons your site may not be doing well in search.
If you're still unable to resolve your issue, please see our Webmaster Help Forum for support.
Sincerely,
Google Search Quality Team0 -
Is there any delay between crawling a page by google and displaying of the ratings in rich snippet of the results in google?
Is there any delay between crawling a page by google and displaying of the ratings in rich snippet of the results in google?
Intermediate & Advanced SEO | | NEWCRAFT0 -
What is the best tool to crawl a site with millions of pages?
I want to crawl a site that has so many pages that Xenu and Screaming Frog keep crashing at some point after 200,000 pages. What tools will allow me to crawl a site with millions of pages without crashing?
Intermediate & Advanced SEO | | iCrossing_UK0 -
1 of the sites i work on keeps having its home page "de-indexed" by google every few months, I then apply for a review and they put it back up. But i have no idea why this keeps happening and its only the home page
1 of the sites i work on (www.eva-alexander.com) keeps having its home page "de-indexed" by google every few months, I then apply for a review and they put it back up. But i have no idea why this keeps happening and its only the home page I have no idea why and have never experienced this before
Intermediate & Advanced SEO | | GMD10 -
How to get a news article / post to show up in a google trend for your keyword?
Does anyone know how google selects the news articles it displays in google trends? EX: http://www.google.com/trends?q=glitch+hop%2C+dubstep&ctab=0&geo=all&date=all&sort=0 See how dubstep has a couple posts that show up when searched in google trends? these are different than regular SERPS as far as i can tell. Does anyone know how google selects them?
Intermediate & Advanced SEO | | adriandg0