Crawler issues
-
Can anyone please suggest why our site is not being crawled by Google at the moment?
Thanks,
-
No problem, happy to help
-
Thats a good point Adam. Thanks for your assistance.
-
Thanks Doug. That is really helpful.
-
Thanks for your help Chris
-
Yep. We did that. Thanks for the suggestion.
-
whoop there it is!
-
Double check your redirects again. When I tried to check your backlinks, I was showing redirects between tasteinc.com, www.tasteinc.com, and www.tasteinc.com/ Every time I tried to find the profile using one of those links, it would tell me that it redirects to another one of those.
I didn't have a problem going to your site, but the way that software was trying to crawl things, I would double check that again. That's the only issue that I saw. Has the site been up long?
-
Tried to crawl your site with Screaming Frog and noticed that your http response header contains a x-robots-tag which is being set to noindex, nofollow.
You can check your http headers on this site: http://web-sniffer.net/
-
I would agree with Mike you can see how here - https://support.google.com/sites/answer/100283?hl=en-GB
How long have you had the site up for? Remember it can take time for it to get crawled. You can ask google to crawl a page but this can still take time.
Good luck.
-
Hi Craig,
Did you submit your sitemap in Google Webmaster Tools?
Mike
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Canonical and Sitemap issue
Hi all, I was told that I could change my homepage Canonical tag to match that of my XML sitemap, this sitemap is being generated for me automatically and shows the homepage as e.g. https://www.mysite.com/index.html, yet my Canonical tag has been set to https://www.mysite.com. Google currently shows as https://www.mysite.com/ being indexed, but https://www.mysite.com/index.html is not currently displayed in search results. Can someone please tell me if I should change the Canonical to the index.html version, or if I should do nothing, or remove the Canonical tag altogether? Thank you for looking.
Web Design | | scarebearz0 -
Redirects Not Working / Issue with Duplicate Page Titles
Hi all We are being penalised on Webmaster Tools and Crawl Diagnostics for duplicate page titles and I'm not sure how to fix it.We recently switched from HTTP to HTTPS, but when we first switched over, we accidentally set a permanent redirect from HTTPS to HTTP for a week or so(!).We now have a permanent redirect going the other way, HTTP to HTTPS, and we also have canonical tags in place to redirect to HTTPS.Unfortunately, it seems that because of this short time with the permanent redirect the wrong way round, Google is confused as sees our http and https sites as duplicate content.Is there any way to get Google to recognise this new (correct) permanent redirect and completely forget the old (incorrect) one?Any ideas welcome!
Web Design | | HireSpace0 -
Breadcrumbs show for some section and dont come for some. What could be the issue
Hello, We have implemented breadcrumbs - which comes up nicely for Blog and Forum in Google Search However, when we look at our 2 other section - New Car Product and Latest News - it does not comes up, is there any issue which can foresee Sample URL of New Car - http://www.mycarhelpline.com/index.php?option=com_newcar&view=product&Itemid=2&id=8&vid=188 Sample URL of News - http://www.mycarhelpline.com/index.php?option=com_latestnews&view=detail&n_id=474&Itemid=10 Its been over 1 month - since breadcrumbs been introduced in both New Car and News. is there any issue like in them as why is is not showing in search. Any recommendations Many Thanks !!
Web Design | | Modi0 -
Pagenation - Crawl Issue
Hi,
Web Design | | semvibe
We have a site with large number of products (6000 +) under each categories and so we have made a page under each category to list out all products (View all page), which lists out product in pagenation setup built on Ajax. The problem is only our 1st page is crawlable and all the other pages beyond 1st page remains hidden,
We need make all our pagenation URL’s crawlable, our requirements are we never want a change in URL as user goes to next page, want to show the user the same URL for all the pagenation numbers. Is there a perfect solution?0 -
International SEO issues for multiple sites
We currently have 3 websites: oursite.co.uk oursite.fr oursite.ch We also own Oursite.com, and that URL currently redirects to Oursite.fr. We are considering a complete site redesign and a possible merge of the 3 sites. Assumptions: ** the 3 sites currently receive organic search traffic to varying degrees
Web Design | | darkgreenguy
** Oursite.ch is almost identical to Oursite.fr in terms of the site content
** Our target market is NOT the USA for English-language searches. It is the UK. With a re-design, we see our options as follows: Merge the 3 sites and make Oursite.com the "main site" and then have subfolders as follows: /uk /fr /ch Keep the 3 sites as they are. We see Option 1 as the best in terms of saving time when updating the site, and saving money paid to the site developers (1 site vs 3 sites). We see Option 2 as the best in terms of ability of the site to rank, as well as confidence of searchers when seeing our site in the search results (in other words, a person searching in France would be more likely to buy and/or submit a form on our site if they saw Oursite.fr vs Oursite.com/fr). I guess we're looking for some suggestions/guidance here. Are we missing any big issues? Does anyone have experience with an issue such as this? Thank you in advance...
-Shawn0 -
Google search issue with exact domain
We had a site from Feb-2011 to Nov-2011 at the domain amcoexterminating.com. The site was pure HTML/CSS and the daily unique visitors steadily increased over that time. So all was fine. We then moved the site to a CMS (Joomla) on Dec. 6th. From that day forward, the daily visitors went into the tank. Before the move, if you typed "amcoexterminating.com" or "amco exterminating" into Google search, the site would be the first result (as you'd expect since those are the words that make up the actua domain). But we tried this yesterday and the site did not come up at all. NOT GOOD. It would work in Yahoo or Bing, but not in Google. So obviously, the problem with Google search directly affected the daily visitors. We just checked Webmaster tools yesterday (yes, this should have been done sooner, lesson learned) and it said "Site has severe health issues - Important page blocked by robots.txt". It listed the "important" page URL and it was just a link to an image. Regardless, I wiped out the Joomla created robots.txt file and added a new one and made it just say... User-agent: *Allow: / About 14 hours later, after the new robots.txt file was recognized by Google, the "severe health" message went away. However if I search in Google for "amcoexterminating.com", it still doesn't show up and the client is concerned (as they should be). Do you think the search engines just need more time to refresh? If so, once it refreshes, should the site show up first again right away? Or is it possible the robots.txt file had nothing to do with the issue? If so, what other things could I check into that might cause Google search to not find a site even if you search for exact domain name? Please share any and all things I should look into as I need to get this site showing in Google search again (as it was before moving to the CMS). Thanks!
Web Design | | MarathonMS0 -
Are my duplicate meta titles and descriptions an issue ?
HelloMy website http://www.gardenbeet.com has been rebuilt using prestacart and there are 158 duplicate title and meta descriptions being reported by google.My developer advised the following Almost all the duplicates are due to the same page being accessible at the root and following the category heading. e.g; /75-vegetable-patio-planter-turquoise.html
Web Design | | GardenBeet
/patio-planters/75-vegetable-patio-planter-turquoise.html This is hard-wired into PrestaShop. Was the Canonical module (now disabled) responsible for the confusion by not including the category name? The Googlebot shouldn't be scanning the root versions now. I don't believe this to be a serious issue but I'd recommend a second opinion from someone more SEO savvy just to be sure.Opinions??0 -
Facing design issue in IE6 and IE7
Hi, I implemented custom search code on my website which is on a test url http://61.246.241.232/. It is working perfectly fine on all the browsers except on IE6 and IE7. Can anyone please let me know the solution to this problem. Thanks & Regards
Web Design | | IM_Learner0