Website Crawl problems
-
I have a feeling that Google doesn't crawl my website.
E.g. this blogpost - I copy a sentence from it and paste it to Google. The page that shows up in search results is www.silvamethodlife.com/page/9/ - which is just a blog page with all the articles listed, not the link to the article itself!
Did anyone ever have this problem? It's definitely some technical issue.
Any advice will be deeply appreciated
Thanks
-
When i use the same tactic as you defined in your question’s explanation i was able to see your page in the search results...
Use this common in search bar to see how many pages are indexed in Google ‘site:example.com’.
May be not all the pages are indexed, for that try and check the following two things:
- Make sure all pages of the website are follow,index for Google and other search engines
- Update your sitemap and submit it to Google webmaster tool and you will see the efficiency in website page index.
Hope this helps!
-
I noticed that your category pages specify noindex,follow and index,follow. Might want to make a decision there.
Google has indexed 676 pages of the 817 on your site and, since their are quite a few URLs that specify nofollow correctly without the additional index like mentioned above, I'd say that is about right.
Did not see the issue you spoke of when I checked some random sentences.
Wouldn't be concerned unless you notice this is coupled with a decrease in organic traffic. If so start looking at off page factors.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Please have a look at my website. I am stuck here.
Here might be the reason. I had loads of unnecessary content so I given them the noindex tag. I tried to change the robot.txt file but that shouldn't be a problem in SEO. First my site had a country specific domain and then a year later I changed it to .Com, as to target globally (Mainly US). My site is ranking well in that specific country (never been close to page 1) on page 3 almost every time. It's not ranking in other countries, despite the fact that I've not targeted it to any specific country since the domain was changed. A month ago, I deleted 404 pages and all the thin content which was indexed in the SERP and also deleted the duplicated contents and as well as the copied contents. Meanwhile I've also tried changing the headings in some of the products articles as they were causing the duplicate heading issue. I've recently switched my hosting from the UK based server to the Us based server because the last hosting has bad downtime. So far until now nothing seems to be working in my favor. I'm just tired of resolving issues and in return finding a zero result. This is my devil site: 10stuffs.com plz check it out and tell me why my site is not ranking at all and what sould I do.
Intermediate & Advanced SEO | | anshu14320 -
How can I stop my facets being crawled?
Hi If my facets are being crawled, how can I stop this? Or set them up so they are SEO friendly - this is new to me as I haven't had to deal with lots of facets in the past. Here's an example of a page on the site - https://www.key.co.uk/en/key/lift-tables Here's an example of a facet URL - https://www.key.co.uk/en/key/lift-tables#facet:-1002779711011711697110,-700000000000001001651484832107103,-700000000000001057452564832109109&productBeginIndex:0&orderBy:5&pageView:list& I've been trying to read up on URL parameters etc, I'm new to it so it's taking some time to understand Any advice would be great!
Intermediate & Advanced SEO | | BeckyKey0 -
Index Problem
Hi guys I have a critical problem with google crawler. Its my website : https://1stquest.com I can't create sitemap with online site map creator tools such as XML-simemap.org Fetch as google tools usually mark as partial MOZ crawler test found both HTTP and HTTPS version on site! and google cant index several pages on site. Is problem regards to "unsafe URL"? or something else?
Intermediate & Advanced SEO | | Okesta0 -
HTTPS website migration and internal links
Hey Moz! I read Moz's guide on migrating websites from http to https, and it seems changing all relative internal links to absolute https is recommended (we currently use relative internal links). But is doing this absolutely necessary if we will already have a redirect in our .htaccess file forcing all http pages to https? Changing all of our internal links to absolute https will be very time consuming, and I'd like to hear your thoughts as to whether it's absolutely recommended/necessary; and if so, why? Thanks!
Intermediate & Advanced SEO | | TheDude0 -
Website completely delisted - reasons?
Hi, I got a request from a potential client as he do not understand why his website cannot be found on Google. I've checked that and found out that the complete website is not listed (complete delist) at all - expect just one pdf file.
Intermediate & Advanced SEO | | TheHecksler
I've checked his robots.txt - but this is ok. I've checked the META Robots - but they are on index,follow ... ok so far. I've checked his backlinks but could not found any massive linking from bad pages - just 6 backlinks and only four of them from designdomains.com which looks like a linklist or so. I've requested access to their GWT account if available in hope to find more infos, but does anyone of you may have a quick idea what els it could be? What could be the issue? I think that they got delisted due to any bad reason ... Let me know your Ideas 🙂 THANX 🙂 Sebi0 -
Rel=canonical an iframed version of the same website?
My issue is that we have two websites with the same content. For the sake of an example lets say they are: jackson.com jacksonboats.com When you go to jacksonboats.com, the website is an iframed version of jackson.com. However all of the companies email addresses are example@jacksonboats.com so a 301 is not possible. What would be the best way to forward over the link juice from jacksonboats.com to jackson.com? I'm thinking a rel=canonical tag, but I wanted to ask first. Thanks,
Intermediate & Advanced SEO | | BenGMKT0 -
Problem w/ 301 Redirect
Here is how I did the configuration of the redirects: I don’t understand why the destination page is different from the one is configured in the apache server. Any ideas? For example: http://www.meliacaribetropical.com/spanish/entertainment/ is being already being 301 redirected to a 404 page (http://www.meliacaribetropical.com/es/index.htmlentertainment/) that does not exist on the Apache server. As you can see, the url was incorrectly written. Another occurrence from the spreadsheet is http://www.meliacaribetropical.com/spanish/gallery/beach.html, which is also being 301 redirected to a 404 page (http://www.meliacaribetropical.com/es/index.htmlgallery/beach.html). This is causing a hard 404 page. Here is my .httpaccess file: <virtualhost 192.168.200.25:80=""></virtualhost> ServerAdmin ecommerce@sol-group.com DocumentRoot /home/www/solgroup/americas/meliacaribetropical.com ServerName www.meliacaribetropical.com ServerAlias meliacaribetropical.com Redirect permanent /spanish/services/ http://www.meliacaribetropical.com/en/services.html Redirect permanent /entertainment/ http://www.meliacaribetropical.com/en/services.html Redirect permanent /press/ http://www.meliacaribetropical.com/en/index.html Redirect permanent /spanish/ http://www.meliacaribetropical.com/es/index.html Redirect permanent /es/restaurantes/ http://www.meliacaribetropical.com/es/gastronomia.html Redirect permanent /spanish/entertainment/ http://www.meliacaribetropical.com/es/servicios.html Redirect permanent /spanish/services/ http://www.meliacaribetropical.com/es/servicios.html Redirect permanent /es/spa/ http://www.meliacaribetropical.com/es/servicios.html Redirect permanent /spanish/accommodations/ http://www.meliacaribetropical.com/es/habitaciones.html Redirect permanent /spanish/spa/ http://www.meliacaribetropical.com/es/servicios.html Redirect permanent /spanish/royal/ http://www.meliacaribetropical.com/es/servicio-real.html Redirect permanent /spanish/dining/ http://www.meliacaribetropical.com/es/gastronomia.html Redirect permanent /spanish/flintstones/ http://www.meliacaribetropical.com/es/index.html Redirect permanent /es/galeria/ http://www.meliacaribetropical.com/es/visor.html Redirect permanent /spanish/gallery/ http://www.meliacaribetropical.com/es/visor.html Redirect permanent /es/reuniones-eventos/ http://www.meliacaribetropical.com/es/grupos.html Redirect permanent /lowest-rate.php http://www.meliacaribetropical.com/en/index.html Redirect permanent /es/los-picapiedra/ http://www.meliacaribetropical.com/es/index.html Redirect permanent /gallery/beach.html http://www.meliacaribetropical.com/en/index.html Redirect permanent /gallery/dining.html http://www.meliacaribetropical.com/en/gastronomy.html Redirect permanent /gallery/pools.html http://www.meliacaribetropical.com/en/index.html Redirect permanent /spanish/sitemap.html http://www.meliacaribetropical.com/es/index.html Redirect permanent /es/galeria/playa.html http://www.meliacaribetropical.com/es/index.html Redirect permanent /es/galeria/restaurantes.html http://www.meliacaribetropical.com/es/gastronomia.html Redirect permanent /es/galeria/piscinas.html http://www.meliacaribetropical.com/es/index.html Redirect permanent /es/prensa/ http://www.meliacaribetropical.com/es/index.html Redirect permanent /spanish/gallery/beach.html http://www.meliacaribetropical.com/es/index.html Redirect permanent /spanish/gallery/pools.html http://www.meliacaribetropical.com/es/index.html Redirect permanent /spanish/gallery/dining.html http://www.meliacaribetropical.com/es/gastronomia.html Redirect permanent /spanish/press/ http://www.meliacaribetropical.com/es/index.html Redirect permanent /en/groups.html http://www.meliacaribetropical.com/en/groups.html Redirect permanent /terms-condition.php http://www.meliacaribetropical.com/en/index.html Redirect permanent /es/all_inclusive.html http://www.meliacaribetropical.com/es/index.html Redirect permanent /es/terms-condition.php http://www.meliacaribetropical.com/es/index.html Redirect permanent /es/prensa/family-facilities-amenities.html http://www.meliacaribetropical.com/es/index.html Redirect permanent /es/outside-us-telephone-listing.php http://www.meliacaribetropical.com/en/index.html Redirect permanent /spanish/press/melia-international-brand-overhaul.html http://www.meliacaribetropical.com/es/index.html Redirect permanent /es/prensa/melia-international-brand-overhaul.html http://www.meliacaribetropical.com/es/index.html Redirect permanent /press/family-facilities-amenities.html http://www.meliacaribetropical.com/en/index.html Redirect permanent /press/melia-international-brand-overhaul.html http://www.meliacaribetropical.com/en/index.html Redirect permanent /es/prensa/melia-caribe-tropical-announces-fall-promotion.html http://www.meliacaribetropical.com/es/index.html Redirect permanent /spanish/press/melia-caribe-tropical-announces-fall-promotion.html http://www.meliacaribetropical.com/es/index.html Redirect permanent /press/melia-caribe-tropical-announces-fall-promotion.html http://www.meliacaribetropical.com/en/index.html
Intermediate & Advanced SEO | | Melia0 -
How to prevent Google from crawling our product filter?
Hi All, We have a crawler problem on one of our sites www.sneakerskoopjeonline.nl. On this site, visitors can specify criteria to filter available products. These filters are passed as http/get arguments. The number of possible filter urls is virtually limitless. In order to prevent duplicate content, or an insane amount of pages in the search indices, our software automatically adds noindex, nofollow and noarchive directives to these filter result pages. However, we’re unable to explain to crawlers (Google in particular) to ignore these urls. We’ve already changed the on page filter html to javascript, hoping this would cause the crawler to ignore it. However, it seems that Googlebot executes the javascript and crawls the generated urls anyway. What can we do to prevent Google from crawling all the filter options? Thanks in advance for the help. Kind regards, Gerwin
Intermediate & Advanced SEO | | footsteps0