Why do old URL format are still being crawled by Rogerbot?
-
Hi,
In the early days of my blog, I used permalinks with the following format:
http://www.mysitesamp.com/2009/02/04/heidi-cortez-photo-shoot/
I then decided to change this format using .htaccess to this format:
http://www.mysitesamp.com//heidi-cortez-photo-shoot/
My question is, why do rogerbot still crawls my old URL format since these urls' no longer exists in my website or blog.
-
Thanks Alan,
That solved my problem...
-
-
Hi Alan,
After disallowing the directory in robots.txt, Rogerbot still includes the non-existing URLs. Here is a sample URL that is being reported by Rogerbot
www.lugaluda.com/2009/08/05/chase-online-banking-chase-checking-bonus/
-
If you give me the url, i can crawl it fior you if you like.
-
Thanks Alan, I really appreciate your help. Gave me an idea since all the old URLs are coming from a virtual 2009 directory, I tried to add a disallow statement for that directory in the robots.txt section. Hopefully this will help solve the problem.
I will let you know the results after rogerbot finishes recrawling my site...
Thanks Dude....
-
You need to search your site, but bots start on a page and follow the links, if the report them then they must of found them, bots like googlebot or bingbot can find them on other sites, but rogerbot is only crawling within your site.
-
How will I know if they still exists on my site? If I tried to access the specific URLs, they are no longer active.
-
The old format must still exist in your site somewhere, bots follow links from your home page though your site.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Redirect old image that has backlinks
Hi Moz Community! I'm doing an audit of a website and did a backlink analysis. In the backlink analysis, there is an image that has 66 backlinks but the image doesn't exist on the website anymore (it was on a website that was created in 2011 - 2 web launches ago). I don't believe a 301 redirect will work for an image that doesn't exist anymore. How would I redirect the image URL (it's WordPress so we have a specific URL that other websites are linking to but get 404 errors) without going to each individual website and requesting they change the URL link? Any advice or recommendations would be great. Thanks!
Intermediate & Advanced SEO | | BradChandler1 -
Schema markup or* format ?
Hello, To present this information on my website Country: France Region: Alsace Price: 1000 USD Should I present that in or format ? what is best for google to understand. Thank you
Intermediate & Advanced SEO | | seoanalytics0 -
Reverting back to old domain name.
I've recently been asked by a client if I can foresee any issues with reverting back to their original domain name. With the original domain name they had a pretty decent DA for their sector which they have now lost. Although I do appreciate that over time this might come back, the CEO is very keen to switch back to the old domain. They do currently have 301 redirects from the old domain to the new and have implemented rel canonical. As yet they have not notified Google of the change of address using Webmaster Tools. Can anyone forsee any issues with returning back to the old domain name? They have only been using the new domain name for a couple of months which currently has a DA for 1.
Intermediate & Advanced SEO | | Macrofireball0 -
Replicating keywords in the URL - bad?
Our site URL structure used to be (example site) frogsforsale.com/cute-frogs-for-sale/blue-frogs wherefrogsforsale.com/cute-frogs-for-sale/ was in front of every URL on the site. We changed it by removing the for-sale part of the URL to be frogsforsale.com/cute-frogs/blue-frogs. Would that have hurt our rankings and traffic by removing the for-sale? Or was having for-sale in the URL twice (once in domain, again in URL) hurting our site? The business wants to change the URLs again to put for-sale back in, but in a new spot such as frogsforsale.com/cute-frogs/blue-frogs-for-sale as they are convinced that is the cause of the rankings and traffic drop. However the entire site was redesigned at the same time, the site architecture is very different, so it is very hard to say whether the traffic drop is due to this or not.
Intermediate & Advanced SEO | | CFSSEO0 -
Are these URL hashtags an SEO issue?
Hi guys - I'm looking at a website which uses hashtags to reveal the relevant content So there's page intro text which stays the same... then you can click a button and the text below that changes So this is www.blablabla.com/packages is the main page - and www.blablabla.com/packages#firstpackage reveals first package text on this page - www.blablabla.com/packages#secondpackage reveals second package text on this same page - and so on. What's the best way to deal with this? My understanding is the URLs after # will not be indexed very easily/atall by Google - what is best practice in this situation?
Intermediate & Advanced SEO | | McTaggart0 -
Company Blog at a different URL
Ok, I have been doing a lot of work over the past 6 months, disavowing low quality links from spammy directories to our company website, etc. However, my efforts seem to have had a negative, not positive effect. This has brought me back to reconsidering what we are doing as we have lost a good amount of traction on the nationwide Google rankings specifically. Considering our company blog - platinumcctv(dot)net - we have used this blog for a long time to inform customers of new products, software developments and then to provide them links to purchase those components. Last week, I revamped the nearly default wordpress theme to another on a piece of advice. However, someone told me that all of our links should be nofollow, even though it is a company blog because we have many links coming from this domain, and it could be found as spammy. Potato/Potato - But before I start the tedious task of changing every link to no follow on a whim, i searched a lot, but have found no CLEAR substantiation of this. Any ideas? Other recommendations appreciated as well! Platinum-CCTV(dot)com
Intermediate & Advanced SEO | | PTCCTV0 -
URL for New Product
Hi, We run an established website (mindflash.com) selling online training software. We are getting ready to launch a new section of the site where our users can sell their own online training programs. This will be branded as the 'marketplace'. This section will have a main page, category pages, tag pages, search and individual course pages. In our URL structure, I'd love to target the word 'training courses' but I don't want to neglect the product brand either. Is it better to use /training-courses in the marketplace urls or to use /marketplace? Or is it better to use both like /marketplace-training-courses or /marketplace/training-courses? Option 1: Example main section page: mindflash.com/training-courses Example category page: mindflash.com/training-courses/software-training Option 2: Example main section page: mindflash.com/marketplace Example category page: mindflash.com/marketplace/software-training Option 3: Example main section page: mindflash.com/marketplace-training-courses Example category page: mindflash.com/marketplace-training-courses/software-training Option 4: Example main section page: mindflash.com/marketplace/training-courses Example category page: mindflash.com/marketplace/training-courses/software-training Which option is better and why?
Intermediate & Advanced SEO | | mindflash0 -
Best way to find all url parameters?
In reference to http://googlewebmastercentral.blogspot.com/2011/07/improved-handling-of-urls-with.html, what is the best way to find all of the parameters that need to be addressed? Thanks!
Intermediate & Advanced SEO | | nicole.healthline0