Welcome to the Q&A Forum

According to me a sitemap is more important than robots.txt as it help a search engine bot in effectively crawling a website. Robots.txt is generally used to request (allow: or disallow:)a crawler not to crawl and index certain section of your website containing sensitive data. This is totally upto the crawler to respect the request by not crawling and indexing that sensitive part. However, it is a general practice among webmasters world wide to have a robots.txt file for each of their sites.  A common robots.txt with permission to access the entire website should look like this:

User-agent: *
Disallow:

Sitemap: http://www.yoursite.com/sitemap.xml

So if you want some section (folders, directories) of your site not to be crawled by a bot then you can use a robots.txt.

Yes logically its the same like having a robots.txt file granting all the access and not having one completely. Its just a difference between like something having 'by default". Having a robots.txt file doesn't guarantee a rank boost in the SERP. Hope it helps. For more understanding please refer these resources:

http://www.seomoz.org/learn-seo/robotstxt

http://www.robotstxt.org/robotstxt.html

Cheers

Hi, I would try to summarize my query through an example. Lets say site A (www.siteA.com) have two sub domain (subdomain1.siteA.com & subdomain2.siteA.com) and another site B ( www.siteB.com ) have no sub domain. Due to some obvious reason we need re direct the site site A (www.siteA.com) to site B ( www.siteB.com ) and one of the sub domain (subdomain1.siteA.com) to site B (subdomain1.siteB.com). Now the question is that in case of ( subdomain2.siteA.com ) can we keep the sub domain to site A even though site A has been re directed to site B ? Reasons for keeping this can be traffic, earnings etc. Is it possible to keep it like that or provision for further optimization? Plz help.

Experts, Thank you for the response.

We have done page to page redirection using asp code. Please find a screenshot attached of a page from the old site. There is no 404 page displayed in the old site as it simply redirects from the old site to the new site. For example, if a user goes to www.oldsite.com/abc.html it redirects to www.newsite.com/abc.html.

Before setting up 301 redirection, Back links from Web 2.0 and other good article directories had been built for the new domain. We started the 301 redirection in September and completed section by section. Rankings for the new domain came up in google and stayed until we filled up the "Change of Address" form on 20th Nov after redirecting the homepage. Please find the screenshot of the analytics attached.

On Nov 28th, many of our keywords disappeared from the rankings and we saw a big drop in traffic. Rankings are not even in 100 at the moment.

1. Will it be appropriate to wait for a week or two to see if google re-indexes the new website.

2. Shall i roll back the entire process and withdraw my request for change of address. Shall i remove 301 from the old site.

3. Have i set it up correctly

Thanks for all your help. I appreciate.

Some images that might help...

GA-drop2.png old-site-code.png GA-drop2.png old-site-code.png

Experts, Thank you for the response.

We have done page to page redirection using asp code. Please find a screenshot attached of a page from the old site. There is no 404 page displayed in the old site as it simply redirects from the old site to the new site. For example, if a user goes to www.oldsite.com/abc.html it redirects to www.newsite.com/abc.html.

Before setting up 301 redirection, Back links from Web 2.0 and other good article directories had been built for the new domain. We started the 301 redirection in September and completed section by section. Rankings for the new domain came up in google and stayed until we filled up the "Change of Address" form on 20th Nov after redirecting the homepage. Please find the screenshot of the analytics attached.

On Nov 28th, many of our keywords disappeared from the rankings and we saw a big drop in traffic. Rankings are not even in 100 at the moment.

1. Will it be appropriate to wait for a week or two to see if google re-indexes the new website.

2. Shall i roll back the entire process and withdraw my request for change of address. Shall i remove 301 from the old site.

3. Have i set it up correctly

Thanks for all your help. I appreciate.

GA-drop2.png old-site-code.png

Hi Experts

We did a 301 redirect from an old site to a new site to get rid of any bad link juice. We recently found a big drop in rankings and traffic after google last indexed the new web pages.

We did 301 using asp at page level coding. The website had 4000 approx. pages and we did 301 section by section. This is how we did as per one of the blog post in seomoz.

1. Create a sitemap for your old domain.
2. Create content (contact information, description of your company, indication of future plans) and something link worthy for the new domain. (You should start trying to build links early)
3. Setup the new domain and make it live.
4. Register and verify your old domain and new domain with Google Webmaster Tools.
5. Create a custom 404 page for old domain which suggests visiting new domain.
6. Old Domain error checking and fixing
7. In a development environment, test the redirects from the old domain to the new domain. Ideally, this will be a 1:1 redirect. (www.example-old-site.com/category/sexy-mustaches.html to www.example-new-site.com/category/sexy-mustaches.html)
8. 301 redirect your old domain to your new domain.
9. Submit your old sitemap to Google and Bing. The submission pages are within Google Webmaster Tools and Bing Webmaster Center (This step will make the engines crawl your old URLs, see that they are 301 redirects and change their index accordingly.)
10. Fill out the Change of Address form in Google Webmaster Tools.
11. Create a new sitemap and submit it to the engines. (This will tell them about any new URLs that were not present on the old domain)
12. Wait until Google Webmaster Tools updates and fix any errors it is indicated in the Diagnostics section.
13. Monitor search engine results to make sure new domain is being properly indexed.

We also did a press release with prweb to announce the new launch. We followed the steps recommended in one of the

I am not sure what to do next. Can anyone suggest if its normal to see a drop and we should wait for some time or if we did something wrong?

We are loosing business with every single day. Please help !

Thanks eyepaq for the tools. However we are already using Raven, Moz and obviously WMT but its nice to know about the search metrics.

Initially you may experience some effects but subsequently it will filtered as a 'Blog Network'. It is not recommended to a large extent however 1 or 2 blogs from a platform can be fine but then again the process must not look promotional.

Hi All,

What are the best ways to track a seo campaign successfully, other or including Google Analytic?

Hi goodlegaladvice,

Thanks for your comment but as I've mentioned the site is not in any CMS, its is in flat html files. Any further help will be appreciated.

Hi all,

We are currently working on a very large site having approximately 5000+ pages and its going to be 301 redirected to a new domain. For this we need to redirect each and every page on a 1:1 basis as mentioned in the Webmaster Central guide. Now the site is in flt file and not in CMS and is becoming very tough to set manually redirection for these pages. The site is hosted in a Windows server and using IIS web config file. Now any help regarding any automated or easy way to do the 1:1 redirection will be appreciated.

Thanks in advance,

According to me a sitemap is more important than robots.txt as it help a search engine bot in effectively crawling a website. Robots.txt is generally used to request (allow: or disallow:)a crawler not to crawl and index certain section of your website containing sensitive data. This is totally upto the crawler to respect the request by not crawling and indexing that sensitive part. However, it is a general practice among webmasters world wide to have a robots.txt file for each of their sites.  A common robots.txt with permission to access the entire website should look like this:

User-agent: *
Disallow:

Sitemap: http://www.yoursite.com/sitemap.xml

So if you want some section (folders, directories) of your site not to be crawled by a bot then you can use a robots.txt.

Yes logically its the same like having a robots.txt file granting all the access and not having one completely. Its just a difference between like something having 'by default". Having a robots.txt file doesn't guarantee a rank boost in the SERP. Hope it helps. For more understanding please refer these resources:

http://www.seomoz.org/learn-seo/robotstxt

http://www.robotstxt.org/robotstxt.html

Cheers

Initially you may experience some effects but subsequently it will filtered as a 'Blog Network'. It is not recommended to a large extent however 1 or 2 blogs from a platform can be fine but then again the process must not look promotional.