Working around Dev Site
-
I am working around development site. All pages are 'nofollow'. What can I advice on, when it come to SEO and is there any good article or checklist that I can go through.
One thing I wanted to know is how can I check for broken links or meta data or any other SEO analytics if the page is 'nofollow' and I cannot crawl it with screaming frog or any other tools.
-
As for an SEO checklist, I'd suggest the Web Developer's Cheat Sheet. It's a good idea to read the Beginner's Guide to SEO, too.
-
Hi George,
Maybe in addition to that make sure you have to enter HTTP Authentication to make sure that none of the pages on the domain are being crawled. Besides that I would indeed go with the recommendation from Lynn.
-
HI,
For screaming frog there is an option in the config -> spider menu that says 'follow internal nofollow'. Checking that box should allow you to crawl the site.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
(Australia) Changing .net.au to .com.au - web dev is refusing to do a 301 redirect and wants to run two sites?
After years using a .net.au site, my client has purchased the .com.au version of the same domain. I've now set up a new, responsive website using a wordpress template with new content, but used a similar page structure. I've asked their web developer to now do a 301 permanent redirect on each old page from .net.au site to it's new .com.au page, but he has refused, saying it would be bad for long term SEO. Instead, he says they should run both sites (which I thought would cause duplicate content issues). Both domains are hosted with the same company. I thought as long as the 301 redirects were done on a page by page basis, there were no issues? I'm no SEO expert, (which he claims to be), so I just wanted to get another opinion on what best practice would be in this instance.
Intermediate & Advanced SEO | | carolineraad0 -
Mobile version of my sites: What is better?
What is the best approach to make my sites ready for mobile, in terms of SEO ? Is it better to create a subdomain called "m.mydomain.com" and redirect mobile users to that domain with a lite version of my sites? Or is it better to just keep the same domain as for my desktop version "mydomain.com" and use a WordPress theme that fits for all gadgets, for example Twenty Fourteen WordPress Theme, that adapts to each device? I see that most big sites use a "m.mydomain.com" subdomain for the mobile version, however, I don't see any sense in creating a subdomain of the site, when you can just use the WP adapting theme in the main domain. Any insight please? Thanks!
Intermediate & Advanced SEO | | BloggerGuy0 -
URL mapping for site migration
Hi all! I'm currently working on a migration for a large e-commerce site. The old one has around 2.5k urls, the new one 7.5k. I now need to sort out the redirects from one to the other. This is proving pretty tricky, as the URL structure has changed site wide. There doesn't seem to be any consistent rules either so using regex doesn't really work. By and large, the copy appears to be the same though. Does anybody know of a tool I can crawl the sites with that will export the crawled url and related copy into a spreadsheet? That way I can crawl both sites and compare the copy to match them up. Thanks!
Intermediate & Advanced SEO | | Blink-SEO0 -
Noindex a meta refresh site
I have a client's site that is a vanity URL, i.e. www.example.com, that is setup as a meta refresh to the client's flagship site: www22.example.com, however we have been seeing Google include the Vanity URL in the index, in some cases ahead of the flagship site. What we'd like to do is to de-index that vanity URL. We have included a no-index meta tag to the vanity URL, however we noticed within 24 hours, actually less, the flagship site also went away as well. When we removed the noindex, both vanity and flagship sites came back. We noticed in Google Webmaster that the flagship site's robots.txt file was corrupt and was also in need of fixing, and we are in process of fixing that - Question: Is there a way to noindex vanity URL and NOT flagship site? Was it due to meta refresh redirect that the noindex moved out the flagship as well? Was it maybe due to my conducting a google fetch and then submitting the flagship home page that the site reappeared? The robots.txt is still not corrected, so we don't believe that's tied in here. To add to the additional complexity, the client is UNABLE to employ a 301 redirect, which was what I recommended initially. Anyone have any thoughts at all, MUCH appreciated!
Intermediate & Advanced SEO | | ACNINTERACTIVE0 -
Similar Sites on Same Class C
Hi there, I asked a similar question a while ago - please pardon the dupe. I figured being more specific may help. Here's the scenario: I have two customers which sell very similar products. They both host with me so they are both on the same class C of ip addresses. Content on sites is similar due to the nature of the business/industry. There are no links between the two sites - they do not link to one another The HTML is about 50% the same, content near zero other than site structure. They have similar category structures. Question - could being on the same Class C adversely effect rankings of either. One site did particularly well until Panda came around and it's sloooowly coming back. Some advise has been given to the client that the IPs being on the same Class C is killing rankings. I am trying to either validate or refute the claim. All help/feedback appreciated.
Intermediate & Advanced SEO | | ChrisInColorado0 -
Should I link my similar sites together?
Hi I currently have two sites within exactly the same market. I've just purchased a third website from someone. Should I link these sites together? (i.e. in the page header should I cross link them or point two of them to the third?) If I do this will it harm them if they are on the same C-Class IP blocks? Is using private domains and different hosting companies considered dodgey in any way? Basically I'm a big wimp and don't want to do anything potentially that might potentially hurt my rankings;)
Intermediate & Advanced SEO | | Blendfish0 -
How do you prevent the mobile site becoming a duplicate of the full browser site?
We have a larger site with 100k+ pages, we need to create a mobile site which gets indexed in the mobile engines but I am afraid that google bot will consider these pages duplicates of the normal site pages. I know I can block it on the robots.txt but I still need it to be indexed for mobile search engines and I think google has a mobile crawler as well. Feel free to give me any other tips that I should follow while trying to optimize the mobile version. Any help would be appreciated 🙂
Intermediate & Advanced SEO | | pulseseo0 -
How to remove bad link to your site?
Hello, Our website www.footballshirtblog.co.uk recently suffered a major Google penalty, wiping out 6 months of hard work. We went from getting 6000-10000 hits a day to absolutely nothing from Google. We have been baffled by the penalty as we couldn't think of anything we've done wrong. After some analysis of Open Site Explorer, it seems I may have found the answer. There is a ton of bad links pointing to us. A few example domains are: ru.gg/ gogopzh.com/ 0575bbs.com/ This is nothing to do with us and so I can only assume some competitor has done this. As we were only about 4-5 months old, I guess Google has punished us. What do we do now? This is not a situation I have experienced before and would really appreciate your expert advice.
Intermediate & Advanced SEO | | ukss19840