WIthout a specific example, there are a couple of options here. I am going to assume that you have an ecommerce site where parameters are being used for sort functions on search results or different options on a given product.

I know you may not be able to do this, but using parameters in this case is just a bad idea to start with. If you can (and I know this can be difficult) find a way to rework this so that your site functions without the use of parameters.

You could use canonicals, but then Google would still be crawling all those pages and then go through the process of using the canonical link to find out what page is canonical. That is a big waste of Google's time. Why waste Googlebots time on crawling a bunch of pages that you do not want to have crawled anyway? I would rather Googlebot focus on crawling your most important pages.

You can use the robots.txt file to stop Google from crawling sections of your site.  The only issue with this is that if some of your pages with a bunch of parameters in them are ranking, once you tell Google to stop crawling it, you would then lose traffic.

It is not that Google does not "like" robot.txt to block them, or that they do not "like" the use of the canonical tag, it is just that there are directives that Google will follow in a certain way and so if not implemented correctly or in the wrong sequence can cause negative results because you have basically told Google to do something without fully understanding what will happen.

Here is what I would do.  Long version for long term success

1. Look at Google Analytics (or other Analytics) and Moz tools and see what pages are ranking and sending you traffic. Make note of your results.

2. Think of the most simple way that you could organize your site that would be logical to your users and would allow Google to crawl every page you deem important. Creating a hierarchical sitemap is a good way to do this. How does this relate to what you found in #1.

3. Rework your URL structure to reflect what you found in #2 without using parameters. If you have to use parameters, then make sure Google can crawl your basic sitemap without using any of the parameters. Use robots.txt to then block the crawling of any parameters on your site. You have now ensured that Google can crawl and will rank pages without parameters and you are not hiding any important pages or page information on a page that uses parameters.

There are other reasons not to use parameters (e.g. easier for users remember, tend to be shorter, etc), so think about if you want to get rid of them.

4. 301 redirect all your main traffic pages from the old URL structure to the new URL structure.  Show 404s for all the old pages including the ones with parameters.  That way all the good pages will move to the new URL structure and the bad ones will go away.

Now, if you are stuck using parameters. I would do a variant of the above.  Still see if there are any important or well ranked pages that use parameters. Consider if there is a way to use the canonical on those pages to get Google to the right page to know what should rank. All the other pages I would use the noindex directive to get them out of the Google index, then later use robots to block Google crawling them.  You want to do this in sequence as if you block Google first, it will never see the noindex directive.

Now, everything I said above is generally "correct" but depending on your situation, things may need to be tweaked. I hope the information I gave might help with you being able to work out the best options for what works for your site and your customers.

Good luck!

We have actually been doing this on one of our sites where we have several thousand articles going all the way back to the late 90s.  Here is what we do / our process (I am not including how to select articles here, just what to do once they are selected).

1. Really take the time to update the article. Ask the questions, "How can we improve it? Can we give better information? Better graphics?  Better references? Can we improve conversion?"  2) Republish with a new date on the page.  Sometimes add an editor's note on how this is an updated version of the older article.  3) Keep the same URL to preserve link equity etc or 301 to new url if needed 4) mix these in with new articles as a part of our publication schedule.

We have done this for years and have not run into issues. I do not think Google sees this as spammy as long as you are really taking the time to improve your articles. John M. and Gary I. have stated unequivocally that Google likes it when you improve your content. We have done the above, it has not been dangerous at all. Our content is better overall. In some cases where we really focused on conversion, we not only got more traffic, but converted better. Doing this will only benefit your visitors, which usually translates into Google liking the result.

I would ask, why take a few months where you only recycle content, to just mixing it up all year long?  If you were going to designate 3 months of the year to just update content, then why not take the 3rd week of the month each month or every Wednesday and do the same thing instead.  You accomplish the same thing, but spread it out. Make it a feature! Flashback Friday etc.

Bonus idea - make sure you get the schema right

We have something new with our process. Previously, we only marked up the publication date in schema. So when we republished, we would change the publication date in the schema as well to the new pub date.  Now that Google requires a pub date and last modified date in schema we have changed our process. When we republish content, we will leave the original publication date as the publication date marked up in schema and then put the new date that the article is being published marked up as last modified in schema. This is a much more clearer and accurate representation to Google as what you are doing with the article.

We are also displaying the last modified date to the user as the primary date, with the publication date made secondary. The intent here is that we want to show that this is an article that has been recently updated to the user so they know the information is current.

To get this to work properly, we had to rework how our CMS interacts with content on both published date and last modified date, but in the end, I think we are giving better signals to Google and users on the statuses of our articles.

I would just follow and say that using Google or any other company that large is not the best example for most businesses.

Google does not need to worry about SEO. It has one of the most authoritative websites on the internet. They have baked in traffic due to android, chrome and map users. Yes the sub domain works for them from a product and technical standpoint. They have different groups using different technologies different servers/resources so they need the subdomains to make that all work.

As I read the question, this person is not in that situation. This brand needs link equity to be shared. Links are still super important, I don't see why they would want to dilute that by using sub domains. Everything they want to accomplish can be done w a sub folder.

Nah - I say subdirectory

You get all the control, plus all the link equity is shared across all site. Rising tides raises all ships. I have done this with independent separate brands under a single brand even. It works.

Yes you can rank for subdomains. But the link equity is not shared. Unless you want all that extra work.

More reading

https://moz.com/community/q/the-great-subdomain-vs-subfolder-debate-what-is-the-best-answer

https://moz.com/blog/subdomains-vs-subfolders-rel-canonical-vs-301-how-to-structure-links-optimally-for-seo-whiteboard-friday

Here is what I would do

1. Disavow the domain that is linking to you from the adult site(s).

2. The fact that Google search console is showing that you have an internal page linking as well makes me want to know a) have you always owned this domain and maybe someone previously did link internally like this or b) you may have been or are hacked

In the case of b) this can be really tricky.  I once had a site that in a crawl it was showing sitewide links to various external sites that we should not be linking to. When I looked at the internal pages via my browser, there was no link as far as I could see even though it showed up on the crawler report.

Here was the trick. The hacker had setup a script to only show the link when a bot was viewing the page. Plus, we were running mirrored servers and they had only hacked one server. So, the links only showed up when you were spidering a specific mirrored instance as a bot.

So thanks to the hacking, not only were we showing bad links to bad sites, we were doing this through cloaking methodology. Two strikes against us. Luckily we picked this up pretty quick and fixed immediately.

Use a spidering program or browser program to show a user agent of Googlebot and go visit your pages that are linking internally. You might be surprised.

Summary

Googlebot has a very long memory. It may be that this was an old issue that was fixed long ago. If that was the case, just show the 404s for the pages that do not exist, and disavow the bad domain and move on. Make sure that you have not been hacked as this would also be why this is showing.

Regardless, the fact that Google did find it at one point, you need to make sure you resolve. Pull all the URLs into a spreadsheet and run Screaming Frog in list mode to check them all to make sure you fix all of it.

I think I understand what you are trying to do, but I may be off with the semantics. The xml sitemap (at it's most basic level) is just a centralized list of links for pages that are on your site to help the search engines find pages without having to crawl all of the pages on your site. They do not contain any anchor text in them.

If you are wanting to crawl your site and find out what all your anchor text is internally, I would use a tool like Screaming Frog.

If you are trying to extract information from your sitemap.xml then you would need to use a combination of a text editor and excel.  Say your URL structure includes product and brand information and you have all the URLs in a sitemap.  You can pull that document into a text editor and then use the "Find and replace" function to get rid of all the extraneous code and then you can be left with just the URLs on one line.  You can then use the excel function "Text to columns" to then split out the URL into sections based on slashes or dashes.  If you have Screaming Frog, it will extract all the URLs for you and then you can split out the info in excel as well.  Use Google to find examples of how to run the tools mentioned above in detail.

If I am totally off, please give a more detailed explanation of what you are trying to do and that will help as well.

Hi Satish,

Not worth the effort. I think it will confuse folks more than anything as your home page is normally in the root directory. If you want your home page to rank for a KW, then just do everything else you should be doing to rank your home page.  Take time to update the content on your home page and build more internal and external links, much better payoff.

Good luck.

Just a correction here.  I agree with all the items above, with one very, very, very, very, very important change.

DO NOT set the corrected urls to disallow in your robots.txt

If you do not allow Google to crawl the pages, Google will not see that the links were removed, that the page is now 4xx, etc. If you were to disallow all those pages, all the clean up work that you have done will not be seen by Google and would be for naught.

If you later want to disallow those pages, that would be fine, but you need to let Google see your clean up work first.

Just to be clear on what Logan said. You have to allow Google to crawl your site by opening up your robots.txt to Google so it can see your noindex directive that is on each of the pages. Otherwise Google will never "see" the noindex directive on your pages.

Likewise, on sitemap.xml.  If you are not allowing Google to crawl the sitemap (because you are blocking it with robots.txt) then Google will not read the sitemap, find all your pages that have the noindex directive on them and then remove those pages from the index.

A great article is here

https://support.google.com/webmasters/answer/93710?hl=en&ref_topic=4598466

From the mouth of Google "Important! For the noindex meta tag to be effective, the page must not be blocked by a robots.txt file. If the page is blocked by a robots.txt file, the crawler will never see the noindex tag, and the page can still appear in search results, for example if other pages link to it."

The other point that logan makes is that Google might list your site if there are enough sites linking to it. The steps above should take care of this, as you are deindexing the page, but here is what I am thinking he is referencing

https://www.youtube.com/watch?v=KBdEwpRQRD0

Google will include a site that is blocked in robots.txt if enough pages link to it, even if they have not crawled the url.

You can go into Search Console and find all the links that they say are pointing to your site. You can also use tools like CognitiveSEO or Ahrefs, Majestic or Moz etc and gather up all of those sites to find links to your site and include those in a disavow file that you put into Search Console and tell Google to ignore all of those links to your site.

Secret bonus method.  Putting a noindex directive in your robots

https://www.deepcrawl.com/knowledge/best-practice/robots-txt-noindex-the-best-kept-secret-in-seo/

This allows you to manage your noindex directives in your robots.txt.  Makes it easier as you can control all your noindex directives from a central location and block whole folders at a time.  This would stop Google from crawling AND indexing pages all in one page and you can just leave the rest of the site alone and not worry about if a noindex tag should or should not be on a certain page.

Good luck!

Howdy!

TLDR - I would estimate Google bot desktop to run at about about 980 pixels, but there is an easy way to test, just mess around with your site by adjusting the width of the browser and see if you can duplicate what you see in Google fetch and render.

http://www.w3schools.com/browsers/browsers_display.asp  97% of browsers have a width of 1024 or greater.  Therefore, if you use that minimum of 1024, your width would be appropriate for pretty much everyone. That said, you might want to go with 980 as the width to account for things like scrolling bars and the fact that most people do not browse in full screen. This is a pretty standard starting point for width.

When you use fetch and render - Google uses one of it's bots depending on the type of page https://support.google.com/webmasters/answer/6066468?hl=en

When Google talks about responsive design https://developers.google.com/webmasters/mobile-sites/mobile-seo/responsive-design it notes, "When the meta viewport element is absent, mobile browsers default to rendering the page at a desktop screen width (usually about 980px, though this varies across devices)."  In other words in some Google documentation they are giving a nod to the 980 pixels being a "standard desktop width"

Having that in mind, I would look at your site and see if you can tell if this jives.  If you have setup the page to look "normal" at greater than 980 pixels, say 1200 pixels, set your width to 1200 pixels in your browser. Then play with the width of the browser and see if you can get it to match what you see in Google fetch and render. If your site looks the same as what you see in fetch and render and your browser is at 980 pixels, then you have a confirmation of the Googlebot desktop viewport size.

You could also setup a simple page and put several images on separate rows that are 950px 980px 1000px 1200px etc wide. Run fetch and render and see what happens, but I like my first suggestion better.

Have fun!

Several ways to go about this. Here is my take.

Buying links - "For only $50 you can get 10,000 links on sites with a PR of 3 or more". People were setting up junk sites and generating a lot of low quality links. It worked for a while, but then Google figured it out and penalized folks for it. Now the reverse has happened, people are having to pay to get links removed or for someone to help them with a disavow, also people have used negative link techniques to build bad links to competitors to hurt them (this has happened to one of my sites).

Buying followers - Similar on twitter, pay to get followers! People setup fake accounts and then sell them so that people can say they have a big number of followers.

Buying likes and shares - I have no idea who Quuu is. No idea who these people are that they use to promote/share something. For the sake of conversation, let's assume that every single person who shares content for Quuuu is legit and that I can pay to have 30,000 shares - will they buy from me? are they really relevant to my niche? On the Quuu website they list that they have followers categorized by topic, but again, unless you are a large company with a wide appeal 99% of the people that they reach will not be relevant IMHO.  Why else do you get a reach of 10k people for $10? You are going to only touch a very small percentage of that 10k who will be interested and buy from you. You can probably curate those people just as easily through your standard social media work etc.

For all the examples above, relevancy is just flushed down the toilet. Sure I have 10,000 links or followers or shares - but if none of those people are interested in buying my product, I have gained nothing but a failed ranking strategy or exposure to people who do not care about my company and my product.

EGOL - can a I have a copy of your continuity plan? I just realized that I need to update my will.

Great answer!  As an example, we have had issues on a site with negative link building from competitors and update 1x to 2x a year.

Invisible to who?  If they are invisible to the search engines (e.g. blocked via robots.txt) they could not be harmful, but if you gain any link or other quality equity through those pages, your will not get any additional domain authority or organic traffic to them.

That said, if the hidden categories were resorts of the main categories on your site then you would want to hide them from the search engine as they would be seen as low quality duplicate content.  Otherwise if they were visible they could negatively impact your site.