Welcome to the Q&A Forum
Browse the forum for helpful insights and fresh discussions about all things SEO.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
I would use your robots.txt file to prevent them from crawling the specific strings / pages. Go into your Google Webmaster Tools and you can see all the information Google has on your site and any issues, you can also specify robots.txt information in there. That would be the best route as Google is obedient with what is on the robots.txt file. If you want more information about robots.txt, go here.
You could check your server stats on who is accessing your site, this should tell you what bots are going to your pages when. I don't know what control panel you are using for your site, but if you are using Cpanel, I am sure there are tutorials online to help you find this information.
You can use the search line on Google: site:www.example.com
This will tell you what Google has indexed on the site.
If you are looking for a specific sub-folder, then use: site:www.example.com/forum
Hope that helps.
Agreed. You now have your work cut out for you Jose.
I would suggest that for each model you create different unique content on the page (as each model should have different features and variants). Now instead of adding a new page for each size of each model, have a drop down box (on the model's page) that lets you select the size, so the user can instantly see the sizes available for the model and then you don't have to scramble to generate new content for each size (which would be difficult). This is a win-win situation for the user and for the client. Hope that helps.
You can use the search line on Google: site:www.example.com
This will tell you what Google has indexed on the site.
If you are looking for a specific sub-folder, then use: site:www.example.com/forum
Hope that helps.
I would use your robots.txt file to prevent them from crawling the specific strings / pages. Go into your Google Webmaster Tools and you can see all the information Google has on your site and any issues, you can also specify robots.txt information in there. That would be the best route as Google is obedient with what is on the robots.txt file. If you want more information about robots.txt, go here.