How to fix these unwanted URLs?
-
Right now i have wordpress, one page website, but google also show wp-content. KIndly check below in google.
site:http://baltimoreelite.com/
How I can fix this issue?
-
Great job, Mark! I can see from this end that nearly all of those unwanted URLs have already dropped out of the results. That's far quicker than even I expected! And the ones that aren't gone are leading to a 403 Forbidden page, which is great.
One last thing you can do if you want. Because you are on HostGator, they are displaying their custom 403 error page, which has their branding all over it (nasty, kinda ugly) You could create your own simple 403 error page, add your own basic branding to it, and for instance add a line that says something like "You don't have permission to view this page or it is blocked for security reasons. Drop by the home page [link to home page] to find what you're looking for, or to conduct a search."
This basic page can be used to replace the one that HostGator provides by default so any visitors that hit it by accident will still feel like they are on your site, and will have a suggestion for what to do next. Your hosting control panel will have instructions for how & where to provide your own custom error pages.
Hope that last little tweak's useful.
Paul
-
Thank You Paul. All is well now.
-
If I were you, Mark, I'd add it right at the top of your htaccess file. I'd also add in a descriptive comment to make the reason for the directive clear. So:
BEGIN Remove ability to read directory indexes
Options -Indexes
END Remove ability to read directory indexes
These lines would be inserted right at the top of the htaccess file. I would also warn though, that I've had situations where caching plugins have overwritten such directives when they update the htaccess themselves. If that happens, you man need to try inserting it after #END WPSuperCache and before # BEGIN WordPress.
Hope that works for you?
Paul
-
Andy and Nishada - don't forget... Adding robots.txt disallows will do nothing to get already indexed URLs out of the search index after the fact.
Paul
-
You have a much bigger problem than what can be solved just with a robots.txt file, Mark.
All of those URLs are showing up because of a misconfiguration of your theme installation (likely caused by the theme developer) is allowing full display of all of the content of each of those directories. In addition to polluting your search results, as you've noticed, it's a also a pretty major security risk. You can see this in action by going to http://baltimoreelite.com/wp-content/themes/sintia/wpv_theme/assets/css/ What should happen when you go to that URL is you see a blank page, or receive a 403-Forbidden warning. Instead, you're seeing a full listing of the directory contents - bad news.
Since I don't know your hosting configuration, the easiest way to fix this issue is to add a line to your .htaccess file at the root of your site. This should correct for all such instances, You need to add this line:
<code>Options -Indexes</code>
If you're not familiar, the .htaccess file is a text file which you can edit with any text editor. You'll need to use an FTP program or the file manager in your hosting control panel to access it at the root of your site. (You may also need to enable "show hidden files" in your program). I always recommend backing up the existing file before editing just in case. You'll add this new line on its own line with a blank line between it and any other lines in the file. It can go near or at the top of your htaccess file.
Now getting those URLs out of the search index is going to take a bit more work. You'll want to implement the robots.txt exclusions like Andy suggested, and then you'll need to go into Google Webmaster Tools and use the Remove URLs tool to specifically request removal of the directories you blocked with the robots.txt and that you also want removed from the search results. (The robots.txt is a critical part of this process, as Google requires it be in place in order to process the removal requests.)
This, combined with the htaccess edit mentioned above, should keep those URLs from showing up again in teh future.
Hope that all makes sense. If not, be sure to ask!
Paul
-
Hi Mark,
I block the same on my site (which is also a single page). Here is the content of my Robots.txt file.
User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /wp-content/ Sitemap: http://www.inetseo.co.uk/sitemap.xml.gz
-Andy
-
Hi Mark,
You need to tell crawlers to not to index those content by modifying the robots.txt file. Below is a good link with some examples and instructions
http://stackoverflow.com/questions/17029811/how-to-set-up-robots-txt-file-for-wordpress
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Technical URL SEO question
Hi All, We sell a product on our site which is displayed in cubic metres, from an SEO perspective is it ok to have /3m³ in the URL or should I use 3m3. Thanks All
Technical SEO | | Redooo0 -
Clean URL vs. Parameter URL and Using Canonical URL...That's a Mouthfull!
Hi Everyone, I a currently migrating a Magento site over to Shopify Plus and have a question about best practices for using the canonical URL. There is a competitor that I believe is not doing it the correct way, so I want to make sure my way is the better choice. With 'Vendor Pages' in Shopify, they show up looking like: https://www.campusprotein.com/collections/vendors?q=Cellucor. Not as clean. Problem is that Shopify also creates https://www.campusprotein.com/collections/cellucor. Same products, same page, just a different more clean URL. I am seeing both indexed in Google. What I want to do is basically create a canonical URL from the URL with the parameter that points to the clean URL. The two pages are very similar. The only difference is that the clean URL page has some additional content at the top of the page. I would say the two pages are 90% the same. Do you see any issue with that?
Technical SEO | | vetofunk0 -
How to fix broken links?
Hi, I use WordPress CMS with Yoast SEO plugin. I have just found out that my 403 errors increased dramatically. It seems that all my tags below of each post are being broken for some reason. When i click on the tags i get the following massage: **403 Forbidden Request forbidden by administrative rules. ** I assume it has something to do with the configuration within Yoast SEO plugin. Dose anyone know how should i fix that? Thanks, Raviv evsGujA
Technical SEO | | Indiatravelz0 -
Title Tags & Url Structure
So I'm working on a website for a client in the Tourism Industry. We've got a comprehensive list of museums & other attractions in a number of cities that have to go online. And we have to come up with the correct url structure, title tags and obviously content. My current line of thought was to work the urls in the following way. http://domain.com/type-of-attraction/city/name-of-attraction/ This is mainly because we think that the type of attraction is far more important then the city (SEO wise) as the country as a whole receives more searches, however we require a city in the url to make it unique because some attractions across cities happen to share names and we don't want to have the names of attractions littered with city names. However for title-tags I wanted to go the other way around, again due to the attraction type being more important then the city. Name of Attraction - Type of Attraction - City - Brand Name or Name of Attraction - Type of Attraction in City - Brand Name I am quite confident in working it this way; however I would appreciate if I receive some feedback on this structure, you think its good or you would make any suggestions / alterations. One last thing, There's the possibility of having many urls ending up with the same city names (For each type of attraction) I would think that just providing a list of links & duplicate text is not enough; would you suggest a canonical pointing to a link containing just information on the city? and using the other pages for user-navigation only? or should i set variables in the text which are replaced by the types of attraction so that the text looks different for each one?
Technical SEO | | jonmifsud0 -
Can I redirect a URL that has a # in it? How?
Hi there - My web developer is saying that I can't do a URL redirect with a "#" in it. Currently, the URL is actually an anchored link within a page (which the URL indicates with a #). I want to change the content to a new URL, but our website links internally to the old URL, so we would need to do a URL redirect (assume 301). Can you tell me if this is possible and how? Thanks!
Technical SEO | | sfecommerce0 -
Urls with or without .html ending
Hello, Can anyone show me some authority info on wheher links are better with or without a .html ending? Thanks is advance
Technical SEO | | sesertin0 -
Strange duplicate url
From your csv report I have this strange issue. This url: elettrodomestici.yeppon.it/climatizzatori/condizionatori-fissi/prodotti/condizionatori-fissi-comfee/ it's a duplicate of this elettrodomestici.yeppon.it/climatizzatori/condizionatori-fissi/prodotti/condizionatori-fissi-comfee/ but the only url that I can see in the website is this one. Why the "-" is transalted some times in "%2D" referrer obviously is elettrodomestici.yeppon.it/climatizzatori/condizionatori-fissi/prodotti/condizionatori-fissi-comfee/solo-disponibili/ I have many duplicate url...Can you help me? Thanks
Technical SEO | | yeppon0