How to get a list of robots.txt file
-
This is my site.
Its in wordpress.I just want to know is there any way I can get the list of blocked URL by Robots.txt
In Google Webmaster its not showing up.Just giving the number of blocked URL's.
Any plugin or Software to extract the list of blocked URL's.
-
If you use Bing Webmaster tools you can see a complete list all URLs blocked by robots.txt. You can export the file and then filter.
Just go to Reports & Data > Crawl Information within your Bing webmaster account. I am not aware of this feature being in Google webmaster tools. Hope this helps.
-
simon_realbuzz buddy If I use this /classifieds/ it means I am blocking all URL starting with it.I want to get a list of all blocked URL's of site.
Example
http://muslim-academy.com/classifieds/
How many URL's associated with this classified are blocked by my robots.txt.
-
I'm sorry I don't follow. If you go to that URL you will see the list of blocked URLs as I've pasted below.
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /forum/viewtopic.php?p=
Disallow: /forum/viewtopic.php?=&p=
Disallow: /forum/viewtopic.php?t=
Disallow: /forum/viewtopic.php?start=
Disallow: /forum/&view=previousDisallow: /forum/&view=next
Disallow: /forum/&sid=
Disallow: /forum/&p=
Disallow: /forum/&sd=a
Disallow: /forum/&start=0
Disallow: /forum/memberlist.php
Disallow: /forum/posting.php
Disallow: /classifieds/
Disallow: /forum/index.php
Disallow: /forum/ucp
Disallow: /http://muslim-academy.com/الا�%A..
Disallow: /http://muslim-academy.com/особенн%D
Disallow: /http://muslim-academy.com/ислам-ка%
Disallow: /http://muslim-academy.com/classifieds/ads/Disallow: /http://muslim-academy.com/значени%D..
Disallow: /.ifieds/
Disallow: /.ifieds/ads/
Disallow: /forum/alternatelogin/al_tw_connect.php?authentication=1
Disallow: /forum/search.php -
simon_realbuzz I need a list of blocked URL's not the robots.txt file path.
-
You can view your robots file simply by appending /robots.txt to your site URL. Just put the following http://muslim-academy.com/robots.txt and you'll be able to view your robots file.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
GA4 account & property not showing in traffic property setup list.
Hi there, I've connected multiple client accounts to GA4 already, but three of our accounts that we have administrator rights to in GA4 are not showing up in our selectable accounts/properties list when logged in via Moz to add to the traffic settings area. Anyone else have this issue and find a fix?
Reporting & Analytics | | luminusagency0 -
Query on Product List Position After doing filters
Hi All, I have implemented enhance ecommerce with tag manager now I am implementing Product List Position so I have query that if visitor do filter or do sorting via highest or lowest price at that time do we have to show List position too? If yes then how and if No then why?
Reporting & Analytics | | Arnold3
Thanks!Mitesh0 -
Is there an automated way to determine which pages of your website are getting 0 traffic?
I'm doing a content audit on my company website and want to identify pages with zero traffic. I can use GA for low traffic, but not zero traffic. I can do this manually, but it would take a long time. Are there any tools to help me determine these pages?
Reporting & Analytics | | Ksink0 -
Tracking Google places (7 pack listing) traffic in google analytics
Is there a way to see Google Places traffic (traffic from users clicking through the 7 pack listings) segmented in Google analytics ? Normally is it just lumped together with the organic traffic ? Can you see the search phrases used to find your site, or do they also show up under 'not provided' when from Google Places. Im aware i can see some limited data in the Google Places analytics, but these seem to be 2 days behind when ever i view them.
Reporting & Analytics | | Sam-P0 -
By linking adwords to ga is it possible to get it done within one profile only?
Hi mozzers, i have a ga account with 40 profiles and none of them are connected to adwords. I would like to connect adwords to just the combined profile and not the other 39 profiles? Is this possible? if it is how would you do it? thanks!
Reporting & Analytics | | Ideas-Money-Art0 -
When will traffic data be working ? also whats with the spike in duplicate listing issues with everyone.
Hi There, We have no traffic data, is this something we are doing wrong or is this an issue with SEOMOZ ? Also duplicate listings have gone sky high, check goggle analytics's and all ok ? Any answers ? Thanks Charlie
Reporting & Analytics | | pro580 -
How do i get Social Media Actions Tracked in GA
Greetings from 17 degrees C wetherby UK 🙂 http://i216.photobucket.com/albums/cc53/zymurgy_bucket/how-do-get-this.jpg The above url pints to dat I'd love to see in my Google analytics account but instead all i can see is this:
Reporting & Analytics | | Nightwing
http://i216.photobucket.com/albums/cc53/zymurgy_bucket/no-socail-media-engagementcopy.jpg What i really want to measure is Facebook Likes etc not just referral traffic from social media sites. So my question is please... "Do i have to add additional tracking code to Google analytics as explained here - https://developers.google.com/analytics/devguides/collection/gajs/gaTrackingSocial " Thanks in advance,
David0 -
Should I delete a page that gets search traffic, that I don't care about?
I have a page on my site that consistently gets traffic, every month. Googlers seems to love it. But I don't like it at all. Webmaster tools shows that google allows us a certain number of search impressions each day. - it flatlines, they are limiting the impressions we get. We also getthe same number of clickthroughs each day. So my question is for anyone who has this same experience, who may have experimented by deleting a page you don't care about. Did you just lose that number of clicks each day or did other pages on your site get displayed and clicked through instead?
Reporting & Analytics | | loopyal0