Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

"Url blocked by robots.txt." on my Video Sitemap

Technical SEO

950

Critical_Mass Subscriber last edited by

I'm getting a warning about "Url blocked by robots.txt." on my video sitemap - but just for youtube videos? Has anyone else encountered this issue, and how did you fix it if so?! Thanks, J
1 Reply Last reply
Reply Quote 0
sergeystefoglo last edited by

Hi there,

It's definitely possible that you are blocking a folder or subfolder in your robots.txt file that is preventing Google from crawling some of your videos. Could you possibly share a link to the robots.txt file so I could look at it specifically? Also, Google Search Console should tell you which URLs are being blocked specifically so that could be useful as well.

Happy to help — just please send some more specific details. Thanks!
1 Reply Last reply
Reply Quote 0

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

URL Change, Old URLs Still In Index

Recently changed URLs on a website to remove dynamic parameters. We 301'd the old dynamic links (canonical version) to the cleaner parameter-free URLs. We then updated the canonical tags to reflect these changes. All pages dropped at least a few ranking positions and now Moz shows both the new page ranking slightly lower in results pages and the old page still in the index. I feel like I'm splitting value between the two page versions until the old one disappears... is there a way to consolidate this quickly?
Technical SEO | | ShawnW

0
Why do some URLs for a specific client have "/index.shtml"?

Reviewing our client's URLs for a 301 redirect strategy, we have noticed that many URLs have "/index.shtml." The part we don'd understand is these URLs aren't the homepage and they have multiple folders followed by "/index.shtml" Does anyone happen to know why this may be occurring? Is there any SEO value in keeping the "/index.shtml" in the URL?
Technical SEO | | FranFerrara

0
Robots.txt best practices & tips

Hey, I was wondering if someone could give me some advice on whether I should block the robots.txt file from the average user (not from googlebot, yandex, etc)? If so, how would I go about doing this? With .htaccess I'm guessing - but not an expert. What can people do with the information in the file? Maybe someone can give me some "best practices"? (I have a wordpress based website) Thanks in advance!
Technical SEO | | JonathanRolande

0
Fix or Block Webmaster Tools URL Errors Not Found Linked from a certain domain?

RE: Webmaster Tool "Not Found" URL Errors are strange links from webstatsdomain.com Should I continue to fix 404 errors for strange links from a website called webstatsdomain.com or is there a way to ask Google Webmaster Tools to ignore them? Most of Webmaster Tools "URL Not Found errors" I find for our website are from this domain. They refer to pages that never existed. For example, one was to www.mydomain.com/virtual. Thanks for your help.
Technical SEO | | zharriet

0
Robots.txt and joomla

Hello, I use joomla for my website and automatically all those files are blocked is that good or bad, so I remove anything and if so why ? User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
Disallow: /xmlrpc/ I also added to my robots.txt files my email address ( is that useful, I am afraid google passes PR to the email address )
and a javascript: void (0) because I have tabs on my webpage ( is that useful )
as well as a .pdf ( is it also useful ) any comments ? does anything need to be changed or is it ok ? Thank you,
Technical SEO | | seoanalytics

0
RegEx help needed for robots.txt potential conflict

I've created a robots.txt file for a new Magento install and used an existing site-map that was on the Magento help forums but the trouble is I can't decipher something. It seems that I am allowing and disallowing access to the same expression for pagination. My robots.txt file (and a lot of other Magento site-maps it seems) includes both: Allow: /*?p= and Disallow: /?p=& I've searched for help on RegEx and I can't see what "&" does but it seems to me that I'm allowing crawler access to all pagination URLs, but then possibly disallowing access to all pagination URLs that include anything other than just the page number? I've looked at several resources and there is practically no reference to what "&" does... Can anyone shed any light on this, to ensure I am allowing suitable access to a shop? Thanks in advance for any assistance
Technical SEO | | MSTJames

0
Video Sitemap Help

MOZ Helpers, Currently our clients videos are hosted on Viemo and that will not change as our client likes the creative/artist vibe and community via Viemo. That being said we need to create a video sitemap. BTW, Our site uses wordpress. When someone in house uploads a video in the future we want them to be able to enter the video title, description, and tags on the video and when they hit "update" the video and information will get added to our video site map. Wistia has this option here http://wistia.com/doc/video-seo , but like I mentioned above our client has all videos via Viemo. I found a Google XML wordpress plugin, but that said it only works for Youtube video's. The Jr. developer is not confident in creating one himself from the Google webmaster instructions and asked me to search for another solution. Check out the attached pic, that is what I am looking for. Is their a plugin or another option where we can use for easy sitemap updating and management for our Viemo videos? Thanks in advance! M video-seo-dialog.png?id=video-seo video-seo-dialog.png?id=video-seo
Technical SEO | | matthew-220200

0
Robots.txt

Hi there, My question relates to the robots.txt file. This statement: /*/trackback Would this block domain.com/trackback and domain.com/fred/trackback ? Peter
Technical SEO | | PeterM22

0