How can I exclude display ads from robots.txt?
-
Google has stated that you can do this to get spiders to content only, and faster. Our IT guy is saying it's impossible.
Do you know how to exlude display ads from robots.txt?Any help would be much appreciated.
-
You'd want to make the URL paths where the display ads live to have the crawl disallowed in your robots.txt, just like any other section of your site. Here's some basics on robots.txt.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Robots.txt error
Moz Crawler is not able to access the robots.txt due to server error. Please advice on how to tackle the server error.
Technical SEO | | Shanidel0 -
CSS Float Top Left Image not displaying
Using CSS/HTML in Dreamweaver CC, I am trying to display a SEO friendly matrix of six images wide by three deep on the Home Page, but cannot get the first (top left) image to display consistently. The page appears correctly in Dreamweaver Design View, but not in the Live View. Equally it works fine in Safari, but not in Firefox nor I.E. I seem to have tried every CSS variation to resolve this issue, without success. Can someone stear me in the right direction, please? The relevant HTML Code ... Celotex PL4050 - 62.5mm insulated wallboard from £11.96 per m² / £34.45 per board SuperFoil SF19 40mm Multi-foil Roll from £5.23 per m² / £98.00 per 18.75 m² Roll Celotex GA4000 General application Rigid PIR from £8.56 per m² / £24.66 per board Rockwool Full-fill 75mm Cavity Wall Slabs from £3.51 per m² / £1.92 per slab Knauf Earthwool 270mm Loft Roll 44 Combi-Cut from £3.80 per m² / £22.51 per 9.93 m² Roll Kingspan Kooltherm K8 75mm Cavity Wall Boards from £16.26 per m² / £8.79 per board Knauf Earthwool 170mm Combi-cut Loft Roll 44 from £3.16 per m² / £25.34 per 8 m² Roll Kingspan Kooltherm K7 Rigid Phenolic Foam - 75mm £13.91 per m² / £40.05 per board ... The associated CSS code ... #popular {
Technical SEO | | JustInsulation
width: 1050px;
height: 800px;
overflow: hidden;
width: auto;
height: auto;
}
#product1 {
background-image: url(../007-graphics/popular/01-prod.png);
background-repeat: no-repeat;
width: 150px;
height: 265px;
margin-left: 24px;
float: left;
overflow: auto;
clear: left;
}
#product2 {
background-image: url(../007-graphics/popular/02-prod.png);
background-repeat: no-repeat;
width: 150px;
height: 265px;
margin-left: 10px;
float: left;
opacity: 0.85;
}
#product3 {
background-image: url(../007-graphics/popular/03-prod.png);
background-repeat: no-repeat;
width: 150px;
height: 265px;
margin-left: 10px;
float: left;
max-height: 275px;
opacity: 0.85;
}
#product4 {
background-image: url(../007-graphics/popular/04-prod.png);
background-repeat: no-repeat;
width: 150px;
height: 265px;
margin-left: 10px;
float: left;
max-height: 275px;
}
#product5 {
background-image: url(../007-graphics/popular/05-prod.png);
background-repeat: no-repeat;
width: 150px;
height: 265px;
margin-left: 10px;
float: left;
max-height: 275px;
opacity: 0.85;
}
#product6 {
background-image: url(../007-graphics/popular/06-prod.png);
background-repeat: no-repeat;
width: 150px;
height: 265px;
margin-left: 10px;
float: left;
max-height: 275px;
opacity: 0.85;
}
#product7 {
clear: both;
background-image: url(../007-graphics/popular/07-prod.png);
background-repeat: no-repeat;
width: 150px;
height: 265px;
margin-left: 24px;
float: left;
max-height: 275px;
opacity: 0.85;
}
#product8 {
background-image: url(../007-graphics/popular/08-prod.png);
background-repeat: no-repeat;
width: 150px;
height: 265px;
margin-left: 10px;
float: left;
max-height: 275px;
}
#product9 {
background-image: url(../007-graphics/popular/09-prod.png);
background-repeat: no-repeat;
width: 150px;
height: 265px;
margin-left: 10px;
float: left;
max-height: 275px;
opacity: 0.85;
} ... The complete code is located at - www.just-insulation.com/index.html0 -
Site is not displaying in Search Engines
My site is www.deoveritas.com it is in magento framework and it has a blog section in wordpress. When I enter Site:www.deoveroitas.com in google it shows all blog links in search result. The homepage and other innerpages are not getting displayed in search results at all. I even tried searching for www.deoveritas.com/about-us and it displays blogs in result. Checked Google webmaster fetch as google and it was index and successful. Can you please help me with this. Is my site de-indexed or banned by Google? the same issue is on Bing and Yahoo search engines too. Please help Thank you.
Technical SEO | | tpt.com0 -
Adding my web link on wikiquote, is it ok?
If i insert my link on wikiquote on a appropriate page, is it ok for seo or negative?
Technical SEO | | rimon56930 -
Same URL in "Duplicate Content" and "Blocked by robots.txt"?
How can the same URL show up in Seomoz Crawl Diagnostics "Most common errors and warnings" in both the "Duplicate Content"-list and the "Blocked by robots.txt"-list? Shouldnt the latter exclude it from the first list?
Technical SEO | | alsvik0 -
Robots.txt not working?
Hello This is my robots.txt file http://www.theprinterdepo.com/Robots.txt However I have 8000 warnings on my dashboard like this:4 What am I missing on the file¿ Crawl Diagnostics Report On-Page Properties <dl> <dt>Title</dt> <dd>Not present/empty</dd> <dt>Meta Description</dt> <dd>Not present/empty</dd> <dt>Meta Robots</dt> <dd>Not present/empty</dd> <dt>Meta Refresh</dt> <dd>Not present/empty</dd> </dl> URL: http://www.theprinterdepo.com/catalog/product_compare/add/product/100/uenc/aHR0cDovL3d3dy50aGVwcmludGVyZGVwby5jb20vaHAtbWFpbnRlbmFjZS1raXQtZm9yLTQtbGo0LWxqNS1mb3ItZXhjaGFuZ2UtcmVmdWJpc2hlZA,,/ 0 Errors No errors found! 1 Warning 302 (Temporary Redirect) Found about 5 hours ago <a class="more">Read More</a>
Technical SEO | | levalencia10 -
Is blocking RSS Feeds with robots.txt necessary?
Is it necessary to block an rss feed with robots.txt? It seems they are automatically not indexed (http://googlewebmastercentral.blogspot.com/2007/12/taking-feeds-out-of-our-web-search.html) And, google says here that it's important not to block RSS feeds (http://googlewebmastercentral.blogspot.com/2009/10/using-rssatom-feeds-to-discover-new.html) I'm just checking!
Technical SEO | | nicole.healthline0 -
Best way to display maintenence mode on a website?
I have a website with lots of traffic and sometimes the backends fail. I want to use lighttpd to show that the website is under mantenence and should be back up shortly. I was thinking of using Soft 503 errors or doing a 302 for every page to /maintenance.html. What would you do (besides fixing the backends, we are already doing that :P) to avoid hurting your SEO efforts? Thanks in advance Mariano
Technical SEO | | marianoSoler980