Screaming Frog tool left me stumped
-
Hi there again,
I found a major cloaking hack in our client's website that is really well camouflaged and all the seo tools that I tried to help me check for cloaking couldn't find it.
I know that screaming frog is a great tool and I want to use it to help me, however, I can't seem to get my way around their system that I downloaded.
Can you help me with the screaming frog program? Do you know where I can make a full site check for cloaking, maybe there are more links that I wasn't notified about?
I would really appreciate if you could help me with that.
Thanks so much, Ruchy
-
Hi Ruchy,
That's fantastic deep crawl is a great tool.
I was talking about two different types of cloaking one using an URL with a C name for hidden cloaking e.g. https://www.rebrandly.com/
Thank you for the kind words. Giving good answers and thumbs up or thumbs down definitely does make a difference I have outlined a lot of it below the official MozPoints URLhttp://moz.com/community/mozpoints
Yes marking anything as a giving thumbs-up is telling somebody they have helped you and in the event of thumbs up is their response was valuable and maybe contributed to answering your question.
In the case, of giving good answer person has answered your question giving as well as thumbs-up is the way anything is a way of saying thank you.
Welcome to the most community, and I look forward to seeing you here.
All the best,
Tom
-
Hi Tom
we finally solved the issue. Thanks for your clear answers, the info really helped me. We bought DeepCrawl and it really was helpful.
What was the C name you were talking about?
Thank you, Ruchy
P.S. i marked good answer by the wrong one, does it make a difference?
-
Hey just wanted to check was any answer helpfull to you?
Is there anything we might've missed or failed to help you solve the issue?
All the best,
Tom
-
-
Thanx Andy
That was a helpful piece of information.
I am still a little confused, can you give me some further advice?
I there any way for me to do a full site cloaking check?
Ruchy
-
Hi Ruchy,
Screaming Frog is an excellent tool. Dan is a great guy to update and the on that.
In my opinion, I have been able to find formation using https://www.deepcrawl.com/
The cost is higher deepcrawl is also a hosted platform.
Still, it can do remarkable things that no other tool that I know of can do.
There are built just for checking cloaking this one is free
http://www.seotools.com/seo-cloaking-checker/
If you have access to the DNS will want to examine this thoroughly.
I have laid out how you would find c name DNS records if you do not have access to the DNS as well.
One other method you might have to use to determine whether or not domain or subdomain has been cloaked as shown here: http://www.ghosturl.com/ if the cloaking happens in a manner described in the URL for this sentence.
So the scenario would be the cloaked URL would be something like
http://www.cloaked.example.com/?=ku2b4B30ijbasT47720sb534Nbq6
( please know I used example.com is the domain because I did not want to pointed that any live site not because of anything to do with cloaking)
Therefore they're using most likely a C name inside the DNS to pointed to http://www.example.com
You will need to use a tool that would be able to look at the IP or C name being created by the DNS.
The free tool I know if that can look for a C name or A record that should not be there is https://www. cloudflare.com the way you would do this is run the site through the DNS wizard that pulls the current DNS including all the records (it can miss some, but it does an excellent job of catching most of all DNS records. (9 times out of 10 will get all of them)
Because most tools make it impossible to see the C name when looking at DNS, it is important to remember there is a free service that is not designed for this purpose but can be used to discover the rogue cloaked URL.
https://www.cloudflare.com/a/sign-up (the free-tier offers the same ability to get DNS information)
http://i.imgur.com/Wm7W0aM.mp4
https://cl.ly/0A3k3d2j401n/Screen Recording 2017-01-26 at 07.09 AM.mov
http://i.imgur.com/lIN9Gsm.png
Hope This helps,
Tom
-
Dan from Screaming Frog has just had a look at this, and although he is on the move at the moment, he said this...
On the road to a meeting, but they can switch user-agent & change to JavaScript rendering mode to test for cloaking. Depends how it's cloaked, though. Some may do via referrer which you can't test for within the Spider (yet!). Will reply a bit later
-Andy
-
Hi there
You can spider the site with screaming frog and then do a bulk export of all the outlinks into a CSV and run through and see if there are links in there that shouldn't be
Have you downloaded the latest version?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Drastic surge of link spam in Webmaster Tools' Link Profile
Hello all I am trying to get some insights/advice on a recent as well as drastic increase in link spam within my Webmaster Tools' Link Profile. Before I get into more detail, I would like to point out, that I did find some relevant MOZ community posts addressing this type of issue. However, my link spam situation may have to be approached from a different angle, as it concerns two sites at the same time and somewhat in the same way. Basically, starting in July 2017, from one day to the other, a multitude of domains (50+) is generating link spam (at least 200 links a month and counting) and to cut a long story short, I believe the sites are hacked. This is because most of the domain names sound legit and load the homepage, but all the sub-pages linking to my site contain "adult" gibberish. In addition, it is interesting to see, that each sub-page follows the same pattern, scraping content from my homepage including the on-page links - that generate the spammy backlinks to my sites - while inserting the adult gibberish in between (basically it's all just text and looks like as if a bot is at work). Therefore, it's not like my link is being inserted "specifically" into pages or to spam me with the same anchor text over and over. So, I am not sure what kind of link spam this really is (or the purpose of it). Some more background information: As mentioned above, this link spam (attack?) is affecting two of my sites and it started off pretty much simultaneously (in addition, the sites focus on a competitive niche). The interesting detail is, that one site suffered a manual penalty years ago, which has been lifted (a disavowal file exists and no further link building campaigns have been undertaken after the cleanup), while the other site has never seen any link building efforts - it is clean, yet the same type of spam is flooding that websites' link profile too. In the webmaster forums the overall opinion is, that Google ignores web spam. All well. However, I am still concerned, that the dozens of spammy links pointing to the website "with a history" may pose a risk (more spam on a daily basis on both sites though). At the same time I wonder, why the other "clean" site is facing the same issue. The clean sites' rankings do not appear to be impacted, while the other website has seen some drops, but I am still observing the situation. Therefore, should I be concerned for both sites or even start an endless disavowal campaign on the site with a history? PS: This MOZ article appears to advice so: https://moz.com/blog/do-we-still-need-to-disavow-penguin "In most cases, sites that have a history of collecting unnatural links tend to continue to collect them. If this is the case for you, then it’s best to disavow those on a regular basis (either monthly or quarterly) so that you can avoid getting another manual action." What is your opinion? Sorry for the long post and many thanks in advance for any help/insight.
White Hat / Black Hat SEO | | Hermski0 -
Unnatural inbound links message from Google Webmaster Tools!
Hi Everyone, I just got this message from GWT(image below) This is probably a penguin Penalty. What is clear is I have to find the best and most efficient way to tackle this issue. We will probably lose tons of traffic in the next couple of weeks so I would like to get the best suggestions and maybe a guideline on how to do this in the most effective way! Thank you! 1a0X2M2a1h0A
White Hat / Black Hat SEO | | Ideas-Money-Art0 -
Website not listing in google - screaming frog shows 500 error? What could the issue be?
Hey, http://www.interconnect.org.uk/ - the site seems to load fine, but for some reason the site is not getting indexed. I tried running the site on screaming frog, and it gives a 500 error code, which suggests it can't access the site? I'm guessing this is the same problem google is having, do you have any ideas as to why this may be and how I can rectify this? Thanks, Andrew
White Hat / Black Hat SEO | | Heehaw0 -
Disavow tool for blocking 4 to 5 sites for Article Republishing
Am finding some very low authority sites (recently picked our articles from ezine and other article sites - written over a year back) and pasted on to there site. The number of articles copies are not 1 or 2, but more than 10-12 in all these domains This has also led to our anchor based url - backlink to us from them (a part of article). Have Wrote down to remove my author profile and articles - but there has been no response from webmaster of these sites. Is Disavow a right approach. The number of such sites are 4 or 5 in nature !!
White Hat / Black Hat SEO | | Modi0 -
When to NOT USE the disavow link tool
Im not here to say this is concrete and should never do this, and please if you disagree with me then lets discuss. One of the biggest things out there today especially after the second wave of Penguin (2.0) is the fear striken web masters who run straight to the disavow tool after they have been hit with Penguin or noticed a drop shortly after. I had a friend who's site who never felt the effects of Penguin 1.0 and thought everything was peachy. Then P2.0 hit and his rankings dropped of the map. I got a call from him that night and he was desperately asking me for help to review his site and guess what might have happened. He then tells me the first thing he did was compile a list of websites back linking to him that might be the issue and create his disavow list and submitted it. I asked him "How long did you research these sites before you came the conclusion they were the problem?" He Said "About an hour" Then I asked him "Did you receive a message in your Google Webmaster Tools about unnatural linking?" He Said "No" I said "Then why are you disavowing anything?" He Said "Um.......I don't understand what you are saying?" In reading articles, forums and even here in the Moz Q/A I tend to think there is some misconceptions about the disavow tool from Google that do not seem to be clearly explained. Some of my findings with the tool and when to use it is purely based on logic IMO. Let me explain When to NOT use the tool If you spent an hour reviewing your back link profile and you are to eager to wait any longer to upload your list. Unless you have less than 20 root domains linking to you, you should spend a lot more than an hour reviewing your back link profile You DID NOT receive a message from GWT informing you that you had some "unnatural" links Ill explain later If you spend a very short amount of time reviewing your back link profile. Did not look at each individual site linking to you and every link that exists, then you might be using it WAY TO SOON. The last thing you want to do is disavow a link that actually might be helping you. Take the time to really look at each link and ask your self this question (Straight from the Google Guidelines) "A good rule of thumb is whether you'd feel comfortable explaining what you've done to a website that competes with you, or to a Google employee" Studying your back link profile We all know when we have cheated. Im sure 99.9% of all of us can admit to it at one point. Most of the time I can find back links from sites and look right at the owner and ask him or her "You placed this back link didn't you?" I can see the guilt immediately in their eyes 🙂 Remember not ALL back links you generate are bad or wrong because you own the site. You need to ask yourself "Was this link necessary and does it apply to the topic at hand?", "Was it relevant?" and most important "Is this going to help other users?". These are some questions you can ask yourself before each link you place. You DID NOT receive a message about unnatural linking This is were I think the most confusing takes place (and please explain to me if I am wrong on this). If you did not receive a message in GWT about unnatural linking, then we can safely say that Google does not think you contain any "fishy" spammy links in which they have determined to be of a spammy nature. So if you did not receive any message yet your rankings dropped, then what could it be? Well it's still your back links that most likely did it, but its more likely the "value" of previous links that hold less or no value at all anymore. So obviously when this value drops, so does your rank. So what do I do? Build more quality links....and watch you rankings come back 🙂
White Hat / Black Hat SEO | | cbielich1 -
Access Denied - 2508 Errors - 403 Response code in webmaster tools
Hello Fellow members, From 9th may I am getting this error messages & these crawl errors is increasing daily. Google is not able to crawl my URLS & getting 403 response code & saying ACCESS Denied Errors in GWT. My all Indexed pages are de-indexed. Why I am receiving this errors ? My website is working fine but why Google is not able to crawl my pages. PLEASE TELL ME what is the ISSUE, I need to resolve ASAP on 9th may I got a message in GWT as well for "http://www.mysitename.co.uk/ Increase in authorization permission errors " Google detected a significant increase in the number of URLs we were blocked from crawling due to authorization permission errors. After this all problem started. Kindly tell what is the issue & how can I solve this. WGsu8pU
White Hat / Black Hat SEO | | sourabhrana390 -
Potential Implications of using the Disavow tool to remove thousands of links
So here's the situation. My companies site has over 30 thousand backlinks from Rippling.info These links all point to 3 product pages, some of which are no longer in production. Apparently a former employee was experimenting with some link farm ideas. My questions are; 1. does anyone here have experience with rippling.info? Is it legit? It seems like a link farm but Google allows adsense ads??? I thought Google was against link farms... 2. if I use the Disavow tool in Webmaster Tools to tell Google these 30k+ incoming links are to be ignored, will there be any consequences? -Google Analytics shows zero referral traffic since jan 1st 2012.
White Hat / Black Hat SEO | | mjmorse0 -
Manage two domains with Webmaster tools
Dear all, I want to create an additional domain in order to: Rank better for a very specific keyword with an exact match domain (I already ask about that, but I did not have my ideas clear at the time); Offer to the user usefull infomation about the topic, without duplicating the content I have in my main domain, just additional and very specific information; Use this domain as landing page, offering a tutorial on "how to" use a specific section of my main domain, including a video tutorial; Link to the related section of my main domain. So, the main idea is, if an user type in google "this specific keyword of ours", they will have in the results "thisspecifickeywordofus.es", they will click and go to the site, where they will find unique and specific information, complementing what I have in my mainsite, and showing how to use my site, so trying to use it for conversion. I want to do only white hat SEO, so first at all, I would like to ask you if you think it is a good idea. The keyword is difficult to rank for, and if I can take advantage of this exact match domain (even if it is nowdays no more so big an advantage), would be great. Second, do you see any problem in managing different domains from the same google account? Newbee question, sorry. Thanks in advance for your help, Daniel
White Hat / Black Hat SEO | | te_c0