Crawler doesn't discover the links in the main nav
-
Hi Moz Community,
We have a headless ecom (Magento) client that I'm trying to crawl the site. During the crawl, the tool (Screaming Frog) cannot discover the sub-category URLs in the main navigation when I start crawling via homepage.
Similarly, when I start crawling with one of the sub-category page, it doesn't crawl any of the product URLs on the sub-category page itself.
When I inspect product and sub-cat URLs through Search Console, they seem as indexed and if I view how Googlebot rendered the sub-category page, I can see the product URLs on the sub-cat page too.
If you have any idea what's the issue with Screaming Frog and would like to help me out, I'd be so grateful!
Thanks in advance
-
Hi Kate,
Thank you!
I followed you on Twitter, my user name is @curetuvana
-
Find me on twitter @katemorris and follow me. Tell me your name and I'll follow you and we can DM.
-
Hi Kate,
Thank you for your time to respond! Is there any way that I can contact you directly?
By the way, I've tried crawling after changing configurations as 'rendering Javascript' however, it still didn't discover product URLs.
Thank you!
-
Ah, I might know your problem. What is your site? We had this issue at my last company, had to do with crawling using JS. If you will send me the site, I can take a look.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does product environment have impact on main website's SEO
We have two environments - product, where login is necessary and where the customers are working. We also have there our help desk, Q&A and knowledge base. Pretty sophisticated page regarding information on a specific topic. We also have our main page where we promote our products, company and events, etc. Main page is www.example.com, where product environment is login.example.com . Does this product environment have an impact on my main page's SEO?
Intermediate & Advanced SEO | | NeringaA0 -
Incorrect Spelling Indexed In Meta Info - Can't Change It
Hi,It would be great if a member of the community could help me to resolve this issue.Google is indexing an incorrect spelling on of our key pages and we can't identify the reason why.- The page in question: https://newbridgesilverware.com/jewelleryAs you can see from the attached image, the Meta Title is rendered to contain the keyword "jewelry" (the American spelling.) We want this to read as "jewellery" - the British-English spelling. Yet in the page source the word is given in the meta title as "jewellery". Nowhere in the page source or on the page itself does the American spelling appear - yet Google still renders it in the Meta Title.Can anyone identify why this is happening and offer any possible solutions?Much appreciatedDhqJp
Intermediate & Advanced SEO | | Johnny_AppleSeed1 -
International Targeting | Language > 'fa-ir' - no return tags
I see this error in search console :International Targeting | Language > 'fa-ir' - no return tagsURLs for your site and alternate URLs in 'fa-ir' that do not have return tags.and it is really increasingi do not know what is the problem and what I have done wrong? Originating URL Crawl date Alternate URL 1 /abadan/%D8%A2%D8%A8%D8%A7%D8%AF%D8%A7%D9%86/browse/vehicles/?place=8,541&v01=0,1&saveLoc=1
Intermediate & Advanced SEO | | divar11/16/16 http://divar.ir/
0 -
When the site's entire URL structure changed, should we update the inbound links built pointing to the old URLs?
We're changing our website's URL structures, this means all our site URLs will be changed. After this is done, do we need to update the old inbound external links to point to the new URLs? Yes the old URLs will be 301 redirected to the new URLs too. Many thanks!
Intermediate & Advanced SEO | | Jade1 -
What sort of content for 'non-niche' website?
Hey guys, had a question with regards to content production. We run an store called Yellow Octopus in Australia and we've literally got thousands of products (4500 skus last count). We've got everything from novelty mugs to kitchen accessories to gag gifts, t-shirts and tech gadgets. I've read a lot of material on creating awesome content to attract backlinks and we are ready to craft our content strategy. We've got a team in place - graphic designer, illustrator and writers to execute that strategy. It's just a matter of formulating the strategy! Largely speaking I have an idea of the quality of content required because I look at a lot of it. The real issue is what type of content is right for us? Most of the articles I have read focus on niche industries i.e. SEO, Piano sales or health foods. Right off the bat I can come up with hundreds of content pieces that work around those niches. However, with such a diverse range of products I'm unsure of what our niche really is, in fact not having a niche is almost our niche. Of course we could do gift guides like '30 Unbelievable Gifts for Foodies' (and we do, do those). However they aren't really the type of posts that are likely to attract back-links. Is the best strategy to split the content into categories? What sort of content pieces would you suggest for a company such as ours? Many thanks in advance!
Intermediate & Advanced SEO | | TheGreatestGoat0 -
Same page Anchor Links vs Internal Link (Cannibalisation)
Hey Mozzers, I have a very long article page that supports several of my sub-category pages. It has sub-headings that link out to the relevant pages. However the article is very long and to make it easier to find the relevant section I was debating adding inpage anchor links in a bullet list at the top of the page for quick navigation. PAGE TITLE Keyword 1 Keyword 2 etc <a name="'Keyword1"></a> Keyword 1 Content
Intermediate & Advanced SEO | | ATP
<a name="'Keyword2"></a> Keyword 2 Content Because of the way my predecessor wrote this article, its section headings are the same as the sub-categories they link out to and boost (not ideal but an issue I will address later). What I wondered is if having the inpage achor would confuse the SERPS because they would be linking with the same keyword. My worry is that by increasing userbility of the article by doing this I also confuse them SERPS First I tell them that this section on my page talk about keyword 1. Then from in that article i tell them that a different page entirely is about the same keyword. Would linking like this confuse SERPS or are inpage anchor links looked upon and dealt with differently?0 -
Been away for a while is SEO really dead ? I don't think so...
I have been struggling with the google updates but recently we started a new project and by using guest blog posts we were able to achieve a top 3 ranking. It delivered traffic and sales so SEO still works. This is my understanding of the current situation - 1. Generic Keywords (forget it) 2. Go niche and long tail (but thats been the case for a while right) 3. Using related searches 4. Incoming links using brands and a wider range of phrases and urls. 5. Content thats sharable 6. Google plus buttons etc This is my current understanding I would love to hear your thoughts.
Intermediate & Advanced SEO | | onlinemediadirect0 -
Preferred domain can't set in Web master Tool
I have put my domain name as xxxxxtours.com without www in web master tool. i have redirect to www version using htaccess file .So I wanna put Preferred domain "Display urls as www.xxxxtours.com .When trying it give error as attached image.but i have verified site the .waiting for expert help . Ar5qx.png
Intermediate & Advanced SEO | | innofidelity0