Good input. I wouldn’t say its the most important… if you don’t link to a page or include it in a sitemap, it won’t get crawled, but you’re certainly onto something.

i had issues recently where more than half of my client’s pages were crawled, but not index. You can guess: thin, lack-of-effort content .

I’ve been spending a lot of time in the new search console. What are your suggestions for pages that are being crawled, but not indexed?

I’ve seen a large increase in this across several websites over the past 90-120 days and I think this is tied into some of Google’s recent algo changes.

I’m wondering if the best approach is to delete these from the website if they were designed as supplemental pages (blogs, newsletters) and are not the core keyword targets?

Hey Alex,

Great observations for sure. Hopefully you got it all fixed up and ranking.

1) speed, but not only server speed but fast loading in general. Especially make images as small as possible. Otherwise they will eat a lot of your crawling capacity. I could see in the search console that after I reduced the average page size the “Kilobytes downloaded per day” in the search console essentially stayed the same (same crawling budget) but the “Pages crawled per day” went up as each page had gotten smaller

2) when I enhanced usability of the site and in general increased the value this site brought on the table for users I could see in analytics that the average time spent on site by USERs went up. At the same time also the number of pages indexed went up. Although there is certainly no simple correlation it is probably safe to say that the more users like your site the more pages google will be willing to crawl and index

3) strong internal linking. Internal linking is even more important than a sitemap.

Tests:

a) new domain with 10,000 pages having a sitemap but no internal linking
b) new domain with 10,000 pages having no sitemap but 1-5 internal links per page

I performed this test several times. In ALL cases b) outperformed a)

4) Rowan mentioned backlinks which is very important for sure. Also an expired domain with strong backlinks coming in will get its pages getting indexed much faster at the start than a new domain as of my findings

Tnx. Will have to take some time to do this.

Great post, I have learned too much today.

Just a bit confused about the crawl budget, will research on this for sure.

Yoast has fixed the issue with the latest version.

Hey Harold,

I think John Mueller is being very transparent on this matter. In the article I mention that people who have crawl budget problems often have site problems. Fixing the root of the problem will ultimately fix the crawl problem.

If this is because your content is low quality, then fixing this will be a bigger help.

If you lack links or have too many links, then fixing this will also have a positive impact. It’s about finding the right balances between these signals.

My personal approach is that crawl budget doesn’t really exist in practice, but only in principle. It’s interesting conceptually, but the main thing is to improve value on the core pages that you want users to engage with.

Hey Terry,

I tackle each problem at the root, and look for whatever is going to achieve the best results with the minimal input and collateral.

Sometimes I’ll use a plugin, other times I will edit the php or liquid files directly. It depends largely on which platform you’re using and the capabilities of it.

Hey Kris,

I personally use several different page speed tools, and load on 4G through my mobile phone.

I’m mostly looking for site speed differences between regions, the client’s target location, and whether site speed is likely a problem for their website.

If your website is consistently performing worse from the United States than any other region; you may wish to tackle this and see if it brings any uplift. Some people will benefit more than others.

In regards to Content Distribution Networks, I work with clients that use tons of different providers. I’ve not seen that one outperforms others from an SEO perspective, but you should definitely look into where the nodes are located and if this aligns with your goals for the CDN.

Remember that pricing and implementation may depend on each website, so this will also be a factor.

Hey Steev,

The Yoast automatic sitemap gets the job done, but it doesn’t really hold any advantages over a simplified sitemap.

This will depend largely on your website. The bigger it is, the more benefit you’re going to have from compartmentalised sitemaps.

The main advantage is that it’s done server side and will adapt based on your page and posts. If you go down the road of using Screaming Frog, then you will need to do it manually whenever you make changes.

Easier to use a plugin. Better to do it from the file level.

I am just wondering whether it’s better/more efficient to control index and crawl from robot.txt/htaccess level, or just us a plugin like Yoast to control the pages manually.

You can use the robots.txt file’s wilkdcard feature to handle many at a time. Or use the redirection plugin if you have a few hours available in a bored afternoon.

As far as distance from Google servers, if your website loads quickly otherwise but shows as slower in Google page speed, would this be an effective measure of being further away?

Is MaxCDN your preferred content distribution network? Do you consider it best practice to have a CDN anyways?

Thank You!

How do I solve this part or do I have to do redirects for all those URLs (there is over 200 of them)?

Thanks
B

Thank you very much for this insightful post.

One other thing that affects crawling and indexing as you rightly said is the quality of the content on the page.

If the contents doesn’t seem to provide value perhaps because they are spun, Google might crawl it but won’t index it