How I fixed Crawl Issues on a 101,356 URLs Website!

Page Indexing Issues
Server Connectivity Issues
Updates Not Visible
Sitemap Issues
Daily Ranking Drops
No New Keyboards

Managing a large eLearning website comes with unique challenges, especially when it has over 100,000 URLs. Recently, we faced a significant issue: search engines weren’t crawling our eLearning website pages effectively, causing us to lose ranking, visibility, and opportunities to reach our audience. Here’s how we identified the issues, tackled the obstacles, and achieved a solid solution that transformed our site’s crawling and indexing performance.

Problems We Faced

♦ Google Wasn’t Showing Our Latest Updates

Despite frequently updating our eLearning content, Courses, Google Search wasn’t reflecting the changes. This impacted the freshness of our content, leading to a drop in relevance.

♦ New Pages and Blogs Weren’t Getting Indexed

New pages, especially eLearning blog posts, weren’t being indexed as quickly as we’d expected. Without indexing, these pages couldn’t appear in search results, affecting our traffic and reach potential learners.

♦ Losing Search Rankings Every Day

Without consistent crawling and indexing, our search engine rankings dropped. As a result, our site was losing visibility, and our daily organic traffic took a hit.

What We Discovered

♦ Server Struggling with Load

Our server was struggling to handle search engine bots, especially Googlebot. This led to numerous failed crawls, where bots tried to access a page but couldn’t load it.

♦ Sitemaps Filled with Unnecessary Pages

Our eLearning websites sitemaps were cluttered with URLs that didn’t add value to search engines or users. This included attachments, units, tags, feeds, author pages, and more. By including these, we were wasting crawl budget on pages that weren’t important for SEO, rather than focusing on high-valued online course pages.

Immediate Solution Considered

Our first thought was a server upgrade. With over 100,000 URLs, our hosting plan was under pressure to keep up with search engine robots’ requests. Changing to a higher-capacity hosting provider seemed like a good option, but it was expensive and would take time to implement.

After weighing the costs and considering the time it would take to migrate and test everything, we decided to put the hosting upgrade on hold and look for alternative solutions that could optimize our current setup.

I am available for Remote or Freelancing SEO projects

Ready to maximize your online presence?

Let’s grow your website’s visibility and attract more clients with expert SEO strategies.

The Real Solution

Instead of making drastic changes, we decided to focus on optimizing our existing setup. We tackled the issues by refining our SEO settings, adjusting robots.txt rules, and managing our hosting cache. Here’s a breakdown of the steps we took:

♦ Optimizing Yoast SEO Settings

We reviewed our Yoast SEO settings and realized we weren’t using certain features efficiently. By turning off specific settings, we reduced the clutter in our sitemaps. Here’s what we disabled:

Units, Tags, and Feeds: These were unnecessary and only added noise to our eLearning website.
Authors and Members: Not relevant to our goals, and their indexing wasn’t beneficial.
Shortlinks and Numbers: Not adding value to our eLearning SEO, so we turned these off.

♦ Creating a New Robots.txt File

We updated our robots.txt file to prevent search engines from crawling unimportant URLs. Blocking these URLs saved crawl budget for more critical pages. Our new robots.txt file was simple but effective, explicitly directing bots to prioritize valuable content.

♦ Clearing Hosting Cache Every Hour

To prevent stale data from slowing down our site, we set up an automated system to clear our hosting cache every hour. This reduced the likelihood of slow load times and ensured that both users and bots could access the latest version of each page.

I would like to thank veteran Mr. Sardar Monsur Ahmad, the marketing and sales genius,

for his valuable guidance in solving our crawl issues.

The Results We Got

♦ Google Bots Now Crawl Our Site Daily

Through managing server load and optimizing accessible URLs, we achieved consistent daily crawls from Google bots, forming the foundation for better visibility and indexing.

♦ Updates Appear in Google Within 1 Hour

Previously, it took days for updates to appear in search results. Now, we’re seeing our changes reflected on Google within an hour of publishing, thanks to efficient crawling.

♦ New Blogs Indexed Within 6 Hours

New content is essential for eLearning SEO growth. After our updates, new blog posts were indexed within 6 hours, significantly boosting their visibility and relevance in search results.

♦ Sitemap Now Focused on Quality Pages

Removing attachments, units, tags, and other non-essential URLs from our sitemaps reduced clutter. Now, Googlebot is focusing its efforts on the pages that matter most, making each crawl count.

♦ Stable Server Performance

With optimized SEO settings and a leaner sitemap, our server load stabilized. We experienced a brief spike during initial testing, but afterward, our server performed smoothly even with frequent crawling.

♦ New Keywords Picking Up on Google

Google improved its crawling and indexing, which helped link our online learning content to more relevant keywords. As a result, our organic search visibility increased, bringing in more traffic from a variety of search terms.

♦ Clicks and Impressions Steadily Rising

The combination of consistent crawling, indexing, and new keyword associations translated to a steady rise in clicks and impressions. Our eLearning websites search rankings improved, and we’re now seeing a more predictable growth in organic traffic.

SEO Projects Done

0 +

Satisfied Clients

0 +

Running Projects

0 +

Years of Experience

0 +

Cups of Coffee

0 +

Clients Review

“Working with Mamunul Islam has been a fantastic experience. His deep understanding of SEO transformed our digital strategy, leading to remarkable growth in our search rankings. He is A true professional”

Stuart Fern

One Education

“Mamunul has done a fantastic job boosting traffic and increasing housekeeping customers, not just in San Francisco but nationwide. His efforts have truly made a difference!”

Vanessa Bossart

GreenTerra Cleaning Service