Donate

Garbage Collection GC.Collect() Method In C# Slowing Web Crawlers Speed

In our web crawlers, we encountered slow crawling of items specially XML feeds. We checked our codes if there are lines generating bottlenecks.

It turned out that we placed GC.Collect() in an inner loop that's processing item per item.

The solution is to comment out GC.Collect() in the inner loop and transfer it to the outer loop statement. Here's the code:
while(!EndOfPage)  
  {  
    do  
    {  
      if(date != String.empty)  
      {  
       //comment this statement, transfer outside inner loop...  
       //GC.Collect();  
       //processing of items code statements...      
      }  
    } while (!(url.Equals(String.Empty)));  
    GC.Collect();  
    PageNo++;  
    if(PageNo > totalPage)   
    {   
      EndOfPage = true;   
    }  
  }
Credits to my fellow developer for the discovery.

Greg

Comments

Donate

Popular Posts From This Blog

WPF CRUD Application Using DataGrid, MVVM Pattern, Entity Framework, And C#.NET

How To Insert Or Add Emojis In Microsoft Teams Status Message

Bootstrap Modal In ASP.NET MVC With CRUD Operations