How to detect search engine crawlers?

Today I was looking for a solution how to detect when a client is a search engine crawler, you can create a fancy solution for this, but in the .NET framework their is already a solution to detect search engine crawler. The property  Request.Browser.Crawler . If you use this property you always get false even if the site is visited by a search engine crawler, that's because it's not configured in a default installation of .NET. 

    In VB.NET:
    
    ''' <summary>
    ''' Check if request is a crawler
    ''' </summary>
    ''' <returns>true if crawler hit</returns>
    Public Shared Function isCrawler() As Boolean
        Dim clientBrowserCaps As System.Web.HttpBrowserCapabilities = System.Web.HttpContext.Current.Request.Browser
        ' Depends on configuration in web.config browserCaps section:
        If DirectCast(clientBrowserCaps, System.Web.Configuration.HttpCapabilitiesBase).Crawler Then
            trace.Write("isCrawler", "Called from crawler: " & System.Web.HttpContext.Current.Request.Browser.ToString, className)
            Return True
        Else
            Dim agent As String = System.Web.HttpContext.Current.Request.ServerVariables("HTTP_USER_AGENT")
            If agent.Contains("Googlebot") Then
                trace.Write("isCrawler", "Google bot detected: " & agent, className)
                Return True
            ElseIf agent.Contains("msnbot") Then
                trace.Write("isCrawler", "MSN bot detected: " & agent, className)
                Return True
            ElseIf agent.Contains("Yahoo") Or agent.Contains("Slurp") Then
                trace.Write("isCrawler", "Yahoo bot detected: " & agent, className)
                Return True
            ElseIf agent.Contains("Mercator") Then
                trace.Write("isCrawler", "Altavista bot detected: " & agent, className)
                Return True
            ElseIf agent.Contains("Baiduspider") Then
                trace.Write("isCrawler", "Baidu bot detected: " & agent, className)
                Return True
            ElseIf agent.Contains("ArchitextSpider") Then
                trace.Write("isCrawler", "Exite bot detected: " & agent, className)
                Return True
            ElseIf agent.Contains("Lycos_Spider") Then
                trace.Write("isCrawler", "Lycos bot detected: " & agent, className)
                Return True
            ElseIf agent.Contains("Ask Jeeves") Then
                trace.Write("isCrawler", "Ask Jeeves bot detected: " & agent, className)
                Return True
            ElseIf agent.Contains(".ibm.com") Then
                trace.Write("isCrawler", "IBM bot detected: " & agent, className)
                Return True
            End If
        End If
        Return False
    End Function

ASP.NET uses the <browsercaps> section in machine.config or web.config to determine the client browser is a crawler or not. In the default installation the crawler filter information is all blank,  that's why you'd always get false. To fix this problem, you should add the search engine crawler filters in the <browsercaps> and add this section to your web.config. Like this:

Tip: you can find more crawler info in your IIS logs ([Windows Folder]\system32\LogFiles)