How do I troubleshoot a dateparsefailure when using Logstash's date plugin?


  • QA Engineer

    I'm getting a date parse failure when trying to parse a log that has the following format:

    172.20.21.10 - - [29/Mar/2022:12:41:27 -0400] "some invalid request" 400 226
    

    The event structure looks like this:

    {
            "request" => "/um/",
               "auth" => "-",
              "ident" => "-",
          "@metadata" => {
            "path" => "C:/net-logs/m4-httpd-logs/wordpress_access.log-20220403",
            "host" => "M6"
        },
               "verb" => "GET",
            "message" => "172.20.21.10 - - [29/Mar/2022:12:44:36 -0400] \"GET /um/ HTTP/1.1\" 404 17787",
               "path" => "C:/net-logs/m4-httpd-logs/wordpress_access.log-20220403",
              "bytes" => "17787",
           "response" => "404",
           "clientip" => "172.20.21.10",
           "@version" => "1",
               "host" => "M6",
        "httpversion" => "1.1",
          "timestamp" => "29/Mar/2022:12:44:36 -0400"
    }
    

    To convert the timestamp field from text to a date I have a filter in the logstash configuration file that looks like this:

    filter {
            grok {
                  match => {"message" => "%{COMMONAPACHELOG}"}
            }
            mutate {
                    remove_field => [ "@timestamp"]
            }
            date {
                  match => ["timestamp", "dd/MM/yyyy:HH:mm:ss Z"]
                  target => ["timestamp"]
            }
    }
    

    I have used the correct syntax for the format according to the https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html for the date filter. I'm not sure whats causing the parse failure since the error msg does not show exactly why the parse failed. It looks like this:

    {
            "request" => "/wp-includes/IXR/login",
               "auth" => "-",
              "ident" => "-",
          "@metadata" => {
            "path" => "C:/net-logs/m4-httpd-logs/wordpress_access.log-20220403",
            "host" => "M6"
        },
               "verb" => "GET",
            "message" => "172.20.21.10 - - [29/Mar/2022:12:47:57 -0400] \"GET /wp-includes/IXR/login HTTP/1.1\" 404 17787",
               "tags" => [
            [0] "_dateparsefailure"
        ],
               "path" => "C:/net-logs/m4-httpd-logs/wordpress_access.log-20220403",
           "response" => "404",
              "bytes" => "17787",
           "clientip" => "172.20.21.10",
           "@version" => "1",
               "host" => "M6",
        "httpversion" => "1.1",
          "timestamp" => "29/Mar/2022:12:47:57 -0400"
    }
    


  • Fixed by creating a mapping for the timestamp field:

    {
      "index_patterns": "httpd-*",
      "mappings": {
        "dynamic": true,
        "properties": {
            "request": {
              "type": "text"
            },
            "auth": {
              "type": "text"
            },
            "ident": {
              "type": "text"
            },
            "verb": {
              "type": "text"
            },
            "response": {
              "type": "text"
            },
            "bytes": {
              "type": "integer"
            },
            "clientip": {
              "type": "ip"
            },
            "httpversion": {
              "type": "text"
            },
            "timestamp": {
              "type": "date",
              "format": "dd/MMM/yyyy:HH:mm:ss Z"
            }
        }
      }
    

Log in to reply
 


Suggested Topics

  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2