Loki: Extracting data from unstructured logs

Data extraction

Grafana/Loki give you the possibility at hand to extract data from your log lines and create valuable panels and/or alerts even when the log is poluted with stack traces and/or unstructured in other ways.

The key is LogQL. Select the labels that where created on ingestion and extract the data. May it by JSON or log_format.

Example

The following is an extract from a Wakfu application log, but could be logging from any other application.

 INFO 09:06:11,636 [AWT-EventQueue-0] (MF:142) - [Error message] The targeted resource is too far away.
 INFO 09:06:49,320 [AWT-EventQueue-0] (Ud:457) - Invalid start cell (-11, -359, 0) for pathfind search : not a suitable position for the mover. 
 INFO 09:07:33,882 [AWT-EventQueue-0] (MF:142) - [Game Log] You have picked up 12x Divine Souper-Glou.
 INFO 09:07:33,883 [AWT-EventQueue-0] (MF:142) - [Game Log] Dajum: +167,365,568 XP points.  Next level in: 1,716,607,661.
 INFO 09:07:33,980 [AWT-EventQueue-0] (MF:142) - [Game Log] You have won the quest "Collaborative: Gobgobstinate."
 INFO 09:07:33,991 [AWT-EventQueue-0] (MF:142) - [Game Log] You have picked up 12x Divine Souper-Glou.
 WARN 09:07:44,992 [AWT-EventQueue-0] (atp:421) - [Animation] Not found (jar:file:./contents/animations/npcs/npcs.jar!/1300071) 0_AnimStatique
atF: [Animation] Not found (jar:file:./contents/animations/npcs/npcs.jar!/1300071) 0_AnimStatique
	at atp.a(SourceFile:421)
	at aaT.a(SourceFile:1125)
	at aaT.a(SourceFile:1039)
	at aaY.a(SourceFile:109)
	at afb.a(SourceFile:253)
	at java.util.ArrayList.forEach(ArrayList.java:1257)
	at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1082)

As you may be only interested in XP points gain for a given char, a LogQL query to get the cummulated sum would look like:

sum(
  sum_over_time(
    {job="wakfu"}
    |= " INFO"
    |= "[Game Log] Dajum" 
    |= " XP points.  Next level"
    | pattern " INFO <_> [<_>] (<_>) - [<_>] <char>: +<xp> XP <_>" 
    | label_format xp=`{{.xp | replace "," ""}}`
    | unwrap xp [$__range]
  )
)

Explaination of the LogQL statements

1. {job="wakfu"}

   All logging with label {job="wakfu"} is taken into account. This is the so named Log stream selection.

2. |= "..."

   Only log lines that match a given filter expression are passed to the next pipeline stage. There are contains, does not contain expressions. Literal and regular expression. These are Line filter expressions.

3. | pattern " INFO <_> ...

   Parser expression can parse and extract labels from the log content. Those extracted labels can then be used for filtering using label filter expressions or for metric aggregations.

   Here we extract the labels char and xp from the log line.

4. | label_format xp='{{.xp ...

   The | label_format expression can rename, modify or add labels. It takes as parameter a comma separated list of equality operations, enabling multiple operations at once.

   As e.g. 167,365,568 is not a number for the included ‘,’ this non- numerical char needs to be removed from the labels value.

5. | unwrap xp

   The unwrap function exposes the value of a label to be used for aggregations. In the above example the values are summed over time.