Log Search


Logentries provides intuitive log search that can be as simple as Google and as powerful as you need. Use search to retrieve log entries using keywords, phrases, regular expressions, logical expression, and field-level comparisons.

On this page there are several examples of searches that you can use and reapply to your specific search requirements. The basic format of a search consists of terms you are looking for inside of your log entries. These can include terms, strings, regular expressions, regular expression with advanced field extraction, boolean, comparison and logical expressions Use search to retrieve events using keywords, phrases, regular expressions, and field-level comparisons.

Quick Examples

In order to illustrate the search examples below, we’ll use the following sample log entries.

2015-01-20T11:21:14+00:00 WG4333_server Kurt Smith ip=123.34.53.111 “request”:”GET” response=200 “user”:”Turing Gödel” level=4 total=500 cash=250 “credit”:”250″

2015-01-20T11:21:14+00:00 WG4333_server Dave Moffit ip=127.0.0.1 “request”:”GET” response=200 “user”:”Turing Gödel” level=4 total=500 cash=250 “credit”:”250″

2015-01-20T11:21:14+00:00 WG4333_server Tom Wilson ip=127.0.0.1 “request”:”GET” response=200 “user”:”Turing Gödel” level=2 total=500 cash=250 “credit”:”250″


Example Search Terms Description
Turing Gödel Search for log entries that contains words Turing and Gödel
“Kurt Gödel” Search for log entries that contains the phrase Kurt Gödel
(GET OR POST) AND (response=200)Search for log entries that contains GET or POST commands with responses of 200 
/[^ ]*server/ Uses Regular Expressions (RE2) to search for a the server name, case insensitive and ungreedy
 response calculate(COUNT) Search for response, and calculates the COUNT.  In this example this will be 2 since there is only one log entry.
total calculate(SUM:total) Search for the word ‘total’ and calculates the Sum based on the key total.  In this example SUM will = 1500.  The format here will be: <search term> calculate(SUM:<key>)
“Kurt Smith” groupby(level) calculate(COUNT) Search for events from a database, where database size is greater than 1MB
source=db AND measure.db_size>=1000000 Search for events from a database, where database size is greater than 1MB

NOTE: Structuring your logs to utilize Key Value Pairs can be expressed with an equals sign (key=value  – no spaces) or using a colon “:” as in “Key”:”Value” (“Key” colon “Value” – no spaces)



Format of a Search

Knowing how to best format a search query will allow you to retrieve the best search results from your log entries.

A search query can consist of a combination search terms, regular expressions, logical expressions, Boolean expressions and functions.

The format of a very simple search takes the form of: <search term> or <search term> <function> 
For instance, a simple search for returning all of the log entries that have the word “server” in them would look like this:

or a simple search returning a count of all of the log entries would that have the word “server”  in them may look like this:

where “server” is the search term and “calculate(COUNT)” is the function.

A more advanced search query looking for Key Value Pairs (KVPs) and normal search of strings, and then calling one or more functions may take the form of:
<search term> <logical expression> <another search term> <function> as in:
ip=127.0.0.1 AND server=001 calculate(COUNT)

where the first search term is “ip=127.0.0.1″ the logical expression is “AND” the second search term is “server=001″ and the function is “calculate(COUNT).

Another more advanced search might take the form of: ip=127.0.0.1 AND server=001 Groupby(response) calculate(COUNT)

QUICK SUMMARY:
A Search term is the string, regular expression, or series words you are looking to find within your log entries or log events.

A Logical Expression is part of your search criteria to allow further refinement of your result set.  Logical expressions used within Logentries consist of AND, OR, or NOT.
Note: Be sure to write all Logical Expressions in uppercase.

A function can be a Groupby() or calculate(COUNT), calculate(SUM), calculate(AVERAGE), calculate(SUM:key) calculate(AVERAGE:key), calculate(UNIQUE:YOUR_KVP) or a combination of both Groupby() and caluculate().

A Regular Expression is a sequence of symbols and characters expressing a string or pattern to be searched for within your log entries / log events.



Log Search Functions

Count

Log search also supports returning a count of matched search results. Append calculate(COUNT) to your search query or press the {+} button (as seen below) to get the number of search results.


Sum

You may use the sum modifier to total the values of your name value pairs. Using the previous screenshot as an example, to get the sum of your sales with a total value of more than 0 search for total>0 calculate(SUM).



Average

The average modifier works the same as sum but it computes the mean of the values matching the search criteria. For instance,  to get the average value of your sales, you might invoke a search like total>0 calculate(AVERAGE).


Unique

The UNIQUE keyword returns the number of unique values for a given key. It takes one parameter: the name of the key. For example, to get the number of unique IP addresses, you might execute a query like /.*/ calculate(UNIQUE:YOUR_KVP)


Grouping

You can perform search functions based on an approximation of the most commonly occurring entries over a certain time period. For example, to get the average response times grouped by source IP address, you might execute a query like fwd!=null groupby(fwd) calcualate(AVERAGE:service), or /.*/ groupby(fwd) calculate(COUNT) to get a ranking of most frequent IP addresses.

All groupby() functions must have a calculate() function after them to work correctly.  This can be calculateCOUNT), calculate(AVERAGE) or calculate(SUM) depending upon your requirement.







Saved Searches

Logentries allows searches to also be saved. You will now see a star and a drop-down symbol to the right of the search box when searching logs.total_average_search_bar In order to save your current log search all you have to do is click on the star. Now if you or any other user in the same account clicks on the dropdown symbol then all the previously saved searches will be displayed. It’s also possible to edit a saved log search in order to give it a name or change the query. Just click on the ‘edit query’ link and you’ll be shown the modal below.



Keyword Expressions

Search keywords to find simple word occurrences in your logs. A keyword is a simple word contained anywhere in the event. A keyword is separated from other keywords via white space or special characters. For example, the event Georg Cantor set_theory contains keywords Georg, Cantor, and set_theory. If you specify multiple keywords, you are searching for events that contains all specified keywords. For example, the search term Cantor Georg will match both “Georg Cantor set theory” as well as “Georg Ferdinand Ludwig Philipp Cantor”.


Logical Expressions

Logentries extends the basic query language with standard boolean operators AND, OR, and NOT. The order of operations is first do any operations in parenthesis, then NOT, then AND, and lastly OR. The operator AND is implicit, i.e. it is assumed if not specified.
Example Matches
Turing AND Gödel Events that contains both Turing and Gödel
Turing Gödel Events that contains both Turing and Gödel, AND is implicit here
Turing OR Gödel Events that contains Turing or Gödel
Turing NOT Gödel Events that contains Turing, but not Gödel
Turing -Gödel Events that contains Turing, but not Gödel, – is a shortcut for NOT
Turing AND (Gödel OR Cantor) Events that contains Turing and one of Gödel or Cantor
Note that logical operators must be capitalized. If you want to search for keywords with the same name as logical operators, quote them in single or double quotes such as “AND” or ‘NOT’.


Regular Expressions

Logentries supports regular expressions enabling you to match events using complex regular language. Regular expressions must be enclosed with slashes / and can include optional flags such as “i”.
Example Matches
/Null/ Events that contains Null, such as NullPointerException
/error/i Events that contains error, case insensitive, such as Error, ERROR
/Exception “.*” at/ Events that contains exception trace with a name
/20[01]/ Events that contains 200 or 201


Regular Expression Field Extraction

If your logs do not contain any Key Value Pairs (KVP), you can create Key Value Pair using Regular Expression Field Extraction.
A typical use for this would be extracting values from Access logs or other logs that do not contain Key Value Pairs.
Regex grouping and naming allows you to identify values in your log events and give these values a name, similar to having a Key value pair in your log events. You can then use this named group to perform more complex search functions.



Comparison Expressions

You can use comparison operators to match a value or a range of values. Comparison expression takes the form of “name?value” where name defines field, ? is an operator, and value represents the value of the comparison. Comparison expressions will only return results with properly formatted fields and values. Please see sections Fields and Escape Characters to learn how fields and values are defined and how they must be formatted to be properly recognized by Logentries.
Example Matches when
code=200 Status code is 200 OK
db.size>1000000 Database size is greater than 1MB
time>=6.7 Operation took longer than 6.7
response!=success Response was not successful

Operators

Operator Example Description
= field=value Field’s value matches the value given
= field=/regexp/ Field’s value matches the regular expression
!= field!=value Field’s value is not the value given
!= field!=/regexp/ Field’s value does not match the regular expression
< field<num Field’s value is less than the number given
<= field<=num Field’s value is less or equal to the number given
> field>num Field’s value is greater than the number given
>= field>=num Field’s value is greater or equal to the number given
Numerical values must be formatted as an integer, floating-point value, or in the scientific notation to be properly recognized by Logentries. Units in field’s value (such as bytes in “842bytes”) are ignored for the purpose of comparison with the exception of equal operator which is string-only. Note that if the field is not present (defined) in the log entry then no operator will match.


Syntax

Keywords

Keywords are words consisting of letters, digits, dots, and underscores. Keywords are stored in indexes which allow the search engine to retrieve results faster. For example the following event:
2013-06-22T11:21:14+00:00 Template rendering complete, render_time=0.2ms
contains information about template rendering which completed in 0.2 milliseconds. This events contains five keywords: Template, rendering, complete, render_time, and 0.2ms.


Fields

Fields are key-value pairs of log entry data. Every field has a name which uniquely identifies type/purpose of the value. Logentries extracts key-value pairs from incoming data automatically. Events need to be of the classical form:
field1=value1 field2=value2 field3=value3
alternatively, you can also separate field-values with comma:
field1=value1, field2=value2, field3=value3
Additionally, Logentries recognizes other formats such as JSON, Ruby, and Perl dictionaries. For example, Logentries will automatically detect fields and values formatted the following ways:
{"version":"1.23"}, "response_time"=>1.23
The following Example:
2013-06-28T10:29:28+00:00 source=postgresql tables=81 db_size=6774660014bytes index-cache-hit-rate=0.99996 avg_response=6.25e-3
represents a single event from the database, defining source as postgresql, the number of tables “tables” as an integer 81, database size “db_size” as an integer with units “6774660014bytes”, cache rate “index-cache-hit-rate” as a float 0.9996, and average response time avg_response as a float in scientific notation 6.25e-3. Logentries will automatically detect these fields and values.


Escape Characters

Field names and values may contain any characters such as letters, digits, dots, or underscores. Field names and values containing spaces or words that conflict with search operators must be treated specially. For example, terms like db_size and measure.cpu_time are correct field names and values. To include non-standard characters in field’s name or value such as spaces, quotes, operators, commas, brackets, or words similar to operators you need to escape them with a slash \ or enclose the whole term in single or double quotes. Examples on how to format field names and values with non standard characters:
  • Values which may contain spaces: name=”Noam Chomsky” or name=Noam\ Chomsky
  • Fields that contain spaces: ‘full name’=’Noam Chomsky’
  • Logical operators: find ‘OR’


Regular Expression Operators

Regular expressions use special characters to enable searching for more advanced patterns. These characters are *, +, ., \, [, ], (, ), {, } If you need to use special characters as ordinary characters, you have to escape them with a backward slash \.

Match something a number of times

Name Example Description
Any a* Star will match zero or more of the previous character.
At least one a+ Plus matches at least one repetition of the previous character.
Exactly a{x} Matches the exact number of the previous character.
From, to a{x,y} Matches the number of the previous character within the range.
Up to a{,y} Matches up to the limit of the previous character.
At least a{x,} Matches at least the limit of the previous character.

Match character set

Name Example Description
Any character . Dot matches any single character.
Any digit \d Matches a digit character, that is 0-9.
Any whitespace \s Matches any whitespace character.
Anything but a digit \D Matches any character that is not a digit.
Anything but a whitespace \S Matches any character except for whitespace.
Given set [abc] Matches any of the characters specified.
Anything but the given set [^abc] Matches any character except for those specified.


Flags

Flags change default behavior of regular expression matching. Flags should be specified at the end of the regular expression, after the closing slash /. This syntax is similar to Perl and other languages.
Case-insensitive search i
By default search with regular expressions is case sensitive. To disable case sensitivity you have to enclose the regular expression in forward slashes and specify the i option at the end.
Multiline m
If you want to match start-end special characters ^ and $ on any lines in the event, use the multiline flag. By Default these special characters matched start and end of the event only.
New lines s
If you want to match across new lines in the event, use the New lines flag. It causes the special character . to match new lines \n by default.
Ungreedy U
By default all universal quantifiers are greedy, i.e. it tries to match as much characters as possible. If you prefer an ungreedy behavior instead, use the Ungreedy flag.

Examples

Pattern Description
ab*c Matches strings ac, abc, abbc.
ab+c Matches strings abc, abbc, but not ac.
ab{2}c Matches abbc, but not abc or abbbc.
ab{1,3}c Matches abc, abbc, and abbbc.
ab{,2}c Matches ac, abc, and abbc.
ab{2,}c Matches abbc, abbbc, but not abc.
a.c Matches strings abc, acc, adc, but not ac.
a\d Matches a0, a1, a2.
a\sb Matches a b.
a\D Matches strings ab, ac, but not a0.
a\Sc Matches strings abc, a0c, but not a c.
/completed/i Matches strings completed as well as Completed, compLeted, and ComplETED.
Back to Top