You can send your logs to Logentries in a number of different ways. You can use syslog forwarding, the Logentries Agent or one of our platform or language plugins.

The Logentries Agent is a simple program that provides a range of useful features. It is opensource so you can see exactly how it works and modify it to suit your needs if required.


Navigate Your Account

Navigate your Logentries account on the command line with the list ls command.
List your hosts, logs, and their attributes.

List your hosts:

le ls hosts

List logs of the host “web”:

le ls hosts/web/

Pull Logs

Export your logs from your Logentries account to your local file system using the pull command.
The pull command enables you to select a time range, a filter, and set a limit to the number of pulled entries.

Pull all GET entries from web server dated April 15th

le pull hosts/web/access.log 'Apr 15' 'GET'

Pull up to 10 entries containing numbered image dated 15th to 17th of April:

le pull hosts/web/access.log 'Apr 15 -> Apr 17' '/\d\.png' 10

 


Resource Usage Information

The agent collects resource usage information from Linux servers when you run the monitor command.

Resource usage information is useful to identify system problems and serves as an extra correlation point for root cause analysis.

Resources


Filtering Sensitive Data

We do not recommend storing user sensitive data in log events. However, we understand that this may be unavoidable from time to time.

The Logentries agent enables you to add filters to logs being sent to Logentries servers so that no sensitive information is stored in your logs at Logentries.

Using regular expressions, patterns (e.g. credit card details) can be identified and removed before they leave your network.

Specify a Python module directory in your configuration (config) by adding a like in the form of::

filters=/opt/le/le_filters

Create empty __init__.py to set up a module. Then add filters.py file which contains filters dictionary. The dictionary informs the agent that for the given log name or UUID the specified filtering function should be used. For example the following dictionary:

filters={
	"example.log": filter_logname,
	"7e518e54-40e4-4c5a-88df-4559d03126e6": filter_loguuid,
}

Where filter_logname and filter_loguuid are functions which filters events for the respective log.

Filtering functions receive a single string containing log entries terminated with a new line. Function can modify lines in any way and return them back for sending to Logentries servers. Do not forget to keep new line termination. The following skeleton displays typical structure of the filtering function:

def filter_example( events):
        # Split the block into individual log entries
        parts = events.split( '\n')[:-1]
        # Collect modified parts
        new_parts = []
        for entry in parts:
                # Do something with entry
                new_entry = entry # XXX
                # Append new entry
                new_parts.append( new_entry)
        # Return modified output
        return ''.join( x+'\n' for x in new_parts)

Typical filtering function is much simpler though. For example the following filtering function removes all occurrences of credit card numbers:

import re

# Credit card number matcher
CREDIT_CARD = re.compile( r'\d{4}-\d{4}-\d{4}-\d{4}')
# Credit card number replacement
CC_REPLACEMENT = 'xxxx-xxxx-xxxx-xxxx'  # '-'.join( ['x'*4]*4) if you prefer

def filter_credit_card( events):
        return CREDIT_CARD.sub( CC_REPLACEMENT, events)

Filtering File Names

If you want to explicitly restrict which files can the agent follow, create the filters module as described in the previous section and define the filter_filenames function.

The filter_filenames function accepts full path to a file which is about bo be followed. The function returns True if the file name is acceptable or False otherwise. The agent will ignore files which does not pass this test.

The following example defines filter which allows the agent to follow log files only:

def filter_filenames( filename):
	return filename.endswith( '.log')

Alternatively, the following example defines filter which denies to follow any file outside /var/log/ directory:

def filter_filenames( filename):
	return filename.startswith( '/var/log/')

Note the examples above do not take into account symbolic links.


Following Logs That Change Their Name

Due to rollover policies logs are often renamed using a sequential number or the current timestamp. Luckily the Logentries agent can handle this for you. The Logentries agent can be pointed at particular folders to gather any active logs from that directory or its subdirectories using wildcards in file names. For example, the following patterns can be used with the follow command to gather logs from the given directories:

/var/log/mysystem/mylog-*.log
C:\Web\logs\local*\*

Using wildcards when specifying the log to follow allows for situations where you need to follow the most recent log in a particular folder. The Logentries agent looks for any active log in the folder and will monitor the events in that log.


Installation

Install The Agent

Copy and paste the following snippet into your terminal to install the agent:

wget https://raw.github.com/logentries/le/master/install/linux/logentries_install.sh && sudo bash logentries_install.sh

Check the list of supported distros in the below.

Or, if you would rather install manually, have a look at these instructions

Select your distro below:

Run the following commands one by one in your terminal:

You will need to provide your Logentries credentials to link the agent to your account.

su -
echo 'deb http://rep.logentries.com/ wheezy main' > /etc/apt/sources.list.d/logentries.list
gpg --keyserver pgp.mit.edu --recv-keys C43C79AD && gpg -a --export C43C79AD | apt-key add -
apt-get update
apt-get install python-setproctitle logentries
le register
apt-get install logentries-daemon

Run the following commands one by one in your terminal:

You will need to provide your Logentries credentials to link the agent to your account.

su -
echo 'deb http://rep.logentries.com/ squeeze main' > /etc/apt/sources.list.d/logentries.list
gpg --keyserver pgp.mit.edu --recv-keys C43C79AD && gpg -a --export C43C79AD | apt-key add -
apt-get update
apt-get install python-setproctitle logentries
le register
apt-get install logentries-daemon

Run the following commands one by one in your terminal:

You will need to provide your Logentries credentials to link the agent to your account. (Note that you will get a `no ssl' warning.)

su -
echo 'deb http://rep.logentries.com/ lenny main' > /etc/apt/sources.list.d/logentries.list
gpg --keyserver pgp.mit.edu --recv-keys C43C79AD && gpg -a --export C43C79AD | apt-key add -
apt-get update
apt-get install logentries
le register
apt-get install logentries-daemon

Run the following commands one by one in your terminal:

You will need to provide your Logentries credentials to link the agent to your account.

sudo -sH
echo 'deb http://rep.logentries.com/ saucy main' > /etc/apt/sources.list.d/logentries.list
gpg --keyserver pgp.mit.edu --recv-keys C43C79AD && gpg -a --export C43C79AD | apt-key add -
apt-get update
apt-get install logentries
le register
apt-get install logentries-daemon

Run the following commands one by one in your terminal:

You will need to provide your Logentries credentials to link the agent to your account.

sudo -sH
echo 'deb http://rep.logentries.com/ raring main' > /etc/apt/sources.list.d/logentries.list
gpg --keyserver pgp.mit.edu --recv-keys C43C79AD && gpg -a --export C43C79AD | apt-key add -
apt-get update
apt-get install logentries
le register
apt-get install logentries-daemon

Run the following commands one by one in your terminal:

You will need to provide your Logentries credentials to link the agent to your account.

sudo -sH
echo 'deb http://rep.logentries.com/ quantal main' > /etc/apt/sources.list.d/logentries.list
gpg --keyserver pgp.mit.edu --recv-keys C43C79AD && gpg -a --export C43C79AD | apt-key add -
apt-get update
apt-get install python-setproctitle logentries
le register
apt-get install logentries-daemon

Run the following commands one by one in your terminal:

You will need to provide your Logentries credentials to link the agent to your account.

sudo -sH
echo 'deb http://rep.logentries.com/ precise main' > /etc/apt/sources.list.d/logentries.list
gpg --keyserver pgp.mit.edu --recv-keys C43C79AD && gpg -a --export C43C79AD | apt-key add -
apt-get update
apt-get install python-setproctitle logentries
le register
apt-get install logentries-daemon

Run the following commands one by one in your terminal:

You will need to provide your Logentries credentials to link the agent to your account.

sudo -sH
echo 'deb http://rep.logentries.com/ oneiric main' > /etc/apt/sources.list.d/logentries.list
gpg --keyserver pgp.mit.edu --recv-keys C43C79AD && gpg -a --export C43C79AD | apt-key add -
apt-get update
apt-get install python-setproctitle logentries
le register
apt-get install logentries-daemon

Run the following commands one by one in your terminal:

You will need to provide your Logentries credentials to link the agent to your account.

sudo -sH
echo 'deb http://rep.logentries.com/ natty main' > /etc/apt/sources.list.d/logentries.list
gpg --keyserver pgp.mit.edu --recv-keys C43C79AD && gpg -a --export C43C79AD | apt-key add -
apt-get update
apt-get install python-setproctitle logentries
le register
apt-get install logentries-daemon

Run the following commands one by one in your terminal:

You will need to provide your Logentries credentials to link the agent to your account.

sudo -sH
echo 'deb http://rep.logentries.com/ maverick main' > /etc/apt/sources.list.d/logentries.list
gpg --keyserver pgp.mit.edu --recv-keys C43C79AD && gpg -a --export C43C79AD | apt-key add -
apt-get update
apt-get install logentries
le register
apt-get install logentries-daemon

Run the following commands one by one in your terminal:

You will need to provide your Logentries credentials to link the agent to your account.

sudo -sH
echo 'deb http://rep.logentries.com/ lucid main' > /etc/apt/sources.list.d/logentries.list
gpg --keyserver pgp.mit.edu --recv-keys C43C79AD && gpg -a --export C43C79AD | apt-key add -
apt-get update
apt-get install logentries
le register
apt-get install logentries-daemon

Run the following commands one by one in your terminal:

You will need to provide your Logentries credentials to link the agent to your account.

sudo -sH
echo 'deb http://rep.logentries.com/ karmic main' > /etc/apt/sources.list.d/logentries.list
gpg --keyserver pgp.mit.edu --recv-keys C43C79AD && gpg -a --export C43C79AD | apt-key add -
apt-get update
apt-get install logentries
le register
apt-get install logentries-daemon

Run the following commands one by one in your terminal:

You will need to provide your Logentries credentials to link the agent to your account.

su -
tee /etc/yum.repos.d/logentries.repo <
    

Run the following commands one by one in your terminal:

You will need to provide your Logentries credentials to link the agent to your account.

su -
yum-config-manager --add-repo http://rep.logentries.com/helpers/fedora/logentries.repo
yum install logentries
le register
yum install logentries-daemon

Run the following commands one by one in your terminal:

You will need to provide your Logentries credentials to link the agent to your account.

su -
yum-config-manager --add-repo http://rep.logentries.com/helpers/fedora20/logentries.repo
yum install logentries
le register
yum install logentries-daemon

Run the following commands one by one in your terminal:

You will need to provide your Logentries credentials to link the agent to your account.

su -
tee /etc/yum.repos.d/logentries.repo <
    

Run the following commands one by one in your terminal:

You will need to provide your Logentries credentials to link the agent to your account.

su -
tee /etc/yum.repos.d/logentries.repo <
    

Run the following commands one by one in your terminal:

You will need to provide your Logentries credentials to link the agent to your account.

su -
tee /etc/yum.repos.d/logentries.repo <
    

Run the following commands one by one in your terminal:

You will need to provide your Logentries credentials to link the agent to your account.

sudo -s
tee /etc/yum.repos.d/logentries.repo <
    

First, you have to download Logentries Overlay.

Create a folder to store the package files:

X=/usr/portage/local/app-admin/le
mkdir -p $X/profiles
wget -r -np -nd  -A "*.ebuild","Manifest" -P $X \
http://rep.logentries.com/gentoo/portage/app-admin/le/
echo 'le' > $X/profiles/repo_name
      

You can also simply add the 'le' folder to an existing overlay, and install the package immediately from there.

Then, you have to add the overlay to Gentoo. Open /etc/make.conf and append the following:

PORTDIR_OVERLAY="${PORTDIR_OVERLAY} /usr/portage/local/app-admin/le"

And install the Logentries package:

emerge app-admin/le

This will install le and all of its components.

Running as a service

To start the service manually, enter as root:

/etc/init.d/logentries start

When you change the configuration, enter as root:

/etc/init.d/logentries restart

Upgrade

Download any newer version of the package:

cd /usr/portage/local/app-admin/le/
wget -r -np -nd -A "*.ebuild","Manifest" -P /usr/portage/local/app-admin/le/ \
http://rep.logentries.com/gentoo/portage/app-admin/le/

And update it:

emerge app-admin/le
Install the Logentries Agent for Windows:

Download and install the Logentries Windows Agent.

Install the Logentries Agent for OSX

Open a terminal in the folder you wish the to install the agent and paste the following snippet:

curl -O https://raw.github.com/logentries/le/master/install/mac/install.sh && sudo sh install.sh
Install the Logentries Agent for SmartOS

Paste the following snippet into your terminal to install the agent:

wget https://raw.github.com/logentries/le/master/install/smartos/install.sh --no-check-certificate && sudo sh install.sh

Current root certificates may not be available. Please check the checksum of the file. sha256 sum will return:

32c444800e4a4a1d769459b32cc66b686c5e761d3e6ab504b85dc9304502c1fb
Run the Logentries Agent from source

Run the following commands in a terminal to start the agent:

git clone git://github.com/logentries/le.git
cd le
./le register
./le monitor &

 

Uninstall the Agent

To remove the agent simply run the following commands.

sudo service logentries stop
sudo apt-get remove logentries-daemon
sudo le clean
sudo apt-get remove logentries

Select your log files

Select which log would you like to follow. For example, if you want to follow an Apache Access log file, run:

le follow /var/log/apache2/access.log --name Access

You can repeat the command for as many logs as you want. Note the –name flag is optional and is used to name the log in your logentries account.


Monitoring and Reloading Configuration

Note we have daemon support for Ubuntu, Debian and SmartOS. For other distributions the agent must be started manually: le monitor.

For Ubuntu and Debian:

service logentries start

For Joyent SmartOS, we use the Solaris service manager:

svcadm enable logentries

Later when you change the configuration, for example using the follow command to add new logs, give a notice to the agent. For Debian Squeeze and Ubuntu:

service logentries force-reload

For Debian Lenny run the init script explicitly:

/etc/init.d/logentries force-reload

For Joyent SmartOS:

svcadm restart logentries

Troubleshooting

Should you experience any difficulties with the agent, please follow these steps to troubleshoot.

Cannot register the machine

This is likely caused by firewall issues. The agent requires to open a connection to logentries.com and api.logentries.com on a secure port 443 or 80. Run the agent with the --debug argument to print the sequence of executed commends.

le --debug register

I don’t see any statistics displayed

Likely, the daemon is not running or there is a firewall on the way. Note that statistic collection works on Linux systems only.

To check that the process is running, run the following command:

ps aux|grep logentries-daemon

The running agent should be displayed. If not, the agent is not running and can be started running this command:

service logentries start

If the agent is running, make sure it can connect to Logentries servers. Firstly, stop the background daemon as root:

service logentries stop

Then, run the monitor command on the command line with debugging enabled. This is similar to daemon mode with a console attached:

le --debug monitor

Any issues will be displayed on console.

Logs are not sent to Logentries

Firstly, make sure that the daemon is running and you see statistics collected.

If you still don’t see any new log entries, make sure that the log file name has been entered correctly. To do that, list the log properties:

le ls hosts/HOSTNAME/LOGNAME/

Change HOSTNAME and LOGNAME to the host name and log name respectively. The output will look like this:

name = Auth
filename = /var/log/auth.log
key = cbe2d8ff-96d1-44e5-bba2-71fcdb162825
type = filename
follow = true

Check that the file with the name given really exists:

ls 'filename'

If you get an error message, the file name is incorrect. Otherwise check that new log entries appears in the log file via:

less 'filename'

and press F to display new lgo entries. If no new entries appear then there are no new entries to be sent to Logentries. Otherwise continue debugging the agent.

Stop the daemon and run it on a command line with debugging enabled:

service logentries stop
le --debug monitor

If you see any error messages, resolve them. If all appears to be ok, check that log entries are captured correctly by enabling dump on console. Break the monitoring with Ctrl+C and run again:

le --debug --debug-events monitor

If no log entries appears on console then the agent cannot retrieve new entries from the file. Otherwise all is OK.

Back to Top