Introduction to CSF/LFD Notifications
- Common Notifications from CSF/LFD
- Common CSF/LFD False Positives and How to Stop The Notifications
ConfigServer Security & Firewall (csf) with Login Failure Daemon (lfd) is “A Stateful Packet Inspection (SPI) firewall, Login/Intrusion Detection and Security application for Linux servers.”official page)) This comes pre-installed on our cPanel servers and has many useful features to assist you with detecting events which might be indicative of security issues in your server.
If it is not already installed in your server with us, for example, if it is a non-cPanel server or if you are having any issues with it, please feel free to open a Support Ticket for assistance.
One of the things that makes CSF/LFD so useful is the notifications that it provides to help you watch for potentially-important events in your server. But sometimes these notifications can be confusing to the uninitiated.
For a very brief overview to get started, we can see the subject lines stored in CSF’s email template files: If desired, these template files can be used to change the format of CSF/LFD’s notifications, but this is not recommended unless you have a specific reason for doing so.
root@host [~]# grep Subject /usr/local/csf/tpl/*.txt
/usr/local/csf/tpl/accounttracking.txt:Subject: lfd on [hostname]: Account modification alert
/usr/local/csf/tpl/alert.txt:Subject: lfd on [hostname]: blocked [ip]
/usr/local/csf/tpl/connectiontracking.txt:Subject: lfd on [hostname]: [ip] blocked with too many connections
/usr/local/csf/tpl/consolealert.txt:Subject: lfd on [hostname]: console login alert
/usr/local/csf/tpl/cpanelalert.txt:Subject: lfd on [hostname]: WHM/cPanel [user] access alert from [ip]
/usr/local/csf/tpl/exploitalert.txt:Subject: lfd on [hostname]: System Exploit checking detected a possible compromise
/usr/local/csf/tpl/filealert.txt:Subject: lfd on [hostname]: Suspicious File Alert
/usr/local/csf/tpl/forkbombalert.txt:Subject: lfd on [hostname]: Fork Bomb detected and killed
/usr/local/csf/tpl/integrityalert.txt:Subject: lfd on [hostname]: System Integrity checking detected a modified system file
/usr/local/csf/tpl/loadalert.txt:Subject: lfd on [hostname]: High [loadavg] minute load average alert - [reportload]
/usr/local/csf/tpl/logalert.txt:Subject: lfd on [hostname]: Log Scanner Report for [hour], (lines:[lines])
/usr/local/csf/tpl/logfloodalert.txt:Subject: lfd on [hostname]: Log file flooding
/usr/local/csf/tpl/netblock.txt:Subject: lfd on [hostname]: Network class [class] [block] has been blocked
/usr/local/csf/tpl/permblock.txt:Subject: lfd on [hostname]: [ip] blocked permanently
/usr/local/csf/tpl/portknocking.txt:Subject: lfd on [hostname]: Port Knocking port opened by [ip]
/usr/local/csf/tpl/portscan.txt:Subject: lfd on [hostname]: [ip] blocked for port scanning
/usr/local/csf/tpl/processtracking.txt:Subject: lfd on [hostname]: Suspicious process running under user [user]
/usr/local/csf/tpl/queuealert.txt:Subject: lfd on [hostname]: Email queue size alert
/usr/local/csf/tpl/relayalert.txt:Subject: lfd on [hostname]: [check] Alert for [ip]
/usr/local/csf/tpl/resalert.txt:Subject: lfd on [hostname]: Excessive resource usage: [user] ([pid])
/usr/local/csf/tpl/reselleralert.txt:Subject: csf UI on [hostname]: reseller [reseller] - [action]
/usr/local/csf/tpl/scriptalert.txt:Subject: lfd on [hostname]: Script Alert for [path]
/usr/local/csf/tpl/sshalert.txt:Subject: lfd on [hostname]: SSH login alert for user [account] from [ip]
/usr/local/csf/tpl/sualert.txt:Subject: lfd on [hostname]: SU login alert - [status] from [from] to [to]
/usr/local/csf/tpl/syslogalert.txt:Subject: lfd on [hostname]: SYSLOG Check Failed
/usr/local/csf/tpl/tracking.txt:Subject: lfd on [hostname]: [account] blocked for [app] access
/usr/local/csf/tpl/uialert.txt:Subject: lfd on [hostname]: UI Alert [ip] [alert]
/usr/local/csf/tpl/uidscan.txt:Subject: lfd on [hostname]: UID [uid] Tracking Hit
/usr/local/csf/tpl/usertracking.txt:Subject: lfd on [hostname]: Excessive processes running under user [user]
/usr/local/csf/tpl/watchalert.txt:Subject: lfd on [hostname]: [file] has changed
/usr/local/csf/tpl/webminalert.txt:Subject: lfd on [hostname]: Webmin login alert for user [account] from [ip]
/usr/local/csf/tpl/x-arf.txt:Subject: abuse report about [ip] - [RFC3339]
We will be going through these approximately by categories.
IP Blocks
LFD (the Login Failure Daemon) will automatically block IP addresses for certain configurable reasons. By default, any time it blocks an IP address, it will send you an email to let you know which IP was blocked, and why it was blocked. If you have looked over the firewall configuration and are confident that it will only block IPs that you want blocked, you may want to disable these alerts so that you don’t get too many notifications distracting your attention from others that might be more important. But at least at first, it is useful to leave these alerts enabled, to help you make sure it is blocking IP addresses for the right reasons. For example, while you are still adjusting the settings for your purposes, you may want to review these notifications to make sure no one is getting blocked that need that access. But once you have determined that the firewall configuration is suitable for your specific situation, you may want to disable firewall settings like LF_EMAIL_ALERT
, LF_PERMBLOCK_ALERT
, LF_NETBLOCK_ALERT
, LF_DISTFTP_ALERT
, LF_DISTSMTP_ALERT
, LT_EMAIL_ALERT
, CT_EMAIL_ALERT
, and/or PS_EMAIL_ALERT
.
Login Failures
One of the most useful features of LFD (Login Failure Daemon) is to block IP addresses that have failed too many login attempts in a short period of time. This helps protect against bruteforce attacks against the accounts, where an attacker will try many different passwords for an account as quickly as possible in hopes of guessing and thereby finding the correct password for that account.
If you get an email from your server with a Subject line that looks like this, it is likely to be a notification resulting from one of these actions:
- “lfd on host.domain.tld: blocked X.X.X.X”
The text of the notification can be changed, but by default, this type of notification will look like this:
From: root
To: root
Subject: lfd on [hostname]: blocked [ip]
Time: [time]
IP: [ip]
Failures: [ipcount]
Interval: [iptick] seconds
Blocked: [block]
Log entries:
[text]
The “To:” and “From:” will usually be changed to “root@<hostname>”, for example, “root@host.domain.tld”. The subject line will tell you which IP was blocked. Within the message is listed the time the notification was sent, the IP address blocked, how many times the IP failed whichever rule was triggered, how long it will be blocked for, and whether it was a temporary or permanent block. After this is a section of log entries, demonstrating the log entries that LFD detected that triggered the firewall block to be created.
This notification is primarily informational in nature and usually does not require any further action to be taken, unless the block should not have been created. If the block should not have been created, the block will need to be removed if it hasn’t already been automatically removed and the LFD rules will need to be adjusted to prevent similarly erroneous blocks from being created. If you need any assistance with either of these, or with disabling the notifications please feel free to open a Support Ticket.
Temp to Perm blocks
To help prevent too many permanent blocks from being created while still allowing repeat offenders to be permanently blocked, LFD has a feature that can be enabled to trigger a permanent block on any IP address that has been temporarily blocked more than a certain number of times within a certain time interval.
If you get an email from your server with a Subject line that looks like this, it is likely to be a notification resulting from this type of action:
- “lfd on host.domain.tld: X.X.X.X blocked permanently”
The text of the notification can be changed, but by default, this type of notification will look like this:
From: root
To: root
Subject: lfd on [hostname]: [ip] blocked permanently
Time: [time]
IP: [ip]
Temporary Blocks: [count]
Temporary blocks that triggered the permanent block:
[blocks]
As before, “root” in the “From:” and “To:” lines will usually be replaced by “root@<hostname>”, like “root@host.domain.tld” or similar. Similarly, “[hostname]” is replaced with your server’s hostname, “[ip]” is replaced with the IP address being blocked, and “[time]” is replaced with the time the permanent block was created. “[count]” is the number of temporary blocks the IP had against it.
As with the previous entry, this type of notification is primarily informational in nature and usually does not require any further action to be taken, unless the block should not have been created, in which case the block would need to be removed and the firewall configuration adjusted accordingly to prevent similarly erroneous blocks from being created. If you need any assistance with either of these, or with disabling the notifications, please feel free to open a Support Ticket.
Netblock blocks
LFD also has a feature to automatically block entire netblocks if too many IPs within that set are blocked too many times within a certain interval. For example, if there are many malicious users located geographically near each other, and if their ISP is not particularly careful at addressing their problematic behavior, there may be larger CIDR ranges that you would want to shut entirely out of your server and give up on there being any wanted visitors from those locations. But it may be better to disable the automatic setting of blocks this large, and instead create the larger blocks manually on a case-by-case basis. But if this feature is enabled, be very careful not to set it too strictly, or you may end up blocking regular visitors.
If you get an email from your server with a Subject line that looks like this, it is likely to be a notification resulting from this type of action:
- “lfd on host.domain.tld: Network class C X.X.X.X/24 has been blocked”
The text of the notification can be changed, but by default, this type of notification will look like this:
From: root
To: root
Subject: lfd on [hostname]: Network class [class] [block] has been blocked
Time: [time]
Block: [block]
Hits: [count]
IP addresses that triggered the block
[ips]
As before, “root” in the “From:” and “To:” lines will usually be replaced by “root@<hostname>”, like “root@host.domain.tld” or similar. Similarly, “[hostname]” will be replaced by the server’s hostname, “[class]” will be replaced by which type of network class, class A, class B, or class C, is being blocked, “[class]” will be replaced by the CIDR range of IPs being blocked, for example, X.X.X.X/24 for a class C range and “[count]” is replaced by the number of IPs that were blocked to cause this range-block. “[ips]” is replaced by timestamped entries of which specific IP addresses caused the range-block and when their respective blocks were.
As with the previous entries, this type of notification is primarily informational in nature and usually does not require any further action to be taken, unless the block should not have been created, in which case the block would need to be removed and the firewall configuration adjusted accordingly to prevent similarly erroneous blocks from being created. If you need any assistance with either of these, or with disabling the notifications, please feel free to open a Support Ticket.
Port Scan
Some types of attack begin with a Port Scan, wherein the attacker will attempt to access many ports in a server, in order to determine which ports are open and what is running on them. Because this process usually results in many connections to closed ports being attempted, it can be useful to watch for these attempts and block IP addresses making too many of these attempts in quick succession. LFD has a feature to watch for this called Port Scan Tracking. But occasionally misconfigurations in commonly-used programs (for example, mail clients) can result in regular users being blocked by this feature. For this reason, it is recommended to leave notifications for this feature enabled, so that if one of your users is affected by this, it can be faster and easier to remove the block, and check for configuration issues that could be adjusted to prevent similarly erroneous blocks from blocking them in the future.
If you get an email from your server with a Subject line that looks like this, it is likely to be a notification resulting from this type of action:
- “lfd on host.domain.tld: X.X.X.X blocked for port scanning”
The text of the notification can be changed, but by default, this type of notification will look like this:
From: root
To: root
Subject: lfd on [hostname]: [ip] blocked for port scanning
Time: [time]
IP: [ip]
Hits: [count]
Blocked: [temp]
Sample of block hits:
[blocks]
As before, “root” in the “From:” and “To:” lines will usually be replaced by “root@<hostname>”. Similarly, “[hostname]” will be replaced by the server’s hostname, “[ip]” will be replaced by the IP that ws blocked, “[time]” will be replaced by the time at which the block was created, “[count]” will be replaced by how many times the IP was seen attempting to access closed ports, and “[temp]” will be replaced by how long the block will be in place. “[blocks]” will be replaced by log lines indicating the specific attempts to access the closed ports.
As with the previous entries, this type of notification is primarily informational in nature and usually does not require any further action to be taken, unless the block should not have been created, in which case the block would need to be removed and the firewall configuration, or the user’s client software configuration, adjusted accordingly to prevent similarly erroneous blocks from being created. If you need any assistance with either of these, or with disabling the notifications, please feel free to open a Support Ticket.
Too Many Connections
Because many simultaneous connections from the same IP address can be indicative of certain types of attack, and because even when it isn’t the result of an actual attack it is still often caused by something that should be investigated in case it might be causing load issues in the server, LFD has a feature for checking for and automatically blocking IP addresses making too many simultaneous connections to the server.
If you get an email from your server with a Subject line that looks like this, it is likely to be a notification resulting from this type of action:
- “lfd on host.domain.tld: X.X.X.X blocked with too many connections”
The text of the notification can be changed, but by default, this type of notification will look like this:
From: root
To: root
Subject: lfd on [hostname]: [ip] blocked with too many connections
Time: [time]
IP: [ip]
Connections: [ipcount]
Blocked: [temp]
Connections:
[iptext]
As before, “root” in the “From:” and “To:” lines will usually be replaced by “root@<hostname>”, like “root@host.domain.tld” or similar. Similarly, “[hostname]” will be replaced by the server’s hostname, “[ip]” will be replaced by the IP addresss that was blocked, “[time]” will be replaced by the time at which the block was created, “[ipcount]” will be replaced by the number of connections at the time the block was created, and “[temp]” will be replaced by the type, ie, temporary or permanent, of block. “[iptext]” will be replaced by lines displaying information about the connections.
As with the previous entries, this type of notification is primarily informational in nature and usually does not require any further action to be taken, unless the block should not have been created, in which case the block would need to be removed and the firewall configuration, or the user’s client software configuration, adjusted accordingly to prevent similarly erroneous blocks from being created. If you need any assistance with either of these, or with disabling the notifications, please feel free to open a Support Ticket.
Too Many POP3/IMAP Connections
Sometimes mail clients are misconfigured to check mailboxes too often, if someone needs notifications of something truly instantly, there may be other more appropriate protocols than email. Although one person’s overambition at keeping up with their mail might not be a problem, if many email users in the server do the same thing, it can cause load issues in the server. For this reason, LFD has a feature to detect when the same IP is logging into the same email account more than a certain number of times within an hour, and block that IP from logging into email, i.e., POP3 or IMAP, for the remainder of the hour.
If you get an email from your server with a Subject line that looks like this, it is likely to be a notification resulting from this type of action:
- “lfd on host.domain.tld: [account] blocked for [app] access”
The text of the notification can be changed, but by default, this type of notification will look like this:
From: root
To: root
Subject: lfd on [hostname]: [account] blocked for [app] access
Time: [time]
Account: [account]
Application: [app]
IP: [ip]
Logins: [logins]
Interval: [timeout]
Allowable: [rate] logins per hour in 3600 second interval
Flushed in: [flush] seconds
As before, “root” in the “From:” and “To:” lines will usually be replaced by “root@<hostname>”, like “root@host.domain.tld” or similar. Similarly, “[hostname]” will be replaced by the server’s hostname, “[ip]” will be replaced by the IP address that was blocked, “[time]” will be replaced by the time at which the block was created, “[account]” will be replaced with which email account had been accessed, “[app]” will be replaced by whether it is POP3 or IMAP that was accessed, “[ip]” will be replaced by the IP address accessing the account, “[logins]” will be replaced by how many times the IP logged into that account over that protocol, “[timeout]” will be replaced by how short a time period it took to exceed the rate limit, “[rate]” will be replaced by the configured allowable logins within one hour, and “[flush]” will be replaced by how long that IP will be prevented from logging into that email account over that protocol, this time period would usually be one hour minus the “[timeout]” length.
Although this type of notification is primarily informational in nature, it may be useful to leave enabled so that you can check with this email user if they have configured their email client correctly. If they don’t realize there is a rate limit configured, they might not realize that checking mail too often can result in them being unable to check mail for a significant portion of an hour at a time. If you need any assistance checking the server logs for more information about this notification, or adjusting the configured limits affecting this feature, or with disabling the notifications, please feel free to open a Support Ticket.
Email Relaying
LFD also has a feature for detecting certain types of issues with mail volume. These will be described in more detail later, but we briefly mention three of them in this section that can result in IP blocks being automatically created in the firewall.
In all three of these cases, the type of mail volume issue is with large numbers of messages being sent into the server from outside of the server. The difference between them is how the messages are entering the server. But for all three, the notification has the same basic format.
If you get an email from your server with a Subject line that looks like one of these, it is likely to be a notification resulting from this type of action:
- “lfd on host.domain.tld: Authrelay Alert for X.X.X.X”
- “lfd on host.domain.tld: Poprelay Alert for X.X.X.X”
- “lfd on host.domain.tld: Relay Alert for X.X.X.X”
The text of the notification can be changed, but by default, this type of notification will look like this:
From: root
To: root
Subject: lfd on [hostname]: [check] Alert for [ip]
Time: [time]
Type: [type] - [ip]
Count: [count] emails relayed
Blocked: [block]
Sample of the first 10 emails:
[emails]
As before, “root” in the “From:” and “To:” lines will usually be replaced by “root@<hostname>”, like “root@host.domain.tld” or similar. Similarly, “[hostname]” will be replaced by the server’s hostname, “[check]” and “[type]” will be replaced by the type of relay alert the notification is, “[ip]” will be replaced by the IP address that sent the messages, “[time]” will be replaced by the time at which the alert was generated, “[count]” will be replaced by the number of email messages sent, “[block]” will be replaced by the type of block temporary or permanent, if any placed against the IP.
It is strongly recommended to leave this type of notification enabled, since it is likely to be indicative of an issue that should be investigated and addressed swiftly. If you need any assistance investigating the mail issue or adjusting the parameters of when this type of alert is sent, please feel free to open a Support Ticket.
Successful Logins
LFD also has several notifications to alert you of successful logins. The purpose of this is to let you know when someone logs into the server, so that you can then check to make sure it is someone who should have access to the server, as opposed to someone else who guessed the password.
WHM/cPanel
LFD can detect logins to WHM or cPanel. If you get an email from your server with a Subject line that looks like this, it is likely to be a notification from LFD due to having detected this: with <user>
being replaced by the cPanel or WHM user detected as having logged in.
- “lfd on host.domain.tld: WHM/cPanel <user> access alert from X.X.X.X”
The text of the notification can be changed, but by default, this type of notification will look like this:
From: root
To: root
Subject: lfd on [hostname]: WHM/cPanel [user] access alert from [ip]
Time: [time]
IP: [ip]
User: [user]
As before, “root” in the “From:” and “To:” lines will usually be replaced by “root@<hostname>”, like “root@host.domain.tld”. Similarly, “[hostname]” will be replaced by the server’s hostname, “[user]” will be replaced by the cPanel or WHM user which was detected logging in, “[ip]” will be replaced by the IP address detected as having logged in, and “[time]” will be replaced by the time the login was detected.
It is strongly recommended to leave this type of notification enabled, so that you can check whether or not someone from these IPs should be logging in as these users. If you need any assistance checking the server logs or adjusting the parameters of when this type of alert is sent, please feel free to open a Support Ticket.
Port Knocking
Port knocking is a special way of allowing access to certain ports only to people who know a specific sequence of port numbers. Since iptables must be configured in a specific way to enable port knocking, csf has a section in its configuration to assist with setting this up. Additoinally, LFD can detect when a user has gained access this way, and can be configured to alert you when this has happened.
If you get an email from your server with a Subject line that looks like this, it is likely to be a notification from LFD that someone has successfully used a port knocking sequence:
- “lfd on host.domain.tld: Port Knocking port opened by X.X.X.X”
The text of the notification can be changed, but by default, this type of notification will look like this:
From: root
To: root
Subject: lfd on [hostname]: Port Knocking port opened by [ip]
Time: [time]
IP: [ip]
Port: [port]
As before, “root” in the “From:” and “To:” lines will usually be replaced by “root@<hostname>”. Similarly, “[hostname]” will be replaced by the server’s hostname, “[ip]” will be replaced by the IP address detected as successfully completing the port knocking sequence, “[time]” will be replaced by the time at which the completed sequence was detected, and “[port]” will be replaced by the port which was accessed via port knocking.
It is strongly recommended to leave this type of notification enabled, so that you can check whether or not someone from these IPs should be accessing the server in this way. If you need any assistance checking the server logs or adjusting the parameters of when this type of alert is sent, please feel free to open a Support Ticket.
SSH
LFD also has a feature to detect when someone has accessed the server via SSH. If you get an email from your server with a Subject line that looks like this, it is likely to be an alert from LFD letting you know that this has occurred:
- “lfd on host.domain.tld: SSH login alert for user <account> from X.X.X.X”
The text of the notification can be changed, but by default, this type of notification will look like this:
From: root
To: root
Subject: lfd on [hostname]: SSH login alert for user [account] from [ip]
Time: [time]
IP: [ip]
Account: [account]
Method: [method] authentication
As before, “root” in the “From:” and “To:” lines will usually be replaced by “root@<hostname>”. Similarly, “[hostname]” will be replaced by the server’s hostname, “[account]” will be replaced by the name of the account which has been accessed, “[ip]” will be replaced by the IP address detected as accessing the account, “[time]” will be replaced by the time the access was detected, and “[method]” will be replaced by the type of authentication that was used to access ssh.
It is strongly recommended to leave this type of notification enabled, so that you can check whether or not someone from these IPs should be accessing the server in this way. If you need any assistance checking the server logs or adjusting the parameters of when this type of alert is sent, please feel free to open a Support Ticket.
SU
LFD also has a feature to detect when someone has used su
to get access to one account from another account. If you get an email from your server with a Subject line that looks like this, it is likely to be an alert from LFD letting you know that this has occurred:
- “lfd on host.domain.tld: SU login alert – Successful login from <user> to <otheruser>”
The text of the notification can be changed, but by default, this type of notification will look like this:
From: root
To: root
Subject: lfd on [hostname]: SU login alert - [status] from [from] to [to]
Time: [time]
From: [from]
To: [to]
Status: [status]
As before, “root” in the “From:” and “To:” lines will usually be replaced by “root@<hostname>”. Similarly, “[hostname]” will be replaced by the server’s hostname, “[status]” will be replaced by whether the SU user change was successful, “[from]” will be replaced by the user that ran the su command, “[to]” will be replaced by the user accessed via the su command, and “[time]” will be replaced by the time at which the su command was detected.
It is strongly recommended to leave this type of notification enabled, so that you can check whether or not someone using this account should be accessing this other account in this way. If you need any assistance checking the server logs or adjusting the parameters of when this type of alert is sent, please feel free to open a Support Ticket.
CSF UI
There exists a web-accessible UI for csf which can be enabled, to allow some of the settings to be changed without having ssh access. It is strongly recommended not to enable this since it adds one more access method that would need to be watched to ensure no malicious activity, and also it is unnecessary for cPanel servers where there is already a web-accessible UI via the WHM plugin. But if it is enabled, there is also a notification that can be enabled to alert you when someone has logged into the CSF UI. If you get an email from your server with a Subject line that looks like this, it is likely to be one of these notifications:
- “lfd on host.domain.tld: UI alert X.X.X.X <alert>”
The text of the notification can be changed, but by default, this type of notification will look like this:
From: root
To: root
Subject: lfd on [hostname]: UI Alert [ip] [alert]
Time: [time]
IP: [ip]
Alert: [alert]
[text]
As before, “root” in the “From:” and “To:” lines will usually be replaced by “root@<hostname>”. Similarly, “[hostname]” will be replaced by the server’s hostname, “[ip]” will be replaced by the IP address detected as having logged in, “
It is strongly recommended to leave this type of notification enabled, so that you can check whether or not someone from these IPs should be accessing the server in this way. If you need any assistance checking the server logs or adjusting the parameters of when this type of alert is sent, please feel free to open a Support Ticket.
Webmin
It is not recommended to install Webmin in a server that already has another panel installed, and we do not support Webmin, but if it is installed in the server, LFD can also detect logins to it. If you get an email from your server with a Subject line that looks like this, it is likely to be a notification from your server about somone having logged into Webmin:
- “lfd on host.domain.tld: Webmin login alert for user <account> from X.X.X.X”
The text of the notification can be changed, but by default, this type of notification looks like this:
From: root
To: root
Subject: lfd on [hostname]: Webmin login alert for user [account] from [ip]
Time: [time]
IP: [ip]
Account: [account]
As before, “root” in the “From:” and “To:” lines will usually be replaced by “root@<hostname>”. Similarly, “[hostname]” will be replaced by the server’s hostname, “[account]” will be replaced by the Webmin account that was accessed, “[ip]” will be replaced by the IP address from which Webmin was accessed, and “[time]” will be replaced by the time when the login was detected.
It is strongly recommended to leave this type of notification enabled, so that you can check whether or not someone from these IPs should be accessing the server in this way. If you need any assistance checking the server logs or adjusting the parameters of when this type of alert is sent, please feel free to open a Support Ticket.
LFD has several features for watching the server’s email activity, to help you make sure no one is abusing the mailserver software installed in your server. It is strongly recommended to leave these alerts enabled, so that you can be alerted of potential issues that should be investigated quickly, to fix potential security issues and reduce potential damage to your IP reputation.
Email queue size alert
LFD has a feature for watching the length of the mail queue. The mail queue is where email messages wait for processing between when the server gets the message either from another server or from one of the configured email users, and when the server is able to deliver the message either locally or to another mailserver (or for the message to fail and generate a bounce message). Most of the time, a message will be delivered within seconds, and the queue will not accumulate messages, but there are certain types of issues that can cause a message’s delivery to be delayed. If many messages are accumulating in the queue, it is important to check what is causing this to occur. Some of the most common issues that would cause many messages to remain in queue are indicative of security issues that should be addressed as quickly as possible.
If you get an email from your server with a Subject line that looks like this, it is likely to be a notification from LFD that the mail queue is long:
- “lfd on host.domain.tld: Email queue size alert”
The text of the notification can be chanbed, but by default, this type of notification looks like this:
From: root
To: root
Subject: lfd on [hostname]: Email queue size alert
Time: [time]
[text]
As before, “root” in the “From:” and “To:” lines will usually be replaced by “root@<hostname>”. Similarly, “[hostname]” will be replaced by the server’s hostname, and “[time]” will be replaced by the time at which the long queue was detected.
It is strongly recommended to leave this type of notification enabled, so that the relevant issues can be found as quickly as possible. If you need any assistance checking the server logs or adjusting the parameters of when this type of alert is sent, please feel free to open a Support Ticket.
Relay Alerts
As briefly mentioned earlier, there are certain types of mailing behavior that LFD is able to detect and notify you of. Earlier, we went over specifically the ones that can result in IP blocks being automatically created in the firewall, but here we go over them in more detail. Here is the generic format of this type of alert. The text of this type of notification can be changed, but by default, it looks like this:
From: root
To: root
Subject: lfd on [hostname]: [check] Alert for [ip]
Time: [time]
Type: [type] - [ip]
Count: [count] emails relayed
Blocked: [block]
Sample of the first 10 emails:
[emails]
As before, “root” in the “From:” and “To:” lines will usually be replaced by “root@<hostname>”. Similarly, “[hostname]” will be replaced by the server’s hostname, “[check]” will be replaced by which type of relay alert this is, “[ip]” will be replaced by the IP address detected if applicable, “[time]” will be replaced by the time the activity was detected, “[count]” will be replaced by the number of messages sent in this way so far, “[block]” will be replaced by whether the block (if applicable) is temporary or permanent, and “[emails]” will be replaced by relevant log lines for the first ten messages of this type detected.
It is strongly recommended to leave these notifications enabled so that you can address any related potential issues as quickly as possible. If you need any assistance checking the server logs or adjusting the parameters of when this type of alert is sent, please feel free to open a Support Ticket.
Relay Alert
A “Relay Alert”, as opposed to the Authrelay, Poprelay, Localrelay, or Localhostrelay alerts, is triggered by “external mail”, that is, messages that are coming from another mailserver. Usually these are incoming messages. Although they are not usually indicative of spam being generated within the server, if enough messages are coming from the same IP address quickly enough to trigger this alert type, it is probably worth looking into why they are sending so much mail, and determine if there is anything that needs to be adjusted.
Authrelay Alert
An “Authrelay Alert” is triggered by “email authenticated by SMTP AUTH”. This is one method logging into the mailserver to send messages. Most modern mail clients log in by this method. If these messages should not be sent or should not be sent this quickly, then that email address is likely to need a new password, since whoever is sending the messages demonstrably has the current password.
Poprelay Alert
A “Poprelay Alert” is triggered by “email authenticated by POP before SMTP”. Some older mail clients authenticate using this method, but it is recommended to use SMTP AUTH instead, in part because it makes the logs clearer which makes it easier to find causes of spam issues or similar if they occur.
Localrelay Alert
A “Localrelay Alert” is triggered by “email sent via /usr/sbin/sendmail
or /usr/sbin/exim
“. This is usually done by scripts. If a script is sending too much mail, it will need to be reconfigured accordingly. If a script is sending mail that it shouldn’t, it will need to be disabled or fixed so that it only sends the messages that it should.
Localhostrelay Alert
A “Localhostrelay Alert” is triggered by “email sent via a local IP address”. This means that the message is coming from within the server. If messages are being sent from within the server without authenticating, then changing email passwords will not prevent them from being sent. If the messages are not authorized, the source of the message will need to be found and stopped.
Script Alert
A common method of sending mail with scripts is to have the script invoke the sendmail or exim binary. When this happens, certain lines will appear in the mail log which LFD is able to detect and notify you of if it happens repeatedly. If you get an email from your server with a Subject line that looks like this, it is likely a notification from lfd that this type of activity has occurred:
- “lfd on host.domain.tld: Script Alert for /home/cpuser/public_html”
The text of the notification can be changed, but by default, it looks like this:
From: root
To: root
Subject: lfd on [hostname]: Script Alert for [path]
Time: [time]
Path: [path]
Count: [count] emails sent
Sample of the first 10 emails:
[emails]
Possible Scripts:
[scripts]
As before, “root” in the “From:” and “To:” lines will usually be replaced by “root@<hostname>”. Similarly, “[hostname]” will be replaced by the server’s hostname, “[path]” will be replaced by the folder path found in the mail log, “[time]” will be replaced by the time the behavior was detected, “[count]” will be replaced by the number of emails detected so far, “[emails]” will be replaced by a few relevant log lines, and “[scripts]” will be replaced by lfd’s guess of which scripts might be involved. Note that this guess is often inaccurate, so it is important to check which scripts are actually involved and in need of attention.
It is strongly recommended to leave this type of alert enabled, so that you can address any related potential issues as quickly as possible. If you need any assistance checking the server logs or adjusting the parameters of when this type of alert is sent, please feel free to open a Support Ticket.
Miscellaneous
LFD has other things that it can check for and alert you of as well. As with the other notifications it is possible for the alert to indicate something that needs to be investigated, but sometimes the alert might be about something that is not a problem. If you are not sure what a particular alert means or what if anything needs to be done about it, please feel free to open a Support Ticket.
Account modification alert
LFD can watch for certain types of modifications to accounts. If you get an email from your server with a Subject line that looks like this, it is likely to be a notification about this type of activity:
- “lfd on host.domain.tld: Account modification alert”
The text of the notification can be changed, but by default, it looks like this:
From: root
To: root
Subject: lfd on [hostname]: Account modification alert
Time: [time]
Reported Modifications:
[report]
As before, “root” in the “From:” and “To:” lines will usually be replaced by “root@<hostname>”. Similarly, “[hostname]” will be replaced by the server’s hostname, “[time]” will be replaced by the time the modification was detected, and “[report]” will be replaced by additional information about the detected change.
It is strongly recommended to leave this type of alert enabled, so that you can check if this modification was expected or if it may be indicative of a problem that may need to be addressed. If you are not sure what a particular alert means or what if anything needs to be done about it, please feel free to open a Support Ticket.
System Exploit checking
There are some specific tests that LFD can run to determine if some specific exploits may have occurred. Not much specific information seems to be made available by ConfigServer as to what exactly these tests entail. If you get an email from your server with a Subject line that looks like this, it is likely to be the result of one or more of these tests:
- “lfd on host.domain.tld: System Exploit checknig detected a possible compromise”
The text of the notification can be changed, but by defalt, it looks like this:
From: root
To: root
Subject: lfd on [hostname]: System Exploit checking detected a possible compromise
Time: [time]
[text]
As before, “root” in the “From:” and “To:” lines will usually be replaced by “root@<hostname>”. Similarly, “[hostname]” will be replaced by the server’s hostname, “[time]” will be replaced by the time the tests detected the possible issue, and “[text]” will be replaced by some information LFD decides is relevant.
It is strongly reocmmended to leave this type of alert enabled, so that you can check whether or not the test results are indicative of issues that may need to be addressed. If you are not sure what a particular alert means or what if anything needs to be done about it, please feel free to open a Support Ticket.
Suspicious File Alert
LFD has a feature to detect files in /tmp
or /dev/shm
that might be suspicious. Since some types of script exploits result in certain types of files being placed in these locations, this can sometimes help detect exploited scripts. If you get an email from your server with a Subject line that looks like this, it is likely to be the result of this test:
- “lfd on host.domain.tld: Suspicious File Alert”
The text of the notification can be changed, but by default, it looks like this:
From: root
To: root
Subject: lfd on [hostname]: Suspicious File Alert
Time: [time]
File: [file]
Reason: [reason]
Owner: [owner]
Action: [action]
As before, “root” in the “From:” and “To:” lines will usually be replaced by “root@<hostname>”. Similarly, “[hostname]” will be replaced by the server’s hostname, “[time]” will be replaced by the time the file was detected, “[file]” will be replaced by the name of the file, “[reason]” will be replaced by the reason it was declared suspicious, “[owner]” will be replaced by the name of the account that owns the file, and “[action]” will be replaced by what LFD did as a result of detecting the file.
It is strongly reocmmended to leave this type of alert enabled, so that you can check whether or not the test results are indicative of issues that may need to be addressed. If you are not sure what a particular alert means or what if anything needs to be done about it, please feel free to open a Support Ticket.
Fork Bomb detected and killed
There is a particular type of attack called a “Fork Bomb”, in which a process forks into many processes, often resulting in high resource usage in the server which may affect the services. Sometimes this also occurs with scripts that are malfunctioning rather than compromised. LFD has a feature intended to detect this type of activity, so that you can act quickly to stop it if needed. If you get an email from your server with a Subject line that looks like this, it is likely to be a result of a potential fork bomb being detected:
- “lfd on host.domain.tld: Fork Bomb detected and killed”
The text of the notification can be changed, but by default, it looks like this:
From: root
To: root
Subject: lfd on [hostname]: Fork Bomb detected and killed
Time: [time]
Trigger: [level]
Processes:
[text]
As before, “root” in the “From:” and “To:” lines will usually be replaced by “root@<hostname>”. Similarly, “[hostname]” will be replaced by the server’s hostname, “[time]” will be replaced by the time the potential fork bomb was detected, and “[text]” will be replaced by some relevant information about the processes that were found and killed.
Since there is a large potential for false positives with this feature, and since the feature will kill the processes that it detects passing its configured thresholds, it is not recommended to have this feature enabled unless you have specific reason to suspect this type of issue is occurring. If you would like assistance with interpreting the notification or adjusting its configuration, or with attempting to determine its cause, please feel free to open a Support Ticket.
System Integrity Alert
LFD has a feature to check certain system files for changes. This can help in detecting certain types of compromises, but will also send you alerts when these files are changed by normal system updates. If you are not sure why the files are being changed, it is important to check the server logs to determine if the changes are expected results of normal updates or other intentional changes, or if there may be a compromise that needs to be addressed. If you get an email from your server with a Subject line that looks like this, it is likely to be the result of this check detecting changes:
- “lfd on host.domain.tld: System Integrity checking detected a modified system file”
The text of the notification can be changed, but by default, it looks like this:
From: root
To: root
Subject: lfd on [hostname]: System Integrity checking detected a modified system file
Time: [time]
The following list of files have FAILED the md5sum comparison test.
This means that the file has been changed in some way.
This could be a result of an OS update or application upgrade.
If the change is unexpected it should be investigated:
[text]
As before, “root” in the “From:” and “To:” lines will usually be replaced by “root@<hostname>”. Similarly, “[hostname]” will be replaced by the server’s hostname, “[time]” will be replaced by the time at which the changes were detected, and “[text]” will be replaced by some information about the detected changes.
It is strongly recommended to leave this type of notification enabled, so that if unexpected changes occur, they can be investigated as soon as possible. Although most of the notifications are likely to be “false positives” resulting from normal updates, it is recommended to check these whenever you are not sure. If you would like assistance checking the server logs or adjusting whether these notifications are sent, please feel free to open a Support Ticket.
High load average alert
LFD also has a feature to watch the server load, and to provide certain information about the currently-running processes when the load is detected to be higher than the configured threshold. Sometimes high server load can be indicative of security issues, if it is unauthorized processes using the server resources. Even when there is not specifically a security issue in the server, high server loads should still be investigated and addressed because they adversely affect server performance. If you get an email from your server with a Subject line that looks like this, it is likely to be the result of LFD detecting high server loads:
- “lfd on host.domain.tld: High 5 minute load average alert – <number>”
The text of the notification can be changed, but by default, it looks like this:
From: root
To: root
Subject: lfd on [hostname]: High [loadavg] minute load average alert - [reportload]
MIME-Version: 1.0
Content-Type: multipart/mixed;
boundary="------------[boundary]"
This is a multi-part message in MIME format.
--------------[boundary]
Content-Type: text/plain;
Content-Transfer-Encoding: 7bit
Time: [time]
1 Min Load Avg: [loadavg1]
5 Min Load Avg: [loadavg5]
15 Min Load Avg: [loadavg15]
Running/Total Processes: [totprocs]
--------------[boundary]
Content-Type: text/plain;
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="ps.txt"
Output from ps:
[processlist]
--------------[boundary]
Content-Type: text/plain;
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="vmstat.txt"
Output from vmstat:
[vmstat]
--------------[boundary]
Content-Type: text/html;
name="apachestatus.html"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="apachestatus.html"
[apache]
--------------[boundary]--
As before, “root” in the “From:” and “To:” lines will usually be replaced by “root@<hostname>”. Similarly, “[hostname]” will be replaced by the server’s hostname, “[loadavg]” will be replaced by the type of load average being used for the check, “[reportload]” will be replaced by how high that load average is, “[time]” will be replaced by the time at which the high load was detected, “[loadavg1]” will be replaced by what the 1-minute load average was at that time, “[loadavg5]” will be replaced by what the 5-minute load average was at that time, “[loadavg15]” will be replaced by what the 15-minute load average was at that time, “[totprocs]” will be replaced by the total number of processes running at that time, “[processlist]” will be replaced by a list of the processes running at that time and some basic information about them, “[vmstat]” will be replaced by some information about the total memory usage at that time, and “[apache]” will be replaced by some information about the current apache connections at that time.
It is strongly recommended to leave this type of notification enabled, so that you can more quickly investigate the cause of the high server load. If you need assistance checking the server logs or adjusting the thresholds for when this type of alert is sent, please feel free to open a Support Ticket.
Log Scanner Report
LFD also has a feature to periodically email you the contents of certain configured log files. It is not recommended to enable this feature unless there is a specific reason you want specific parts of specific logs to be emailed to you. If you get an email from your server with a Subject line that looks like this, it is likely to be a report from this feature:
- “lfd on host.domain.tld: Log Scanner Report for 11:00, (lines:9)”
The text of the notification can be changed, but by default, it will look like this:
From: root
To: root
Subject: lfd on [hostname]: Log Scanner Report for [hour], (lines:[lines])
[text]
As before, “root” in the “From:” and “To:” lines will usually be replaced by “root@<hostname>”. Similarly, “[hostname]” will be replaced by the server’s hostname, “[hour]” will be replaced by the time the notification is for, “[lines]” will be replaced by the number of log lines being reported, and “[text]” will be replaced by the lines of logs being included in the report.
Unless you have a specific reason for wanting specific parts of specific log files emailed to you periodically, it is not recommended to enable this feature. If you want assistance interpreting any parts of any server logs, or with adjusting the settings of this feature, please feel free to open a Support Ticket.
Log file flooding
Because LFD relies so heavily on various server logs in order to detect any of the things it seeks to detect, it can become substantially less effective if the logs themselves are having problems. One such problem can be if the logs are “flooded” with too many lines in a row that are too similar. If you get an email from the server with a Subject line that looks like this, it is likely to be a notification from LFD about log flooding having occurred:
- “lfd on host.domain.tld: Log file flooding”
The text of the notification can be changed, but by default, it will look like this:
From: root
To: root
Subject: lfd on [hostname]: Log file flooding
Time: [time]
Alert: [text]
As before, “root” in the “From:” and “To:” lines will usually be replaced by “root@<hostname>”. Similarly, “[hostname]” will be replaced by the server’s hostname, “[time]” will be replaced by the time the log flooding was detected, and “[text]” will be replaced by somewhat more detailed information about the detected log flood event.
If you need assistance interpreting the alert, checking server logs, or adjusting the conditions under which the alert is sent, please feel free to open a Support Ticket.
Suspicous process alert
There are a few situations in which LFD will detect a process as being “Suspicious”, for example if it has network connections open or is running from a deleted excutable. If you get an email from the server with a Subject line that looks like this, it is likely to be a notification from LFD about a process it considers suspicious:
- “lfd on host.domain.tld: Suspicious process running under user <username>”
The text of the notification can be changed, but by default, it will look like this:
From: root
To: root
Subject: lfd on [hostname]: Suspicious process running under user [user]
Time: [time]
PID: [pid]
Account: [user]
Uptime: [uptime] seconds
Executable:
[exe]
Command Line (often faked in exploits):
[cmdline]
Network connections by the process (if any):
[sockets]
Files open by the process (if any):
[files]
Memory maps by the process (if any):
[maps]
As before, “root” in the “From:” and “To:” lines will usually be replaced by “root@<hostname>”. Similarly, “[hostname]” will be replaced by the server’s hostname, “[user]” will be replaced by the user the process is running under, “[time]” will be replaced by the time at which the process was detected as suspicious, “[pid]” will be replaced by the Process ID of the process, “[uptime]” will be replaced by how long the process has been running, “[exe]” will be replaced by the executable the process is running from, “[cmdline]” will be replaced by the command line command associated with the process, “[sockets]” will be replaced by information about any network connections the process has open, “[files]” will be replaced by a list of files the process has open, and “[maps]” will be replaced by a list of memory maps the process has open.
It is strongly recommended to leave this type of notification enabled, so that you can check whether or not the processes are indeed suspicious. If you need assistance interpreting the notification, checking server logs, or adjusting the conditions under which this notification is sent, please feel free to open a Support Ticket.
Excessive resource usage
LFD also has a feature to watch the running processes to see if they are using too many resources. For some of these resources, how much counts as “too much” is configurable. Sometimes, if a process is using more resources than expected, it can be indicative of a security issue. Even when it is not indicative of a security issue, it should still be investigated to check if something is misconfigured or otherwise likely to cause load issues in the server. If you get an email from your server with a Subject line that looks like this, it is likely to be a notification from LFD regarding a process it decided is using too many resources:
- “lfd on host.domain.tld: Excessive resource usage: <username> (<pid>)”
The text of the notification can be changed, but by default, it will look like this:
From: root
To: root
Subject: lfd on [hostname]: Excessive resource usage: [user] ([pid])
Time: [time]
Account: [user]
Resource: [resource]
Exceeded: [level]
Executable: [exe]
Command Line: [cmd]
PID: [pid]
Killed: [kill]
As before, “root” in the “From:” and “To:” lines will usually be replaced by “root@<hostname>”. Similarly, “[hostname]” will be replaced by the server’s hostname, “[user]” will be replaced by the user the process is running under, “[pid]” will be replaced by the Process ID of the process, “[time]” will be replaced by the time at which the process was detected as using too many resources, “[resource]” will be replaced by which resource the process seems to be using too much of, “[level]” will be replaced by how much of that resource the process was detected as using, “[exe]” will be replaced by the executable the process is running from, “[cmd]” will be replaced by the command line command of the process, and “[kill]” will be replaced by whether or not LFD attempted to kill the process.
It is strongly recommended to leave this type of notification enabled, so that you can check whether or not it is expected for that process to use that much of whichever resource it is using a lot of, in case there are any needed changes. If you need assistance interpreting the notification, checking server logs, or adjusting the conditions under which this notification is sent, please feel free to open a Support Ticket.
SYSLOG alert
Because LFD relies so heavily upon system logging, it has a check to make sure the log is still accepting entries and to make sure LFD is still able to read the system log. Although this check sometimes has false positives, it is still a good idea to investigate why LFD was unable to read its attempted addition to the logfile, in case there are any logging issues that may reduce the effectiveness of LFD. If you get an email from your server with a Subject line that looks like this, it is likely to be an alert from LFD that this check has failed:
- “lfd on host.domain.tld: SYSLOG Check Failed”
The text of the notification can be changed, but by default, it will look like this:
From: root
To: root
Subject: lfd on [hostname]: SYSLOG Check Failed
Time: [time]
Error: Failed to detect code [[code]] in SYSLOG_LOG [[log]]
SYSLOG may not be running correctly on [hostname]
As before, “root” in the “From:” and “To:” lines will usually be replaced by “root@<hostname>”. Similarly, “[hostname]” will be replaced by the server’s hostname, “[time]” will be replaced by the time at which the check failed, “[code]” will be replaced by the string LFD tried to add to the system log, and “[log]” will be replaced by the log file LFD tried to add the string to and was unable to read the string in.
It is strongly recommended to leave this type of notification enabled, so that you can check whether or not there are indeed any logging issues in the server that may need to be addressed. If you need assistance interpreting the notification, checking server logs, or adjusting the conditions under which this notification is sent, please feel free to open a Support Ticket.
Excessive process alert
LFD also has a feature to check whether a particular user has too many processes open simultaneously. If a user has more simultaneous processes open than expected, it can be indicative of an issue likely to cause resource issues in the server, and sometimes can even be indicative of security issues that may need to be addressed. If you get an email from your server with a Subject line like this, it is likely to be a notification from LFD letting you know that a user has more processes open than the configured threshold:
- “lfd on host.domain.tld: Excessive processes running under user <username>”
The text of the notification can be changed, but by default, it will look like this:
From: root
To: root
Subject: lfd on [hostname]: Excessive processes running under user [user]
Time: [time]
Account: [user]
Process Count: [count]
Process Information:
[text]
As before, “root” in the “From:” and “To:” lines will usually be replaced by “root@<hostname>”. Similarly, “[hostname]” will be replaced by the server’s hostname, “[user]” will be replaced by the user with too many processes detected, “[time]” will be replaced by the time at which the user was detected as having too many processes open, “[count]” will be replaced by the number of processes the user was detected as having hopen, and “[text]” will be replaced by some information about the processes detected as running under that user at that time.
It is strongly recommended to leave this type of notification enabled, so that you can check whether or not it is expected for this user to have so many processes open at once, and so that you can check if these processes are causing any resource issues in the server. If you need assistance interpreting the notification, checking server logs, or adjusting the conditions under which this notification is sent, please feel free to open a Support Ticket.
File change alert
In addition to watching system files for changes, LFD can be configured to watch the files of any directory for any changes. It is not recommended to enable this unless there is a specific reason you want to watch for changes to files in a specific directory. If you get an email from your server with a Subject line that looks like this, it is likely to be a notification from LFD about files in one of these directories being changed:
- “lfd on host.domain.tld: <file> has changed”
The text of the notification can be changed, but by default, it will look like this:
From: root
To: root
Subject: lfd on [hostname]: [file] has changed
Time: [time]
File: [file] has changed
Output:
[output]
As before, “root” in the “From:” and “To:” lines will usually be replaced by “root@<hostname>”. Similarly, “[hostname]” will be replaced by the server’s hostname, “[file]” will be replaced by the name of the file detected as having changed, “[time]” will be replaced by the time at which the change was detected, and “[output]” will be replaced by some additional information.
It is not recommended to enable this feature unless there is a specific reason you want to watch for changes to files in a specific directory. If you need assistance interpreting the notification, checking server logs, or adjusting the conditions under which this notification is sent, please feel free to open a Support Ticket.
X-ARF alert
LFD also has a feature to send you X-ARF-formatted messages for any firewall block created by LFD, in case after reviewing the cause of the block you decide you would like to send an abuse report to someone responsible for that IP address. It is not recommended to enable these, and even the csf/lfd configuration file states the recommendation not to enable this feature:
# We do not recommend enabling this option. Abuse reports should be checked and
# verified before being forwarded to the abuse contact
If any messages are generated by this feature, it is strongly recommended to disable them. If you need assistance interpreting the notification, checking server logs, or adjusting the conditions under which this notification is sent, please feel free to open a Support Ticket.