How to Protect a LAMP Server Against nf_conntrack Flood Attacks

An AWS hosted website went offline at 02:00 this morning. It was running on a t2.nano Debian 9 instance. I was unable to log into the affected server, and a reboot was the only available course of action. Logging in and checking the logs afterwards revealed thousands of errors like this in the kernel log file, from 2:00 AM onward:

nf_conntrack: nf_conntrack: table full, dropping packet

The cause was a denial of service attack, coming from a couple of IP addresses seemingly in Iran. However, it was a little unusual for a couple of reasons. This article explains more about the attack vector and presents a solution to guard against future attacks. (In summary: block IP addresses, tune the kernel).

Continue reading

How to Set Local Search Provider in Android Firefox

In Android Firefox, you can perform a search by typing directly into the address bar (aka the “awesome bar“). Results are provided by the default search engine, usually google.com. That’s fine, but you might prefer the results to come from a more local source, such as google.co.uk, or a completely different provider, such as bing.com.

Use the following procedure to change the default search engine in Android Firefox. “Awesome bar” searches will then be performed by your provider of choice.

Continue reading

Redirecting Firewall Messages in Linux

The Linux firewall is a great way to secure a server, especially one that is Internet facing. Together with ipset and an appropriate blacklist, it can protect your server from the worst the Internet can throw at it. However, netfilter (aka iptables), can generate a lot of messages. By default they go to the kernel logging channel, flooding out log files such as messages, syslog and kern.log.

It is important to keep log files clear so that system issues are not missed. For example, a hardware or memory error message might be written to kern.log, but could be difficult to notice due to many thousands of firewall messages. Worse, over time, the important message will be moved into a historical log files due to the action of logrotate.

This article explains how to send firewall messages to their own log file, using the example of a Raspberry Pi running Raspbian 9 (Debian Stretch). After a small configuration change, netfilter messages go to their own file instead of clogging up the general logs.

Continue reading

How to Identify the Build Time of a Red Hat System

To determine when a Red Hat or CentOS server was first built, use rpm -qi basesystem:

[fred@rhel7 ~]$ rpm -qi basesystem
Name        : basesystem
Version     : 10.0
Release     : 7.el7
Architecture: noarch
Install Date: Thu 16 Apr 2015 18:16:04 CEST    <----- here
Group       : System Environment/Base
Size        : 0
License     : Public Domain
Signature   : RSA/SHA256, Tue 01 Apr 2014 15:23:16 CEST, Key ID 199e3a91fd554c52
Source RPM  : basesystem-10.0-7.el7.src.rpm
Build Date  : Fri 27 Dec 2013 18:22:15 CET
Build Host  : xxxxxxxxxx
Relocations : (not relocatable)
Packager    : Red Hat, Inc. 
Vendor      : Red Hat, Inc.
Summary     : The skeleton package which defines a simple Red Hat Enterprise Linux system
Description :
Basesystem defines the components of a basic Red Hat Enterprise Linux
system (for example, the package installation order to use during
bootstrapping). Basesystem should be in every installation of a system,
and it should never be removed.

Note: The date returned, 16th April in the example above, reflects the time that the operating system was installed. This is usually the same as the system creation time, except in cases where a system has been cloned or built from a pre-installed image. In the last case, the date returned will reflect the build date of the source image, rather than the target system.

SSH Proxying and Agent Forwarding

SSH allows secure connections from one host to another. All traffic is encrypted. Authentication is usually by means of a key pair, where the private key resides on your local machine, and the public key is imported to the remote system. SSH keys have become particularly important for cloud computing, where users need to access cloud servers over a potentially hostile Internet.

Sometimes, the requirement is to access one system via another. You “hop” through the first system to reach the second. For example, an AWS server through a gateway or “jump” system. The following article shows how to do that, in a secure way, either by proxying or agent forwarding, without having to place a private SSH key onto the middle system. Continue reading

Using Address Ranges and Port Ranges with Iptables

Iptables is the name of the firewall built into the Linux kernel. It is also the tool used for firewall configuration. This post explains how to use iptables with a range of IP addresses and/or ports. It could be used, for example, to allow SSH traffic from a number of systems. Or to open up a range of ports with a single firewall rule.

Note: This article is not about blacklisting. If you are looking to set up a blacklist, perhaps to protect your server from a number of unrelated IP addresses, my related procedure on how to protect your webserver with IPset might be more appropriate.

The Linux firewall (part of the Netfilter project) is important on Internet facing systems, “edge” servers and “jump” boxes. Particularly when they do not sit behind another protective network element such as a load balancer or discrete firewall. For example, standaline cloud instances that are not part of a protected VPC infrastructure. Continue reading

Predicting a Red Hat/CentOS Point Release Upgrade

Upgrading a Red Hat or CentOS system isn’t difficult. Just type yum update, it’s easy. So easy, in fact, that is quite possible to upgrade a system you didn’t intend to, or to upgrade a system further than was wanted, which is the subject of this post.

Point Upgrades

Yum update, used on a system which has not been updated for months or years, will cause hundreds of packages to be upgraded. It will also result in a point release upgrade. For example, a system running Red Hat/CentOS 6.2 might change to 6.5 after the update, or even to 6.7 or 6.9. Continue reading

Patching for Dirty Cow on Debian, Red Hat, Ubuntu, CentOS

“Dirty Cow” is the common name given to Linux vulnerability CVE-2016-5195. It is a “privilege escalation” that allows a non-root user to gain root access on a system. An attacker must have system access first, as a normal user. Then they use the bug to obtain root rights. It is dangerous and should be patched.

The Linux kernel itself was fixed in October 2016. Since then, Linux vendors have all released patches. Many Internet articles suggest addressing the bug by doing a general system update. While that might be fine for a test machine or Linux desktop, it isn’t ideal for a production server. This article describes how to fix the bug in the least invasive way possible – by updating the kernel only. Continue reading

Protect Your Web Server With Ipset

The Linux packet filter provides an easy way to protect against unwanted network intrusions. Often referred to simply as “iptables“, it is a basic firewall built into the Linux kernel. Iptables is most useful, perhaps, on those servers most susceptible to attack, such as LAMP systems, content management servers and blogging platforms like WordPress, especially where they are Internet facing.

Ipset is a fairly recent addition to Linux, having been introduced into kernel version 2.6.32. This means it is supported in Debian 7 and 8, as well as Red Hat 6 onwards. In short, ipset allows a large number of IP addresses to be blocked in an efficient way, as demonstrated below. Continue reading

Linux Device Change Breaks Encrypted Swap

Linux disk partition names such as /dev/sda1, /dev/sda2 are not as fixed as they once were. From time to time they can change. Perhaps due to a hardware change or kernel upgrade, or sometimes for no apparent reason. If and when this happens on your system, things can break. In this case, an encrypted swap partition had been configured as “/dev/sda6”, and failed to activate following a change in the the /dev/sdXX partition names. This article describes the symptoms and a fix.

The article also discusses a bug affecting Ubuntu 14.04 based distributions, which can make it more difficult to recover an encrypted swap configuration broken by a device name change. Continue reading