Debian 8 (jessie) was released in April 2015, ten months ago at the time of writing. Some sites will have upgraded by now, and others might be considering the move. This article provides a list of issues encountered after upgrading several servers from Debian 7 (wheezy) to Debian 8. These are system administration issues in the most part, though some might also be of interest to application support staff.
Upgrade Procedure
This article does not focus on the upgrade procedure itself (which was performed as a simple re-install), but on the aftermath. Some issues were obvious immediately, while others did not show up for several weeks.
Logrotate Schedule
Logrotate is a Linux command for managing log files. It rolls log files over, compresses the older ones and deletes the oldest. It therefore manages the disk space occupied by log files, and prevents partitions, specifically the /var partition, from filling up. Logrotate is called from cron.
In Debian 8, the default schedule has been reduced greatly. For example, Apache logs are rotated every day rather than every week as they were under Debian 7. The explanation is in /etc/logrotate.d/apache2. At Debian 7, here are the first few lines of the file:
/var/log/apache2/*.log {
weekly
missingok
rotate 52
Compare that with the same file under Debian 8:
/var/log/apache2/*.log {
daily
missingok
rotate 14
So at Debian 7, the log goes through 52 weekly rotations before being removed, storing 53 log files containing a year’s worth of data. On Debian 8 it is rotated daily and removed after 14 days, so only 2 weeks’ worth is retained, and only 2 days of that uncompressed, and 15 logs in all are retained (14 rotated plus one current).
Systems administrators and security staff might want to retain more data. To do that, change “daily” to “weekly” in the Debian 8 file, and “rotate 14” to “rotate 52“. In other words, make it look like the Debian 7 file.
root@debian8:~# vi /etc/logrotate.d/apache2
After saving the file, there is nothing to restart to make the change take effect, since logrotate is called from cron rather than being a daemon. As a check, run the logrotate command now. No errors should be generated, and neither should any Apache2 logs be rotated.
root@debian8:~# /etc/cron.daily/logrotate
A month or so after making the above change I checked the logs again. As expected, log files had been retained in the proper way, for a longer period.
Apache Configuration Files Renamed
There have been several significant changes to Apache. The version shipped with Debian 7 was Apache 2.2. However, it is version 2.4 that comes with Debian 8. Quite a big jump.
And the first change is a bit of a show stopper. All site configuration files must now have names ending in “.conf“. A file not so named will be ignored. This will impact all running web sites on an upgraded system.
If you have upgraded an existing web server from Debian 7 to Debian 8, go to /etc/apache2/sites-available and rename every configuration file in there to the same name, but with “.conf” on the end. Then go to /etc/apache2/sites-enabled and rename each corresponding link in the same way.
For example:
root@debian8:~# cd /etc/apache2/sites-available
root@debian8:~# mv mysite.co.uk mysite.co.uk.conf
root@debian8:~# mv anothersite.com anothersite.com.conf
root@debian8:~# cd ../sites-enabled
root@debian8:~# mv mysite.co.uk mysite.co.uk.conf
root@debian8:~# mv anothersite.com anothersite.com.conf
And don’t forget to restart apache:
root@debian8:~# service apache2 restart
Apache Rewrite Directives Replaced
With the move to Apache 2.4, “rewrite” directives have been superceded by “Loglevel”. You might be affected if you have lines such as these in any of your site configuration files:
RewriteLog "/tmp/rewrite.log" RewriteLogLevel 3
I had previously added the above lines while debugging a WordPress web site. However, they will cause Apache 2.4 to throw an error. The module mod_rewrite has been updated, and the above directives have been replaced with a per-module logging configuration.
You might want to just comment out the offending directives if they are no longer of use. On the other hand, if rewrite logging is needed, you will have to implement the new per-logging configuration as detailed on the link.
Apache AuthGroupFile Moved
The AuthGroupFile directive is now implemented in the module mod_authz_groupfile. If your web server is not running this module, and the directive appears somewhere in your configuration, eg. in an .htaccess file, an error appears when you try to start Apache:
"/var/www/html/mysite.co.uk/.htaccess: Invalid command 'AuthGroupFile', perhaps misspelled or defined..."
Again, you have the option of commenting out AuthGroupFile if it isn’t serving any purpose, or of installing mod_authz_groupfile. In either case, don’t foget to restart Apache.
root@debian8:~# vi /var/www/html/mysite.co.uk/.htaccess
or
root@debian8:~# a2enmod authz_groupfile
Considering dependency authz_core for authz_groupfile:
Module authz_core already enabled
Enabling module authz_groupfile.
To activate the new configuration, you need to run:
service apache2 restart
root@debian8:~# service apache2 restart
For more information, see the section “Letting more than one person in” in https://httpd.apache.org/docs/2.4/howto/auth.html
IPV6 Enabled by Default
This isn’t really a gotcha. Just an observation. IPV6 is enabled in Debian 8 by default, as confirmed by ifconfig. First in Debian 7:
root@debian7:~# ifconfig -a
eth0 Link encap:Ethernet HWaddr b9:22:ed:d2:54:d3
inet addr:192.168.1.100 Bcast:192.168.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:77159 errors:0 dropped:41 overruns:0 frame:0
TX packets:22710 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:65880875 (62.8 MiB) TX bytes:4020331 (3.8 MiB)
Compare that with Debian 8. Note the extra inet6 line:
root@debian8:~# ifconfig -a
eth0 Link encap:Ethernet HWaddr b9:22:ed:d2:54:d3
inet addr:192.168.1.100 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::c108:8220:7c74:f919/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:141485 errors:0 dropped:23 overruns:0 frame:0
TX packets:80136 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:119798997 (114.2 MiB) TX bytes:14053509 (13.4 MiB)
If you want to turn off IPV6, follow the procedure at the bottom of the Debian IPV6 page. It involves adding a few variables to /etc/sysctl.conf and doing a sysctl -p (no reboot needed).
Mailx and Mime Types
Mailx is a lightweight mail client which is often used to send automated emails. For example, systems administrators might use it to send notification emails from backups, regular jobs or cron invoked scripts.
With the move from Debian 7 to 8, mailx, more specifically heirloom-mailx, has been upgraded from version 12.5.2 to 12.5.4. No major change you would think. However, mailx now behaves slightly differently (more correctly) in the way it encodes emails.
How to Disable DHCP in Debian 8
In some Debian 8 distributions, for example Raspbian, DHCP is enabled by default. This is perhaps fine for a desktop system but less desirable in the case of a server.
Static IP addresses are configured in the file /etc/network/interfaces. Meanwhile, /etc/dhcpd.conf contains the dhcp settings. At the time of writing (1st May 2016), there is little good documentation on the way that these two interact, triggering some forum discussions. But disabling DHCP is quite easy. After you have set a static IP address (by editing /etc/network/interfaces), disable dhcp altogether with
# systemctl disable dhcpcd.service
And reboot the system. Your static IP address will then be set permanently.
If dhcp is not disabled, a situation arises where the network interface has two IP addresses, as confirmed with the ip addr command. One is your static IP address and the other (usually secondary) is the dhcp address received from the local network. An undesirable situation, and one which can confuse other systems on the local network.
Embedded Control Characters
An email composed of plain text but containing a few control characters is now, under mailx 12.5.2, interpreted as binary data and encoded as an attachment (octet stream). The older version of mailx was more tolerant, and would send the same email as plain text, control characters and all. It can be verified by examining the full email headers at the receiving end. The “Content-Type” is set to “octet stream” instead of “text/plain”.
What this means for sysadmins is that automated jobs that previously sent emails (eg regular backup reports) might no longer be able to send those emails correctly. Or at least, not in a way that is easy to read.
I had a cron job that performed regular updates on a Debian 7 system. It used the apt-get command. But because apt-get formats its output with invisible control characters, it stopped working properly.
Several solutions are possible, including changing the script to filter control chars, switching to a different mail client such as mutt, or using sendmail like a proper sysadmin. In the end I chose to simply filter the output through the “strings” command. Here’s an excerpt from the relevant cron tab:
23 16 * * 6 /home/admin/scripts/update.sh | /usr/bin/strings | /usr/bin/mailx -s "debian 8 Debian update" sysadmin@mycompany.co.uk 2>&1
Which worked fine. Examination of the received headers confirmed the desired encoding, ie. text/plain.
SSH PermitRootLogin
By default, sshd PermitRootLogin is set to “without password”, meaning root can login over SSH, but only by using an SSH key. This is good security practice really, and not something you would normally want to change. The recommended way to obtain root access in Debian 8 is to login with a non-root account then “su” to root or (better) use sudo.
If you wish to change the setting, is in /etc/ssh/sshd_config. Edit the file and restart ssh.
Changing PermitRootLogin to “no” would offer greater security, in that root would not be allowed to log in at all, even with an SSH key. But the “without password” option is sometimes used because it allows, for example, remotely triggered backup and rsync operations to operate as root.
END