All posts by Phil

“letsencrypt.sh isn’t renewing my certs!”

There’s been a change at some point to the JSON format that Let’s Encrypt returns challenges in.

If you have an “old” installation of letsencrypt.sh that pre-dates v0.2.0, letsencrypt.sh is probably doing this:

$ ./letsencrypt.sh -c
# INFO: Using main config file /home/letsencrypt/letsencrypt.sh/config.sh
Processing somedomain.netcalibre.net
 + Checking domain name(s) of existing cert... unchanged.
 + Checking expire date of existing cert...
 + Valid till Jun 22 16:24:00 2016 GMT (Less than 30 days). Renewing!
 + Signing domains...
 + Generating signing request...
 + Requesting challenge for somedomain.netcalibre.net...
$

and then it silently exits.

Update it from git, move your config file to the new location:

git pull
mv config.sh config
./letsencrypt.sh -c

Using DNS challenge with Let’s Encrypt (and migrating from the official client)

Edited 09/03/2020: Tweaked to reflect the change in the project status and repo change
Edited 23/06/2018: Long overdue update to reflect the fact that the project has changed name from letsencrypt.sh to dehydrated

Edited 30/03/2016: Tweaked to reflect where config.sh.example can be found in letsencrypt.sh’s repo these days
Edited 02/06/2016: Tweaked again to reflect the change from “config.sh” to “config”, and an addition about securing private_key.pem

Let’s be honest – Let’s Encrypt is a great project with lofty ideals, but their official client is just awful bloatware.

I ended up duct-taping functionality I needed by wrapping their client in a shell script that could look at a list of domains and take the appropriate renew actions as required.

With the advent of support for DNS challenges in the Let’s Encrypt infrastructure but apparently no plans to add it to the standard client.. I found myself looking for alternatives.

Enter letsencrypt.sh dehydrated – essentially a 28KB bash wrapper for standard OpenSSL binaries (compare to the 14MB of crap that the official client entails) that doesn’t insist on putting things into /etc/ by default, and includes the functionality to manage a list of hostnames from a text file.

This lightweight client has added support for the DNS-01 challenge mechanism, and calls a hook script during the challenge/completion stage to achieve the actual DNS record changes (and certificate installation) necessary. You can obviously write this hook script in whatever language you see fit. There are plenty of sample hookscripts here.

A chunk of this guide will include the effort required to migrate from an existing Let’s Encrypt install, but you can just skip those bits if you are setting up a “green field” dehydrated deployment.

Getting dehydrated

First, get dehydrated downloaded, copy the sample config file, and start a blank domains.txt file (this file tracks the domains you have/want certs for – more on this later):

git clone https://github.com/dehydrated-io/dehydrated dehydrated
cd dehydrated
cp docs/examples/config config
touch domains.txt

Edit config with your preferred editor; if, like me, you want to use the dns-01 challenge mechanism, make sure you set:

CHALLENGETYPE=”dns-01″

and

HOOK=”$BASEDIR/myhookscript.php”

(or wherever your hook script/binary will be)

You’ll probably also want to set CONTACT_EMAIL to something sensible if you’re setting up a new install and want to be able to recover your Let’s Encrypt ‘account’ if something goes wrong somehow (presumably, if you lose your private key). If you’re importing an existing Let’s Encrypt install and key, the contact email, if you set one, was determined when you created your first certificate with that installation.

Everything else can be left commented out unless you know that you need something specific for your environment.

Migrating from the Let’s Encrypt Python client

Moving from the official client to dehydrated is pretty straightforward with a few migration scripts (source). I offer a copy of the scripts here for expediency and as a backup record in case something happens to the originals, but if the versions here don’t work for you, check the source link to see if there’s an updated version available.

Import existing keys/certs/etc

First, let’s import your existing private keys, Let’s Encrypt issued certs, etc – create a script called import.sh in your dehydrated directory with the following contents:

#!/usr/bin/env bash

set -e
set -u
set -o pipefail

umask 077 # paranoid umask, we're creating private keys

SCRIPTDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
BASEDIR="${SCRIPTDIR}"
LETSENCRYPT="/etc/letsencrypt"

eval "$("${SCRIPTDIR}/dehydrated" --env)"

if [[ ! -e "${LETSENCRYPT}" ]]; then
  echo "No existing letsencrypt files found."
  exit 1
fi

if [[ -e "${BASEDIR}/domains.txt" ]]; then
  DOMAINS_TXT="${BASEDIR}/domains.txt"
elif [[ -e "${SCRIPTDIR}/domains.txt" ]]; then
  DOMAINS_TXT="${SCRIPTDIR}/domains.txt"
else
  echo "You have to create a domains.txt file listing the domains you want certificates for. Have a look at domains.txt.example."
  echo "For the purpose of this import script the file can be empty, but it has to exist."
  exit 1
fi

for certdir in "${LETSENCRYPT}/live/"*; do
  domain="$(basename "${certdir}")"
  echo "Processing ${domain}"

  # Check if we already have a certificate for the same (main) domain
  if [ -e "${BASEDIR}/certs/${domain}" ]; then
    echo " + Skipping: Found existing certificate directory, don't want to delete anything."
    continue
  fi

  # Check if private-key, certificate and fullchain exist
  if [[ ! -e "${certdir}/privkey.pem" ]]; then
    echo " + Skipping: Private key is missing."
    continue
  fi
  if [[ ! -e "${certdir}/cert.pem" ]]; then
    echo " + Skipping: Certificate is missing."
    continue
  fi
  if [[ ! -e "${certdir}/fullchain.pem" ]]; then
    echo " + Skipping: Chain is missing."
    continue
  fi

  # Check if certificate still valid
  if ! openssl x509 -checkend 0 -noout -in "${certdir}/cert.pem" >/dev/null 2>&1; then
    echo " + Skipping: Certificate is expired."
    continue
  fi

  # Import certificate
  timestamp="$(date +%s)"

  echo " + Adding list of domains to ${DOMAINS_TXT}"
  SAN="$(openssl x509 -in "${certdir}/cert.pem" -noout -text | grep -A1 "Subject Alternative Name" | grep "DNS")"
  SAN="${SAN//DNS:/}"
  SAN="${SAN//, / }"
  altnames="${domain}"
  for altname in ${SAN}; do
    if [[ ! "${altname}" = "${domain}" ]]; then
      altnames="${altnames} ${altname}"
    fi
  done
  echo "${altnames}" >> "${DOMAINS_TXT}"

  mkdir -p "${BASEDIR}/certs/${domain}"

  echo " + Importing private key"
  cat "${certdir}/privkey.pem" > "${BASEDIR}/certs/${domain}/privkey-${timestamp}.pem"
  ln -s "privkey-${timestamp}.pem" "${BASEDIR}/certs/${domain}/privkey.pem"

  echo " + Importing certificate"
  cat "${certdir}/cert.pem" > "${BASEDIR}/certs/${domain}/cert-${timestamp}.pem"
  ln -s "cert-${timestamp}.pem" "${BASEDIR}/certs/${domain}/cert.pem"

  echo " + Importing chain"
  cat "${certdir}/fullchain.pem" > "${BASEDIR}/certs/${domain}/fullchain-${timestamp}.pem"
  ln -s "fullchain-${timestamp}.pem" "${BASEDIR}/certs/${domain}/fullchain.pem"
done

and run it to import all your keys/certs/domains into your dehydrated environment.

bash import.sh

Certs, Keys and the like will end up in individual dehydrated/certs/<domain> directories so make sure you update any paths to the certs in your Apache/nginx/Exim/whatever configuration to reflect this.

Importing your Let’s Encrypt account private key

Then to import your private key – create a script called exportkey.pl next to your private_key.json file in your existing Let’s Encrypt installation – by default, you’ll find this in a unique-to-you subdirectory under /etc/letsencrypt/accounts/acme-v01.api.letsencrypt.org/directory/ (eg: /etc/letsencrypt/accounts/acme-v01.api.letsencrypt.org/directory/3ca73eb3d1e1c3ea9123a3ecb278d43e)

NB: You’ll need the perl JSON library for this script to work. ( Debian users can just apt-get install libjson-perl )

#!/usr/bin/env perl

use strict;

use Crypt::OpenSSL::RSA;
use Crypt::OpenSSL::Bignum;
use JSON;
use File::Slurp;
use MIME::Base64;

my $json_file = "private_key.json";
my $json_content = read_file($json_file);
$json_content =~ tr/-/+/;
$json_content =~ tr/_/\//;

my $json = decode_json($json_content);

my $n = Crypt::OpenSSL::Bignum->new_from_bin(decode_base64($json->{n}));
my $e = Crypt::OpenSSL::Bignum->new_from_bin(decode_base64($json->{e}));
my $d = Crypt::OpenSSL::Bignum->new_from_bin(decode_base64($json->{d}));
my $p = Crypt::OpenSSL::Bignum->new_from_bin(decode_base64($json->{p}));
my $q = Crypt::OpenSSL::Bignum->new_from_bin(decode_base64($json->{q}));

my $rsa = Crypt::OpenSSL::RSA->new_key_from_parameters($n, $e, $d, $p, $q);

print($rsa->get_private_key_string());

and execute it:

perl exportkey.pl

This will spit out your private key in the standard PEM format. Cut and paste the output into a new file – private_key.pem – in your dehydrated directory.

Setting up your DNS challenge hook script

With the DNS-01 challenge mechanism, you need some way of creating/deleting DNS records on the fly. If you use a DNS service that provides an API or uses a database back-end, this should be nice and easy for you to implement. If you don’t, well, you’re going to need to figure out that stuff on your own.

If you use one of the DNS services covered by the sample scripts I linked to earlier, you can just download, tweak, and away you go.

If you’re writing your own hook script, your script needs to take arguments on the command line:

$1 = operation
$2 = hostname
$3 = challenge token (not used in DNS-01)
$4 = challenge response

$1 has three possible values indicating the required action from your script:

  • deploy_challenge – create the DNS record
  • clean_challenge – delete the DNS record (cleaning up after the challenge attempt)
  • deploy_cert – deploy the new certificate (you may or may not need to do anything here depending on your environment; such as reload daemons to load the new cert)

The DNS record you need to create/delete is then, effectively:

_acme-challenge.$2. IN TXT “$4”

How best you do this is left as an exercise to you, the reader. If you want to “try out” the DNS challenge mechanism, you can create a manual hook script that simply outputs the record to create/delete and waits for you to press enter to continue – that way you can run dehydrated, the hook script will spit out the DNS record, you can create it, reload your nameserver if necessary, and then press enter in the hookscript when you’re ready to continue:

#!/bin/bash

echo "Method: $1"
echo "_acme-challenge.$2.  IN  TXT \"$4\""

read -s -r -e < /dev/tty

Using dehydrated

NB: If you haven’t imported an existing account key (and perhaps even if you have?) you might need to run the following to accept LEs terms and conditions (read them first!)

./dehydrated --register --accept-terms

Run dehydrated -c once to see what happens (it should check your existing certs and renew any less than 30 days old)

./dehydrated -c

Run a cron, say, once a month, that executes dehydrated -c to refresh any certs that have less than 30 days left on them.

Getting a cert for a new domain

Add your domain to the end of domains.txt followed by any SANs you want, such as the below:

mydomain.com wiki.mydomain.com support.mydomain.com

Remember that combining hostnames into one cert means using the same private key and cert (duh) for all of those hostnames – consider if this is an acceptable risk to you!

Then run

dehydrated -c

to create the request, do the challenge/response – done!

Postscript:

I’ve just realised that on my test systems when migrating, private_key.pem was created world readable. Given this key is used to identify you to Let’s Encrypt, you should definitely consider changing the permissions to only be readable by the user you run dehydrated as:

chmod go-r private_key.pem

Internal IPMI error / Stopping kipmi0

As previously documented on this site, I use Nagios extensively. I’ve used the check_ipmi_sensor plugin for a while now, but have had problems on a Centos 6.6 Supermicro box that I had installed it on.

I’d regularly get hit with failures caused by the IPMI failing to return full payloads, and the freeipmi tools frequently dumping out halfway through execution stating that there was an “Internal IPMI Error” – here’s an example running check_ipmi_sensor from the command line:

# ./check_ipmi_sensor -H localhost -fc 5 -v
ID   | Name            | Type              | State    | Reading    | Units | Event
4    | CPU1 Temp       | Temperature       | Nominal  | 38.00      | C     | 'OK'
71   | CPU2 Temp       | Temperature       | Nominal  | 40.00      | C     | 'OK'
138  | PCH Temp        | Temperature       | Nominal  | 31.00      | C     | 'OK'
205  | System Temp     | Temperature       | Nominal  | 26.00      | C     | 'OK'
272  | Peripheral Temp | Temperature       | Nominal  | 41.00      | C     | 'OK'
339  | Vcpu1VRM Temp   | Temperature       | Nominal  | 33.00      | C     | 'OK'
406  | Vcpu2VRM Temp   | Temperature       | Nominal  | 39.00      | C     | 'OK'
473  | VmemABVRM Temp  | Temperature       | Nominal  | 29.00      | C     | 'OK'
540  | VmemCDVRM Temp  | Temperature       | Nominal  | 26.00      | C     | 'OK'
607  | VmemEFVRM Temp  | Temperature       | Nominal  | 38.00      | C     | 'OK'
674  | VmemGHVRM Temp  | Temperature       | Nominal  | 32.00      | C     | 'OK'
741  | P1-DIMMA1 Temp  | Temperature       | Nominal  | 27.00      | C     | 'OK'
808  | P1-DIMMB1 Temp  | Temperature       | Nominal  | 27.00      | C     | 'OK'
875  | P1-DIMMC1 Temp  | Temperature       | Nominal  | 27.00      | C     | 'OK'
942  | P1-DIMMD1 Temp  | Temperature       | Nominal  | 26.00      | C     | 'OK'
1009 | P2-DIMME1 Temp  | Temperature       | Nominal  | 28.00      | C     | 'OK'
1076 | P2-DIMMF1 Temp  | Temperature       | Nominal  | 29.00      | C     | 'OK'
1143 | P2-DIMMG1 Temp  | Temperature       | Nominal  | 28.00      | C     | 'OK'
1210 | P2-DIMMH1 Temp  | Temperature       | Nominal  | 29.00      | C     | 'OK'
1411 | FAN3            | Fan               | Nominal  | 6500.00    | RPM   | 'OK'
1478 | FAN4            | Fan               | Nominal  | 6400.00    | RPM   | 'OK'
ipmi_sensor_read: internal IPMI error

-> Execution of /usr/sbin/ipmi-sensors failed with return code 1.
-> /usr/sbin/ipmi-sensors was executed with the following parameters:
   sudo /usr/sbin/ipmi-sensors --quiet-cache --sdr-cache-recreate --interpret-oem-data --output-sensor-state --ignore-not-available-sensors

This obviously isn’t especially ideal – and I was seeing this every 10-30 minutes when Nagios was running its regular checks. First glances, it looked like it could be caused by defective hardware, but fortunately, that wasn’t the case – just my checks colliding with kipmid.

Centos 6 has built ipmi_si into the kernel by default – and kipmid starts on boot and starts polling any detected IPMI devices (you’ll probably see [kipmi0] running). Given I’m configuring Nagios for monitoring, I’ve no interest in the kernel helper polling my IPMI and tying it up, and if you’re reading this and you have a kernel ipmid thread, chances are, neither do you. Alternatively, it’s possible you’re reading this because your kipmi0 process is claiming to use 100% of a CPU core, perhaps because you did a bmc cold reboot or similar – in which case, this is probably useful for you too, even if you want to keep kipmid running.

You can stop kipmid in its tracks by hot-removing the IPMI device from it’s list of detected devices; first, get the parameters for the detected device like this:

# cat /proc/ipmi/0/params
kcs,i/o,0xca2,rsp=1,rsi=1,rsh=0,irq=0,ipmb=0

Then, take the output from the above, prefix it with “remove,” and use /sys/module/ipmi_si/parameters/hotmod to remove the device:

# echo "remove,kcs,i/o,0xca2,rsp=1,rsi=1,rsh=0,irq=0,ipmb=0" > /sys/module/ipmi_si/parameters/hotmod

The kipmid thread will be cleaned up immediately without requiring a reboot. freeipmi’s various utilities do not use ipmi_si/kipmid and will continue to work just fine.

If all you wanted to do was restart kipmid for some reason, you could then re-add the device by instead prefixing with “add,”:

# echo "add,kcs,i/o,0xca2,rsp=1,rsi=1,rsh=0,irq=0,ipmb=0" > /sys/module/ipmi_si/parameters/hotmod

Whereupon kipmi0 will resume.

If stopping kipmid fixes your issue (it provided immediate relief in my case), make sure it stays gone by adding the following to your kernel options in /boot/grub/menu.lst (or, wherever your bootloader configuration is, if you’re not using the default grub environment)

ipmi_si.force_kipmid=0

Weirdly, kipmid doesn’t seem to cause problems with Ubuntu 14.04 (on a Dell R200) or Debian 8 (on a different Supermicro board), so perhaps this is a centos specific issue.

Tips for Configuring Nagios3 Efficiently – part 1

Back when I started using Nagios (I think ~1.2 or earlier) I don’t remember many options for being all that efficient in terms of “lines of config written” – certainly, any options for being efficient that there may have been ended up being overlooked in the rush to get it up and running, and I’ve been largely been using the same configuration files (and style) ever since – though I did start using host and service templates as soon as I became aware of them some time back in the 2.x branch days.

In the spirit of self-improvement, I’ve been revisiting the Nagios configuration syntax as part of rolling out a fresh monitoring host based on Nagios3, and have significantly reduced the number of lines of config my Nagios installation depends on as a result.

Continue reading Tips for Configuring Nagios3 Efficiently – part 1

The LCHost Debian package mirror

As part of giving back to the community LCHost runs a Debian package mirror (including backports) (for i386 and amd64) which was recently added to the official mirrors list.

I spent a little time making it automatically detect changes at our source mirror within a couple of minutes and pull down, so at most it’ll never be more than 5 minutes behind the top-level mirrors.

To use our debian mirror, simply replace your regular mirror definition in /etc/apt/sources.list with:

deb http://mirror.lchost.net/debian/ stable main contrib
deb-src http://mirror.lchost.net/debian/ stable main contrib

(You may choose to use the release name in place of “stable” to ensure you never accidentally go between major releases – so for wheezy, simply replace the word stable with wheezy)

If you use backports, you can get those packages from us too, using something like:

deb http://mirror.lchost.net/debian/ wheezy-backports main

Installing Nagios3 on Debian Wheezy

It’s pretty straightforward to install Nagios on a Debian system but if you want to be able to use the web interface to control the nagios process a little more work is required.

Starting with a blank slate (apt/dpkg will ensure any required prerequisites will be installed):

# apt-get install nagios3 apache2-suexec

You’ll be asked to set a password for the nagiosadmin user for the web interface.

Enable check_external_commands in Nagios to enable the ability to mute alarms, make comments, restart the nagios process etc from the web interface (pretty much invaluable, but be aware of the inherent risks in enabling the ability to influence the process from “outside”)

# sed -i -e 's/check_external_commands=0/check_external_commands=1/' /etc/nagios3/nagios.cfg
# /etc/init.d/nagios3 restart

Edit the nagios3 apache2 config include to make the web interface scripts run as the nagios user so that the web interface can write to the nagios command pipe; inserting the following at the top of /etc/nagios3/apache2.conf:

User nagios
Group nagios

Restart apache..

# /etc/init.d/apache2 restart

And you’re pretty much done! You can go to http://YOUR_HOST_NAME/nagios3/ and log in with your nagiosadmin password you set up when prompted at the start of this process.

Now, you can get started with creating host and service configuration files in /etc/nagios3/conf.d/ to monitor your servers/network/etc

New(ish)

So, I used to post this stuff onto a tumblr blog at http://cluebyfour.tumblr.com but decided that – frankly – I’m not exactly short of server resource, so I’ve moved what little content there was in-house.

Lately I have found that poor documentation – or simply vast documentation – has been irking me more and more and I’ll likely start documenting things I figure out for both my own sanity and that of others who follow in my footsteps.

Reset virtualhost / domain file permissions plesk 9.x (and possibly 10.x?) linux

Having just completed a rescue job on a customer’s Plesk 9 server, all of the files in /var/www/vhosts had incorrect permissions on them resulting in visitors to any of the domains receiving a 403 Permission Denied error. The Plesk disaster recoveryKB article (KB 112699) has a section on fixing ownership of transferred files (step 9) but nothing on fixing vhost permissions.

The Parallels knowledgebase has this article (KB 6572) on restoring permissions on ONE domain, but that’s not all that helpful when you need to restore permissions on EVERY domain on the server. Plesk Windows users get the ability to batch “Check Permissions” in the control panel interface, but Linux users get no such love.

Instead, you can execute the following (as root!) to reset the permissions for every virtual host on the server:

# mysql -uadmin -p`cat /etc/psa/.psa.shadow` -Dpsa -Ns -e”select domains.name, sys_users.login as username from domains left join hosting on domains.id=hosting.dom_id left join sys_users on hosting.sys_user_id=sys_users.id left join accounts on  sys_users.account_id=accounts.id where htype=’vrt_hst’;” | awk ‘{print “/usr/local/psa/admin/sbin/vhostmng —install-vhost —vhost-name=” $1 ” —user-name=” $2 ” —set-content-permissions”}’ | sh -x

It should be noted that I have only tested this on a 9.x server but I think the psa database structure is similar on 10.x so you may find this works for you. The trickiest part of this was figuring out how to get a listing of virtual host domains mapped to system users out of the psa database as the schema is a bit convoluted – hat tip to rackerbox.com for having something I could adapt.

Debian 6 (Debian Squeeze) & Debian 7 (Debian Wheezy) reboot… doesn’t.

Someone made kexec-tools handle reboot requests by default seemingly. This allows the system to skip BIOS/POST etc and just drop to a minimal runlevel and start a kernel again.

This is great if you only have debian on your system and particularly great if you spend a lot of time changing kernels – when you issue reboot, or shutdown -r now (etc) kexec-tools intercepts the command and does a warm-restart rather than resetting the machine cold – if you don’t need to, why wait through all the BIOS checks, bootroms, etc, right?

Except some of us reboot because we want to change OS. I’d argue that it should perhaps be the default behaviour to cold-reboot (and the installer could, perhaps, ask!) or that KDE should have a button for “warm restart” and one for “cold reboot” or whatever, but anyway.

If you want to make reboot actually reboot the system you’ll want to:

# dpkg-reconfigure kexec-tools

And tell it to not use kexec-tools to handle reboots. If you’re never going to want kexec-tools, you can probably uninstall it using apt, but I just disabled it. It’s useful on the odd occasion I do want to just upgrade the kernel to enable it, reboot, and disable it again, I suppose.

Some SEO, perhaps?

Debian 6 Squeeze won’t reboot
Debian 6 Squeeze reboot doesn’t go to grub
Debian 6 Squeeze reboot dualboot
Debian 7 Wheezy won’t reboot
Debian 7 Wheezy reboot doesn’t go to bios