Automating (Wildcard) cerrtificate renewal with Acme.sh, DeSec.io and some DNS magic

Automatically requesting TLS/SSL certificates using the ACME protocol is quite nice. Usually to just get a regular TLS certificate, you just have to run an ACME capable tool or webserver, which will place a textfile in /.well-known/acme-challenge on your webserver, which is then used to validate control over the domain.

Alternatively you can use DNS based validation. This is even required when requesting wildcard certificates, for example, when requesting a certificate for '*.sig-io.nl'.

When validating using DNS, you need to have your ACME client create a DNS record '_acme-challenge' in your DNS domain. This can be done using various ACME clients, and I myself have written modules/plugins for Dehydrated for various DNS providers over the years.

Recently I've switched from Dehydrated as my preferred ACME client to Acme.sh. This client has a massive amount of DNS providers supported in the main codebase. So it will usually have an interface to whatever DNS system you might use.

Some of my customers however use domains hosted by (for example) TransIP.nl, which does have an API to update DNS-records, but usually takes very long (5-10 minutes) to update records, and also has no granulated access-control to their API. If you have API access, you can do everything with all domains under that account. This means you can't really leave credentials to this account in config-files somewhere.

However, I've now found a solution to these problems that I'm quite satisfied with. I'm using deSEC.io, a new non-profit, community-oriented DNS provider, with a nice API to do DNS validations. With deSEC I can create an API key that is limited in time and source-network. And since deSEC is free, I can create a seperate account just for acme-validation on an otherwise unused domain.

So to create a certificate for a customer-domain, I just have them create a CNAME entry in whatever DNS system they use, which points to my acme-validation only domain-name hosted at deSEC.io.

For example, for sig-io.nl I could create a CNAME record '_acme-challenge.sig-io.nl' pointing to '_acme-challenge.sig-io.nl.acme.example.com'. And then (given that example.com is a domain I manage with deSEC.io) use Acme.sh to create certificates using this alias domain using:

export DOMAIN="sig-io.nl" acme.sh --issue -d "*.${DOMAIN}" --challenge-alias "${DOMAIN}.acme.example.com" --server letsencrypt --dns dns_desec

Various SSL/TLS related sites and services

Upto this week (end January 2020), a friend of mine used to run a couple of SSL/TLS related websites (cipherli.st, certificatemonitor.org, ssldecoder.org) which I used quite frequently. Sadly he has decided to no longer host these sites, for reasons undisclosed. I've taken it apon me to host alternative versions of these services, as I myself use them a lot, and also think they shouldn't disappear from the internet.

Since I do not control the original domain-names, these services can now be found under the following new URL's:

This was quite easy and quick to get running, as Raymii was so friendly to have all the source-code to these services on his github. I've made personal forks of the various repositories, where I will try to keep them somewhat maintained.

Contributions and additions are more then welcome, please visit the relevant sites for links to their github pages.

The CfgMgmtCamp 2019 Recording Setup

As in earlier years, Sig-I/O takes care of some of the lecture recordings at various conferences like CfgMgmtCamp, LOADays, HackerHotel, Eth0 and Techtalks at IT-Gilde, Revspace and Bitlair.

During one of these recording sessions, JJ Asghar (@jjasghar), asked about the video setup and asked if there was a blog-post about how it worked. This will be that blog-post.

Basics

The basics for the recording setup being used is founded on the use of Open Source / Free Software and affordable hardware. Originally a Firewire based camera setup and dv-grab were being used, but since new machines with firewire have been getting harder and harder to find, a switch to something more modern was needed.

The current recording setup consists of the following:

  • 2 USB3 HDMI capture cards

  • A camera with HDMI output, and preferably some form of audio input and connected to the HDMI capture card.

  • HDMI Splitter between the presenter's laptop and the projector, with the second output going to our HDMI capture card.

  • A reasonably fast/modern laptop (i5-6th gen or newer) with USB3 ports running Linux.

  • Open Broadcasting Studio software (obs-studio.org)

The Camera

Currently, 3 different camera's are being used, with their own pro's and con's. For our requirements, the following features are ranked most important:

  • HDMI Output (SDI would also be appropriate, but it much more expensive)

  • XLR Audio inputs (for connecting to the sound-mixer / microphones)

  • Dual SD-Card slots for continuous-recording (nice to have)

  • Low-weight, so the camera can be oriented vertically (9x16)

  • AC/DC power input, so we can run continuously without using batteries

The specific camera's in use:

  • Canon XA10, nice and small, internal 64GB flash and dual SD-slots, 2 XLS's and minijack audio input, basically everything we need. (No option to have XLR+internal mic running at the same time though).

  • Panasonic AG-AC90, large, featurefull, good optics, 2 XLR's and internal mic, all switchable. 2 SD-cards and good audio tuning/interface.

  • Sony (model unknown), records to tape only so no backup to SD-cards, HDMI output, proprietary batteries and hard to find an AC-adapter for, not recommended.

Audio

In many larger venue's amplified audio is essential, so an audio hookup using an XLR cable is preferred (with the mic-signal from the speaker). This will allow us to get a clear and crisp audio input for our recording.

For smaller venues or if there is no audio equiptment present, we usually use some Samson Concert 88 series wireless microphones/beltpacks, as these are decent and affordable, though not as good as the (way to expensive) senheisers.

When a speaker used video and/or audio in their presentation, this will also be received when it is sent over the HDMI port (not always the case). One or more seperate handheld microphones are nice to have, so the presenter can be introduced or mic's can be put in the room for questions.

USB HDMI Capture

Affordable USB-3 HDMI capture cards are available from the usual chinese websites, and are mostly clones of Magewell capture cards. Expect to pay about $60 to $110 for them, and steer clear of the $20 ones, as these are SD-TV capture cards with an HDMI port. The quality on those cheap ones is beyond useless.

The 'good' ones will be marked with words such as: HDMI USB3.0 1080P HDMI Video Capture Card for Windows/Linux/Mac USB UVC UAC.

Basically, these are HDMI input ports which present themselves as generic USB webcam's and audio sources to the computer, and they work with the generic webcam drivers (USB Video Class, USB Audio Class) available in Windows, Mac and Linux.

Open Broadcasting Studio

The final piece of the pie is OBS, a bit of open-source software meant for webcam streaming. In my setup I usually hook up the 2 video/audio sources, name one 'Camera' and the other 'Speaker' or 'Projector', add a logo and titles (using the chatlog feature).

By orienting the camera vertically, the 2 video-streams can be combined more efficiently, leaving out less empty space. OBS allows you to configure various scenes beforehand, and quickly change between these at runtime. It's handy to have a full-screen projector version, some picture-in-picture scenes, and a default scene with everything.

In OBS you can also configure your livestream, for example to Youtube or Twitch (and various other sites/methods).

At most events (with enough bandwidth) we livestream to youtube and simultaniously record to disk. The camera itself is also recording constantly, and this recording can be used in case of issues with the computer-recording or as an audio-source then needed.

Encoding

After the event is completed, the recordings are usually edited using kdenlive to cut out the idle bits before and after the talks, long questions where there isn't a microphone in the room and for delays in demo's / mistakes.

A title-card is added which will give some info about the video and the event. When editing is complete, the video is re-encoded as VP9/Webm for upload to youtube and the event website.

Upload to youtube

Uploading to youtube is done manually, with texts and titles copied from the event schedule. In case of CfgMgmtCamp, most speakers have a twitter-account, and a tweet is sent out with the link, title and hashtags for the video.

When the edited video's have been uploaded to youtube, the livestream versions are usually taken offline, as these are of lower quality.

Improvements for next time

For the next conference, some improvements that can be made:

  • Recodings in more rooms, even if it's just a static unmanned camera, as long as it's on a tripod or fixed mount, audio input would also be preferred, unless the camera is really up close and personal, so the internal mic would have enough signal.

  • Whitelist MAC-addresses for wired uplinks beforehand (Venue blocked youtube)

Working on a Linux System Administration Book

With the release of the first beta version of RHEL-8, and more Linux trainings coming up in the coming months, I've decided to start writing some educational documentation for Linux System Administration on modern versions of CentOS, RHEL, Debian and Ubuntu. The plan is to do this as an Open-Source project and modern open-source methodologies.

The book is begin written using reStructuredText (just like this blog), to be processed into webpages and PDF documents by Sphinx.

To check out the current status (which is still quite limited, as I started this project 2 days ago), see the latest rendered version at https://linuxsysadminbook.sigio.nl/

Currently the source is hosted in a private repository on Bitbucket, but if the project gets a bit more mature, it will most likely be moved to either a public Bitbucket repository, or more likely a public github repository.

Getting (fast) IPv6 at home

IPv6

I value IPv6 access as quite important, however T-Mobile Thuis doesn't offer any official form of IPv6 on their network at this time. I looked into various options to get decent IPv6 for my home network:

  • 6to4

  • HE.net tunnelbroker

  • VPN based access

6in4

6to4 is a deprecated form of IPv6 access, but something that would work for any connection with a fixed IPv4 address. Configuring 6to4 on OpenWRT is quickly done, and it was easy to delegate a /64 subnet to my local network. 6to4 however results in a quite slow connection (of around 10 to 20 megabit), and reachability was spotty at best.

HE.net tunnelbroker

Upto a couple of years ago there used to be various public tunnelbrokers for 6in4 tunnels, and I used the sixxs.net tunnels for almost 10 years myself, however, almost all public tunnel-services seem to have shut down over the years. Only HE.net's tunnelbroker seems to be alive.

I have used a couple of HE.net tunnels throughout the years, and while they work fine in general, they seem quite limited in available bandwidth. This became even worse as sixxs.net shutdown, as most sixxs users migrated to HE.net.

Configuring OpenWRT to use a HE.net tunnel is quite easy, just a matter of copy/pasting the values from your HE.net account into OpenWRT's webinterface and providing credentials for the dynamic updating of the tunnel.

I did some speed-tests and these confirmed my experience, I seemed to be limited to maybe 20mbit, which is a bit low when using a 700+ mbit connection, but this is to be expected for a free service. If your needs are limited, HE.net is a good and free solution.

VPN based access

The third and final method of getting IPv6 that I tried was tunneling over IPv4 to my own server in a public datacenter. At that server (hosted at Hetzner) I have a /56 of IPv6 space, This is not standard, but available on request, and you get a /64 standard with every server or VPS.

I configured wireguard on OpenWRT (client) and my server in the datacenter, and route a /60 subnet of IPv6 space to my home-network. This range can then be split further for a couple of subnets. The remainder of the /56 can be used for some more VPN's and tunnels.

/images/speedtest-v6.png

Using the Wireguard VPN and testing speed on IPv6 I could easily get over 200mbit per second and I have even seen it hit 500mbit.

Todo

  • Try and get IP-TV working, though this doesn't have much priority for me

  • Cancelling voice and tv-subscriptions before the discount runs out :)

OpenWRT Config

Various bits of relevant configuration for OpenWRT:

The configuration of the switchports:

config switch
  option name 'switch0'
  option reset '1'
  option enable_vlan '1'

config switch_vlan
  option device 'switch0'
  option vlan '1'
  option ports '1 2 3 4 6t'
  option vid '1'

config switch_vlan
  option device 'switch0'
  option vlan '300'
  option vid '300'
  option ports '0t 6t'

config switch_vlan
  option device 'switch0'
  option vlan '640'
  option vid '640'
  option ports '0t 6t'

config switch_vlan
  option device 'switch0'
  option vlan '100'
  option ports '0t 6t'
  option vid '100'

Configuration of the Wireguard VPN for IPv6 tunneling

config interface 'WG6'
  option proto 'wireguard'
  option private_key '<wireguard-private-key-base64>'
  list addresses '2001:xxxx:xxx:xxx::3/60'

config wireguard_WG6
  list allowed_ips '::/0'
  option endpoint_host 'ipv4-of-wireguard-server'
  option endpoint_port 'wireguard-portnumber'
  option persistent_keepalive '25'
  option description 'Wireguard-ipv6-tunnel-name'
  option public_key '<public-key-of-wireguard-server-in-base64>'
  option route_allowed_ips '1'

config route6
  option interface 'WG6'
  option target '0::/0'
  option gateway '2001:xxxx:xxx:xxx::1'

T-Mobile Thuis (fiber) with a custom router

I have been using Tweak.nl as my ISP for a few years now, since getting fiber-to-the-home, but Tweak doesn't have their own (non-KPN (incumbant dutch telco)) infrastructure in my area. This means they are limited in offering products provided by KPN, at prices mostly dominated by what they have to pay KPN to get access to the last-mile. In area's where they do have their own fiber infrastructure they can offer gigabit connections at very nice prices (less then €400 per year) and 10-gigabit even, but this is a bit overkill, especially since traffic is then quite limited :)

T-Mobile Thuis (which used to be Vodafone Thuis, but had to be split-off from Vodafone due to anti-compete measures) does have it's own infrastructure in my area, which means they can provide their own networking products. This translates into getting a symmetrical 750 mbit connection for €40,- per month (Sold as €50,- with a permanent 10,- discount). It can also include voice and TV-services, but these cost extra, and would only with if you let T-Mobile control your network by using their router.

/images/speedtest-v4.png

Sadly they still don't provide any form of native IPv6 connectivity, and I'm not too fond of letting ISP's control my routers and internet-infrastucture, so I looked into ways to get a fast and affordable connection whule using my own router, preferably running OpenWRT.

After some research on the Tweakers.net and T-Mobile Thuis web forums I was sure that it wouldn't be too hard to get a fast internet-connection on T-Mobile Thuis using my trusty OpenWRT routers. The requirements basically come down to:

  • Some way to connect to the fiber-connection

  • A fast enough OpenWRT based router that can handle gigabit speeds

  • Support for VLAN's

I initially used a RouterBoard RB750Gr3, since I was already using that as a router for my Tweak connection, but I also had a few Edgerouter ER-X's around. Both routers basically are the same chipset, but with slightly different peripherals. The RB750 has a low amount of flash, but USB and MicroSD ports, so you can add external storage, the ER-X has no USB or MicroSD, but has 256MB of built-in flash, which is more then enough for everything you might want to install on it.

Both routers would be more then sufficient and powerful enough to route a gigabit connection, as they have 256MB of RAM, and a quad-core Mips24 800Mhz cpu. In the end I swapped out the RB750 for the ER-X, since I had a few of those and only one RB750 and would have no use for the USB port on the router.

Installing OpenWRT 18.06.1 is outside of the scope of this article, but I've written about installing it in an earlier post, and documentation is on the OpenWRT wiki

Connecting to the fiber

The first step is finding some way to connect the router to the fiber. There are basically 2 methods to do this:

  • Get a router with an SFP port,and use the SFP module that is provided by T-Mobile, as their own router also uses an SFP-port. This is most useful when the T-Mobile/Guidion mechanic has set-up your connection.

  • Use a media-converter. In my case, there was already a media-converter present, since this was the solution used by my two previous ISP's. This box connects to the fiber, and outputs the conneciton over a RJ45 connection. I used this connection to hoop up to the OpenWRT routers WAN port using a CAT5e cable.

If you want to go for the SFP method, be sure to get a router with SFP-ports, like the ER-X-SFP or the HEX_S

VLAN Configuration

T-Mobile Thuis uses a few different VLAN's, but for our use we only need to use the regular internet VLAN, which is vlan 300. Besides this vlan there is also vlan 100, which is used for T-Mobile's management and vlan 640, which is used for TV.

Configure the WAN port or port that's used for connecting to T-Mobile with the 3 tagged vlan's

/images/tmobile-wan-vlan.png

The internet vlan (300) will give you a public IPv4 address using a DHCP-request. This will also be the default gateway. The Management-lan (100) wil also respond to DHCP-requests, but only return adresses in 10.66.0.0/16 ip-space. There is some traffic on this network, but I haven't looked into it too much yet.

Television

Posts on various forums informed me that IP-TV is normally configured on the 640 vlan. I myself don't use much TV, so I haven't done any configuration yet. T-Mobile also lets you use TV-Anywhere, which is a mobile application (IOS/Android) for streaming TV on any internet-connection, so this can be used as a zero-configuration alternative. I might update this post or publish a new one when I get TV-Streaming working, but I'll probably cancel the TV and voice subscriptions before too long, I only took them because it was cheaper with then without (the first 6 months).

Todo

  • Try and get IP-TV working, though this doesn't have much priority for me

  • Cancelling voice and tv-subscriptions before the discount runs out :)

OpenWRT Config

Various bits of relevant configuration for OpenWRT:

The configuration of the switchports:

config switch
  option name 'switch0'
  option reset '1'
  option enable_vlan '1'

config switch_vlan
  option device 'switch0'
  option vlan '1'
  option ports '1 2 3 4 6t'
  option vid '1'

config switch_vlan
  option device 'switch0'
  option vlan '300'
  option vid '300'
  option ports '0t 6t'

config switch_vlan
  option device 'switch0'
  option vlan '640'
  option vid '640'
  option ports '0t 6t'

config switch_vlan
  option device 'switch0'
  option vlan '100'
  option ports '0t 6t'
  option vid '100'

Sig-I/O now 9 years old, debian-lts sponsor for 3

This week marks the 9th year in Sig-I/O's existance. Looking back over the past nine years shows a nice growth in the number of clients and a collection of interesting assignments Sig-I/O has been involved in.

While the services that Sig-I/O provides have shifted somewhat over these 9 years, some have also remained the same, with some clients already with us from the very beginning. The past few years have been mostly about Managed Hosting, Linux-, Ansible- and Security-Consulting and since a few years also Training, via a partnership with IT-Gilde which has been going steady for 3 years now.

/images/Debian-LTS-2-small.png

This month also marks the 3rd complete year that Sig-I/O has been a sponsor of the Debian LTS project. This sponsorship has also been renewed for the coming year. The Debian LTS project could use some more sponsors, so if your organisation uses Debian servers extensively, it might be beneficial to sponsor them. Debian LTS will support Debian 7 until May of 2019, and will then continue with LTS support for Debian 8 into the 2020's

The next few years will undoubtedly bring many more interesting challenges and opportunities.

Migrated website to a static site powered by Nikola

As you might have noticed if you visited my site before, the entire look and feel has been changed. The site is now powered by the Nikola static-site-generator. The most-recent articles have been migated over, the older articles from the previous website will be restored when they are still relevant.

I had been thinking about using a Static-Site-Generator before, but wordpress was working quite well for me. Recently however I managed to lose my wordpress database, and this privided me with a good opportunity to re-do the site using Nikola.

I'm still getting the hang of writing reStucturedText, and still need to update some pages, but at least the website is back from the abyss. Most articles have been restored from the WayBack-Machine operated by Archive.org.

Using Nikola

In case you are interested in using a static-site generator, this is my workflow:

  • Stream some nice relaxing music

  • Install python3, setup a virtualenv for Nikola

  • Pip install "Nikola[Extras]"

  • nikola init mywebsite

  • Version the newly created site in git

  • Make some changes to the config-file

  • Choose and download a theme

  • Write some posts and pages (nikola new_post -e)

  • Git add all your changes, push to a remote server

  • Nikola build

  • Rsync the output directory to a webserver

OpenWRT/LEDE On a Routerboard RB750Gr3 (Hex3)

The Routerboard RB750Gr3 (aka Hex3) is a nice and very affordable (~$60) hardware platform based on the MediaTek MT7621AT. It features gigabit ethernet ports, and a relatively fast multi-core CPU. However, out of the box it runs RouterOS. While this is a feature-full platform, I found configuration difficult and not very pleasant. Hoping that a build of OpenWRT/LEDE would soon become available, I bought an RB750Gr3 a couple of months ago.

A build became available, but it couldn’t be flashed on the RB750Gr3 without external hardware like a bus-pirate. But this has recently changed, and it’s now possible to TFTP boot a LEDE runtime, which allows access to write to the flash/mtd. Using this method it’s now possible to install OpenWRT/LEDE and upgrade uboot on the device without external hardware.

The OpenWRT site documents this procedure nicely, however, it’s still quite a lot of work to build all the required images and files. So I’ve deciced to host the images that I’ve created here, so other people can skip this rather tedious step of building the image.

The files can be found on https://rb750gr3.sigio.nl/

Please update factory.bin with your own MAC address at offset E000. Currently it’s set to 64:D1:54:AA:BB:CC

  • You can then netboot/tftpboot using the file ‘vmlinux-initramfs.elf’ (boot the RB750 and hold RESET until the leds stop flashing)

  • Wait for the system to come up on 192.168.1.1

  • Login as root

  • Copy the factory.bin uboot.bin uboot-env.bin and lede-17.01.2-ramips-mt7621-rb750gr3-squashfs-sysupgrade.bin to the system (in /tmp for example)

  • mtd write (without reboot) the 4 files to the relevant partitions (cat /proc/mtd to see the names/devices)

Monitoring S.M.A.R.T. attributes in Nagios/Icinga

One of my customers is using various (Samsung) SSD’s in their servers, and the first of these have started reaching their end-of-life. SSD’s have a somewhat different failure scenario then spinning metal disks, so monitoring their life-expectancy can be interesting.

Besides just logging and graphing the SMART attributes, it is also handy to have some alerting on when certain thresholds are crossed. To do this, I’ve written a simple nagios/icinga script which will alert on interesting SMART attributes, and will also calculate the percentage of total guaranteed writes on the SSD’s. Since the guaranteed TBW value will differ between various SSD vendors and product-ranges, this value needs to be specified on the command-line by the user.

/images/2016-03-11-153110_1581x46_scrot.png

I’ve integrated this check-script into my normal monitoring-scripts, but it can off-course also be used as a stand-alone tool. If has options to specify the device to smartctl, so disks behind raid-controllers can also be monitored.

The script can be found in my sysadmin repository on github: check_ssd_attribs