OpenBSD Network Gateway on EdgeRouter Lite

EdgeRouter Lite is a great device to run at the edge of a home network. It becomes even better when it's running OpenBSD. This guide documents how to setup such a gateway. There are accompanying git repos to somewhat automate the process as well.

Why?

This is a follow up to FreeBSD Network Gateway on EdgeRouter Lite. I observed some issues, possibly related to network drivers, where the network would become unresponsive after some time. I first tried OpenWRT on the same hardware and didn't observe any unresponsiveness. I wanted to use a BSD instead of Linux so tried OpenBSD.

OpenBSD has all the required ingredients to make a secure and easy-to-administer firewall. I find the syntax of pf much better than iptables and would always prefer it if I had a choice.

Before you read this guide make sure you have read these resources which I used as well for this guide.

Required Tools

The following two cables connect to each other so you can connect from USB on your Mac to the serial port on ERL3.

  • USB to Serial interface cable
  • Serial to RJ45 Console Adapter Cable for Cisco Routers

You also need:

  • screen installed on your Mac
  • Phillips head screwdriver size 0
  • Wired network with DHCP server to use during install

Download OpenBSD

At the time of writing the latest release of OpenBSD was 6.1 so that's what I downloaded.

$ curl -O http://ftp.openbsd.org/pub/OpenBSD/6.1/octeon/miniroot61.fs

Hardware Setup and First Boot

Once you've downloaded the file it's time to write it to a USB drive. There are two options: overwrite the original drive in the ERL or buy a new drive.

I tried the second option first and wrote to a new Sandrive Ultra Fit 32GB USB 3.0 Flash Drive (SDCZ43-032G-GAM46). It did not work and I later found on some blog that those drives do not work. I bought another drive -- Samsung 32GB USB 3.0 Flash Drive Fit (MUF-32BB/AM) -- and that did work.

It is easy to take the drive out of the ERL enclosure. Take out three screws from the back to open the enclosure. Wiggle the drive up and down with a little more than gentle force and it'll slide out.

Before overwriting the original drive I created a backup image using dd. On my macOS it showed up as /dev/disk2 upon insertion. Replace rdrive596870 with the name of the drive on your machine.

$ diskutil list
$ diskutil unmountdrive /dev/rdrive596870
$ sudo dd if=/dev/rdrive596870 of=original-erl.img

With the backup created it was easy to overwrite the drive with the downloaded file.

$ sudo dd if=miniroot61.fs of=/dev/rdrive596870 bs=1m && sync
$ diskutil eject /dev/rdrive596870

Reinsert the drive, put back the enclosure, and put back the screws.

DO NOT power on ERL3 just yet.

Connect an ethernet cable to eth0 that's connected to your network. We need a DHCP server to give an IP address to ERL3.

Connect to the serial port.

Check which device your Mac has identified for the serial connection. In my case it was /dev/tty.usbserial.

$ ls -ltr /dev/*usb* | grep tty

Start a screen session on port 115200.

$ screen /dev/tty.usbserial 115200

Power on ERL3 and watch it boot on the screen session.

Install OpenBSD

Once the device is powered up and ERL3 has booted successfully, you'll see this prompt.

Octeon ubnt_e100#

Load the miniroot.

Octeon ubnt_e100# fatload usb 0 $loadaddr bsd.rd

Boot the image.

Octeon ubnt_e100# bootoctlinux

After a lot of text scrolling across you'll be asked to make a choice from the following menu.

Welcome to the OpenBSD/octeon 6.1 installation program.
(I)nstall, (U)pgrade, (A)utoinstall or (S)hell?

Follow each prompt and answer appropriately. A sample session of my install is below.

Welcome to the OpenBSD/octeon 6.1 installation program.
(I)nstall, (U)pgrade, (A)utoinstall or (S)hell? i
At any prompt except password prompts you can escape to a shell by
typing '!'. Default answers are shown in []'s and are selected by
pressing RETURN.  You can exit this program at any time by pressing
Control-C, but this can leave your system in an inconsistent state.

Terminal type? [vt220]
System hostname? (short form, e.g. 'foo') erl

Available network interfaces are: cnmac0 cnmac1 cnmac2.
Which network interface do you wish to configure? (or 'done') [cnmac0]
IPv4 address for cnmac0? (or 'dhcp' or 'none') [dhcp]
cnmac0: no link ..... got link
DHCPDISCOVER on cnmac0 - interval 1
DHCPOFFER from 10.10.10.1 (REDACTED_FOR_PRIVACY)
DHCPREQUEST on cnmac0 to 255.255.255.255
DHCPACK from 10.10.10.1 (REDACTED_FOR_PRIVACY)
bound to 10.10.10.74 -- renewal in 21600 seconds.
IPv6 address for cnmac0? (or 'rtsol' or 'none') [none] rtsol
Available network interfaces are: cnmac0 cnmac1 cnmac2.
Which network interface do you wish to configure? (or 'done') [done]
DNS domain name? (e.g. 'bar.com') [my.domain] my.example.com
Using DNS nameservers at 10.10.10.1 8.8.8.8

Password for root account? (will not echo)
Password for root account? (again)
Start sshd(8) by default? [yes]
Setup a user? (enter a lower-case loginname, or 'no') [no] ubnt
Full name for user ubnt? [ubnt]
Password for user ubnt? (will not echo)
Password for user ubnt? (again)
WARNING: root is targeted by password guessing attacks, pubkeys are safer.
Allow root ssh login? (yes, no, prohibit-password) [no] yes
What timezone are you in? ('?' for list) [America/Los_Angeles]

Available disks are: sd0.
Which disk is the root disk? ('?' for details) [sd0]
Disk: sd0       geometry: 487/255/63 [7831552 Sectors]
Offset: 0       Signature: 0xAA55
            Starting         Ending         LBA Info:
#: id      C   H   S -      C   H   S [       start:        size ]
-------------------------------------------------------------------------------
*0: 0C      0   1   2 -      1 103  38 [          64:       22528 ] FAT32L
1: 00      0   0   0 -      0   0   0 [           0:           0 ] unused
2: 00      0   0   0 -      0   0   0 [           0:           0 ] unused
3: 00      0   0   0 -      0   0   0 [           0:           0 ] unused
Use (W)hole disk or (E)dit the MBR? [whole]
Creating a FAT partition and an OpenBSD partition for rest of sd0...done.
/dev/rsd0i: 65372 sectors in 16343 FAT16 clusters (2048 bytes/cluster)
bps=512 spc=4 res=1 nft=2 rde=512 mid=0xf8 spf=64 spt=63 hds=255 hid=64 bsec=65536
The auto-allocated layout for sd0 is:
#                size           offset  fstype [fsize bsize   cpg]
a:           887.8M            65600  4.2BSD   2048 16384     1 # /
b:           255.6M          1883808    swap
c:          3824.0M                0  unused
d:          2269.6M          2407296  4.2BSD   2048 16384     1 # /usr
e:           378.9M          7055488  4.2BSD   2048 16384     1 # /home
i:            32.0M               64   MSDOS
Use (A)uto layout, (E)dit auto layout, or create (C)ustom layout? [a]
Rounding size to bsize (32 sectors): 1818208
Rounding offset to bsize (32 sectors): 2407296
Rounding size to bsize (32 sectors): 4648192
Rounding size to bsize (32 sectors): 776032
/dev/rsd0a: 887.8MB in 1818208 sectors of 512 bytes
5 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each
newfs: reduced number of fragments per cylinder group from 48496 to 48304 to enlarge last cylinder group
/dev/rsd0e: 378.9MB in 776032 sectors of 512 bytes
5 cylinder groups of 94.34MB, 6038 blocks, 12160 inodes each
/dev/rsd0d: 2269.6MB in 4648192 sectors of 512 bytes
12 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each
/dev/sd0a (5e1cbc1a46a4044c.a) on /mnt type ffs (rw, asynchronous, local)
/dev/sd0e (5e1cbc1a46a4044c.e) on /mnt/home type ffs (rw, asynchronous, local, nodev, nosuid)
/dev/sd0d (5e1cbc1a46a4044c.d) on /mnt/usr type ffs (rw, asynchronous, local, nodev)

Let's install the sets!
Location of sets? (disk http nfs or 'done') [http]
HTTP proxy URL? (e.g. 'http://proxy:8080', or 'none') [none]
HTTP Server? (hostname, list#, 'done' or '?') [ftp.OpenBSD.org]
Server directory? [pub/OpenBSD/6.1/octeon]

Select sets by entering a set name, a file name pattern or 'all'. De-select
sets by prepending a '-' to the set name, file name pattern or 'all'. Selected
sets are labelled '[X]'.
    [X] bsd           [X] base61.tgz    [X] game61.tgz    [X] xfont61.tgz
    [X] bsd.rd        [X] comp61.tgz    [X] xbase61.tgz   [X] xserv61.tgz
    [ ] bsd.mp        [X] man61.tgz     [X] xshare61.tgz
Set name(s)? (or 'abort' or 'done') [done] -game61.tgz
    [X] bsd           [X] base61.tgz    [ ] game61.tgz    [X] xfont61.tgz
    [X] bsd.rd        [X] comp61.tgz    [X] xbase61.tgz   [X] xserv61.tgz
    [ ] bsd.mp        [X] man61.tgz     [X] xshare61.tgz
Set name(s)? (or 'abort' or 'done') [done]
Get/Verify SHA256.sig   100% |**************************|  1376       00:00
Signature Verified
Get/Verify bsd          100% |**************************|  5277 KB    00:07
Get/Verify bsd.rd       100% |**************************|  8324 KB    00:11
Get/Verify base61.tgz   100% |**************************| 51721 KB    05:41
Get/Verify comp61.tgz   100% |**************************| 48758 KB    01:08
Get/Verify man61.tgz    100% |**************************|  8721 KB    00:29
Get/Verify xbase61.tgz  100% |**************************| 16463 KB    00:19
Get/Verify xshare61.tgz 100% |**************************|  4407 KB    00:10
Get/Verify xfont61.tgz  100% |**************************| 39350 KB    00:45
Get/Verify xserv61.tgz  100% |**************************|  5043 KB    00:05
Installing bsd          100% |**************************|  5277 KB    00:02
Installing bsd.rd       100% |**************************|  8324 KB    00:03
Installing base61.tgz   100% |**************************| 51721 KB    01:57
Extracting etc.tgz      100% |**************************|   189 KB    00:00
Installing comp61.tgz   100% |**************************| 48758 KB    01:51
Installing man61.tgz    100% |**************************|  8721 KB    00:27
Installing xbase61.tgz  100% |**************************| 16463 KB    00:32
Extracting xetc.tgz     100% |**************************|  7007       00:00
Installing xshare61.tgz 100% |**************************|  4407 KB    00:27
Installing xfont61.tgz  100% |**************************| 39350 KB    00:54
Installing xserv61.tgz  100% |**************************|  5043 KB    00:06
Location of sets? (disk http nfs or 'done') [done]
Time appears wrong.  Set to 'Tue Jun  6 15:22:59 PDT 2017'? [yes]
Saving configuration files...done.
Making all device nodes...done.

CONGRATULATIONS! Your OpenBSD install has been successfully completed!
To boot the new system, enter 'reboot' at the command prompt.
When you login to your new system the first time, please read your mail
using the 'mail' command.

INSTALL.octeon describes how to configure U-Boot to boot OpenBSD.

#

I configured root SSH access because I use Ansible later on and found it worked better when it logged in as root. Feel free to disable root SSH for better security.

I created a regular user ubnt to use for SSH access.

I did not install game61.tgz just because. I didn't feel it necessary to install it.

Time to reboot.

# reboot

Swap the Bootloader

Once ERL3 has rebooted you'll be at the Octeon ubnt_e100# prompt again because the bootloader is not at the expected location.

We'll swap the bootloader. But first, let's get a list of the existing environment variables. Save this output and keep in a safe place. You may need to undo risky steps ahead.

Octeon ubnt_e100# printenv
baudrate=115200
download_baudrate=115200
nuke_env=protect off $(env_addr) +$(env_size);erase $(env_addr) +$(env_size)
autoload=n
ethact=octeth0
bootdelay=5
bootcmd=fatload usb 0 $loadaddr vmlinux.64;bootoctlinux $loadaddr coremask=0x3 root=/dev/sda2 rootdelay=15 rw rootsqimg=squashfs.img rootsqwdir=w mtdparts=phys_mapped_flash:512k(boot0),512k(boot1),64k@1024k(eeprom)
loadaddr=0x9f00000
numcores=2
stdin=serial
stdout=serial
stderr=serial
env_addr=0x1fbfe000
env_size=0x2000
flash_base_addr=0x1f800000
flash_size=0x400000
uboot_flash_addr=0x1f880000
uboot_flash_size=0x70000
flash_unused_addr=0x1f8f0000
flash_unused_size=0x310000
bootloader_flash_update=bootloaderupdate

Notice bootcmd above. That's what we'll swap.

Octeon ubnt_e100# setenv old_bootcmd "${bootcmd}"
Octeon ubnt_e100# setenv bootcmd 'fatload usb 0 $loadaddr bsd;bootoctlinux rootdev=/dev/sd0'
Octeon ubnt_e100# setenv bootdelay 5
Octeon ubnt_e100# saveenv
Octeon ubnt_e100# reset

After ERL3 reboots it should happily boot OpenBSD.

A sample session is below.

U-Boot 1.1.1 (UBNT Build ID: 4670715-gbd7e2d7) (Build time: May 27 2014 - 11:16:22)

BIST check passed.
UBNT_E100 r1:2, r2:18, f:4/71, serial #: REDACTED_FOR_PRIVACY
MPR 13-00318-18
Core clock: 500 MHz, DDR clock: 266 MHz (532 Mhz data rate)
DRAM:  512 MB
Clearing DRAM....... done
Flash:  4 MB
Net:   octeth0, octeth1, octeth2

USB:   (port 0) scanning bus for devices... 1 USB Devices found
    scanning bus for storage devices...
Device 0: Vendor:          Prod.: USB DISK 2.0     Rev: PMAP
            Type: Removable Hard Disk
            Capacity: 3824.0 MB = 3.7 GB (7831552 x 512)                 0
reading bsd
...........................

5404313 bytes read
ELF file is 64 bit
Allocating memory for ELF segment: addr: 0xffffffff81000000 (adjusted to: 0x1000000), size 0x52fe80
Allocated memory for ELF segment: addr: 0xffffffff81000000, size 0x52fe80
Processing PHDR 0
Loading 4a2978 bytes at ffffffff81000000
Clearing 8d508 bytes at ffffffff814a2978
## Loading Linux kernel with entry point: 0xffffffff81000000 ...
Bootloader: Done loading app on coremask: 0x1
bootmem desc 0x24108 version 3.0
avail phys mem 0x0000000000100290 - 0x0000000000fffb50
avail phys mem 0x000000000152fe80 - 0x0000000008100000
avail phys mem 0x0000000008100010 - 0x000000000fffdc00
avail phys mem 0x0000000410000000 - 0x000000041ff00000
Total DRAM Size 0x0000000020000000
mem_layout[0] page 0x0000000000000041 -> 0x00000000000003FF
mem_layout[1] page 0x000000000000054C -> 0x0000000000002040
mem_layout[2] page 0x0000000000002041 -> 0x0000000000003FFF
mem_layout[3] page 0x0000000000104000 -> 0x0000000000107FC0
boot_desc->argv[0] = bootoctlinux
boot_desc->argv[1] = rootdev=/dev/sd0
Initial setup done, switching console.
boot_desc->desc_ver:7
boot_desc->desc_size:400
boot_desc->stack_top:0
boot_desc->heap_start:0
boot_desc->heap_end:0
boot_desc->argc:2
boot_desc->flags:0x5
boot_desc->core_mask:0x1
boot_desc->dram_size:512
boot_desc->phy_mem_desc_addr:0
boot_desc->debugger_flag_addr:0xa44
boot_desc->eclock:500000000
boot_desc->boot_info_addr:0x1001f0
boot_info->ver_major:1
boot_info->ver_minor:2
boot_info->stack_top:0
boot_info->heap_start:0
boot_info->heap_end:0
boot_info->boot_desc_addr:0
boot_info->exception_base_addr:0x1000
boot_info->stack_size:0
boot_info->flags:0x5
boot_info->core_mask:0x1
boot_info->dram_size:512
boot_info->phys_mem_desc_addr:0x24108
boot_info->debugger_flags_addr:0
boot_info->eclock:500000000
boot_info->dclock:266000000
boot_info->board_type:20002
boot_info->board_rev_major:2
boot_info->board_rev_minor:18
boot_info->mac_addr_count:3
boot_info->cf_common_addr:0
boot_info->cf_attr_addr:0
boot_info->led_display_addr:0
boot_info->dfaclock:0
boot_info->config_flags:0x8
Copyright (c) 1982, 1986, 1989, 1991, 1993
        The Regents of the University of California.  All rights reserved.
Copyright (c) 1995-2017 OpenBSD. All rights reserved.  https://www.OpenBSD.org

OpenBSD 6.1 (GENERIC) #0: Mon Apr  3 07:47:02 UTC 2017
    visa@octeon:/usr/src/sys/arch/octeon/compile/GENERIC
real mem = 536870912 (512MB)
avail mem = 524255232 (499MB)
warning: no entropy supplied by boot loader
mainbus0 at root
cpu0 at mainbus0: CN50xx CPU rev 0.1 500 MHz, Software FP emulation
cpu0: cache L1-I 32KB 4 way D 8KB 64 way, L2 128KB 8 way
clock0 at mainbus0: int 5
iobus0 at mainbus0
dwctwo0 at iobus0 base 0x1180068000000 irq 56
usb0 at dwctwo0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "Octeon DWC2 root hub" rev 2.00/1.00 addr 1
octrng0 at iobus0 base 0x1400000000000 irq 0
cn30xxgmx0 at iobus0 base 0x1180008000000
cnmac0 at cn30xxgmx0: RGMII, address REDACTED_FOR_PRIVACY
atphy0 at cnmac0 phy 7: AR8035 10/100/1000 PHY, rev. 2
cnmac1 at cn30xxgmx0: RGMII, address REDACTED_FOR_PRIVACY
atphy1 at cnmac1 phy 6: AR8035 10/100/1000 PHY, rev. 2
cnmac2 at cn30xxgmx0: RGMII, address REDACTED_FOR_PRIVACY
atphy2 at cnmac2 phy 5: AR8035 10/100/1000 PHY, rev. 2
uar: ns16550a, 64 byte fifo
com0: console
com1 at uartbus0 base 0x1180000000c00 irq 35: ns16550a, 64 byte fifo
/dev/ksyms: Symbol table not valid.
umass0 at uhub0 port 1 configuration 1 interface 0 "vendor 0x13fe USB DISK 2.0" rev 2.00/1.00 addr 2
umass0: using SCSI over Bulk-Only
scsibus0 at umass0: 2 targets, initiator 0
sd0 at scsibus0 targ 1 lun 0: <, USB DISK 2.0, PMAP> SCSI4 0/direct removable serial.REDACTED_FOR_PRIVACY
sd0: 3824MB, 512 bytes/sector, 7831552 sectors
vscsi0 at root
scsibus1 at vscsi0: 256 targets
softraid0 at root
scsibus2 at softraid0: 256 targets
boot device: sd0
root on sd0a (5e1cbc1a46a4044c.a) swap on sd0b dump on sd0b
WARNING: No TOD clock, believing file system.
WARNING: CHECK AND RESET THE DATE!
Automatic boot in progress: starting file system checks.
/dev/sd0a (5e1cbc1a46a4044c.a): file system is clean; not checking
/dev/sd0e (5e1cbc1a46a4044c.e): file system is clean; not checking
/dev/sd0d (5e1cbc1a46a4044c.d): file system is clean; not checking
setting tty flags
pf enabled
starting network
cnmac0: link is not up, the packet was dropped
DHCPREQUEST on cnmac0 to 255.255.255.255
DHCPACK from 10.10.10.1 (REDACTED_FOR_PRIVACY)
bound to 10.10.10.190 -- renewal in 15768000 seconds.
starting early daemons: syslogd pflogd ntpd.
starting RPC daemons:.
savecore: /bsd: kvm_read: version misread
checking quotas: done.
kvm_mkdb: can't open /dev/ksyms
clearing /tmp
kern.securelevel: 0 -> 1
creating runtime link editor directory cache.
preserving editor files.
starting network daemons: sshd smtpd sndiod.
starting local daemons: cron.
Tue Jun  6 15:22:59 PDT 2017

OpenBSD/octeon (octeon1) (console)

login:

Network Design

Comcast is my internet service provider (ISP) and thus the Ansible stuff is configured to work with it. Be especially careful with the IPv6 stuff since your provider may be different.

I wanted both IPv4 and IPv6 running in my network and to reach out to the internet.

For the wireless LAN (WLAN) or WiFi part of the network I'm using a simple access point (AP) that provides no routing, DHCP, or other services. It acts as a dumb AP providing WiFi services only.

Networking

ERL3 has three network interfaces: eth0, eth1, and eth2. They are recognized by OpenBSD as cnmac0, cnmac1, and cnmac2 respectively.

I'll use cnmac0 (eth0) as the WAN port. I'll bridge cnmac1 (eth1) and cnmac2 (eth2) for LAN.

cnmac0

To troubleshoot my internet service I use a USB-to-ethernet dongle and its MAC address is what's configured with Comcast on my cable modem. I use the same MAC address to override any gateway device that connects to the cable modem since I can swap devices without having to contact Comcast to update settings at their end.

I override the MAC address of cnmac0 in my config but you don't have to.

I was confused for a while and got it wrong on getting an IPv6 non-temporary address (NA) via dhcp6c. Just like I would expect a client OS to ask a DHCPv6 server to hand it an IPv6 address, I was trying to do it on this router. I've explained what I was doing wrong in a section below.

What I needed to understand was that the WAN interface (cnmac0) only needed IPv6 autoconf to get a link-local address going. Then it was a matter of adding a default route. With this setup ERL could reach any globally routable address through the ISP-provided gateway.

# vi /etc/hostname.cnmac0
dhcp lladdr OV:ER:RI:DE:00:00
up
inet6 autoconf
!/sbin/route add ::/0 -ifp cnmac0 fe80::

It took me a while to get this right but it's very important that you understand why I did it this way.

cnmac1

# vi /etc/hostname.cnmac1
up

cnmac2

# vi /etc/hostname.cnmac2
up

vether0

Create a virtual ethernet interface to be used in bridging cnmac1 and cnmac2.

# vi /etc/hostname.vether0
inet 192.168.1.1 255.255.255.0 192.168.1.255

bridge0

Create a bridge for LAN ports.

# vi /etc/hostname.bridge0
add vether0
add cnmac1
add cnmac2
blocknonip vether0
blocknonip cnmac1
blocknonip cnmac2
up

dhcp6c

Install dhcp6.

# pkg_add -U wide-dhcpv6

Add a line to /etc/rc.conf.local. Leave other lines, if present, untouched.

# vi /etc/rc.conf.local
dhcp6c_flags=cnmac0

Append a line (!/usr/sbin/rcctl restart dhcp6c) at the end of /etc/hostname.cnmac0. Leave other lines untouched.

That makes the entire contents of /etc/hostname.cnmac0 to be.

# vi /etc/hostname.cnmac0
dhcp lladdr OV:ER:RI:DE:00:00
up
inet6 autoconf
!/sbin/route add ::/0 -ifp cnmac0 fe80::
!/usr/sbin/rcctl restart dhcp6c

Make sure /etc/dhcp6c.conf looks like below for Comcast. You may need to alter some settings depending on your network and ISP.

interface cnmac0 {
    send ia-pd 0;
};

id-assoc pd 0 {
    prefix ::/64 infinity;
    prefix-interface vether0 {
        sla-id 1;
        sla-len 0;
    };
};

Create an init script.

# vi /etc/rc.d/dhcp6c
#!/bin/sh

daemon="/usr/local/sbin/dhcp6c"

. /etc/rc.d/rc.subr

rc_reload=NO

rc_cmd $1

Change permissions of the init script.

# chmod ugo+rx /etc/rc.d/dhcp6c

Enable and start dhcp6c.

# rcctl enable dhcp6c
# rcctl start dhcp6c

The "Wrong" Config

This is the "wrong" way to configure dhcp6c.conf for a router.

interface cnmac0 {
    send ia-pd 0;
    send ia-na 1;
};

id-assoc na 1 {
};

id-assoc pd 0 {
    prefix ::/64 infinity;
    prefix-interface vether0 {
        sla-id 1;
        sla-len 0;
    };
};

Correspondingly, /etc/hostname.cnmac0 was configured thusly.

# vi /etc/hostname.cnmac0
dhcp lladdr OV:ER:RI:DE:00:00
up
inet6 autoconf
!/usr/sbin/rcctl restart dhcp6c

Notice how I have send ia-na 1; and id-assoc na 1 { }; in the "wrong" config above? You need these when the device is on the edge of the network, for example a server or a desktop. This tells the DHCPv6 server to assign your WAN a non-temporary IPv6 address. You would use this address to access the device (client) from the WAN side. You don't necessarily have to do the same on a router, primarily because you can access it from the LAN (vether0) side.

I reached out to misc@ mailing list with a cry for help (WAN interface loses IPv6 NA address after pltime/vltime expire).

I was running into a problem where -- with the above config -- WAN was getting an IPv6 address (IA_NA) when dhcp6c was started. The remaining lease time was provided by pltime and vltime values in ifconfig cnmac0. When that time expired, dhcp6c would renew the address but it never got applied to the network interface (cnmac0). This resulted in my entire network losing IPv6 connectivity to the outside world. All devices would have an IPv6 address but they couldn't get to the Internet. When I restarted dhcp6c (rcctl restart dhcp6c), cnmac0 would get an IPv6 address again and things would start working. I ended up creating a cron entry to restart dhcp6c every hour.

As Stuart Henderson notes in his reply to my email,

There was a problem with vltime/pltime exported from the kernel being incorrect which was fixed after 6.1

I appeared to have been running into this bug since I was using OpenBSD 6.1.

I'll try wide-dhcpv6 on OpenBSD 6.2 or later when they become available to see if my original problem is fixed. Otherwise it may be time to give dhcpcd a try instead, as suggested by Henderson. It seems to be a more actively maintained project and thus merits a deeper look.

Ideally, WAN should also be assigned a globally routable IPv6 address by the ISP so it can be reached from outside. It works for me when I run LEDE (OpenWRT fork) or RouterOS. It does work in this case, too, but obviously there are deficiencies and bugs. Once the kinks are ironed out I'd consider this to not be the "wrong" config. It's "wrong" as long as there are problems using it.

Unbound DNS Server

I decided on Unbound for DNS services in my LAN. Big thanks to Unbound DNS Tutorial for all the help it provided in configuring Unbound.

Create /var/unbound/etc/unbound.conf with these contents. Modify as needed.

# vi /var/unbound/etc/unbound.conf
## Authoritative, validating, recursive caching DNS
## modified form of unbound.conf from https://calomel.org retrieved on 2016-10-24
#
server:
  # log verbosity
    verbosity: 1

  # specify the interfaces to answer queries from by ip-address.  The default
  # is to listen to localhost (127.0.0.1 and ::1).  specify 0.0.0.0 and ::0 to
  # bind to all available interfaces.  specify every interface[@port] on a new
  # 'interface:' labeled line.  The listen interfaces are not changed on
  # reload, only on restart.
    interface: 127.0.0.1
    interface: ::1
    interface: 192.168.1.1

  # port to answer queries from
    port: 53

  # Enable IPv4, "yes" or "no".
    do-ip4: yes

  # Enable IPv6, "yes" or "no".
    do-ip6: yes

  # Enable UDP, "yes" or "no".
    do-udp: yes

  # Enable TCP, "yes" or "no". If TCP is not needed, Unbound is actually
  # quicker to resolve as the functions related to TCP checks are not done.i
  # NOTE: you may need tcp enabled to get the DNSSEC results from *.edu domains
  # due to their size.
    do-tcp: yes

  # control which client ips are allowed to make (recursive) queries to this
  # server. Specify classless netblocks with /size and action.  By default
  # everything is refused, except for localhost.  Choose deny (drop message),
  # refuse (polite error reply), allow (recursive ok), allow_snoop (recursive
  # and nonrecursive ok)
    access-control: 10.0.0.0/16 allow
    access-control: 127.0.0.0/8 allow
    access-control: 192.168.0.0/16 allow

  # Read  the  root  hints from this file. Default is nothing, using built in
  # hints for the IN class. The file has the format of  zone files,  with  root
  # nameserver  names  and  addresses  only. The default may become outdated,
  # when servers change,  therefore  it is good practice to use a root-hints
  # file.  get one from ftp://FTP.INTERNIC.NET/domain/named.cache
    #root-hints: "/var/unbound/etc/root.hints"

  # enable to not answer id.server and hostname.bind queries.
    hide-identity: yes

  # enable to not answer version.server and version.bind queries.
    hide-version: yes

  # Will trust glue only if it is within the servers authority.
  # Harden against out of zone rrsets, to avoid spoofing attempts.
  # Hardening queries multiple name servers for the same data to make
  # spoofing significantly harder and does not mandate dnssec.
    harden-glue: yes

  # Require DNSSEC data for trust-anchored zones, if such data is absent, the
  # zone becomes  bogus.  Harden against receiving dnssec-stripped data. If you
  # turn it off, failing to validate dnskey data for a trustanchor will trigger
  # insecure mode for that zone (like without a trustanchor).  Default on,
  # which insists on dnssec data for trust-anchored zones.
    harden-dnssec-stripped: yes

  # Use 0x20-encoded random bits in the query to foil spoof attempts.
  # http://tools.ietf.org/html/draft-vixie-dnsext-dns0x20-00
  # While upper and lower case letters are allowed in domain names, no significance
  # is attached to the case. That is, two names with the same spelling but
  # different case are to be treated as if identical. This means calomel.org is the
  # same as CaLoMeL.Org which is the same as CALOMEL.ORG.
    use-caps-for-id: yes

  # the time to live (TTL) value lower bound, in seconds. Default 0.
  # If more than an hour could easily give trouble due to stale data.
    cache-min-ttl: 3600

  # the time to live (TTL) value cap for RRsets and messages in the
  # cache. Items are not cached for longer. In seconds.
    cache-max-ttl: 86400

  # perform prefetching of close to expired message cache entries.  If a client
  # requests the dns lookup and the TTL of the cached hostname is going to
  # expire in less than 10% of its TTL, unbound will (1st) return the ip of the
  # host to the client and (2nd) pre-fetch the dns request from the remote dns
  # server. This method has been shown to increase the amount of cached hits by
  # local clients by 10% on average.
    prefetch: yes

  # number of threads to create. 1 disables threading. This should equal the number
  # of CPU cores in the machine. Our example machine has 4 CPU cores.
    num-threads: 2


  ## Unbound Optimization and Speed Tweaks ###

  # the number of slabs to use for cache and must be a power of 2 times the
  # number of num-threads set above. more slabs reduce lock contention, but
  # fragment memory usage.
    #msg-cache-slabs: 4
    #rrset-cache-slabs: 4
    #infra-cache-slabs: 4
    #key-cache-slabs: 4

  # Increase the memory size of the cache. Use roughly twice as much rrset cache
  # memory as you use msg cache memory. Due to malloc overhead, the total memory
  # usage is likely to rise to double (or 2.5x) the total cache memory. The test
  # box has 4gig of ram so 256meg for rrset allows a lot of room for cacheed objects.
    #rrset-cache-size: 256m
    #msg-cache-size: 128m

  # buffer size for UDP port 53 incoming (SO_RCVBUF socket option). This sets
  # the kernel buffer larger so that no messages are lost in spikes in the traffic.
    #so-rcvbuf: 1m

  ## Unbound Optimization and Speed Tweaks ###


  # Enforce privacy of these addresses. Strips them away from answers.  It may
  # cause DNSSEC validation to additionally mark it as bogus.  Protects against
  # 'DNS Rebinding' (uses browser as network proxy).  Only 'private-domain' and
  # 'local-data' names are allowed to have these private addresses. No default.
    private-address: 192.168.0.0/16
    private-address: 172.16.0.0/12
    private-address: 10.0.0.0/8

  # Allow the domain (and its subdomains) to contain private addresses.
  # local-data statements are allowed to contain private addresses too.
    private-domain: home.lan

  # If nonzero, unwanted replies are not only reported in statistics, but also
  # a running total is kept per thread. If it reaches the threshold, a warning
  # is printed and a defensive action is taken, the cache is cleared to flush
  # potential poison out of it.  A suggested value is 10000000, the default is
  # 0 (turned off). We think 10K is a good value.
    unwanted-reply-threshold: 10000

  # IMPORTANT FOR TESTING: If you are testing and setup NSD or BIND  on
  # localhost you will want to allow the resolver to send queries to localhost.
  # Make sure to set do-not-query-localhost: yes . If yes, the above default
  # do-not-query-address entries are present.  if no, localhost can be queried
  # (for testing and debugging).
    do-not-query-localhost: no

  # File with trusted keys, kept up to date using RFC5011 probes, initial file
  # like trust-anchor-file, then it stores metadata.  Use several entries, one
  # per domain name, to track multiple zones. If you use forward-zone below to
  # query the Google DNS servers you MUST comment out this option or all DNS
  # queries will fail.
  # auto-trust-anchor-file: "/var/unbound/etc/root.key"

  # Should additional section of secure message also be kept clean of unsecure
  # data. Useful to shield the users of this validator from potential bogus
  # data in the additional section. All unsigned data in the additional section
  # is removed from secure messages.
    val-clean-additional: yes

  # Blocking Ad Server domains. Google's AdSense, DoubleClick and Yahoo
  # account for a 70 percent share of all advertising traffic. Block them.
  # local-zone: "doubleclick.net" redirect
  # local-data: "doubleclick.net A 127.0.0.1"
  # local-zone: "googlesyndication.com" redirect
  # local-data: "googlesyndication.com A 127.0.0.1"
  # local-zone: "googleadservices.com" redirect
  # local-data: "googleadservices.com A 127.0.0.1"
  # local-zone: "google-analytics.com" redirect
  # local-data: "google-analytics.com A 127.0.0.1"
  # local-zone: "ads.youtube.com" redirect
  # local-data: "ads.youtube.com A 127.0.0.1"
  # local-zone: "adserver.yahoo.com" redirect
  # local-data: "adserver.yahoo.com A 127.0.0.1"
  # local-zone: "ask.com" redirect
  # local-data: "ask.com A 127.0.0.1"


  # Unbound will not load if you specify the same local-zone and local-data
  # servers in the main configuration as well as in this "include:" file. We
  # suggest commenting out any of the local-zone and local-data lines above if
  # you suspect they could be included in the unbound_ad_servers servers file.
  #include: "/etc/unbound/unbound_ad_servers"

  # locally served zones can be configured for the machines on the LAN.

    local-zone: "home.lan." static

    local-data: device1.home.lan.  IN A 192.168.1.55
    local-data: device2.home.lan.  IN A 192.168.1.97

    local-data-ptr: 192.168.1.55 device1.home.lan
    local-data-ptr: 192.168.1.97 device2.home.lan

  # Use the following forward-zone to forward all queries to Google DNS,
  # OpenDNS.com or your local ISP's dns servers for example. To test resolution
  # speeds use "drill calomel.org @8.8.8.8" and look for the "Query time:" in
  # milliseconds.
  #
   forward-zone:
      name: "."
      forward-addr: 50.116.40.226        # OpenDNS
      forward-addr: 8.8.4.4        # Google
      forward-addr: 2604:180:1:22a::8c53        # OpenDNS

Append these lines to /etc/rc.conf.local. Leave other lines untouched.

# vi /etc/rc.conf.local
unbound_flags=""

Edit /etc/dhclient.conf so it doesn't overwrite the local nameserver. Leave other lines untouched.

# vi /etc/dhclient.conf
ignore domain-name-servers;

Enable and start unbound.

# rcctl enable unbound
# rcctl start unbound

DHCP

Append these lines to /etc/rc.conf.local. Leave other lines untouched.

# vi /etc/rc.conf.local
dhcpd_flags="vether0"

Create /etc/dhcpd.conf with these settings. I like to have static assignments for IPv4 addresses based on MAC addresses. Modify as needed.

authoritative;
option domain-name "my.example.com";
default-lease-time 43200;
max-lease-time 90000;
option routers 192.168.1.1;
option broadcast-address 192.168.1.255;
option domain-name-servers 192.168.1.1, 8.8.8.8;

subnet 192.168.1.0 netmask 255.255.255.0 {
    range 192.168.1.21 192.168.1.200;
}

host device1 {
    hardware ethernet 00:00:00:00:be:ef;
    fixed-address 192.168.1.238;
}

host device2 {
    hardware ethernet 00:00:00:01:be:ef;
    fixed-address 192.168.1.244;
}

Notice that the range for dynamic assignment does not include the IPs provided in static assignment. This is on purpose since otherwise dhcpd complains that the same address has been assigned twice.

Enable and start dhcpd.

# rcctl enable dhcpd
# rcctl start dhcpd

Routing

Enable IPv4 and IPv6 forwarding.

Append these lines to /etc/sysctl.conf. Leave other lines untouched.

# vi /etc/sysctl.conf
net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1

Enable these settings.

# xargs sysctl < /etc/sysctl.conf

Append these lines to /etc/resolv.conf.tail. Leave other lines untouched. Modify as needed.

# vi /etc/resolv.conf.tail
search my.example.com
nameserver 127.0.0.1
nameserver 8.8.8.8

rtadvd

Configure rtadvd with an empty file in /etc/rtadvd.conf. The reason is that it is able to use the delegated prefixes assigned by dhcp6c to network interfaces to advertise downstream. When I tried using a non-empty config file, IPv6 didn't work as expected.

Append these lines to /etc/rc.conf.local. Leave other lines untouched.

# vi /etc/rc.conf.local
rtadvd_flags="vether0"

Enable and start rtadvd.

# rcctl enable rtadvd
# rcctl start rtadvd

Firewall with pf

Replace /etc/pf.conf with these settings, updated as needed. I have curated these rules from many sources, primarily The modern OpenBSD home router. I also make no claim that these rules have created a secure firewall.

# vi /etc/pf.conf
### ~~~ Interface layout ~~~ ###

# cnmac0: 802.3ab (ethernet) to cable modem
# cnmac1: 802.3ab (ethernet) to internal switch
# cnmac2: 802.3ab (ethernet) to internal switch
# vether0: persists address 192.168.1.0/255.255.255.0
# bridge0: Ethernet bridge over cnmac1 and cnmac2
# pflog0: target interface for blocked packets

### ~~~ Constants and variables ~~~ ###

# All addresses associated with this host
self = "{ (egress), (vether0) }"

self_lan = "{ (vether0) }"

# RFC 6890: Special-Purpose IP Address Registries:
# https://www.iana.org/assignments/iana-ipv4-special-registry/
# https://www.iana.org/assignments/iana-ipv6-special-registry/

# Included below are all address blocks with either Forwardable = False,
# Global = False, or both, but excluding 2001::/23 because it is often
# superseded by more specific allocations, as of 2015-08-05.

table <martians> const { \
    0.0.0.0/8, \
    10.0.0.0/8, \
    100.64.0.0/10, \
    127.0.0.0/8, \
    169.254.0.0/16, \
    172.16.0.0/12, \
    192.0.0.0/24, \
    192.0.2.0/24, \
    192.168.0.0/16, \
    198.18.0.0/15, \
    198.51.100.0/24, \
    203.0.113.0/24, \
    240.0.0.0/4, \
    255.255.255.255/32, \
    ::1/128, \
    ::/128, \
    ::ffff:0:0/96, \
    100::/64, \
    2001::/32, \
    2001:2::/48, \
    2001:db8::/32, \
    fc00::/7, \
    fe80::/10 \
}

### ~~~ Default rules ~~~ ###

# Never touch loopback interfaces
set skip on lo

# Normalise packets, especially IPv4 DF and Identification
match in all scrub (no-df random-id)

# Limit the MSS on PPPoE to 1440 octets
# match on pppoe0 scrub (max-mss 1440)

# Block all packets by default, logging them to pflog0
block log

### ~~~ Link-scoped services ~~~ ###

# DHCPv6 client: make IA_PD requests and receive responses to them
pass out quick on egress inet6 proto udp from (egress) to ff02::1:2 port dhcpv6-server
pass in quick on egress inet6 proto udp to (egress) port dhcpv6-client

### ~~~ Bulk pass rules ~~~ ###

# Pass all traffic on internal interfaces
# vether0 is necessary here, but bridge0 is not
pass quick on { vether0 cnmac1 cnmac2 }

# Pass all outbound IPv6 traffic
pass out quick on egress inet6 from { egress, (vether0:network) } modulate state

# Pass all outbound IPv4 traffic from this host
pass out quick on egress inet from egress modulate state

# NAT all outbound IPv4 traffic from the rest of our network
pass out quick on egress inet from (vether0:network) nat-to (egress) modulate state

### ~~~ Block undesirable traffic ~~~ ###

# These rules must not precede the DHCPv6 client or NAT rules above
block log quick on egress from { no-route, urpf-failed, <martians> }
block log quick on egress to { no-route, <martians> }

### ~~~ Pass some ICMP and ICMPv6 traffic ~~~ ####

# Pass all inbound ICMP echo requests
pass in quick on egress inet proto icmp icmp-type echoreq
pass in quick on egress inet6 proto icmp6 icmp6-type echoreq

# RFC 4890: Recommendations for Filtering ICMPv6 Messages in Firewalls
pass quick on egress inet6 proto icmp6 icmp6-type { 1, 2, 128, 129, 133, 134, 135, 136, 137 }
#pass quick on egress inet6 proto icmp6 icmp6-type { 1, 2, 128, 129 }
pass quick on egress inet6 proto icmp6 icmp6-type 3 code 0
pass quick on egress inet6 proto icmp6 icmp6-type 3 code 1
pass quick on egress inet6 proto icmp6 icmp6-type 4 code 0
pass quick on egress inet6 proto icmp6 icmp6-type 4 code 1
pass quick on egress inet6 proto icmp6 icmp6-type 4 code 2

### ~~~ Open services on this router ~~~ ###

# OpenSSH server
pass in on egress proto { tcp, udp } to $self_lan port ssh

# Allow LAN to ping us
pass in on { vether0 } inet proto icmp to any icmp-type { echoreq, echorep }

# Allow LAN to access DNS and NTP
pass quick on egress inet proto udp from (vether0:network) to any port { 53, 67, 123 }
pass quick on egress inet6 proto udp from (vether0:network) to any port { 53, 67, 123 }
pass quick on egress inet proto udp from $self to any port { 53, 67, 123 }
pass quick on egress inet6 proto udp from $self to any port { 53, 67, 123 }

Reload rules.

# pfctl -f /etc/pf.conf

Ansible

You can use Ansible roles I created for this post if you're so inclined. They're available on GitHub - openbsd-on-erl.

Conclusion

I'm pretty happy with ERL and OpenBSD. There is great community documentation on how to configure all the pieces of software that make a OpenBSD-based home network gateway possible. I can tweak things as needed and upgrade when newer versions become available.

My plan on upgrading the base OS is to get a third party USB drive that works, write a newer OpenBSD image to it, and replace the drive in the ERL enclosure. This way I can keep a bunch of drives in rotation. Upgrades to newer builds or reverts to last known good version are as easy as swapping USB drives.

Configuration with Ansible means I don't have to manually do things again and again. As the configs change they'll be tracked in git so I get version control as well.