v6 autoconf broken in bsd.rd

[Christian Weisgerber]

i386 snapshot, Build date: 1500601432 - Fri Jul 21 01:43:52 UTC 2017

I tried to run an install with...

------------------->
IPv6 address for em0? (or 'autoconf' or 'none') [none] autoconf
...
DNS nameservers? (IP address list or 'none') [none] fddd:28ee:243::1
<-------------------

... but there is something wrong with SLAAC. Here's what's advertised
on the network:

------------------->
Router Advertisement from fe80::20d:b9ff:fe41:568%vlan0
received: 2017-07-21 17:16:26.496289572; 152.154965150s ago
Cur Hop Limit: 64, M: 0, O: 0, Router Lifetime: 1800s
Default Router Preference: Medium
Reachable Time: 0ms, Retrans Timer: 0ms
prefix: 2003:e5:cbc0:a01::/64
On-link: 1, Autonomous address-configuration: 1
vltime: 2592000, pltime: 604800
prefix: fddd:28ee:243::/64
On-link: 1, Autonomous address-configuration: 1
vltime: 2592000, pltime: 604800
<-------------------

After network configuration, the installer ends up with this:

------------------->
# ifconfig egress
em0: flags=208843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,AUTOCONF6> mtu 1500
lladdr 00:0d:b9:41:06:a4
llprio 3
groups: netboot egress
media: Ethernet autoselect (1000baseT full-duplex,rxpause,txpause)
status: active
inet 172.16.0.2 netmask 0xffffff00 broadcast 172.16.0.255
inet6 fe80::20d:b9ff:fe41:6a4%em0 prefixlen 64 scopeid 0x1
inet6 fddd:28ee:243:0:384d:91e7:fefc:b697 prefixlen 64 autoconf autoconfprivacy pltime 85665 vltime 604477
<-------------------

This is missing a non-privacy address for one prefix, and any addresses
for the other prefix. The routing table also looks suspicious:

------------------->
# route -n show -inet6
Routing tables

Internet6:
Destination Gateway Flags Refs Use Mtu Prio Iface
default fe80::20d:b9ff:fe41:568%em0 UGS 0 63 - 56 em0
::1 ::1 UHPl 0 0 32768 1 lo0
fddd:28ee:243:0:384d:91e7:fefc:b697 00:0d:b9:41:06:a4 UHLPl 0 75 - 1 em0
fe80::%em0/64 fe80::20d:b9ff:fe41:6a4%em0 UCPn 3 3 - 4 em0
fe80::20d:b9ff:fe41:568%em0 00:0d:b9:41:05:68 UHLcP 0 183 - 3 em0
fe80::20d:b9ff:fe41:568%em0 link#1 UHLchP 1 3 - 3 em0
fe80::20d:b9ff:fe41:6a4%em0 00:0d:b9:41:06:a4 UHLPl 0 89 - 1 em0
fe80::fab1:56ff:feb6:dcc8%em0 f8:b1:56:b6:dc:c8 UHLcP 0 13 - 3 em0
fe80::1%lo0 fe80::1%lo0 UHPl 0 0 32768 1 lo0
ff01::%em0/32 fe80::20d:b9ff:fe41:6a4%em0 Um 0 2 - 4 em0
ff01::%lo0/32 ::1 Um 0 1 32768 4 lo0
ff02::%em0/32 fe80::20d:b9ff:fe41:6a4%em0 Um 0 77 - 4 em0
ff02::%lo0/32 ::1 Um 0 1 32768 4 lo0
<-------------------

There is a default route, but no routes for the advertised prefixes.

[Florian Obser]

Post by Christian Weisgerber
i386 snapshot, Build date: 1500601432 - Fri Jul 21 01:43:52 UTC 2017
I tried to run an install with...
------------------->
IPv6 address for em0? (or 'autoconf' or 'none') [none] autoconf
...
DNS nameservers? (IP address list or 'none') [none] fddd:28ee:243::1
<-------------------
... but there is something wrong with SLAAC. Here's what's advertised
------------------->
Router Advertisement from fe80::20d:b9ff:fe41:568%vlan0
received: 2017-07-21 17:16:26.496289572; 152.154965150s ago
Cur Hop Limit: 64, M: 0, O: 0, Router Lifetime: 1800s
Default Router Preference: Medium
Reachable Time: 0ms, Retrans Timer: 0ms
prefix: 2003:e5:cbc0:a01::/64
On-link: 1, Autonomous address-configuration: 1
vltime: 2592000, pltime: 604800
prefix: fddd:28ee:243::/64
On-link: 1, Autonomous address-configuration: 1
vltime: 2592000, pltime: 604800
<-------------------
------------------->
# ifconfig egress
em0: flags=208843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,AUTOCONF6> mtu 1500
lladdr 00:0d:b9:41:06:a4
llprio 3
groups: netboot egress
media: Ethernet autoselect (1000baseT full-duplex,rxpause,txpause)
status: active
inet 172.16.0.2 netmask 0xffffff00 broadcast 172.16.0.255
inet6 fe80::20d:b9ff:fe41:6a4%em0 prefixlen 64 scopeid 0x1
inet6 fddd:28ee:243:0:384d:91e7:fefc:b697 prefixlen 64 autoconf autoconfprivacy pltime 85665 vltime 604477
<-------------------
This is missing a non-privacy address for one prefix, and any addresses

curious, rpe pointed out the oposite some time ago, the normal address
was there but the privacy was missing. I see the same thing as rpre.
It's probably racing somewhere, not entirely sure where though since
it first configures the normal one and then the privacy one.

reenabling debug output on the ramdisk I see this:

engine_dispatch_frontend: IMSG_PROPOSAL_ACK: 1 - 25467
configure_dfr: 1
engine_dispatch_frontend: IMSG_PROPOSAL_ACK: 2 - 25467
configure_address: 1
engine_dispatch_frontend: IMSG_PROPOSAL_ACK: 3 - 25467
configure_address: 1
configure_interface: vio0
configure_interface: vio0
fatal in main: SIOCAIFADDR_IN6: File exists
# RTM_NEWADDR: vio0[1]
RTM_NEWADDR: vio0[1]
RTM_DELADDR: vio0[1]
engine exiting
fatal in engine: msgbuf_write: Broken pipe

Maybe you can only have one v6 address on the ramdisk?

Post by Christian Weisgerber
------------------->
# route -n show -inet6
Routing tables
Destination Gateway Flags Refs Use Mtu Prio Iface
default fe80::20d:b9ff:fe41:568%em0 UGS 0 63 - 56 em0
::1 ::1 UHPl 0 0 32768 1 lo0
fddd:28ee:243:0:384d:91e7:fefc:b697 00:0d:b9:41:06:a4 UHLPl 0 75 - 1 em0
fe80::%em0/64 fe80::20d:b9ff:fe41:6a4%em0 UCPn 3 3 - 4 em0
fe80::20d:b9ff:fe41:568%em0 00:0d:b9:41:05:68 UHLcP 0 183 - 3 em0
fe80::20d:b9ff:fe41:568%em0 link#1 UHLchP 1 3 - 3 em0
fe80::20d:b9ff:fe41:6a4%em0 00:0d:b9:41:06:a4 UHLPl 0 89 - 1 em0
fe80::fab1:56ff:feb6:dcc8%em0 f8:b1:56:b6:dc:c8 UHLcP 0 13 - 3 em0
fe80::1%lo0 fe80::1%lo0 UHPl 0 0 32768 1 lo0
ff01::%em0/32 fe80::20d:b9ff:fe41:6a4%em0 Um 0 2 - 4 em0
ff01::%lo0/32 ::1 Um 0 1 32768 4 lo0
ff02::%em0/32 fe80::20d:b9ff:fe41:6a4%em0 Um 0 77 - 4 em0
ff02::%lo0/32 ::1 Um 0 1 32768 4 lo0
<-------------------
There is a default route, but no routes for the advertised prefixes.
--

[Christian Weisgerber]

Post by Florian Obser
engine_dispatch_frontend: IMSG_PROPOSAL_ACK: 1 - 25467
configure_dfr: 1
engine_dispatch_frontend: IMSG_PROPOSAL_ACK: 2 - 25467
configure_address: 1
engine_dispatch_frontend: IMSG_PROPOSAL_ACK: 3 - 25467
configure_address: 1
configure_interface: vio0
configure_interface: vio0
fatal in main: SIOCAIFADDR_IN6: File exists
# RTM_NEWADDR: vio0[1]
RTM_NEWADDR: vio0[1]
RTM_DELADDR: vio0[1]
engine exiting
fatal in engine: msgbuf_write: Broken pipe
Maybe you can only have one v6 address on the ramdisk?

Well, I can manually configure additional addresses:

------------------->
# ifconfig egress
em0: flags=208843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,AUTOCONF6> mtu 1500
lladdr 00:0d:b9:41:06:a4
llprio 3
groups: netboot egress
media: Ethernet autoselect (1000baseT full-duplex,rxpause,txpause)
status: active
inet 172.16.0.2 netmask 0xffffff00 broadcast 172.16.0.255
inet6 fe80::20d:b9ff:fe41:6a4%em0 prefixlen 64 scopeid 0x1
inet6 fddd:28ee:243:0:384d:91e7:fefc:b697 prefixlen 64 autoconf autoconfprivacy pltime 2171 vltime 520983
inet6 fddd:28ee:243::2 prefixlen 64
inet6 2003:e5:cbc0:a01::2 prefixlen 64
<-------------------

And the corresponding /64s show up in the routing table, too.

However, I can't send packets beyond the local segment. tcpdump
on the gateway shows that it receives a steady stream of neighbor
solicitations for its advertised default router address. The replies
don't seem to have any effect.

[Florian Obser]

Post by Christian Weisgerber

Post by Florian Obser
engine_dispatch_frontend: IMSG_PROPOSAL_ACK: 1 - 25467
configure_dfr: 1
engine_dispatch_frontend: IMSG_PROPOSAL_ACK: 2 - 25467
configure_address: 1
engine_dispatch_frontend: IMSG_PROPOSAL_ACK: 3 - 25467
configure_address: 1
configure_interface: vio0
configure_interface: vio0
fatal in main: SIOCAIFADDR_IN6: File exists
# RTM_NEWADDR: vio0[1]
RTM_NEWADDR: vio0[1]
RTM_DELADDR: vio0[1]
engine exiting
fatal in engine: msgbuf_write: Broken pipe
Maybe you can only have one v6 address on the ramdisk?

------------------->
# ifconfig egress
em0: flags=208843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,AUTOCONF6> mtu 1500
lladdr 00:0d:b9:41:06:a4
llprio 3
groups: netboot egress
media: Ethernet autoselect (1000baseT full-duplex,rxpause,txpause)
status: active
inet 172.16.0.2 netmask 0xffffff00 broadcast 172.16.0.255
inet6 fe80::20d:b9ff:fe41:6a4%em0 prefixlen 64 scopeid 0x1
inet6 fddd:28ee:243:0:384d:91e7:fefc:b697 prefixlen 64 autoconf autoconfprivacy pltime 2171 vltime 520983
inet6 fddd:28ee:243::2 prefixlen 64
inet6 2003:e5:cbc0:a01::2 prefixlen 64
<-------------------

This is a problem with the kernel routing table not being mpatch
capable, you can see this with v4, too:

# ifconfig vio0
vio0: flags=248843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,INET6_NOPRIVACY,AUTOCONF6> mtu 1500
lladdr 52:54:00:c7:09:1b
llprio 3
groups: egress
media: Ethernet autoselect
status: active
inet 192.168.178.178 netmask 0xffffff00 broadcast 192.168.178.255
inet6 fe80::5054:ff:fec7:91b%vio0 prefixlen 64 scopeid 0x1
# route -n show -inet
Routing tables

Internet:
Destination Gateway Flags Refs Use Mtu Prio Iface
default 192.168.178.1 UGS 0 4 - 8 vio0
127/8 127.0.0.1 UGRS 0 0 32768 8 lo0
127.0.0.1 127.0.0.1 UHhPl 1 2 32768 1 lo0
192.168.178/24 192.168.178.178 UCPn 1 0 - 4 vio0
192.168.178.1 54:67:51:de:e7:ce UHLchP 1 2 - 3 vio0
192.168.178.178 52:54:00:c7:09:1b UHLPl 0 16 - 1 vio0
192.168.178.255 192.168.178.178 UHPb 0 0 - 1 vio0

# ifconfig vio0 inet alias 192.168.178.179/24
ifconfig: SIOCAIFADDR: File exists
# ifconfig vio0
vio0: flags=248843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,INET6_NOPRIVACY,AUTOCONF6> mtu 1500
lladdr 52:54:00:c7:09:1b
llprio 3
groups: egress
media: Ethernet autoselect
status: active
inet 192.168.178.178 netmask 0xffffff00 broadcast 192.168.178.255
inet6 fe80::5054:ff:fec7:91b%vio0 prefixlen 64 scopeid 0x1
# route -n show -inet
Routing tables

Internet:
Destination Gateway Flags Refs Use Mtu Prio Iface
127/8 127.0.0.1 UGRS 0 0 32768 8 lo0
127.0.0.1 127.0.0.1 UHhPl 1 2 32768 1 lo0
192.168.178.178 52:54:00:c7:09:1b UHLPl 0 16 - 1 vio0

note how we lost a bunch of routes, including 192.168.178.0/24
now you can add 192.168.178.179/24:

# ifconfig vio0 inet alias 192.168.178.179/24
# route -n show -inet
Routing tables

Internet:
Destination Gateway Flags Refs Use Mtu Prio Iface
127/8 127.0.0.1 UGRS 0 0 32768 8 lo0
127.0.0.1 127.0.0.1 UHhPl 1 2 32768 1 lo0
192.168.178/24 192.168.178.179 UCPn 1 0 - 4 vio0
192.168.178.178 52:54:00:c7:09:1b UHLPl 0 16 - 1 vio0
192.168.178.179 52:54:00:c7:09:1b UHLPl 0 1 - 1 vio0
192.168.178.252 08:62:66:8b:84:20 UHLcP 0 1 - 3 vio0
192.168.178.255 192.168.178.179 UHPb 0 0 - 1 vio0

# ifconfig vio0
vio0: flags=248843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,INET6_NOPRIVACY,AUTOCONF6> mtu 1500
lladdr 52:54:00:c7:09:1b
llprio 3
groups: egress
media: Ethernet autoselect
status: active
inet 192.168.178.178 netmask 0xffffff00 broadcast 192.168.178.255
inet6 fe80::5054:ff:fec7:91b%vio0 prefixlen 64 scopeid 0x1
inet 192.168.178.179 netmask 0xffffff00 broadcast 192.168.178.255


------------------------------------------------------------------------
here is the same thing for IPv6:

# ifconfig vio0 inet6 2001:db8::1/64
# route -n show -inet6
Routing tables

Internet6:
Destination Gateway Flags Refs Use Mtu Prio Iface
::1 ::1 UHPl 0 0 32768 1 lo0
2001:db8::/64 2001:db8::1 UCPn 0 0 - 4 vio0
2001:db8::1 52:54:00:c7:09:1b UHLPl 0 0 - 1 vio0
fe80::%vio0/64 fe80::5054:ff:fec7:91b%vio0 UCPn 0 0 - 4 vio0
fe80::5054:ff:fec7:91b%vio0 52:54:00:c7:09:1b UHLPl 0 0 - 1 vio0
fe80::1%lo0 fe80::1%lo0 UHPl 0 0 32768 1 lo0
ff01::%vio0/32 fe80::5054:ff:fec7:91b%vio0 Um 0 1 - 4 vio0
ff01::%lo0/32 ::1 Um 0 1 32768 4 lo0
ff02::%vio0/32 fe80::5054:ff:fec7:91b%vio0 Um 0 1 - 4 vio0
ff02::%lo0/32 ::1 Um 0 1 32768 4 lo0
# ifconfig vio0
vio0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 52:54:00:c7:09:1b
llprio 3
groups: egress
media: Ethernet autoselect
status: active
inet 192.168.178.178 netmask 0xffffff00 broadcast 192.168.178.255
inet6 fe80::5054:ff:fec7:91b%vio0 prefixlen 64 scopeid 0x1
inet6 2001:db8::1 prefixlen 64
# ifconfig vio0 inet6 2001:db8::2/64
ifconfig: SIOCAIFADDR: File exists
# route -n show -inet6
Routing tables

Internet6:
Destination Gateway Flags Refs Use Mtu Prio Iface
::1 ::1 UHPl 0 0 32768 1 lo0
2001:db8::1 52:54:00:c7:09:1b UHLPl 0 0 - 1 vio0
fe80::%vio0/64 fe80::5054:ff:fec7:91b%vio0 UCPn 0 0 - 4 vio0
fe80::5054:ff:fec7:91b%vio0 52:54:00:c7:09:1b UHLPl 0 0 - 1 vio0
fe80::1%lo0 fe80::1%lo0 UHPl 0 0 32768 1 lo0
ff01::%vio0/32 fe80::5054:ff:fec7:91b%vio0 Um 0 2 - 4 vio0
ff01::%lo0/32 ::1 Um 0 1 32768 4 lo0
ff02::%vio0/32 fe80::5054:ff:fec7:91b%vio0 Um 0 2 - 4 vio0
ff02::%lo0/32 ::1 Um 0 1 32768 4 lo0
# ifconfig vio0
vio0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 52:54:00:c7:09:1b
llprio 3
groups: egress
media: Ethernet autoselect
status: active
inet 192.168.178.178 netmask 0xffffff00 broadcast 192.168.178.255
inet6 fe80::5054:ff:fec7:91b%vio0 prefixlen 64 scopeid 0x1
inet6 2001:db8::1 prefixlen 64
# ifconfig vio0 inet6 2001:db8::2/64
# route -n show -inet6
Routing tables

Internet6:
Destination Gateway Flags Refs Use Mtu Prio Iface
::1 ::1 UHPl 0 0 32768 1 lo0
2001:db8::/64 2001:db8::2 UCPn 0 0 - 4 vio0
2001:db8::1 52:54:00:c7:09:1b UHLPl 0 0 - 1 vio0
2001:db8::2 52:54:00:c7:09:1b UHLPl 0 0 - 1 vio0
fe80::%vio0/64 fe80::5054:ff:fec7:91b%vio0 UCPn 0 0 - 4 vio0
fe80::5054:ff:fec7:91b%vio0 52:54:00:c7:09:1b UHLPl 0 0 - 1 vio0
fe80::1%lo0 fe80::1%lo0 UHPl 0 0 32768 1 lo0
ff01::%vio0/32 fe80::5054:ff:fec7:91b%vio0 Um 0 3 - 4 vio0
ff01::%lo0/32 ::1 Um 0 1 32768 4 lo0
ff02::%vio0/32 fe80::5054:ff:fec7:91b%vio0 Um 0 3 - 4 vio0
ff02::%lo0/32 ::1 Um 0 1 32768 4 lo0
# ifconfig vio0
vio0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 52:54:00:c7:09:1b
llprio 3
groups: egress
media: Ethernet autoselect
status: active
inet 192.168.178.178 netmask 0xffffff00 broadcast 192.168.178.255
inet6 fe80::5054:ff:fec7:91b%vio0 prefixlen 64 scopeid 0x1
inet6 2001:db8::1 prefixlen 64
inet6 2001:db8::2 prefixlen 64

As a work around you could put in inet6 -autoconfprivacy into the
hostname.if since the privacy address is in the same /64

Btw. with kernel based slaac this used to work since the
kernel didn't go through the ioctl path but just fiddled directly with
the interface and skipped rt_ifa_add() which is failing here.

Post by Christian Weisgerber
And the corresponding /64s show up in the routing table, too.
However, I can't send packets beyond the local segment. tcpdump
on the gateway shows that it receives a steady stream of neighbor
solicitations for its advertised default router address. The replies
don't seem to have any effect.

I haven't figured out what's going on here. If I delete the default
route and set the exact same one with route(8)it works. I suspect
another mpath issue, since that seems to be the main difference
between bsd.rd and bsd.

Post by Christian Weisgerber
--

[Martin Pieuchot]

Post by Florian Obser
[...]
# ifconfig vio0
vio0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 52:54:00:c7:09:1b
llprio 3
groups: egress
media: Ethernet autoselect
status: active
inet 192.168.178.178 netmask 0xffffff00 broadcast 192.168.178.255
inet6 fe80::5054:ff:fec7:91b%vio0 prefixlen 64 scopeid 0x1
inet6 2001:db8::1 prefixlen 64
inet6 2001:db8::2 prefixlen 64
As a work around you could put in inet6 -autoconfprivacy into the
hostname.if since the privacy address is in the same /64
Btw. with kernel based slaac this used to work since the
kernel didn't go through the ioctl path but just fiddled directly with
the interface and skipped rt_ifa_add() which is failing here.

Because the kernel based slaac was inserting a single prefix. This was
a limitation of the code.

The correct fix is to enable MPATH on ramdisk. This is mostly a matter
of changing all the "#ifdef ART" into "#ifndef RADIX" and deal with the
ramdisk grow.

[Theo de Raadt]

Post by Martin Pieuchot
The correct fix is to enable MPATH on ramdisk. This is mostly a matter
of changing all the "#ifdef ART" into "#ifndef RADIX" and deal with the
ramdisk grow.

It sounds like v6 requires MPATH.