flow queue problem on -current : not able to send packets anymore

[Sebastien Marie]

Hi,

With last snapshot (Jul 5), I experimented problems with networking:
I couldn't send packets.

As example:

# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: sendmsg: No buffer space available
ping: wrote 8.8.8.8 64 chars, ret=-1
^C

My previous (working) snap was Jun 26.

I tracked a bit the problem until I found that commenting the flow queue
line in pf.conf resolv my problem.

The offended line (on my laptop) was:
queue fq on bge0 flows 1024 default


I tried to reproduce on other host (i386), recompiling from source,
using up-to-date cvs tree. The problem is still here, and is the same:
reloading pf rules with flow queue makes network to become unusable: "No
buffer space available".

The queue statistic shows that qlength is full.

# pfctl -sq -v
queue fq on bce0 flows 1024 default
[ pkts: 0 bytes: 0 dropped pkts: 7 bytes: 650 ]
[ qlength: 50/ 50 avg delay: 0.000ms std-dev: 0.000ms flows: 1 ]

The dmesg below is my test host (with bce0), not my laptop (with bge0).

Thanks.
--
Sebastien Marie


OpenBSD 6.1-current (GENERIC.MP) #24: Thu Jul 6 13:34:29 CEST 2017
***@bert.local:/home/openbsd/src/sys/arch/i386/compile/GENERIC.MP
cpu0: Genuine Intel(R) CPU T2400 @ 1.83GHz ("GenuineIntel" 686-class) 1.83 GHz
cpu0: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,NXE,SSE3,MWAIT,VMX,EST,TM2,xTPR,PDCM,PERF,SENSOR
real mem = 2137354240 (2038MB)
avail mem = 2083151872 (1986MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: date 06/13/07, BIOS32 rev. 0 @ 0xffa10, SMBIOS rev. 2.4 @ 0xf7980 (44 entries)
bios0: vendor Dell Inc. version "A17" date 06/13/2007
bios0: Dell Inc. MM061
acpi0 at bios0: rev 0
acpi0: sleep states S0 S3 S4 S5
acpi0: tables DSDT FACP HPET APIC MCFG SLIC BOOT SSDT
acpi0: wakeup devices LID_(S3) PBTN(S4) MBTN(S5) PCI0(S3) USB0(S0) USB1(S0) USB2(S0) USB3(S0) EHCI(S0) AZAL(S3) PCIE(S4) RP01(S4) RP02(S3) RP03(S3) RP04(S3) RP05(S3) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpihpet0 at acpi0: 14318179 Hz
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 166MHz
cpu0: mwait min=64, max=64, C-substates=0.2.2.2.2, IBE
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Genuine Intel(R) CPU T2400 @ 1.83GHz ("GenuineIntel" 686-class) 1.83 GHz
cpu1: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,NXE,SSE3,MWAIT,VMX,EST,TM2,xTPR,PDCM,PERF,SENSOR
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins
acpimcfg0 at acpi0 addr 0xf0000000, bus 0-63
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus -1 (AGP_)
acpiprt2 at acpi0: bus 3 (PCIE)
acpiprt3 at acpi0: bus 11 (RP01)
acpiprt4 at acpi0: bus -1 (RP02)
acpiprt5 at acpi0: bus -1 (RP03)
acpiprt6 at acpi0: bus 12 (RP04)
acpiprt7 at acpi0: bus -1 (RP05)
acpiprt8 at acpi0: bus -1 (RP06)
acpicpu0 at acpi0: !C3(***@85 ***@0x1015), !C2(***@1 ***@0x1014), C1(***@1 halt), PSS
acpicpu1 at acpi0: !C3(***@85 ***@0x1015), !C2(***@1 ***@0x1014), C1(***@1 halt), PSS
acpitz0 at acpi0: critical temperature is 126 degC
"*pnp0c14" at acpi0 not configured
acpiac0 at acpi0: AC unit online
acpibat0 at acpi0: BAT0 model " DELLUD2649" serial 878 type LION oem "Sanyo"
acpibtn0 at acpi0: LID_
acpibtn1 at acpi0: PBTN
acpibtn2 at acpi0: SBTN
"PNP0C32" at acpi0 not configured
"PNP0F13" at acpi0 not configured
"PNP0303" at acpi0 not configured
acpivideo0 at acpi0: VID_
acpivideo1 at acpi0: VID_
acpivout0 at acpivideo1: LCD_
acpivideo2 at acpi0: VID2
bios0: ROM list: 0xc0000/0xe800! 0xce800/0x1800
cpu0: Enhanced SpeedStep 1829 MHz: speeds: 1833, 1333, 1000 MHz
pci0 at mainbus0 bus 0: configuration mode 1 (bios)
pchb0 at pci0 dev 0 function 0 "Intel 82945GM Host" rev 0x03
inteldrm0 at pci0 dev 2 function 0 "Intel 82945GM Video" rev 0x03
drm0 at inteldrm0
intagp0 at inteldrm0
agp0 at intagp0: aperture at 0xd0000000, size 0x10000000
inteldrm0: apic 2 int 16
inteldrm0: 848x480, 32bpp
wsdisplay0 at inteldrm0 mux 1: console (std, vt100 emulation)
wsdisplay0: screen 1-5 added (std, vt100 emulation)
"Intel 82945GM Video" rev 0x03 at pci0 dev 2 function 1 not configured
azalia0 at pci0 dev 27 function 0 "Intel 82801GB HD Audio" rev 0x01: msi
azalia0: codecs: Sigmatel STAC9200, Conexant/0x2bfa, using Sigmatel STAC9200
audio0 at azalia0
ppb0 at pci0 dev 28 function 0 "Intel 82801GB PCIE" rev 0x01: apic 2 int 16
pci1 at ppb0 bus 11
wpi0 at pci1 dev 0 function 0 "Intel PRO/Wireless 3945ABG" rev 0x02: msi, MoW2, address 00:13:02:2e:8b:46
ppb1 at pci0 dev 28 function 3 "Intel 82801GB PCIE" rev 0x01: apic 2 int 19
pci2 at ppb1 bus 12
uhci0 at pci0 dev 29 function 0 "Intel 82801GB USB" rev 0x01: apic 2 int 20
uhci1 at pci0 dev 29 function 1 "Intel 82801GB USB" rev 0x01: apic 2 int 21
uhci2 at pci0 dev 29 function 2 "Intel 82801GB USB" rev 0x01: apic 2 int 22
uhci3 at pci0 dev 29 function 3 "Intel 82801GB USB" rev 0x01: apic 2 int 23
ehci0 at pci0 dev 29 function 7 "Intel 82801GB USB" rev 0x01: apic 2 int 20
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb2 at pci0 dev 30 function 0 "Intel 82801BAM Hub-to-PCI" rev 0xe1
pci3 at ppb2 bus 3
bce0 at pci3 dev 0 function 0 "Broadcom BCM4401B1" rev 0x02: apic 2 int 17, address 00:15:c5:0b:8b:7a
bmtphy0 at bce0 phy 1: BCM4401 10/100baseTX PHY, rev. 0
"Ricoh 5C832 Firewire" rev 0x00 at pci3 dev 1 function 0 not configured
sdhc0 at pci3 dev 1 function 1 "Ricoh 5C822 SD/MMC" rev 0x19: apic 2 int 18
sdhc0: SDHC 1.0, 33 MHz base clock
sdmmc0 at sdhc0: 4-bit, sd high-speed, mmc high-speed
"Ricoh 5C843 MMC" rev 0x01 at pci3 dev 1 function 2 not configured
"Ricoh 5C592 Memory Stick" rev 0x0a at pci3 dev 1 function 3 not configured
"Ricoh 5C852 xD" rev 0x05 at pci3 dev 1 function 4 not configured
ichpcib0 at pci0 dev 31 function 0 "Intel 82801GBM LPC" rev 0x01: PM disabled
pciide0 at pci0 dev 31 function 2 "Intel 82801GBM SATA" rev 0x01: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility
wd0 at pciide0 channel 0 drive 0: <ST96812AS>
wd0: 16-sector PIO, LBA48, 57241MB, 117231408 sectors
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 5
atapiscsi0 at pciide0 channel 1 drive 0
scsibus1 at atapiscsi0: 2 targets
cd0 at scsibus1 targ 0 lun 0: <_NEC, DVD+-RW ND-6650A, 102C> ATAPI 5/cdrom removable
cd0(pciide0:1:0): using PIO mode 4, Ultra-DMA mode 2
ichiic0 at pci0 dev 31 function 3 "Intel 82801GB SMBus" rev 0x01: apic 2 int 17
iic0 at ichiic0
spdmem0 at iic0 addr 0x50: 1GB DDR2 SDRAM non-parity PC2-5300CL5 SO-DIMM
spdmem1 at iic0 addr 0x52: 1GB DDR2 SDRAM non-parity PC2-5300CL5 SO-DIMM
usb1 at uhci0: USB revision 1.0
uhub1 at usb1 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb2 at uhci1: USB revision 1.0
uhub2 at usb2 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb3 at uhci2: USB revision 1.0
uhub3 at usb3 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb4 at uhci3: USB revision 1.0
uhub4 at usb4 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1
isa0 at ichpcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pms0: Synaptics touchpad, firmware 6.2
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
vmm0 at mainbus0: VMX
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
root on wd0a (055d82c92ce5e61f.a) swap on wd0b dump on wd0b

[Stuart Henderson]

Remove "default" from the rule.

Post by Sebastien Marie
Hi,
I couldn't send packets.
# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: sendmsg: No buffer space available
ping: wrote 8.8.8.8 64 chars, ret=-1
^C
My previous (working) snap was Jun 26.
I tracked a bit the problem until I found that commenting the flow queue
line in pf.conf resolv my problem.
queue fq on bge0 flows 1024 default
I tried to reproduce on other host (i386), recompiling from source,
reloading pf rules with flow queue makes network to become unusable: "No
buffer space available".
The queue statistic shows that qlength is full.
# pfctl -sq -v
queue fq on bce0 flows 1024 default
[ pkts: 0 bytes: 0 dropped pkts: 7 bytes: 650 ]
[ qlength: 50/ 50 avg delay: 0.000ms std-dev: 0.000ms flows: 1 ]
The dmesg below is my test host (with bce0), not my laptop (with bge0).
Thanks.
--
Sebastien Marie
OpenBSD 6.1-current (GENERIC.MP) #24: Thu Jul 6 13:34:29 CEST 2017
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,NXE,SSE3,MWAIT,VMX,EST,TM2,xTPR,PDCM,PERF,SENSOR
real mem = 2137354240 (2038MB)
avail mem = 2083151872 (1986MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
@ 0xf7980 (44 entries)
bios0: vendor Dell Inc. version "A17" date 06/13/2007
bios0: Dell Inc. MM061
acpi0 at bios0: rev 0
acpi0: sleep states S0 S3 S4 S5
acpi0: tables DSDT FACP HPET APIC MCFG SLIC BOOT SSDT
acpi0: wakeup devices LID_(S3) PBTN(S4) MBTN(S5) PCI0(S3) USB0(S0) USB1(S0)
USB2(S0) USB3(S0) EHCI(S0) AZAL(S3) PCIE(S4) RP01(S4) RP02(S3) RP03(S3)
RP04(S3) RP05(S3) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpihpet0 at acpi0: 14318179 Hz
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 166MHz
cpu0: mwait min=64, max=64, C-substates=0.2.2.2.2, IBE
cpu1 at mainbus0: apid 1 (application processor)
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,NXE,SSE3,MWAIT,VMX,EST,TM2,xTPR,PDCM,PERF,SENSOR
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins
acpimcfg0 at acpi0 addr 0xf0000000, bus 0-63
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus -1 (AGP_)
acpiprt2 at acpi0: bus 3 (PCIE)
acpiprt3 at acpi0: bus 11 (RP01)
acpiprt4 at acpi0: bus -1 (RP02)
acpiprt5 at acpi0: bus -1 (RP03)
acpiprt6 at acpi0: bus 12 (RP04)
acpiprt7 at acpi0: bus -1 (RP05)
acpiprt8 at acpi0: bus -1 (RP06)
acpitz0 at acpi0: critical temperature is 126 degC
"*pnp0c14" at acpi0 not configured
acpiac0 at acpi0: AC unit online
acpibat0 at acpi0: BAT0 model " DELLUD2649" serial 878 type LION oem "Sanyo"
acpibtn0 at acpi0: LID_
acpibtn1 at acpi0: PBTN
acpibtn2 at acpi0: SBTN
"PNP0C32" at acpi0 not configured
"PNP0F13" at acpi0 not configured
"PNP0303" at acpi0 not configured
acpivideo0 at acpi0: VID_
acpivideo1 at acpi0: VID_
acpivout0 at acpivideo1: LCD_
acpivideo2 at acpi0: VID2
bios0: ROM list: 0xc0000/0xe800! 0xce800/0x1800
cpu0: Enhanced SpeedStep 1829 MHz: speeds: 1833, 1333, 1000 MHz
pci0 at mainbus0 bus 0: configuration mode 1 (bios)
pchb0 at pci0 dev 0 function 0 "Intel 82945GM Host" rev 0x03
inteldrm0 at pci0 dev 2 function 0 "Intel 82945GM Video" rev 0x03
drm0 at inteldrm0
intagp0 at inteldrm0
agp0 at intagp0: aperture at 0xd0000000, size 0x10000000
inteldrm0: apic 2 int 16
inteldrm0: 848x480, 32bpp
wsdisplay0 at inteldrm0 mux 1: console (std, vt100 emulation)
wsdisplay0: screen 1-5 added (std, vt100 emulation)
"Intel 82945GM Video" rev 0x03 at pci0 dev 2 function 1 not configured
azalia0 at pci0 dev 27 function 0 "Intel 82801GB HD Audio" rev 0x01: msi
azalia0: codecs: Sigmatel STAC9200, Conexant/0x2bfa, using Sigmatel STAC9200
audio0 at azalia0
ppb0 at pci0 dev 28 function 0 "Intel 82801GB PCIE" rev 0x01: apic 2 int 16
pci1 at ppb0 bus 11
wpi0 at pci1 dev 0 function 0 "Intel PRO/Wireless 3945ABG" rev 0x02: msi,
MoW2, address 00:13:02:2e:8b:46
ppb1 at pci0 dev 28 function 3 "Intel 82801GB PCIE" rev 0x01: apic 2 int 19
pci2 at ppb1 bus 12
uhci0 at pci0 dev 29 function 0 "Intel 82801GB USB" rev 0x01: apic 2 int 20
uhci1 at pci0 dev 29 function 1 "Intel 82801GB USB" rev 0x01: apic 2 int 21
uhci2 at pci0 dev 29 function 2 "Intel 82801GB USB" rev 0x01: apic 2 int 22
uhci3 at pci0 dev 29 function 3 "Intel 82801GB USB" rev 0x01: apic 2 int 23
ehci0 at pci0 dev 29 function 7 "Intel 82801GB USB" rev 0x01: apic 2 int 20
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb2 at pci0 dev 30 function 0 "Intel 82801BAM Hub-to-PCI" rev 0xe1
pci3 at ppb2 bus 3
bce0 at pci3 dev 0 function 0 "Broadcom BCM4401B1" rev 0x02: apic 2 int 17,
address 00:15:c5:0b:8b:7a
bmtphy0 at bce0 phy 1: BCM4401 10/100baseTX PHY, rev. 0
"Ricoh 5C832 Firewire" rev 0x00 at pci3 dev 1 function 0 not configured
sdhc0 at pci3 dev 1 function 1 "Ricoh 5C822 SD/MMC" rev 0x19: apic 2 int 18
sdhc0: SDHC 1.0, 33 MHz base clock
sdmmc0 at sdhc0: 4-bit, sd high-speed, mmc high-speed
"Ricoh 5C843 MMC" rev 0x01 at pci3 dev 1 function 2 not configured
"Ricoh 5C592 Memory Stick" rev 0x0a at pci3 dev 1 function 3 not configured
"Ricoh 5C852 xD" rev 0x05 at pci3 dev 1 function 4 not configured
ichpcib0 at pci0 dev 31 function 0 "Intel 82801GBM LPC" rev 0x01: PM disabled
pciide0 at pci0 dev 31 function 2 "Intel 82801GBM SATA" rev 0x01: DMA,
channel 0 wired to compatibility, channel 1 wired to compatibility
wd0 at pciide0 channel 0 drive 0: <ST96812AS>
wd0: 16-sector PIO, LBA48, 57241MB, 117231408 sectors
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 5
atapiscsi0 at pciide0 channel 1 drive 0
scsibus1 at atapiscsi0: 2 targets
cd0 at scsibus1 targ 0 lun 0: <_NEC, DVD+-RW ND-6650A, 102C> ATAPI 5/cdrom removable
cd0(pciide0:1:0): using PIO mode 4, Ultra-DMA mode 2
ichiic0 at pci0 dev 31 function 3 "Intel 82801GB SMBus" rev 0x01: apic 2 int 17
iic0 at ichiic0
spdmem0 at iic0 addr 0x50: 1GB DDR2 SDRAM non-parity PC2-5300CL5 SO-DIMM
spdmem1 at iic0 addr 0x52: 1GB DDR2 SDRAM non-parity PC2-5300CL5 SO-DIMM
usb1 at uhci0: USB revision 1.0
uhub1 at usb1 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb2 at uhci1: USB revision 1.0
uhub2 at usb2 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb3 at uhci2: USB revision 1.0
uhub3 at usb3 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb4 at uhci3: USB revision 1.0
uhub4 at usb4 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1
isa0 at ichpcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pms0: Synaptics touchpad, firmware 6.2
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
vmm0 at mainbus0: VMX
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
root on wd0a (055d82c92ce5e61f.a) swap on wd0b dump on wd0b

[Sebastien Marie]

Post by Stuart Henderson
Remove "default" from the rule.

I am unsure how to interprete the sentence: if I remove "default",
shouldn't the queue apply only on states with "set queue fq" ? And in my
ruleset there are none like that. As I want to use the fq queue on all
my states, using "default" seems right.

But I agree that testing show a different behaviour...

If I define a bandwidth-queue without "default", pfctl complains:
# pfctl -f /etc/pf.conf
pfctl: no default queue specified

If I define a flow-queue without "default", pfctl doesn't complains, and
queue statistic seems to follow activity (so the flow-queue to be used).

If I define a flow-queue with "default", pfctl doesn't complains, and I
couldn't send packets...

It is a bit weird for me, but at least it works without "default".

Thanks Stuart.
--
Sebastien Marie

Post by Stuart Henderson

Post by Sebastien Marie
Hi,
I couldn't send packets.
# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: sendmsg: No buffer space available
ping: wrote 8.8.8.8 64 chars, ret=-1
^C
My previous (working) snap was Jun 26.
I tracked a bit the problem until I found that commenting the flow queue
line in pf.conf resolv my problem.
queue fq on bge0 flows 1024 default
I tried to reproduce on other host (i386), recompiling from source,
reloading pf rules with flow queue makes network to become unusable: "No
buffer space available".
The queue statistic shows that qlength is full.
# pfctl -sq -v
queue fq on bce0 flows 1024 default
[ pkts: 0 bytes: 0 dropped pkts: 7 bytes: 650 ]
[ qlength: 50/ 50 avg delay: 0.000ms std-dev: 0.000ms flows: 1 ]
The dmesg below is my test host (with bce0), not my laptop (with bge0).
Thanks.
--
Sebastien Marie

[Stuart Henderson]

Post by Sebastien Marie

Post by Stuart Henderson
Remove "default" from the rule.

I am unsure how to interprete the sentence: if I remove "default",
shouldn't the queue apply only on states with "set queue fq" ? And in my
ruleset there are none like that. As I want to use the fq queue on all
my states, using "default" seems right.
But I agree that testing show a different behaviour...
# pfctl -f /etc/pf.conf
pfctl: no default queue specified
If I define a flow-queue without "default", pfctl doesn't complains, and
queue statistic seems to follow activity (so the flow-queue to be used).
If I define a flow-queue with "default", pfctl doesn't complains, and I
couldn't send packets...
It is a bit weird for me, but at least it works without "default".
Thanks Stuart.

As far as I know, with a flow-queue on an interface, you don't assign
traffic, it applies to all traffic outbound of that interface.
mikeb has some more bits to add, but the current status that if you use
"default" it misconfigures the queue. (he's aware of that, he's away at
the moment, which is why I jumped in rather than waiting for him to
answer :)

Other parts are in https://github.com/mbelop/src/commits/hfsq
and add bandwidth control to the flow queue, e.g.

queue hfsq-em1 on em1 flows 1024 bandwidth 1500K max 1500K quantum 400 qlimit 1000 default

I had a deadlock with IPsec with this but otherwise it was working
very well for me, latency under load is very good indeed.