Hrvoje Popovski
2017-07-03 23:49:44 UTC
Hi all,
with cvs tree (with or without bluhm@ uipc_syscalls patch) fetched few
minutes ago i'm getting panic below. would show witness be of any good?
Setup is quite standard, carp, pf, pfsync, isakmpd ....
# panic: kernel diagnostic assertion "_kernel_lock_held()" failed: file
"/usr/src/sys/netinet/ip_output.c", line 234
Stopped at db_enter+0x9: leave
TID PID UID PRFLAGS PFLAGS CPU COMMAND
*324761 84163 0 0x14000 0x200 3 pfpurge
db_enter(ffffffff817aac8e,ffff80002a197af0,10,ffff80002a197ad0,282,8) at
db_ent
er+0x9
panic(ffffffff817b0eb8,ffff80002a197c30,ffff80002a197c90,ffffff00cb60b040,ffff8
0002a197ca0,ffff8000009e5000) at panic+0x102
__assert(ffffffff8177cd9f,ffffffff819b0848,ea,ffffffff8180a1e2,ffff80002a197ca0
,f6548594e2a225ee) at __assert+0x35
ip_output(ffffff00bee50200,0,0,2,ffff8000005306d0,0) at ip_output+0xc66
pfsync_sendout(2,6,ffffff031e278b00,ffff800000530020,ffff800000530000,ffffffff8
17bc49a) at pfsync_sendout+0x4a3
pfsync_q_ins(ffffff031e278b00,2,0,0,a7,87) at pfsync_q_ins+0x80
pf_remove_state(ffffff031e278b00,ffffff02cfe65af8,ffffff031e278c38,ffffff031e27
8b00,ffffff031e278b00,4b9) at pf_remove_state+0x46
pf_purge_expired_states(a7,ffffffff819b6540,ffff80002a1828e8,ffffffff811f8a40,a
,ffff80002a197ed0) at pf_purge_expired_states+0x94
pf_purge_thread(ffff80002a1828e8,8,0,2a2,0,0) at pf_purge_thread+0x7f
end trace frame: 0x0, count: 6
https://www.openbsd.org/ddb.html describes the minimum info required in bug
reports. Insufficient info makes it difficult to find and fix bugs.
ddb{3}>
ddb{3}> ps
PID TID PPID UID S FLAGS WAIT COMMAND
95333 408859 31895 68 3 0x90 select isakmpd
31895 176251 1 0 3 0x80 netio isakmpd
63436 504646 39586 0 3 0x100083 ttyin ksh
39586 262651 92535 1000 3 0x10008b pause ksh
92535 357987 8833 1000 3 0x90 select sshd
8833 112150 26268 0 3 0x82 poll sshd
31387 328233 97173 0 3 0x100080 netio tcpdump
42290 171829 1 0 3 0x100083 ttyin ksh
65439 248358 1 0 3 0x100083 ttyin getty
21248 239925 1 0 3 0x100083 ttyin getty
52429 205382 1 0 3 0x100083 ttyin getty
8335 298412 1 0 3 0x100083 ttyin getty
15444 268378 1 0 3 0x100083 ttyin getty
1219 519319 21071 0 3 0x100082 piperd logger
97173 355107 21071 76 3 0x100092 bpf tcpdump
21071 69502 20470 0 3 0x10008a pause sh
20470 452736 41956 0 3 0x100090 piperd cron
41956 118037 1 0 3 0x100098 poll cron
77911 43754 46055 623 3 0x90 nanosleep zabbix_agentd
90124 154706 46055 623 3 0x90 select zabbix_agentd
36985 476197 46055 623 3 0x90 nanosleep zabbix_agentd
46055 147897 1 623 3 0x90 wait zabbix_agentd
66988 155598 88699 606 3 0x90 kqread ladvd
88699 289592 1 0 3 0x80 kqread ladvd
14520 422918 91264 95 3 0x100092 kqread smtpd
47626 399875 91264 103 3 0x100092 kqread smtpd
53425 115027 91264 95 3 0x100092 kqread smtpd
72522 448888 91264 95 3 0x100092 kqread smtpd
42825 513503 91264 95 3 0x100092 kqread smtpd
80815 73826 91264 95 3 0x100092 kqread smtpd
91264 521300 1 0 3 0x100080 kqread smtpd
41105 230373 1 77 3 0x100090 poll dhcpd
17016 195580 1 0 3 0x80 kqread snmpd
16112 301694 1 91 3 0x92 kqread snmpd
77397 285973 1 91 3 0x92 kqread snmpd
26268 209370 1 0 3 0x80 select sshd
75434 386174 22720 83 3 0x100092 poll ntpd
22720 30117 50775 83 3 0x100092 poll ntpd
50775 233390 1 0 3 0x100080 poll ntpd
64659 285289 51210 74 3 0x100090 bpf pflogd
51210 109018 1 0 3 0x100080 netio pflogd
55462 143783 72659 73 3 0x100090 kqread syslogd
72659 395721 1 0 3 0x100082 netio syslogd
5517 124236 0 0 3 0x14200 pgzero zerothread
4113 392916 0 0 3 0x14200 aiodoned aiodoned
70989 126677 0 0 3 0x14200 syncer update
10794 65054 0 0 3 0x14200 cleaner cleaner
15625 371308 0 0 3 0x14200 reaper reaper
40060 90383 0 0 3 0x14200 pgdaemon pagedaemon
49299 364764 0 0 3 0x14200 bored crynlk
46049 427455 0 0 3 0x14200 bored crypto
*84163 324761 0 0 7 0x14200 pfpurge
20770 460440 0 0 3 0x14200 usbtsk usbtask
78669 340878 0 0 3 0x14200 usbatsk usbatsk
64678 117299 0 0 3 0x40014200 acpi0 acpi0
14851 204792 0 0 7 0x40014200 idle7
78460 399928 0 0 7 0x40014200 idle6
79148 150294 0 0 7 0x40014200 idle5
99590 11699 0 0 7 0x40014200 idle4
20300 73152 0 0 3 0x40014200 idle3
57788 69361 0 0 7 0x40014200 idle2
10347 454769 0 0 7 0x40014200 idle1
90158 269554 0 0 3 0x14200 bored sensors
10993 165403 0 0 3 0x14200 netlock softnet
5144 88161 0 0 3 0x14200 bored systqmp
28138 325820 0 0 3 0x14200 netlock systq
14696 172035 0 0 3 0x40014200 netlock softclock
91848 440922 0 0 7 0x40014200 idle0
46949 473780 0 0 3 0x14200 bored sbar
1 275196 0 0 3 0x82 wait init
0 0 -1 0 3 0x10200 scheduler swapper
ddb{3}>
ddb{3}> mach ddbcpu 0
Stopped at db_enter+0x9: leave
db_enter(ffffffff819c5380,ffffffff81468446,ffff80002a143d80,10,ffff80002a143d28
,286) at db_enter+0x9
x86_ipi_handler(ffffffff819c5380,ffff80000001ee00,ffff80000001b440,0,ffff800000
01b440,0) at x86_ipi_handler+0x85
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1c
--- interrupt ---
acpicpu_idle(ffff80002a12e470,ffffffff819c5380,ffffffff819c5390,0,f6548594e2a22
5ee,286) at acpicpu_idle+0xec
cpu_idle_cycle(ffffffff819c5380,8,0,ffffffff819c5380,ffffffff81477200,0)
at cpu
_idle_cycle+0x10
end trace frame: 0x0, count: 10
ddb{0}> trace
db_enter(ffffffff819c5380,ffffffff81468446,ffff80002a143d80,10,ffff80002a143d28
,286) at db_enter+0x9
x86_ipi_handler(ffffffff819c5380,ffff80000001ee00,ffff80000001b440,0,ffff800000
01b440,0) at x86_ipi_handler+0x85
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1c
--- interrupt ---
acpicpu_idle(ffff80002a12e470,ffffffff819c5380,ffffffff819c5390,0,f6548594e2a22
5ee,286) at acpicpu_idle+0xec
cpu_idle_cycle(ffffffff819c5380,8,0,ffffffff819c5380,ffffffff81477200,0)
at cpu
_idle_cycle+0x10
end trace frame: 0x0, count: -5
ddb{0}>
ddb{0}> mach ddbcpu 1
Stopped at db_enter+0x9: leave
db_enter(ffff80000003d000,ffffffff81b0a768,ffff80000003d000,10,ffff80002a161d28
,286) at db_enter+0x9
x86_ipi_handler(ffff80000003d000,ffff80000001ef00,ffff80000001b460,0,ffffffff81
b0a768,0) at x86_ipi_handler+0x85
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1c
--- interrupt ---
acpicpu_idle(ffff80002a12f1c0,ffff80000003d000,ffff80000003d010,0,f6548594e2a22
5ee,286) at acpicpu_idle+0xec
cpu_idle_cycle(ffff80000003d000,8,0,ffff80000003d000,ffffffff81477200,0)
at cpu
_idle_cycle+0x10
end trace frame: 0x0, count: 10
ddb{1}> trace
db_enter(ffff80000003d000,ffffffff81b0a768,ffff80000003d000,10,ffff80002a161d28
,286) at db_enter+0x9
x86_ipi_handler(ffff80000003d000,ffff80000001ef00,ffff80000001b460,0,ffffffff81
b0a768,0) at x86_ipi_handler+0x85
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1c
--- interrupt ---
acpicpu_idle(ffff80002a12f1c0,ffff80000003d000,ffff80000003d010,0,f6548594e2a22
5ee,286) at acpicpu_idle+0xec
cpu_idle_cycle(ffff80000003d000,8,0,ffff80000003d000,ffffffff81477200,0)
at cpu
_idle_cycle+0x10
end trace frame: 0x0, count: -5
ddb{1}>
ddb{1}> mach ddbcpu 2
Stopped at db_enter+0x9: leave
db_enter(ffff80000003f000,ffffffff81b0a768,ffff80000003f000,10,ffff80002a166d28
,286) at db_enter+0x9
x86_ipi_handler(ffff80000003f000,ffff80000001e000,ffff80000001b4a0,0,1c0,0)
at x
86_ipi_handler+0x85
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1c
--- interrupt ---
acpicpu_idle(ffff80002a12f3f8,ffff80000003f000,ffff80000003f010,0,f6548594e2a22
5ee,286) at acpicpu_idle+0xec
cpu_idle_cycle(ffff80000003f000,8,0,ffff80000003f000,ffffffff81477200,0)
at cpu
_idle_cycle+0x10
end trace frame: 0x0, count: 10
ddb{2}> trace
db_enter(ffff80000003f000,ffffffff81b0a768,ffff80000003f000,10,ffff80002a166d28
,286) at db_enter+0x9
x86_ipi_handler(ffff80000003f000,ffff80000001e000,ffff80000001b4a0,0,1c0,0)
at x
86_ipi_handler+0x85
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1c
--- interrupt ---
acpicpu_idle(ffff80002a12f3f8,ffff80000003f000,ffff80000003f010,0,f6548594e2a22
5ee,286) at acpicpu_idle+0xec
cpu_idle_cycle(ffff80000003f000,8,0,ffff80000003f000,ffffffff81477200,0)
at cpu
_idle_cycle+0x10
end trace frame: 0x0, count: -5
ddb{2}>
ddb{2}> mach ddbcpu 4
Stopped at db_enter+0x9: leave
db_enter(ffff800000041000,ffffffff81b0a768,ffff800000041000,10,ffff80002a170d28
,286) at db_enter+0x9
x86_ipi_handler(ffff800000041000,ffff80000001e600,ffff80000001b4e0,0,ffffffff81
b0a768,0) at x86_ipi_handler+0x85
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1c
--- interrupt ---
acpicpu_idle(ffff80002a12f868,ffff800000041000,ffff800000041010,0,f6548594e2a22
5ee,286) at acpicpu_idle+0xec
cpu_idle_cycle(ffff800000041000,8,0,ffff800000041000,ffffffff81477200,0)
at cpu
_idle_cycle+0x10
end trace frame: 0x0, count: 10
ddb{4}> trace
db_enter(ffff800000041000,ffffffff81b0a768,ffff800000041000,10,ffff80002a170d28
,286) at db_enter+0x9
x86_ipi_handler(ffff800000041000,ffff80000001e600,ffff80000001b4e0,0,ffffffff81
b0a768,0) at x86_ipi_handler+0x85
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1c
--- interrupt ---
acpicpu_idle(ffff80002a12f868,ffff800000041000,ffff800000041010,0,f6548594e2a22
5ee,286) at acpicpu_idle+0xec
cpu_idle_cycle(ffff800000041000,8,0,ffff800000041000,ffffffff81477200,0)
at cpu
_idle_cycle+0x10
end trace frame: 0x0, count: -5
ddb{4}>
ddb{4}> mach ddbcpu 5
Stopped at db_enter+0x9: leave
db_enter(ffff800000042000,ffffffff81b0a768,ffff800000042000,10,ffff80002a177d28
,286) at db_enter+0x9
x86_ipi_handler(ffff800000042000,ffff80000001e700,ffff80000001b500,0,ffffffff81
b0a768,0) at x86_ipi_handler+0x85
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1c
--- interrupt ---
acpicpu_idle(ffff80002a12faa0,ffff800000042000,ffff800000042010,0,f6548594e2a22
5ee,286) at acpicpu_idle+0xec
cpu_idle_cycle(ffff800000042000,8,0,ffff800000042000,ffffffff81477200,0)
at cpu
_idle_cycle+0x10
end trace frame: 0x0, count: 10
ddb{5}> trace
db_enter(ffff800000042000,ffffffff81b0a768,ffff800000042000,10,ffff80002a177d28
,286) at db_enter+0x9
x86_ipi_handler(ffff800000042000,ffff80000001e700,ffff80000001b500,0,ffffffff81
b0a768,0) at x86_ipi_handler+0x85
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1c
--- interrupt ---
acpicpu_idle(ffff80002a12faa0,ffff800000042000,ffff800000042010,0,f6548594e2a22
5ee,286) at acpicpu_idle+0xec
cpu_idle_cycle(ffff800000042000,8,0,ffff800000042000,ffffffff81477200,0)
at cpu
_idle_cycle+0x10
end trace frame: 0x0, count: -5
ddb{5}>
ddb{5}> mach ddbcpu 6
Stopped at db_enter+0x9: leave
db_enter(ffff800000044000,ffffffff81b0a768,ffff800000044000,10,ffff80002a17cd28
,286) at db_enter+0x9
x86_ipi_handler(ffff800000044000,ffff80000001e800,ffff80000001b520,0,ffffffff81
b0a768,0) at x86_ipi_handler+0x85
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1c
--- interrupt ---
acpicpu_idle(ffff80002a12fcd8,ffff800000044000,ffff800000044010,0,f6548594e2a22
5ee,286) at acpicpu_idle+0xec
cpu_idle_cycle(ffff800000044000,8,0,ffff800000044000,ffffffff81477200,0)
at cpu
_idle_cycle+0x10
end trace frame: 0x0, count: 10
ddb{6}> trace
db_enter(ffff800000044000,ffffffff81b0a768,ffff800000044000,10,ffff80002a17cd28
,286) at db_enter+0x9
x86_ipi_handler(ffff800000044000,ffff80000001e800,ffff80000001b520,0,ffffffff81
b0a768,0) at x86_ipi_handler+0x85
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1c
--- interrupt ---
acpicpu_idle(ffff80002a12fcd8,ffff800000044000,ffff800000044010,0,f6548594e2a22
5ee,286) at acpicpu_idle+0xec
cpu_idle_cycle(ffff800000044000,8,0,ffff800000044000,ffffffff81477200,0)
at cpu
_idle_cycle+0x10
end trace frame: 0x0, count: -5
ddb{6}>
ddb{6}> mach ddbcpu 7
Stopped at db_enter+0x9: leave
db_enter(ffff800000045000,ffffffff81b0a768,ffff800000045000,10,ffff80002a181d28
,286) at db_enter+0x9
x86_ipi_handler(ffff800000045000,ffff80000001e900,ffff80000001b540,0,ffffffff81
b0a768,0) at x86_ipi_handler+0x85
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1c
--- interrupt ---
acpicpu_idle(ffff80002a182008,ffff800000045000,ffff800000045010,0,f6548594e2a22
5ee,286) at acpicpu_idle+0xec
cpu_idle_cycle(ffff800000045000,8,0,ffff800000045000,ffffffff81477200,0)
at cpu
_idle_cycle+0x10
end trace frame: 0x0, count: 10
ddb{7}> trace
db_enter(ffff800000045000,ffffffff81b0a768,ffff800000045000,10,ffff80002a181d28
,286) at db_enter+0x9
x86_ipi_handler(ffff800000045000,ffff80000001e900,ffff80000001b540,0,ffffffff81
b0a768,0) at x86_ipi_handler+0x85
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1c
--- interrupt ---
acpicpu_idle(ffff80002a182008,ffff800000045000,ffff800000045010,0,f6548594e2a22
5ee,286) at acpicpu_idle+0xec
cpu_idle_cycle(ffff800000045000,8,0,ffff800000045000,ffffffff81477200,0)
at cpu
_idle_cycle+0x10
end trace frame: 0x0, count: -5
ddb{7}>
ddb{7}> dmesg
OpenBSD 6.1-current (GENERIC.MP) #10: Tue Jul 4 01:29:35 CEST 2017
***@fw02bcbn.srce.hr:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 12854988800 (12259MB)
avail mem = 12359168000 (11786MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.6 @ 0xcf49c000 (84 entries)
bios0: vendor Dell Inc. version "6.4.0" date 07/23/2013
bios0: Dell Inc. PowerEdge R610
acpi0 at bios0: rev 2
acpi0: sleep states S0 S4 S5
acpi0: tables DSDT FACP APIC SPCR HPET DM__ MCFG WD__ SLIC ERST HEST
BERT EINJ S
RAT TCPA
acpi0: wakeup devices PCI0(S5)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 32 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E5630 @ 2.53GHz, 2527.31 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,C
FLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,V
MX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,PAGE1
GB,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: TSC frequency 2527312550 Hz
cpu0: smt 0, core 0, package 1
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 132MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.1, IBE
cpu1 at mainbus0: apid 0 (application processor)
cpu1: Intel(R) Xeon(R) CPU E5630 @ 2.53GHz, 2527.00 MHz
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,C
FLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,V
MX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,PAGE1
GB,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 0, core 0, package 0
cpu2 at mainbus0: apid 34 (application processor)
[138/4944]
cpu2: Intel(R) Xeon(R) CPU E5630 @ 2.53GHz, 2527.00 MHz
cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,C
FLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,V
MX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,PAGE1
GB,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
cpu2: 256KB 64b/line 8-way L2 cache
cpu2: smt 0, core 1, package 1
cpu3 at mainbus0: apid 2 (application processor)
cpu3: Intel(R) Xeon(R) CPU E5630 @ 2.53GHz, 2527.00 MHz
cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,C
FLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,V
MX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,PAGE1
GB,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
cpu3: 256KB 64b/line 8-way L2 cache
cpu3: smt 0, core 1, package 0
cpu4 at mainbus0: apid 50 (application processor)
cpu4: Intel(R) Xeon(R) CPU E5630 @ 2.53GHz, 2527.00 MHz
cpu4:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,C
FLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,V
MX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,PAGE1
GB,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
cpu4: 256KB 64b/line 8-way L2 cache
cpu4: smt 0, core 9, package 1
cpu5 at mainbus0: apid 18 (application processor)
cpu5: Intel(R) Xeon(R) CPU E5630 @ 2.53GHz, 2527.00 MHz
cpu5:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,C
FLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,V
MX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,PAGE1
GB,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
cpu5: 256KB 64b/line 8-way L2 cache
cpu5: smt 0, core 9, package 0
cpu6 at mainbus0: apid 52 (application processor)
cpu6: Intel(R) Xeon(R) CPU E5630 @ 2.53GHz, 2527.00 MHz
cpu6:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,C
FLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,V
MX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,PAGE1
GB,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
cpu6: 256KB 64b/line 8-way L2 cache
cpu6: smt 0, core 10, package 1
cpu7 at mainbus0: apid 20 (application processor)
cpu7: Intel(R) Xeon(R) CPU E5630 @ 2.53GHz, 2526.99 MHz
cpu7:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,C
FLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,V
MX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,PAGE1
GB,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
cpu7: 256KB 64b/line 8-way L2 cache
cpu7: smt 0, core 10, package 0
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 20, 24 pins
[90/4944]
ioapic1 at mainbus0: apid 1 pa 0xfec80000, version 20, 24 pins
acpihpet0 at acpi0: 14318179 Hz
acpimcfg0 at acpi0 addr 0xe0000000, bus 0-255
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 1 (PEX1)
acpiprt2 at acpi0: bus 2 (PEX3)
acpiprt3 at acpi0: bus -1 (PEX4)
acpiprt4 at acpi0: bus -1 (PEX5)
acpiprt5 at acpi0: bus -1 (PEX6)
acpiprt6 at acpi0: bus 4 (PEX7)
acpiprt7 at acpi0: bus 5 (PEX9)
acpiprt8 at acpi0: bus -1 (PEXA)
acpiprt9 at acpi0: bus 3 (SBEX)
acpiprt10 at acpi0: bus 6 (COMP)
acpicpu0 at acpi0: C1(@1 halt!)
acpicpu1 at acpi0: C1(@1 halt!)
acpicpu2 at acpi0: C1(@1 halt!)
acpicpu3 at acpi0: C1(@1 halt!)
acpicpu4 at acpi0: C1(@1 halt!)
acpicpu5 at acpi0: C1(@1 halt!)
acpicpu6 at acpi0: C1(@1 halt!)
acpicpu7 at acpi0: C1(@1 halt!)
"PNP0C33" at acpi0 not configured
"ACPI000D" at acpi0 not configured
"PNP0501" at acpi0 not configured
"PNP0501" at acpi0 not configured
"IPI0001" at acpi0 not configured
"PNP0C14" at acpi0 not configured
ipmi at mainbus0 not configured
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 5500 Host" rev 0x13
ppb0 at pci0 dev 1 function 0 "Intel X58 PCIE" rev 0x13
pci1 at ppb0 bus 1
bnx0 at pci1 dev 0 function 0 "Broadcom BCM5709" rev 0x20: apic 1 int 4
bnx1 at pci1 dev 0 function 1 "Broadcom BCM5709" rev 0x20: apic 1 int 16
ppb1 at pci0 dev 3 function 0 "Intel X58 PCIE" rev 0x13
pci2 at ppb1 bus 2
bnx2 at pci2 dev 0 function 0 "Broadcom BCM5709" rev 0x20: apic 1 int 0
bnx3 at pci2 dev 0 function 1 "Broadcom BCM5709" rev 0x20: apic 1 int 10
ppb2 at pci0 dev 7 function 0 "Intel X58 PCIE" rev 0x13: msi
pci3 at ppb2 bus 4
ppb3 at pci0 dev 9 function 0 "Intel X58 PCIE" rev 0x13: msi
pci4 at ppb3 bus 5
"Intel X58 Misc" rev 0x13 at pci0 dev 20 function 0 not configured
"Intel X58 GPIO" rev 0x13 at pci0 dev 20 function 1 not configured
"Intel X58 RAS" rev 0x13 at pci0 dev 20 function 2 not configured
uhci0 at pci0 dev 26 function 0 "Intel 82801I USB" rev 0x02: apic 0 int 17
uhci1 at pci0 dev 26 function 1 "Intel 82801I USB" rev 0x02: apic 0 int 18
ehci0 at pci0 dev 26 function 7 "Intel 82801I USB" rev 0x02: apic 0 int 19
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev
2.00/1.00 a
ddr 1
ppb4 at pci0 dev 28 function 0 "Intel 82801I PCIE" rev 0x02
pci5 at ppb4 bus 3
[36/4944]
mfi0 at pci5 dev 0 function 0 "Symbios Logic SAS1078" rev 0x04: apic 0
int 16
mfi0: "PERC 6/i Integrated", firmware 6.3.3.0002, 256MB cache
scsibus1 at mfi0: 64 targets
sd0 at scsibus1 targ 0 lun 0: <DELL, PERC 6/i, 1.22> SCSI3 0/direct
fixed naa.6
a4badb0331002001357370807b4917a
sd0: 139392MB, 512 bytes/sector, 285474816 sectors
uhci2 at pci0 dev 29 function 0 "Intel 82801I USB" rev 0x02: apic 0 int 21
uhci3 at pci0 dev 29 function 1 "Intel 82801I USB" rev 0x02: apic 0 int 20
ehci1 at pci0 dev 29 function 7 "Intel 82801I USB" rev 0x02: apic 0 int 21
usb1 at ehci1: USB revision 2.0
uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev
2.00/1.00 a
ddr 1
ppb5 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x92
pci6 at ppb5 bus 6
vga1 at pci6 dev 3 function 0 "Matrox MGA G200eW" rev 0x0a
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
pcib0 at pci0 dev 31 function 0 "Intel 82801IB LPC" rev 0x02
pciide0 at pci0 dev 31 function 2 "Intel 82801I SATA" rev 0x02: DMA,
channel 0 c
onfigured to native-PCI, channel 1 configured to native-PCI
pciide0: using apic 0 int 23 for native-PCI interrupt
atapiscsi0 at pciide0 channel 0 drive 0
scsibus2 at atapiscsi0: 2 targets
cd0 at scsibus2 targ 0 lun 0: <TEAC, DVD-ROM DV-28SW, R.2A> ATAPI
5/cdrom remov
able
cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 5
usb2 at uhci0: USB revision 1.0
uhub2 at usb2 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 a
ddr 1
usb3 at uhci1: USB revision 1.0
uhub3 at usb3 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 a
ddr 1
usb4 at uhci2: USB revision 1.0
uhub4 at usb4 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 a
ddr 1
usb5 at uhci3: USB revision 1.0
uhub5 at usb5 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 a
ddr 1
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
uhub6 at uhub0 port 3 configuration 1 interface 0 "Standard Microsystems
produc
t 0x2514" rev 2.00/0.00 addr 2
uhidev0 at uhub4 port 2 configuration 1 interface 0 "Avocent USB
Composite Devi
ce-0" rev 1.10/0.00 addr 2
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 variable keys, 6 key codes
wskbd1 at ukbd0 mux 1
wskbd1: connecting to wsdisplay0
uhidev1 at uhub4 port 2 configuration 1 interface 1 "Avocent USB
Composite Devi
ce-0" rev 1.10/0.00 addr 2
uhidev1: iclass 3/1
ums0 at uhidev1: 3 buttons, Z dir
wsmouse0 at ums0 mux 0
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (bd0f9f53c67ec5d2.a) swap on sd0b dump on sd0b
bnx0: address 00:22:19:6b:e0:8c
brgphy0 at bnx0 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
bnx1: address 00:22:19:6b:e0:8e
brgphy1 at bnx1 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
bnx2: address 00:22:19:6b:e0:90
brgphy2 at bnx2 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
bnx3: address 00:22:19:6b:e0:92
brgphy3 at bnx3 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
<2>carp1: state transition: BACKUP -> MASTER
<2>carp2: state transition: BACKUP -> MASTER
with cvs tree (with or without bluhm@ uipc_syscalls patch) fetched few
minutes ago i'm getting panic below. would show witness be of any good?
Setup is quite standard, carp, pf, pfsync, isakmpd ....
# panic: kernel diagnostic assertion "_kernel_lock_held()" failed: file
"/usr/src/sys/netinet/ip_output.c", line 234
Stopped at db_enter+0x9: leave
TID PID UID PRFLAGS PFLAGS CPU COMMAND
*324761 84163 0 0x14000 0x200 3 pfpurge
db_enter(ffffffff817aac8e,ffff80002a197af0,10,ffff80002a197ad0,282,8) at
db_ent
er+0x9
panic(ffffffff817b0eb8,ffff80002a197c30,ffff80002a197c90,ffffff00cb60b040,ffff8
0002a197ca0,ffff8000009e5000) at panic+0x102
__assert(ffffffff8177cd9f,ffffffff819b0848,ea,ffffffff8180a1e2,ffff80002a197ca0
,f6548594e2a225ee) at __assert+0x35
ip_output(ffffff00bee50200,0,0,2,ffff8000005306d0,0) at ip_output+0xc66
pfsync_sendout(2,6,ffffff031e278b00,ffff800000530020,ffff800000530000,ffffffff8
17bc49a) at pfsync_sendout+0x4a3
pfsync_q_ins(ffffff031e278b00,2,0,0,a7,87) at pfsync_q_ins+0x80
pf_remove_state(ffffff031e278b00,ffffff02cfe65af8,ffffff031e278c38,ffffff031e27
8b00,ffffff031e278b00,4b9) at pf_remove_state+0x46
pf_purge_expired_states(a7,ffffffff819b6540,ffff80002a1828e8,ffffffff811f8a40,a
,ffff80002a197ed0) at pf_purge_expired_states+0x94
pf_purge_thread(ffff80002a1828e8,8,0,2a2,0,0) at pf_purge_thread+0x7f
end trace frame: 0x0, count: 6
https://www.openbsd.org/ddb.html describes the minimum info required in bug
reports. Insufficient info makes it difficult to find and fix bugs.
ddb{3}>
ddb{3}> ps
PID TID PPID UID S FLAGS WAIT COMMAND
95333 408859 31895 68 3 0x90 select isakmpd
31895 176251 1 0 3 0x80 netio isakmpd
63436 504646 39586 0 3 0x100083 ttyin ksh
39586 262651 92535 1000 3 0x10008b pause ksh
92535 357987 8833 1000 3 0x90 select sshd
8833 112150 26268 0 3 0x82 poll sshd
31387 328233 97173 0 3 0x100080 netio tcpdump
42290 171829 1 0 3 0x100083 ttyin ksh
65439 248358 1 0 3 0x100083 ttyin getty
21248 239925 1 0 3 0x100083 ttyin getty
52429 205382 1 0 3 0x100083 ttyin getty
8335 298412 1 0 3 0x100083 ttyin getty
15444 268378 1 0 3 0x100083 ttyin getty
1219 519319 21071 0 3 0x100082 piperd logger
97173 355107 21071 76 3 0x100092 bpf tcpdump
21071 69502 20470 0 3 0x10008a pause sh
20470 452736 41956 0 3 0x100090 piperd cron
41956 118037 1 0 3 0x100098 poll cron
77911 43754 46055 623 3 0x90 nanosleep zabbix_agentd
90124 154706 46055 623 3 0x90 select zabbix_agentd
36985 476197 46055 623 3 0x90 nanosleep zabbix_agentd
46055 147897 1 623 3 0x90 wait zabbix_agentd
66988 155598 88699 606 3 0x90 kqread ladvd
88699 289592 1 0 3 0x80 kqread ladvd
14520 422918 91264 95 3 0x100092 kqread smtpd
47626 399875 91264 103 3 0x100092 kqread smtpd
53425 115027 91264 95 3 0x100092 kqread smtpd
72522 448888 91264 95 3 0x100092 kqread smtpd
42825 513503 91264 95 3 0x100092 kqread smtpd
80815 73826 91264 95 3 0x100092 kqread smtpd
91264 521300 1 0 3 0x100080 kqread smtpd
41105 230373 1 77 3 0x100090 poll dhcpd
17016 195580 1 0 3 0x80 kqread snmpd
16112 301694 1 91 3 0x92 kqread snmpd
77397 285973 1 91 3 0x92 kqread snmpd
26268 209370 1 0 3 0x80 select sshd
75434 386174 22720 83 3 0x100092 poll ntpd
22720 30117 50775 83 3 0x100092 poll ntpd
50775 233390 1 0 3 0x100080 poll ntpd
64659 285289 51210 74 3 0x100090 bpf pflogd
51210 109018 1 0 3 0x100080 netio pflogd
55462 143783 72659 73 3 0x100090 kqread syslogd
72659 395721 1 0 3 0x100082 netio syslogd
5517 124236 0 0 3 0x14200 pgzero zerothread
4113 392916 0 0 3 0x14200 aiodoned aiodoned
70989 126677 0 0 3 0x14200 syncer update
10794 65054 0 0 3 0x14200 cleaner cleaner
15625 371308 0 0 3 0x14200 reaper reaper
40060 90383 0 0 3 0x14200 pgdaemon pagedaemon
49299 364764 0 0 3 0x14200 bored crynlk
46049 427455 0 0 3 0x14200 bored crypto
*84163 324761 0 0 7 0x14200 pfpurge
20770 460440 0 0 3 0x14200 usbtsk usbtask
78669 340878 0 0 3 0x14200 usbatsk usbatsk
64678 117299 0 0 3 0x40014200 acpi0 acpi0
14851 204792 0 0 7 0x40014200 idle7
78460 399928 0 0 7 0x40014200 idle6
79148 150294 0 0 7 0x40014200 idle5
99590 11699 0 0 7 0x40014200 idle4
20300 73152 0 0 3 0x40014200 idle3
57788 69361 0 0 7 0x40014200 idle2
10347 454769 0 0 7 0x40014200 idle1
90158 269554 0 0 3 0x14200 bored sensors
10993 165403 0 0 3 0x14200 netlock softnet
5144 88161 0 0 3 0x14200 bored systqmp
28138 325820 0 0 3 0x14200 netlock systq
14696 172035 0 0 3 0x40014200 netlock softclock
91848 440922 0 0 7 0x40014200 idle0
46949 473780 0 0 3 0x14200 bored sbar
1 275196 0 0 3 0x82 wait init
0 0 -1 0 3 0x10200 scheduler swapper
ddb{3}>
ddb{3}> mach ddbcpu 0
Stopped at db_enter+0x9: leave
db_enter(ffffffff819c5380,ffffffff81468446,ffff80002a143d80,10,ffff80002a143d28
,286) at db_enter+0x9
x86_ipi_handler(ffffffff819c5380,ffff80000001ee00,ffff80000001b440,0,ffff800000
01b440,0) at x86_ipi_handler+0x85
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1c
--- interrupt ---
acpicpu_idle(ffff80002a12e470,ffffffff819c5380,ffffffff819c5390,0,f6548594e2a22
5ee,286) at acpicpu_idle+0xec
cpu_idle_cycle(ffffffff819c5380,8,0,ffffffff819c5380,ffffffff81477200,0)
at cpu
_idle_cycle+0x10
end trace frame: 0x0, count: 10
ddb{0}> trace
db_enter(ffffffff819c5380,ffffffff81468446,ffff80002a143d80,10,ffff80002a143d28
,286) at db_enter+0x9
x86_ipi_handler(ffffffff819c5380,ffff80000001ee00,ffff80000001b440,0,ffff800000
01b440,0) at x86_ipi_handler+0x85
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1c
--- interrupt ---
acpicpu_idle(ffff80002a12e470,ffffffff819c5380,ffffffff819c5390,0,f6548594e2a22
5ee,286) at acpicpu_idle+0xec
cpu_idle_cycle(ffffffff819c5380,8,0,ffffffff819c5380,ffffffff81477200,0)
at cpu
_idle_cycle+0x10
end trace frame: 0x0, count: -5
ddb{0}>
ddb{0}> mach ddbcpu 1
Stopped at db_enter+0x9: leave
db_enter(ffff80000003d000,ffffffff81b0a768,ffff80000003d000,10,ffff80002a161d28
,286) at db_enter+0x9
x86_ipi_handler(ffff80000003d000,ffff80000001ef00,ffff80000001b460,0,ffffffff81
b0a768,0) at x86_ipi_handler+0x85
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1c
--- interrupt ---
acpicpu_idle(ffff80002a12f1c0,ffff80000003d000,ffff80000003d010,0,f6548594e2a22
5ee,286) at acpicpu_idle+0xec
cpu_idle_cycle(ffff80000003d000,8,0,ffff80000003d000,ffffffff81477200,0)
at cpu
_idle_cycle+0x10
end trace frame: 0x0, count: 10
ddb{1}> trace
db_enter(ffff80000003d000,ffffffff81b0a768,ffff80000003d000,10,ffff80002a161d28
,286) at db_enter+0x9
x86_ipi_handler(ffff80000003d000,ffff80000001ef00,ffff80000001b460,0,ffffffff81
b0a768,0) at x86_ipi_handler+0x85
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1c
--- interrupt ---
acpicpu_idle(ffff80002a12f1c0,ffff80000003d000,ffff80000003d010,0,f6548594e2a22
5ee,286) at acpicpu_idle+0xec
cpu_idle_cycle(ffff80000003d000,8,0,ffff80000003d000,ffffffff81477200,0)
at cpu
_idle_cycle+0x10
end trace frame: 0x0, count: -5
ddb{1}>
ddb{1}> mach ddbcpu 2
Stopped at db_enter+0x9: leave
db_enter(ffff80000003f000,ffffffff81b0a768,ffff80000003f000,10,ffff80002a166d28
,286) at db_enter+0x9
x86_ipi_handler(ffff80000003f000,ffff80000001e000,ffff80000001b4a0,0,1c0,0)
at x
86_ipi_handler+0x85
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1c
--- interrupt ---
acpicpu_idle(ffff80002a12f3f8,ffff80000003f000,ffff80000003f010,0,f6548594e2a22
5ee,286) at acpicpu_idle+0xec
cpu_idle_cycle(ffff80000003f000,8,0,ffff80000003f000,ffffffff81477200,0)
at cpu
_idle_cycle+0x10
end trace frame: 0x0, count: 10
ddb{2}> trace
db_enter(ffff80000003f000,ffffffff81b0a768,ffff80000003f000,10,ffff80002a166d28
,286) at db_enter+0x9
x86_ipi_handler(ffff80000003f000,ffff80000001e000,ffff80000001b4a0,0,1c0,0)
at x
86_ipi_handler+0x85
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1c
--- interrupt ---
acpicpu_idle(ffff80002a12f3f8,ffff80000003f000,ffff80000003f010,0,f6548594e2a22
5ee,286) at acpicpu_idle+0xec
cpu_idle_cycle(ffff80000003f000,8,0,ffff80000003f000,ffffffff81477200,0)
at cpu
_idle_cycle+0x10
end trace frame: 0x0, count: -5
ddb{2}>
ddb{2}> mach ddbcpu 4
Stopped at db_enter+0x9: leave
db_enter(ffff800000041000,ffffffff81b0a768,ffff800000041000,10,ffff80002a170d28
,286) at db_enter+0x9
x86_ipi_handler(ffff800000041000,ffff80000001e600,ffff80000001b4e0,0,ffffffff81
b0a768,0) at x86_ipi_handler+0x85
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1c
--- interrupt ---
acpicpu_idle(ffff80002a12f868,ffff800000041000,ffff800000041010,0,f6548594e2a22
5ee,286) at acpicpu_idle+0xec
cpu_idle_cycle(ffff800000041000,8,0,ffff800000041000,ffffffff81477200,0)
at cpu
_idle_cycle+0x10
end trace frame: 0x0, count: 10
ddb{4}> trace
db_enter(ffff800000041000,ffffffff81b0a768,ffff800000041000,10,ffff80002a170d28
,286) at db_enter+0x9
x86_ipi_handler(ffff800000041000,ffff80000001e600,ffff80000001b4e0,0,ffffffff81
b0a768,0) at x86_ipi_handler+0x85
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1c
--- interrupt ---
acpicpu_idle(ffff80002a12f868,ffff800000041000,ffff800000041010,0,f6548594e2a22
5ee,286) at acpicpu_idle+0xec
cpu_idle_cycle(ffff800000041000,8,0,ffff800000041000,ffffffff81477200,0)
at cpu
_idle_cycle+0x10
end trace frame: 0x0, count: -5
ddb{4}>
ddb{4}> mach ddbcpu 5
Stopped at db_enter+0x9: leave
db_enter(ffff800000042000,ffffffff81b0a768,ffff800000042000,10,ffff80002a177d28
,286) at db_enter+0x9
x86_ipi_handler(ffff800000042000,ffff80000001e700,ffff80000001b500,0,ffffffff81
b0a768,0) at x86_ipi_handler+0x85
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1c
--- interrupt ---
acpicpu_idle(ffff80002a12faa0,ffff800000042000,ffff800000042010,0,f6548594e2a22
5ee,286) at acpicpu_idle+0xec
cpu_idle_cycle(ffff800000042000,8,0,ffff800000042000,ffffffff81477200,0)
at cpu
_idle_cycle+0x10
end trace frame: 0x0, count: 10
ddb{5}> trace
db_enter(ffff800000042000,ffffffff81b0a768,ffff800000042000,10,ffff80002a177d28
,286) at db_enter+0x9
x86_ipi_handler(ffff800000042000,ffff80000001e700,ffff80000001b500,0,ffffffff81
b0a768,0) at x86_ipi_handler+0x85
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1c
--- interrupt ---
acpicpu_idle(ffff80002a12faa0,ffff800000042000,ffff800000042010,0,f6548594e2a22
5ee,286) at acpicpu_idle+0xec
cpu_idle_cycle(ffff800000042000,8,0,ffff800000042000,ffffffff81477200,0)
at cpu
_idle_cycle+0x10
end trace frame: 0x0, count: -5
ddb{5}>
ddb{5}> mach ddbcpu 6
Stopped at db_enter+0x9: leave
db_enter(ffff800000044000,ffffffff81b0a768,ffff800000044000,10,ffff80002a17cd28
,286) at db_enter+0x9
x86_ipi_handler(ffff800000044000,ffff80000001e800,ffff80000001b520,0,ffffffff81
b0a768,0) at x86_ipi_handler+0x85
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1c
--- interrupt ---
acpicpu_idle(ffff80002a12fcd8,ffff800000044000,ffff800000044010,0,f6548594e2a22
5ee,286) at acpicpu_idle+0xec
cpu_idle_cycle(ffff800000044000,8,0,ffff800000044000,ffffffff81477200,0)
at cpu
_idle_cycle+0x10
end trace frame: 0x0, count: 10
ddb{6}> trace
db_enter(ffff800000044000,ffffffff81b0a768,ffff800000044000,10,ffff80002a17cd28
,286) at db_enter+0x9
x86_ipi_handler(ffff800000044000,ffff80000001e800,ffff80000001b520,0,ffffffff81
b0a768,0) at x86_ipi_handler+0x85
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1c
--- interrupt ---
acpicpu_idle(ffff80002a12fcd8,ffff800000044000,ffff800000044010,0,f6548594e2a22
5ee,286) at acpicpu_idle+0xec
cpu_idle_cycle(ffff800000044000,8,0,ffff800000044000,ffffffff81477200,0)
at cpu
_idle_cycle+0x10
end trace frame: 0x0, count: -5
ddb{6}>
ddb{6}> mach ddbcpu 7
Stopped at db_enter+0x9: leave
db_enter(ffff800000045000,ffffffff81b0a768,ffff800000045000,10,ffff80002a181d28
,286) at db_enter+0x9
x86_ipi_handler(ffff800000045000,ffff80000001e900,ffff80000001b540,0,ffffffff81
b0a768,0) at x86_ipi_handler+0x85
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1c
--- interrupt ---
acpicpu_idle(ffff80002a182008,ffff800000045000,ffff800000045010,0,f6548594e2a22
5ee,286) at acpicpu_idle+0xec
cpu_idle_cycle(ffff800000045000,8,0,ffff800000045000,ffffffff81477200,0)
at cpu
_idle_cycle+0x10
end trace frame: 0x0, count: 10
ddb{7}> trace
db_enter(ffff800000045000,ffffffff81b0a768,ffff800000045000,10,ffff80002a181d28
,286) at db_enter+0x9
x86_ipi_handler(ffff800000045000,ffff80000001e900,ffff80000001b540,0,ffffffff81
b0a768,0) at x86_ipi_handler+0x85
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1c
--- interrupt ---
acpicpu_idle(ffff80002a182008,ffff800000045000,ffff800000045010,0,f6548594e2a22
5ee,286) at acpicpu_idle+0xec
cpu_idle_cycle(ffff800000045000,8,0,ffff800000045000,ffffffff81477200,0)
at cpu
_idle_cycle+0x10
end trace frame: 0x0, count: -5
ddb{7}>
ddb{7}> dmesg
OpenBSD 6.1-current (GENERIC.MP) #10: Tue Jul 4 01:29:35 CEST 2017
***@fw02bcbn.srce.hr:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 12854988800 (12259MB)
avail mem = 12359168000 (11786MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.6 @ 0xcf49c000 (84 entries)
bios0: vendor Dell Inc. version "6.4.0" date 07/23/2013
bios0: Dell Inc. PowerEdge R610
acpi0 at bios0: rev 2
acpi0: sleep states S0 S4 S5
acpi0: tables DSDT FACP APIC SPCR HPET DM__ MCFG WD__ SLIC ERST HEST
BERT EINJ S
RAT TCPA
acpi0: wakeup devices PCI0(S5)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 32 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E5630 @ 2.53GHz, 2527.31 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,C
FLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,V
MX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,PAGE1
GB,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: TSC frequency 2527312550 Hz
cpu0: smt 0, core 0, package 1
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 132MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.1, IBE
cpu1 at mainbus0: apid 0 (application processor)
cpu1: Intel(R) Xeon(R) CPU E5630 @ 2.53GHz, 2527.00 MHz
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,C
FLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,V
MX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,PAGE1
GB,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 0, core 0, package 0
cpu2 at mainbus0: apid 34 (application processor)
[138/4944]
cpu2: Intel(R) Xeon(R) CPU E5630 @ 2.53GHz, 2527.00 MHz
cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,C
FLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,V
MX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,PAGE1
GB,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
cpu2: 256KB 64b/line 8-way L2 cache
cpu2: smt 0, core 1, package 1
cpu3 at mainbus0: apid 2 (application processor)
cpu3: Intel(R) Xeon(R) CPU E5630 @ 2.53GHz, 2527.00 MHz
cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,C
FLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,V
MX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,PAGE1
GB,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
cpu3: 256KB 64b/line 8-way L2 cache
cpu3: smt 0, core 1, package 0
cpu4 at mainbus0: apid 50 (application processor)
cpu4: Intel(R) Xeon(R) CPU E5630 @ 2.53GHz, 2527.00 MHz
cpu4:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,C
FLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,V
MX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,PAGE1
GB,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
cpu4: 256KB 64b/line 8-way L2 cache
cpu4: smt 0, core 9, package 1
cpu5 at mainbus0: apid 18 (application processor)
cpu5: Intel(R) Xeon(R) CPU E5630 @ 2.53GHz, 2527.00 MHz
cpu5:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,C
FLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,V
MX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,PAGE1
GB,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
cpu5: 256KB 64b/line 8-way L2 cache
cpu5: smt 0, core 9, package 0
cpu6 at mainbus0: apid 52 (application processor)
cpu6: Intel(R) Xeon(R) CPU E5630 @ 2.53GHz, 2527.00 MHz
cpu6:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,C
FLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,V
MX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,PAGE1
GB,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
cpu6: 256KB 64b/line 8-way L2 cache
cpu6: smt 0, core 10, package 1
cpu7 at mainbus0: apid 20 (application processor)
cpu7: Intel(R) Xeon(R) CPU E5630 @ 2.53GHz, 2526.99 MHz
cpu7:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,C
FLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,V
MX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,PAGE1
GB,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
cpu7: 256KB 64b/line 8-way L2 cache
cpu7: smt 0, core 10, package 0
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 20, 24 pins
[90/4944]
ioapic1 at mainbus0: apid 1 pa 0xfec80000, version 20, 24 pins
acpihpet0 at acpi0: 14318179 Hz
acpimcfg0 at acpi0 addr 0xe0000000, bus 0-255
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 1 (PEX1)
acpiprt2 at acpi0: bus 2 (PEX3)
acpiprt3 at acpi0: bus -1 (PEX4)
acpiprt4 at acpi0: bus -1 (PEX5)
acpiprt5 at acpi0: bus -1 (PEX6)
acpiprt6 at acpi0: bus 4 (PEX7)
acpiprt7 at acpi0: bus 5 (PEX9)
acpiprt8 at acpi0: bus -1 (PEXA)
acpiprt9 at acpi0: bus 3 (SBEX)
acpiprt10 at acpi0: bus 6 (COMP)
acpicpu0 at acpi0: C1(@1 halt!)
acpicpu1 at acpi0: C1(@1 halt!)
acpicpu2 at acpi0: C1(@1 halt!)
acpicpu3 at acpi0: C1(@1 halt!)
acpicpu4 at acpi0: C1(@1 halt!)
acpicpu5 at acpi0: C1(@1 halt!)
acpicpu6 at acpi0: C1(@1 halt!)
acpicpu7 at acpi0: C1(@1 halt!)
"PNP0C33" at acpi0 not configured
"ACPI000D" at acpi0 not configured
"PNP0501" at acpi0 not configured
"PNP0501" at acpi0 not configured
"IPI0001" at acpi0 not configured
"PNP0C14" at acpi0 not configured
ipmi at mainbus0 not configured
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 5500 Host" rev 0x13
ppb0 at pci0 dev 1 function 0 "Intel X58 PCIE" rev 0x13
pci1 at ppb0 bus 1
bnx0 at pci1 dev 0 function 0 "Broadcom BCM5709" rev 0x20: apic 1 int 4
bnx1 at pci1 dev 0 function 1 "Broadcom BCM5709" rev 0x20: apic 1 int 16
ppb1 at pci0 dev 3 function 0 "Intel X58 PCIE" rev 0x13
pci2 at ppb1 bus 2
bnx2 at pci2 dev 0 function 0 "Broadcom BCM5709" rev 0x20: apic 1 int 0
bnx3 at pci2 dev 0 function 1 "Broadcom BCM5709" rev 0x20: apic 1 int 10
ppb2 at pci0 dev 7 function 0 "Intel X58 PCIE" rev 0x13: msi
pci3 at ppb2 bus 4
ppb3 at pci0 dev 9 function 0 "Intel X58 PCIE" rev 0x13: msi
pci4 at ppb3 bus 5
"Intel X58 Misc" rev 0x13 at pci0 dev 20 function 0 not configured
"Intel X58 GPIO" rev 0x13 at pci0 dev 20 function 1 not configured
"Intel X58 RAS" rev 0x13 at pci0 dev 20 function 2 not configured
uhci0 at pci0 dev 26 function 0 "Intel 82801I USB" rev 0x02: apic 0 int 17
uhci1 at pci0 dev 26 function 1 "Intel 82801I USB" rev 0x02: apic 0 int 18
ehci0 at pci0 dev 26 function 7 "Intel 82801I USB" rev 0x02: apic 0 int 19
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev
2.00/1.00 a
ddr 1
ppb4 at pci0 dev 28 function 0 "Intel 82801I PCIE" rev 0x02
pci5 at ppb4 bus 3
[36/4944]
mfi0 at pci5 dev 0 function 0 "Symbios Logic SAS1078" rev 0x04: apic 0
int 16
mfi0: "PERC 6/i Integrated", firmware 6.3.3.0002, 256MB cache
scsibus1 at mfi0: 64 targets
sd0 at scsibus1 targ 0 lun 0: <DELL, PERC 6/i, 1.22> SCSI3 0/direct
fixed naa.6
a4badb0331002001357370807b4917a
sd0: 139392MB, 512 bytes/sector, 285474816 sectors
uhci2 at pci0 dev 29 function 0 "Intel 82801I USB" rev 0x02: apic 0 int 21
uhci3 at pci0 dev 29 function 1 "Intel 82801I USB" rev 0x02: apic 0 int 20
ehci1 at pci0 dev 29 function 7 "Intel 82801I USB" rev 0x02: apic 0 int 21
usb1 at ehci1: USB revision 2.0
uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev
2.00/1.00 a
ddr 1
ppb5 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x92
pci6 at ppb5 bus 6
vga1 at pci6 dev 3 function 0 "Matrox MGA G200eW" rev 0x0a
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
pcib0 at pci0 dev 31 function 0 "Intel 82801IB LPC" rev 0x02
pciide0 at pci0 dev 31 function 2 "Intel 82801I SATA" rev 0x02: DMA,
channel 0 c
onfigured to native-PCI, channel 1 configured to native-PCI
pciide0: using apic 0 int 23 for native-PCI interrupt
atapiscsi0 at pciide0 channel 0 drive 0
scsibus2 at atapiscsi0: 2 targets
cd0 at scsibus2 targ 0 lun 0: <TEAC, DVD-ROM DV-28SW, R.2A> ATAPI
5/cdrom remov
able
cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 5
usb2 at uhci0: USB revision 1.0
uhub2 at usb2 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 a
ddr 1
usb3 at uhci1: USB revision 1.0
uhub3 at usb3 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 a
ddr 1
usb4 at uhci2: USB revision 1.0
uhub4 at usb4 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 a
ddr 1
usb5 at uhci3: USB revision 1.0
uhub5 at usb5 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 a
ddr 1
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
uhub6 at uhub0 port 3 configuration 1 interface 0 "Standard Microsystems
produc
t 0x2514" rev 2.00/0.00 addr 2
uhidev0 at uhub4 port 2 configuration 1 interface 0 "Avocent USB
Composite Devi
ce-0" rev 1.10/0.00 addr 2
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 variable keys, 6 key codes
wskbd1 at ukbd0 mux 1
wskbd1: connecting to wsdisplay0
uhidev1 at uhub4 port 2 configuration 1 interface 1 "Avocent USB
Composite Devi
ce-0" rev 1.10/0.00 addr 2
uhidev1: iclass 3/1
ums0 at uhidev1: 3 buttons, Z dir
wsmouse0 at ums0 mux 0
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (bd0f9f53c67ec5d2.a) swap on sd0b dump on sd0b
bnx0: address 00:22:19:6b:e0:8c
brgphy0 at bnx0 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
bnx1: address 00:22:19:6b:e0:8e
brgphy1 at bnx1 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
bnx2: address 00:22:19:6b:e0:90
brgphy2 at bnx2 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
bnx3: address 00:22:19:6b:e0:92
brgphy3 at bnx3 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
<2>carp1: state transition: BACKUP -> MASTER
<2>carp2: state transition: BACKUP -> MASTER