Discussion:
Suspend-to-full-encrypted-disk is broken (by KARL?)
Natasha Kerensikova
2017-07-18 14:55:02 UTC
Permalink
Synopsis: Suspend-to-full-encrypted-disk is broken (by KARL?)
Category: <PR category (one line)>
System : OpenBSD 6.1
Details : OpenBSD 6.1-current (GENERIC.MP) #94: Mon Jul 10 18:20:35 MDT 2017
***@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP

Architecture: OpenBSD.amd64
Machine : amd64
On my Thinkpad X220 (with Core i5) with full disk encryption,
OpenBSD doesn't resume after suspend to disk since my latest snanpshot
update (July 11th snapshot).
However, when forcing the boot on /bsd.booted, resume works
as expected.
Suspend-to-disk a live OpenBSD will full disk encryption. On next
boot, it should resume from disk, but instead it starts a standard
boot with dirty filesystems.
Probably something in the bootloader?

As far as I understand it, the new KARL feature has to be worked
around when resuming from hibernation, because in this case the
exact same kernel has to be used. To that end, the bootloader
has to determine whether this is a real cold boot (and use /bsd)
or a resume from hibernation (and usr /bsd.booted), but it
doesn't seem to work on my system, maybe because of FDE.

It was suspected that the determination might come too early in
the process, before the passphrase is available, but we couldn't
work into the code deep enough to be certain one way or the
other.

dmesg:
OpenBSD 6.1-current (GENERIC.MP) #94: Mon Jul 10 18:20:35 MDT 2017
***@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 4156157952 (3963MB)
avail mem = 4024406016 (3837MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.6 @ 0xdae9c000 (65 entries)
bios0: vendor LENOVO version "8DET58WW (1.28 )" date 02/14/2012
bios0: LENOVO 4287CTO
acpi0 at bios0: rev 2
acpi0: sleep states S0 S3 S4 S5
acpi0: tables DSDT FACP SLIC SSDT SSDT SSDT HPET APIC MCFG ECDT ASF! TCPA SSDT SSDT UEFI UEFI UEFI
acpi0: wakeup devices LID_(S3) SLPB(S3) IGBE(S4) EXP4(S4) EXP7(S4) EHC1(S3) EHC2(S3) HDEF(S4)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpihpet0 at acpi0: 14318179 Hz
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Core(TM) i5-2520M CPU @ 2.50GHz, 2492.25 MHz
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: TSC frequency 2492246240 Hz
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 99MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.1.2, IBE
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Intel(R) Core(TM) i5-2520M CPU @ 2.50GHz, 2491.90 MHz
cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 1, core 0, package 0
cpu2 at mainbus0: apid 2 (application processor)
cpu2: Intel(R) Core(TM) i5-2520M CPU @ 2.50GHz, 2491.90 MHz
cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
cpu2: 256KB 64b/line 8-way L2 cache
cpu2: smt 0, core 1, package 0
cpu3 at mainbus0: apid 3 (application processor)
cpu3: Intel(R) Core(TM) i5-2520M CPU @ 2.50GHz, 2491.90 MHz
cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
cpu3: 256KB 64b/line 8-way L2 cache
cpu3: smt 1, core 1, package 0
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins
acpimcfg0 at acpi0 addr 0xf8000000, bus 0-63
acpiec0 at acpi0
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus -1 (PEG_)
acpiprt2 at acpi0: bus 2 (EXP1)
acpiprt3 at acpi0: bus 3 (EXP2)
acpiprt4 at acpi0: bus 5 (EXP4)
acpiprt5 at acpi0: bus 13 (EXP5)
acpiprt6 at acpi0: bus -1 (EXP7)
acpicpu0 at acpi0: C3(***@104 ***@0x415), C1(***@1 halt), PSS
acpicpu1 at acpi0: C3(***@104 ***@0x415), C1(***@1 halt), PSS
acpicpu2 at acpi0: C3(***@104 ***@0x415), C1(***@1 halt), PSS
acpicpu3 at acpi0: C3(***@104 ***@0x415), C1(***@1 halt), PSS
acpipwrres0 at acpi0: PUBS, resource for EHC1, EHC2
acpitz0 at acpi0: critical temperature is 99 degC
acpibtn0 at acpi0: LID_
acpibtn1 at acpi0: SLPB
"PNP0303" at acpi0 not configured
"LEN0020" at acpi0 not configured
"SMO1200" at acpi0 not configured
acpibat0 at acpi0: BAT0 model "42T4861" serial 12194 type LION oem "SANYO"
acpiac0 at acpi0: AC unit online
acpithinkpad0 at acpi0
"PNP0C14" at acpi0 not configured
"PNP0C14" at acpi0 not configured
acpidock0 at acpi0: GDCK not docked (0)
acpivideo0 at acpi0: VID_
acpivout at acpivideo0 not configured
acpivideo1 at acpi0: VID_
cpu0: Enhanced SpeedStep 2492 MHz: speeds: 2501, 2500, 2200, 2000, 1800, 1600, 1400, 1200, 1000, 800 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel Core 2G Host" rev 0x09
inteldrm0 at pci0 dev 2 function 0 "Intel HD Graphics 3000" rev 0x09
drm0 at inteldrm0
inteldrm0: msi
inteldrm0: 1366x768, 32bpp
wsdisplay0 at inteldrm0 mux 1: console (std, vt100 emulation)
wsdisplay0: screen 1-5 added (std, vt100 emulation)
"Intel 6 Series MEI" rev 0x04 at pci0 dev 22 function 0 not configured
em0 at pci0 dev 25 function 0 "Intel 82579LM" rev 0x04: msi, address f0:de:f1:f3:16:ab
ehci0 at pci0 dev 26 function 0 "Intel 6 Series USB" rev 0x04: apic 2 int 16
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
azalia0 at pci0 dev 27 function 0 "Intel 6 Series HD Audio" rev 0x04: msi
azalia0: codecs: Conexant CX20590, Intel/0x2805, using Conexant CX20590
audio0 at azalia0
ppb0 at pci0 dev 28 function 0 "Intel 6 Series PCIE" rev 0xb4: msi
pci1 at ppb0 bus 2
ppb1 at pci0 dev 28 function 1 "Intel 6 Series PCIE" rev 0xb4: msi
pci2 at ppb1 bus 3
iwn0 at pci2 dev 0 function 0 "Intel WiFi Link 1000" rev 0x00: msi, MIMO 1T2R, BGS, address 74:e5:0b:f3:73:8a
ppb2 at pci0 dev 28 function 3 "Intel 6 Series PCIE" rev 0xb4: msi
pci3 at ppb2 bus 5
ppb3 at pci0 dev 28 function 4 "Intel 6 Series PCIE" rev 0xb4: msi
pci4 at ppb3 bus 13
sdhc0 at pci4 dev 0 function 0 "Ricoh 5U823 SD/MMC" rev 0x04: apic 2 int 16
sdhc0: SDHC 3.0, 50 MHz base clock
sdmmc0 at sdhc0: 4-bit, sd high-speed, mmc high-speed, dma
ehci1 at pci0 dev 29 function 0 "Intel 6 Series USB" rev 0x04: apic 2 int 23
usb1 at ehci1: USB revision 2.0
uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
pcib0 at pci0 dev 31 function 0 "Intel QM67 LPC" rev 0x04
ahci0 at pci0 dev 31 function 2 "Intel 6 Series AHCI" rev 0x04: msi, AHCI 1.3
ahci0: port 0: 6.0Gb/s
scsibus1 at ahci0: 32 targets
sd0 at scsibus1 targ 0 lun 0: <ATA, ST1000LM035-1RK1, SDM1> SCSI3 0/direct fixed naa.5000c5009cd96696
sd0: 953869MB, 512 bytes/sector, 1953525168 sectors
ichiic0 at pci0 dev 31 function 3 "Intel 6 Series SMBus" rev 0x04: apic 2 int 18
iic0 at ichiic0
spdmem0 at iic0 addr 0x50: 4GB DDR3 SDRAM PC3-10600 SO-DIMM
isa0 at pcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
aps0 at isa0 port 0x1600/31
uhub2 at uhub0 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev 2.00/0.00 addr 2
uhub3 at uhub1 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev 2.00/0.00 addr 2
umodem0 at uhub3 port 4 configuration 1 interface 1 "Lenovo F5521gw" rev 2.00/0.00 addr 3
umodem0: data interface 2, has CM over data, has break
umodem0: status change notification available
ucom0 at umodem0
umodem1 at uhub3 port 4 configuration 1 interface 3 "Lenovo F5521gw" rev 2.00/0.00 addr 3
umodem1: data interface 4, has CM over data, has break
umodem1: status change notification available
ucom1 at umodem1
umodem2 at uhub3 port 4 configuration 1 interface 9 "Lenovo F5521gw" rev 2.00/0.00 addr 3
umodem2: data interface 10, has CM over data, has break
umodem2: status change notification available
ucom2 at umodem2
ugen0 at uhub3 port 4 configuration 1 "Lenovo F5521gw" rev 2.00/0.00 addr 3
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
softraid0: sd1 was not shutdown properly
sd1 at scsibus3 targ 1 lun 0: <OPENBSD, SR CRYPTO, 006> SCSI2 0/direct fixed
sd1: 953866MB, 512 bytes/sector, 1953519473 sectors
root on sd1a (63848a4fade4a944.a) swap on sd1b dump on sd1b
WARNING: / was not properly unmounted
uhub2 detached
uhub0 detached
ucom0 detached
umodem0 detached
ucom1 detached
umodem1 detached
ucom2 detached
umodem2 detached
ugen0 detached
uhub3 detached
uhub1 detached
uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
uhub2 at uhub0 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev 2.00/0.00 addr 2
uhub3 at uhub1 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev 2.00/0.00 addr 2
umodem0 at uhub3 port 4 configuration 1 interface 1 "Lenovo F5521gw" rev 2.00/0.00 addr 3
umodem0: data interface 2, has CM over data, has break
umodem0: status change notification available
ucom0 at umodem0
umodem1 at uhub3 port 4 configuration 1 interface 3 "Lenovo F5521gw" rev 2.00/0.00 addr 3
umodem1: data interface 4, has CM over data, has break
umodem1: status change notification available
ucom1 at umodem1
umodem2 at uhub3 port 4 configuration 1 interface 9 "Lenovo F5521gw" rev 2.00/0.00 addr 3
umodem2: data interface 10, has CM over data, has break
umodem2: status change notification available
ucom2 at umodem2
ugen0 at uhub3 port 4 configuration 1 "Lenovo F5521gw" rev 2.00/0.00 addr 3
Theo de Raadt
2017-07-18 15:14:04 UTC
Permalink
Probably something in the bootloader?

Yes, exactly. The bootloader handles the selection of /bsd.booted

Please create a transcript of your default boot. You should see a
'%' character next to your boot disk. If not, update your bootblocks.

I've heard a few reports of failure, but people haven't been exact about
tracking down where the failures comes from.


CVSROOT: /cvs
Module name: src
Changes by: ***@cvs.openbsd.org 2017/06/21 19:26:28

Modified files:
sys/arch/i386/stand/libsa: diskprobe.c
sys/arch/amd64/stand/libsa: diskprobe.c

Log message:
Only print the magic '&' letter on a disk if the hibernate partition is
valid. This is an undocumented little debugging diagnostic, much like
the '*'.
Natasha Kerensikova
2017-07-18 19:01:55 UTC
Permalink
Hello,
Post by Theo de Raadt
Please create a transcript of your default boot.
Here it is, handcopied but hopefully without typo. I mistyped the
passphrase on the first attempt, because muscle memory was impaired by
the camera, but I'm showing the text as it appears without tampering. If
you need a transcript with the correct passphrase on the first try, I
can try to record another one.


Using drive 0, partition 3.
Loading......
probing: pc0 mem[630K 511M 510M 2471M 486M a20=on]
disk: hd0+ sr0*
Post by Theo de Raadt
OpenBSD/amd64 BOOT 3.33
Passphrase:
incorrect passphrase or keydisk
open(sr0a:/etc/boot.conf): Operation not permitted
boot>
Passphrase:
booting sr0a:/bsd: 7782584+2319368+264992+0+675840 [72+751104+512939]=0bbec78
entry point at 0x1000158
( using 1264760 bytes of bsd ELF symbol table )
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
Copyright (c) 1995-2017 OpenBSD. All rights reserved. https://www.OpenBSD.org

OpenBSD 6.1-current (GENERIC.MP) #94: Mon Jul 10 18:20:35 MDT 2017
***@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP

and I believe the rest is accounted for in the dmesg.
Post by Theo de Raadt
You should see a
'%' character next to your boot disk. If not, update your bootblocks.
I updated the whole system by downloading the snapshot bsd.rd,
checking the signature, booting on it, and selecting (U)pgrade. I have
been told it does update the bootblocks.

I ran `installboot -v sd1` anyway, and got exactly the same behavior
with the same transcript as above.



Sorry for having made you yell,
Natasha
Sebastien Marie
2017-07-18 17:01:19 UTC
Permalink
Hi,

(First sorry for the thread break, I have some problems in receiving
mails actually.)

I dig a bit inside the code of bootloader, and I think it is possible
that the bootloader is unable to see hibernate signature when FDE is in
use.

Taking boot(bootdev) as entrypoint (stand/boot/boot.c +60):

it calls machdep() (arch/amd64/stand/libsa/machdep.c +48)
and call each probe routines.

In the list of probe routines, there is diskprobe()
(arch/amd64/stand/libsa/diskprobe.c +246), which call:

- hardprobe() (line 122),
- and only after sr_probe() (line 270)

It is hardprobe() that will call check_hibernate():

- allocating dip struct
- running check_hibernate() to set BDI_HIBVALID if HIBERNATE_MAGIC is
found.

To read the value, it uses dip->strategy() function.

arch/amd64/stand/libsa/diskprobe.c
491 error = dip->strategy(dip, F_READ, (daddr32_t)sec, sizeof hib, &hib, NULL);
492 if (error == 0 && hib.magic == HIBERNATE_MAGIC) {
493 /* Hibernate present */
494 dip->bios_info.flags |= BDI_HIBVALID;
495 printf("&");
496 }

the actual code is biosstrategy() (arch/amd64/stand/libsa/biosdev.c +722)
which will call sr_strategy() if dip->sr_vol is not NULL.

arch/amd64/stand/libsa/biosdev.c
721 /* Intercept strategy for softraid volumes. */
722 if (dip->sr_vol)
723 return sr_strategy(dip->sr_vol, rw, blk, size, buf, rsize);


If I correctly understood the code, dip->sr_vol is initialized on first
biosopen() call. But at this point, we didn't call it. Additionally,
sr_probe() isn't called either.

So dip->strategy() will read the encrypted data and not decrypted one,
so it is unable to found HIBERNATE_MAGIC (as it is encrypted).

Thanks.
--
Sebastien Marie
Theo de Raadt
2017-07-18 17:10:00 UTC
Permalink
Possibly.

cc'ing yasuoka who helped me with this method of disk probing.

But anyways, beyond a code review we need someone to SHOW THEIR BOOT LOADER
OUTPUT, so that we can see the hibernate marker letter is missing.
Post by Sebastien Marie
(First sorry for the thread break, I have some problems in receiving
mails actually.)
I dig a bit inside the code of bootloader, and I think it is possible
that the bootloader is unable to see hibernate signature when FDE is in
use.
it calls machdep() (arch/amd64/stand/libsa/machdep.c +48)
and call each probe routines.
In the list of probe routines, there is diskprobe()
- hardprobe() (line 122),
- and only after sr_probe() (line 270)
- allocating dip struct
- running check_hibernate() to set BDI_HIBVALID if HIBERNATE_MAGIC is
found.
To read the value, it uses dip->strategy() function.
arch/amd64/stand/libsa/diskprobe.c
491 error = dip->strategy(dip, F_READ, (daddr32_t)sec, sizeof hib, &hib, NULL);
492 if (error == 0 && hib.magic == HIBERNATE_MAGIC) {
493 /* Hibernate present */
494 dip->bios_info.flags |= BDI_HIBVALID;
495 printf("&");
496 }
the actual code is biosstrategy() (arch/amd64/stand/libsa/biosdev.c +722)
which will call sr_strategy() if dip->sr_vol is not NULL.
arch/amd64/stand/libsa/biosdev.c
721 /* Intercept strategy for softraid volumes. */
722 if (dip->sr_vol)
723 return sr_strategy(dip->sr_vol, rw, blk, size, buf, rsize);
If I correctly understood the code, dip->sr_vol is initialized on first
biosopen() call. But at this point, we didn't call it. Additionally,
sr_probe() isn't called either.
So dip->strategy() will read the encrypted data and not decrypted one,
so it is unable to found HIBERNATE_MAGIC (as it is encrypted).
Thanks.
--
Sebastien Marie
Theo Buehler
2017-07-18 18:40:53 UTC
Permalink
Post by Theo de Raadt
Possibly.
cc'ing yasuoka who helped me with this method of disk probing.
But anyways, beyond a code review we need someone to SHOW THEIR BOOT LOADER
OUTPUT, so that we can see the hibernate marker letter is missing.
Using FDE from MBR on my T420, I can confirm Natasha's findings.

Automatic kernel selection to unhibernate fails, but manually selecting
/bsd.booted works fine.

I see the exact same thing on reboot and on boot into unhibernate:
(manually transcribed, no & sign)

Using drive 0, partition 3.
Loading......
probing: pc0 mem[630K 511M 510M 2471M 486M a20=on]
disk: hd0+ sr0*
Post by Theo de Raadt
OpenBSD/amd64 BOOT 3.33
Passphrase: _
Sebastien Marie
2017-07-18 19:15:54 UTC
Permalink
Post by Theo Buehler
Post by Theo de Raadt
Possibly.
cc'ing yasuoka who helped me with this method of disk probing.
But anyways, beyond a code review we need someone to SHOW THEIR BOOT LOADER
OUTPUT, so that we can see the hibernate marker letter is missing.
Using FDE from MBR on my T420, I can confirm Natasha's findings.
Automatic kernel selection to unhibernate fails, but manually selecting
/bsd.booted works fine.
(manually transcribed, no & sign)
Using drive 0, partition 3.
Loading......
probing: pc0 mem[630K 511M 510M 2471M 486M a20=on]
disk: hd0+ sr0*
Post by Theo de Raadt
OpenBSD/amd64 BOOT 3.33
Passphrase: _
To compare, here my output on suspended boot for a laptop (i386)
*without* FDE:

Using drive 0, partition 3.
Loading......
probing: pc0 pci mem[636K 2037M a20=on]
disk: hd0+&
Post by Theo Buehler
Post by Theo de Raadt
OpenBSD/i386 BOOT 3.31
unhibernate detected: switching to /bsd.booted
boot>
booting hd0a:/bsd.booted: 8122376+....
entry point at 0x2000d4
...

So here the & is present, and an explicit line is echoed.
--
Sebastien Marie
Theo de Raadt
2017-07-18 20:56:45 UTC
Permalink
Post by Natasha Kerensikova
Post by Theo de Raadt
Please create a transcript of your default boot.
Here it is, handcopied but hopefully without typo. I mistyped the
passphrase on the first attempt, because muscle memory was impaired by
the camera, but I'm showing the text as it appears without tampering. If
you need a transcript with the correct passphrase on the first try, I
can try to record another one.
Using drive 0, partition 3.
Loading......
probing: pc0 mem[630K 511M 510M 2471M 486M a20=on]
disk: hd0+ sr0*
Post by Theo de Raadt
OpenBSD/amd64 BOOT 3.33
incorrect passphrase or keydisk
open(sr0a:/etc/boot.conf): Operation not permitted
boot>
booting sr0a:/bsd: 7782584+2319368+264992+0+675840 [72+751104+512939]=0bbec78
So it looks like the problem has been identified, and hibernate signature
analysis will need to be deferred to a slightly later phase.

That is pretty difficult to do right.
Theo de Raadt
2017-07-18 21:48:48 UTC
Permalink
Post by Sebastien Marie
Post by Theo Buehler
Post by Theo de Raadt
Possibly.
cc'ing yasuoka who helped me with this method of disk probing.
But anyways, beyond a code review we need someone to SHOW THEIR BOOT LOADER
OUTPUT, so that we can see the hibernate marker letter is missing.
Using FDE from MBR on my T420, I can confirm Natasha's findings.
Automatic kernel selection to unhibernate fails, but manually selecting
/bsd.booted works fine.
(manually transcribed, no & sign)
Using drive 0, partition 3.
Loading......
probing: pc0 mem[630K 511M 510M 2471M 486M a20=on]
disk: hd0+ sr0*
Post by Theo de Raadt
OpenBSD/amd64 BOOT 3.33
Passphrase: _
To compare, here my output on suspended boot for a laptop (i386)
Using drive 0, partition 3.
Loading......
probing: pc0 pci mem[636K 2037M a20=on]
disk: hd0+&
Post by Theo Buehler
Post by Theo de Raadt
OpenBSD/i386 BOOT 3.31
unhibernate detected: switching to /bsd.booted
boot>
booting hd0a:/bsd.booted: 8122376+....
entry point at 0x2000d4
...
So here the & is present, and an explicit line is echoed.
I'll explain what is going on here, and what needs to probably be done.

The changes introduced into the bootloader are largely correct. They
make the right decisions. They just don't make them at the right
points in time. The IO operations should be moved further forward, to
after the key operations.

Havinge done it this way, yasuoka and I were able to create a workable
model, but the operations need to be moved forward.

Someone want to dive in?
YASUOKA Masahiko
2017-07-20 12:45:55 UTC
Permalink
This post might be inappropriate. Click to display it.
Theo Buehler
2017-07-20 18:44:53 UTC
Permalink
Post by YASUOKA Masahiko
On Tue, 18 Jul 2017 15:48:48 -0600
Post by Theo de Raadt
Post by Sebastien Marie
Post by Theo Buehler
Post by Theo de Raadt
Possibly.
cc'ing yasuoka who helped me with this method of disk probing.
But anyways, beyond a code review we need someone to SHOW THEIR BOOT LOADER
OUTPUT, so that we can see the hibernate marker letter is missing.
Using FDE from MBR on my T420, I can confirm Natasha's findings.
Automatic kernel selection to unhibernate fails, but manually selecting
/bsd.booted works fine.
(manually transcribed, no & sign)
Using drive 0, partition 3.
Loading......
probing: pc0 mem[630K 511M 510M 2471M 486M a20=on]
disk: hd0+ sr0*
Post by Theo de Raadt
OpenBSD/amd64 BOOT 3.33
Passphrase: _
To compare, here my output on suspended boot for a laptop (i386)
Using drive 0, partition 3.
Loading......
probing: pc0 pci mem[636K 2037M a20=on]
disk: hd0+&
Post by Theo Buehler
Post by Theo de Raadt
OpenBSD/i386 BOOT 3.31
unhibernate detected: switching to /bsd.booted
boot>
booting hd0a:/bsd.booted: 8122376+....
entry point at 0x2000d4
...
So here the & is present, and an explicit line is echoed.
I'll explain what is going on here, and what needs to probably be done.
The changes introduced into the bootloader are largely correct. They
make the right decisions. They just don't make them at the right
points in time. The IO operations should be moved further forward, to
after the key operations.
Havinge done it this way, yasuoka and I were able to create a workable
model, but the operations need to be moved forward.
Someone want to dive in?
The diff is to support selecting bsd.booted for FED.
disk: hd0 sr0*
Post by Theo de Raadt
Post by Sebastien Marie
OpenBSD/amd64 BOOTX64 3.33
Passphrase: <--- Enter the passphrase
unhibernate detected: switching to /bsd.booted
boot>
booting sr0a:/bsd.booted: xxxx...
we can't add "&" sign on the "disk" line. Since the bootloader gets
the passpharase after that line.
The boot codes other than BOOTX64.EFI are not tested yet.
I tried to pxeboot into bsd.rd to install a snapshot with your diff
(with FDE on MBR) and this leads to a hang after password entry.
Therefore I didn't dare to install the bootloader, as I need this
machine for work.

More precisely, it looks like check_hibernate() doesn't return on a
normally rebooted machine:

probing: pc0 pxe![2.1] mem[630K 511M 518M 2471M 486M a20=on]
disk: hd0+ sr0*
net: mac aa:bb:cc:dd:ee:ff, ip 192.168.10.11, server 192.168.10.10
Post by YASUOKA Masahiko
Post by Theo de Raadt
OpenBSD/amd64 PXEBOOT 3.28
Passphrase:
start check_hibernate()
_


and the machine hangs here.

I used the following simple modification of your diff:

Index: libsa/biosdev.c
===================================================================
RCS file: /var/cvs/src/sys/arch/amd64/stand/libsa/biosdev.c,v
retrieving revision 1.30
diff -u -p -r1.30 biosdev.c
--- libsa/biosdev.c 18 Sep 2016 15:13:10 -0000 1.30
+++ libsa/biosdev.c 20 Jul 2017 17:21:05 -0000
@@ -548,6 +548,9 @@ biosopen(struct open_file *f, ...)
if (sr_getdisklabel(bv, &dip->disklabel))
return ERDLAB;
dip->bios_info.flags &= ~BDI_BADLABEL;
+ printf("start check_hibernate()\n");
+ check_hibernate(dip);
+ printf("stop check_hibernate()\n");
}

bv->sbv_part = part + 'a';
Theo Buehler
2017-07-20 21:26:57 UTC
Permalink
Post by Theo Buehler
Post by YASUOKA Masahiko
On Tue, 18 Jul 2017 15:48:48 -0600
Post by Theo de Raadt
Post by Sebastien Marie
Post by Theo Buehler
Post by Theo de Raadt
Possibly.
cc'ing yasuoka who helped me with this method of disk probing.
But anyways, beyond a code review we need someone to SHOW THEIR BOOT LOADER
OUTPUT, so that we can see the hibernate marker letter is missing.
Using FDE from MBR on my T420, I can confirm Natasha's findings.
Automatic kernel selection to unhibernate fails, but manually selecting
/bsd.booted works fine.
(manually transcribed, no & sign)
Using drive 0, partition 3.
Loading......
probing: pc0 mem[630K 511M 510M 2471M 486M a20=on]
disk: hd0+ sr0*
Post by Theo de Raadt
OpenBSD/amd64 BOOT 3.33
Passphrase: _
To compare, here my output on suspended boot for a laptop (i386)
Using drive 0, partition 3.
Loading......
probing: pc0 pci mem[636K 2037M a20=on]
disk: hd0+&
Post by Theo Buehler
Post by Theo de Raadt
OpenBSD/i386 BOOT 3.31
unhibernate detected: switching to /bsd.booted
boot>
booting hd0a:/bsd.booted: 8122376+....
entry point at 0x2000d4
...
So here the & is present, and an explicit line is echoed.
I'll explain what is going on here, and what needs to probably be done.
The changes introduced into the bootloader are largely correct. They
make the right decisions. They just don't make them at the right
points in time. The IO operations should be moved further forward, to
after the key operations.
Havinge done it this way, yasuoka and I were able to create a workable
model, but the operations need to be moved forward.
Someone want to dive in?
The diff is to support selecting bsd.booted for FED.
disk: hd0 sr0*
Post by Theo de Raadt
Post by Sebastien Marie
OpenBSD/amd64 BOOTX64 3.33
Passphrase: <--- Enter the passphrase
unhibernate detected: switching to /bsd.booted
boot>
booting sr0a:/bsd.booted: xxxx...
we can't add "&" sign on the "disk" line. Since the bootloader gets
the passpharase after that line.
The boot codes other than BOOTX64.EFI are not tested yet.
I tried to pxeboot into bsd.rd to install a snapshot with your diff
(with FDE on MBR) and this leads to a hang after password entry.
Therefore I didn't dare to install the bootloader, as I need this
machine for work.
More precisely, it looks like check_hibernate() doesn't return on a
probing: pc0 pxe![2.1] mem[630K 511M 518M 2471M 486M a20=on]
disk: hd0+ sr0*
net: mac aa:bb:cc:dd:ee:ff, ip 192.168.10.11, server 192.168.10.10
Post by YASUOKA Masahiko
Post by Theo de Raadt
OpenBSD/amd64 PXEBOOT 3.28
start check_hibernate()
_
and the machine hangs here.
Unfortunately, regular boot is also broken the same way as PXE boot with
this diff. I made a fresh FDE on MBR install to a memory stick. After
the install I rebooted from the memory stick, boot decrypts the volume
but hangs in check_hibernate().

(with the same diff as in my previous mail):

Using drive 0, partition 3
Loading......
probing: pc0 mem[630K 511M 518M 2471M 486M a20=on]
disk hd0+ hd1+ sr0* sr1*
Post by Theo Buehler
Post by YASUOKA Masahiko
OpenBSD/amd64 BOOT 3.33
Passphrase:
start check_hibernate()
Post by Theo Buehler
Index: libsa/biosdev.c
===================================================================
RCS file: /var/cvs/src/sys/arch/amd64/stand/libsa/biosdev.c,v
retrieving revision 1.30
diff -u -p -r1.30 biosdev.c
--- libsa/biosdev.c 18 Sep 2016 15:13:10 -0000 1.30
+++ libsa/biosdev.c 20 Jul 2017 17:21:05 -0000
@@ -548,6 +548,9 @@ biosopen(struct open_file *f, ...)
if (sr_getdisklabel(bv, &dip->disklabel))
return ERDLAB;
dip->bios_info.flags &= ~BDI_BADLABEL;
+ printf("start check_hibernate()\n");
+ check_hibernate(dip);
+ printf("stop check_hibernate()\n");
}
bv->sbv_part = part + 'a';
YASUOKA Masahiko
2017-07-20 23:41:04 UTC
Permalink
Thank you for the test. Let me update the diff.

On Thu, 20 Jul 2017 23:26:57 +0200
Post by Theo Buehler
Post by Theo Buehler
I tried to pxeboot into bsd.rd to install a snapshot with your diff
(with FDE on MBR) and this leads to a hang after password entry.
Therefore I didn't dare to install the bootloader, as I need this
machine for work.
More precisely, it looks like check_hibernate() doesn't return on a
probing: pc0 pxe![2.1] mem[630K 511M 518M 2471M 486M a20=on]
disk: hd0+ sr0*
net: mac aa:bb:cc:dd:ee:ff, ip 192.168.10.11, server 192.168.10.10
Post by Theo de Raadt
OpenBSD/amd64 PXEBOOT 3.28
start check_hibernate()
_
and the machine hangs here.
Unfortunately, regular boot is also broken the same way as PXE boot with
this diff. I made a fresh FDE on MBR install to a memory stick. After
the install I rebooted from the memory stick, boot decrypts the volume
but hangs in check_hibernate().
Using drive 0, partition 3
Loading......
probing: pc0 mem[630K 511M 518M 2471M 486M a20=on]
disk hd0+ hd1+ sr0* sr1*
Post by Theo Buehler
OpenBSD/amd64 BOOT 3.33
start check_hibernate()
Index: sys/arch/amd64/stand/efiboot/efidev.c
===================================================================
RCS file: /cvs/src/sys/arch/amd64/stand/efiboot/efidev.c,v
retrieving revision 1.26
diff -u -p -r1.26 efidev.c
--- sys/arch/amd64/stand/efiboot/efidev.c 16 May 2017 02:56:23 -0000 1.26
+++ sys/arch/amd64/stand/efiboot/efidev.c 20 Jul 2017 23:37:58 -0000
@@ -625,6 +625,7 @@ efiopen(struct open_file *f, ...)
if (sr_getdisklabel(bv, &dip->disklabel))
return ERDLAB;
dip->bios_info.flags &= ~BDI_BADLABEL;
+ check_hibernate(dip);
}

bv->sbv_part = part + 'a';
Index: sys/arch/amd64/stand/libsa/biosdev.c
===================================================================
RCS file: /cvs/src/sys/arch/amd64/stand/libsa/biosdev.c,v
retrieving revision 1.30
diff -u -p -r1.30 biosdev.c
--- sys/arch/amd64/stand/libsa/biosdev.c 18 Sep 2016 15:13:10 -0000 1.30
+++ sys/arch/amd64/stand/libsa/biosdev.c 20 Jul 2017 23:37:58 -0000
@@ -535,6 +535,7 @@ biosopen(struct open_file *f, ...)
if (bv->sbv_diskinfo == NULL) {
dip = alloc(sizeof(struct diskinfo));
bzero(dip, sizeof(*dip));
+ dip->strategy = biosstrategy;
bv->sbv_diskinfo = dip;
dip->sr_vol = bv;
dip->bios_info.flags |= BDI_BADLABEL;
@@ -548,6 +549,7 @@ biosopen(struct open_file *f, ...)
if (sr_getdisklabel(bv, &dip->disklabel))
return ERDLAB;
dip->bios_info.flags &= ~BDI_BADLABEL;
+ check_hibernate(dip);
}

bv->sbv_part = part + 'a';
Index: sys/arch/amd64/stand/libsa/disk.h
===================================================================
RCS file: /cvs/src/sys/arch/amd64/stand/libsa/disk.h,v
retrieving revision 1.5
diff -u -p -r1.5 disk.h
--- sys/arch/amd64/stand/libsa/disk.h 2 Sep 2015 01:52:26 -0000 1.5
+++ sys/arch/amd64/stand/libsa/disk.h 20 Jul 2017 23:37:58 -0000
@@ -59,4 +59,6 @@ extern struct disklist_lh disklist;

void dump_diskinfo(void);

+void check_hibernate(struct diskinfo *);
+
#endif /* _DISK_H */
Index: sys/arch/amd64/stand/libsa/diskprobe.c
===================================================================
RCS file: /cvs/src/sys/arch/amd64/stand/libsa/diskprobe.c,v
retrieving revision 1.19
diff -u -p -r1.19 diskprobe.c
--- sys/arch/amd64/stand/libsa/diskprobe.c 22 Jun 2017 01:26:28 -0000 1.19
+++ sys/arch/amd64/stand/libsa/diskprobe.c 20 Jul 2017 23:37:58 -0000
@@ -55,7 +55,6 @@

/* Local Prototypes */
static int disksum(int);
-static void check_hibernate(struct diskinfo *);

int bootdev_has_hibernate(void); /* export for loadfile() */

@@ -176,6 +175,8 @@ hardprobe(void)
dip->bsddev = dip->bios_info.bsd_dev =
MAKEBOOTDEV(type, 0, 0, bsdunit, RAW_PART);
check_hibernate(dip);
+ if (dip->bios_info.flags & BDI_HIBVALID)
+ printf("&");

/* Add to queue of disks */
TAILQ_INSERT_TAIL(&disklist, dip, list);
@@ -232,6 +233,8 @@ efi_hardprobe(void)
dip->bsddev = dip->bios_info.bsd_dev =
MAKEBOOTDEV(type, 0, 0, bsdunit, RAW_PART);
check_hibernate(dip);
+ if (dip->bios_info.flags & BDI_HIBVALID)
+ printf("&");

/* Add to queue of disks */
TAILQ_INSERT_TAIL(&disklist, dip, list);
@@ -472,7 +475,7 @@ bootdev_has_hibernate(void)
return ((bootdev_dip->bios_info.flags & BDI_HIBVALID)? 1 : 0);
}

-static void
+void
check_hibernate(struct diskinfo *dip)
{
daddr_t sec;
@@ -489,9 +492,6 @@ check_hibernate(struct diskinfo *dip)
(sizeof(union hibernate_info) / DEV_BSIZE);

error = dip->strategy(dip, F_READ, (daddr32_t)sec, sizeof hib, &hib, NULL);
- if (error == 0 && hib.magic == HIBERNATE_MAGIC) {
- /* Hibernate present */
- dip->bios_info.flags |= BDI_HIBVALID;
- printf("&");
- }
+ if (error == 0 && hib.magic == HIBERNATE_MAGIC)
+ dip->bios_info.flags |= BDI_HIBVALID; /* Hibernate present */
}
Index: sys/arch/i386/stand/libsa/biosdev.c
===================================================================
RCS file: /cvs/src/sys/arch/i386/stand/libsa/biosdev.c,v
retrieving revision 1.95
diff -u -p -r1.95 biosdev.c
--- sys/arch/i386/stand/libsa/biosdev.c 18 Sep 2016 15:13:10 -0000 1.95
+++ sys/arch/i386/stand/libsa/biosdev.c 20 Jul 2017 23:37:58 -0000
@@ -536,6 +536,7 @@ biosopen(struct open_file *f, ...)
if (bv->sbv_diskinfo == NULL) {
dip = alloc(sizeof(struct diskinfo));
bzero(dip, sizeof(*dip));
+ dip->strategy = biosstrategy;
bv->sbv_diskinfo = dip;
dip->sr_vol = bv;
dip->bios_info.flags |= BDI_BADLABEL;
@@ -549,6 +550,7 @@ biosopen(struct open_file *f, ...)
if (sr_getdisklabel(bv, &dip->disklabel))
return ERDLAB;
dip->bios_info.flags &= ~BDI_BADLABEL;
+ check_hibernate(dip);
}

bv->sbv_part = part + 'a';
Index: sys/arch/i386/stand/libsa/disk.h
===================================================================
RCS file: /cvs/src/sys/arch/i386/stand/libsa/disk.h,v
retrieving revision 1.6
diff -u -p -r1.6 disk.h
--- sys/arch/i386/stand/libsa/disk.h 2 Sep 2015 04:09:24 -0000 1.6
+++ sys/arch/i386/stand/libsa/disk.h 20 Jul 2017 23:37:58 -0000
@@ -59,4 +59,6 @@ extern struct disklist_lh disklist;

void dump_diskinfo(void);

+void check_hibernate(struct diskinfo *);
+
#endif /* _DISK_H */
Index: sys/arch/i386/stand/libsa/diskprobe.c
===================================================================
RCS file: /cvs/src/sys/arch/i386/stand/libsa/diskprobe.c,v
retrieving revision 1.43
diff -u -p -r1.43 diskprobe.c
--- sys/arch/i386/stand/libsa/diskprobe.c 22 Jun 2017 01:26:28 -0000 1.43
+++ sys/arch/i386/stand/libsa/diskprobe.c 20 Jul 2017 23:37:58 -0000
@@ -55,7 +55,6 @@

/* Local Prototypes */
static int disksum(int);
-static void check_hibernate(struct diskinfo *);

int bootdev_has_hibernate(void); /* export for loadfile() */

@@ -176,6 +175,8 @@ hardprobe(void)
dip->bsddev = dip->bios_info.bsd_dev =
MAKEBOOTDEV(type, 0, 0, bsdunit, RAW_PART);
check_hibernate(dip);
+ if (dip->bios_info.flags & BDI_HIBVALID)
+ printf("&");

/* Add to queue of disks */
TAILQ_INSERT_TAIL(&disklist, dip, list);
@@ -232,6 +233,8 @@ efi_hardprobe(void)
dip->bsddev = dip->bios_info.bsd_dev =
MAKEBOOTDEV(type, 0, 0, bsdunit, RAW_PART);
check_hibernate(dip);
+ if (dip->bios_info.flags & BDI_HIBVALID)
+ printf("&");

/* Add to queue of disks */
TAILQ_INSERT_TAIL(&disklist, dip, list);
@@ -472,7 +475,7 @@ bootdev_has_hibernate(void)
return ((bootdev_dip->bios_info.flags & BDI_HIBVALID)? 1 : 0);
}

-static void
+void
check_hibernate(struct diskinfo *dip)
{
daddr_t sec;
@@ -489,9 +492,6 @@ check_hibernate(struct diskinfo *dip)
(sizeof(union hibernate_info) / DEV_BSIZE);

error = dip->strategy(dip, F_READ, (daddr32_t)sec, sizeof hib, &hib, NULL);
- if (error == 0 && hib.magic == HIBERNATE_MAGIC) {
- /* Hibernate present */
- dip->bios_info.flags |= BDI_HIBVALID;
- printf("&");
- }
+ if (error == 0 && hib.magic == HIBERNATE_MAGIC)
+ dip->bios_info.flags |= BDI_HIBVALID; /* Hibernate present */
}
YASUOKA Masahiko
2017-07-20 23:44:06 UTC
Permalink
FYI,

On Fri, 21 Jul 2017 08:41:04 +0900 (JST)
Post by YASUOKA Masahiko
Thank you for the test. Let me update the diff.
The previous diff lacks
Post by YASUOKA Masahiko
Index: sys/arch/i386/stand/libsa/biosdev.c
===================================================================
RCS file: /cvs/src/sys/arch/i386/stand/libsa/biosdev.c,v
retrieving revision 1.95
diff -u -p -r1.95 biosdev.c
--- sys/arch/i386/stand/libsa/biosdev.c 18 Sep 2016 15:13:10 -0000 1.95
+++ sys/arch/i386/stand/libsa/biosdev.c 20 Jul 2017 23:37:58 -0000
@@ -536,6 +536,7 @@ biosopen(struct open_file *f, ...)
if (bv->sbv_diskinfo == NULL) {
dip = alloc(sizeof(struct diskinfo));
bzero(dip, sizeof(*dip));
+ dip->strategy = biosstrategy;
bv->sbv_diskinfo = dip;
dip->sr_vol = bv;
dip->bios_info.flags |= BDI_BADLABEL;
this initialization of dip ->strategy

--yasuoka
Theo Buehler
2017-07-21 02:08:55 UTC
Permalink
Post by YASUOKA Masahiko
Thank you for the test. Let me update the diff.
I see that you already committed the diff, thanks!

It seems to work quite well. pxeboot and boot do what they should do.

I did 5 ZZZ/unhibernate cycles and observed the following:

Three times I got the passphrase right the first time around and it
selected bsd.booted, as it should.

In two attempts, I had to enter the passphrase a second time and both
times it selected sr0a:/bsd instead of bsd.booted:

Using drive 0, partition 3
Loading......
probing: pc0 mem[630K 511M 518M 2471M 486M a20=on]
disks: hd0+ sr0*
Post by YASUOKA Masahiko
OpenBSD/amd64 BOOT 3.33
Passphrase:
incorrect passphrase or keydisk
open(sr0a:/etc/boot.conf): Operation not permitted
boot> <- hit <enter>
Passphrase:
booting sr0a:/bsd: 77...etcetc

Loading...