Installing ESXi 5.5 U2 on PCEngines apu1d4

AlertI started writing this article on 10th January, 2015. But I didn’t have so much time to finish it… Now time has come, so I decided to finish this work and publish it. Many time has passed by, and many things have changed. A new ESXi version has been carried out, and new APU hardware is coming soon. But I think that this article can be a good starting point also for headless installation of new ESXi and other *nix based headless devices.

Unfortunately installation was not successfully, or better speaking installation correctly ended, but device didn’t turned up as I expected it would. I attempted many ways to overcome this problem, but I haven’t found any of them to correctly work. I posted all of this history here, in the hope that someone would find what I missed.

Introduction

In this article I’m introducing a step by step guide on how to install ESXi 5.5 U2 on PCEngines apu1d4 board.
apu1d4 is a small computer board that can be used for routers, firewalls, VOIP, dedicated servers, special purpose network plumbing, education tools… It’s equipped with an AMD G series T40E, 1 GHz dual core with 64 bit support processor, 4GB RAM, three Gigabit Ethernet ports and a serial console. You can read all of its specification here.

This setup guide it’s also suitable to install many other Linux distributions.

Prerequisites

As apu1d4 does not have a video output, setup must be performed via serial console. This is the major issue when installing ESXi, as the plain setup is intended to be used with a video card. But as it is Linux-based, we can turn on the serial console redirection to handle all of the setup process from here.

In this guide I’m assuming to install ESXi on a mSATA SSD installed on the board. Please make sure the SSD is detected by board BIOS before starting the setup process (you should see your mSATA ID into the BIOS boot messages on the serial console).

You need the following hardware:

  • an apu1d4 board already boxed (for passive cooling to work), with an internal mSATA drive installed
  • an empty USB pen drive (at least 1GB)
  • a null modem cable
  • a PC with a serial port or a USB to serial converter (you may omit the null modem cable if you use a USB to DCE converter like this one, that is intended to be used to connect directly to a console port)

and the following software:

  • ESXi 5.5 U2 install ISO (you don’t need to burn it onto a CD)
  • a utility to create a bootable USB key from an ISO. There are many utilities to do that; here I’m using Rufus
  • a serial communication software which allows you to connect to console port; here I’m using PuTTY
  • a utility to open .zip files; here I’m using 7-Zip
  • a general purpose text editor; here I’m using PSPad
  • the two file contained into this ESXi setup patch: net-r816-drivers.zip

A note on the patch: apu1d4 board integrates three Gigabit Ethernet channels based on the Realtek RTL8111E chipset. As RTL8111E support has been removed from ESXi setup since version 5.5, but it was still present in 5.1 setup, we need to extract the drivers from 5.1 installer and integrate them into the 5.5 one. This is needed as without installing and activating the drivers, you will end your setup into a “No network hardware detected” screen. How to integrate this driver in the setup is better explained later.

Step by step guide

Installation process can be resumed in the following steps:

  1. create ESXi 5.5 U2 USB boot pen drive (called pen drive in the following steps)
    1. unpack ISO to pen drive and make it bootable
    2. patch pen drive to load RTL8111 drivers
    3. patch pen drive to work with serial console
  2. install ESXi
  3. finalize setup

Create ESXi 5.5 U2 USB boot pen drive

Any tool you can find to extract an ISO image to a pen drive and make it bootable is suitable to create ESXi setup pen drive. I frufusound Rufus to be easy and clear to use. It’s also OpenSource and sources can be grabbed from it’s home page. From Rufus main window you have to select the USB drive related to the pen drive you want to use, the partitioning method (usually the default value is correct) and obviously the ESXi ISO image to store on key.

If should be warned that the menu.c32 file which comes on the ISO is obsolete and you should substitute it with a freshly downloaded version. Please accept this suggestion and  let Rufus patch this file.

You should end up in few minutes with the key ready to be patched.

Patch pen drive to load RTL8111 drivers

As said ESXi 5.5 U2 setup does not include RTL8111 drivers. If you do not patch the pen drive with those drivers, installation cannot be performed and you’ll receive a “No network card present” message. Usually drivers can be added later installing the so called VIB packages, but as the network card is considered to be part of the core setup, if no network card can be detected during setup phase, process will be stopped.

To avoid this problem a clean and correct way should be to install ESXi 5.1 version and then perform a version upgrade, but this could take long. A faster-and-quite-clean method is to include ESXi 5.1 drivers directly on 5.5 U2 setup, and that’s what we are going to do now.

When googling around this problem you’ll find several .vib files packaged into a .zip file, that should unpack to upload .vibs to your ESXi machine. Basically each .vib file is in turn a .zip package, containing the driver(s) itself and some descriptors. I simply extracted and repacked the two drivers included in the following .vib:

VMware_bootbank_net-r8168_8.013.00-3vmw.510.0.0.799733.vib
VMware_bootbank_net-r8169_6.011.00-2vmw.510.0.0.799733.vib

into a single .zip file that you’ll find attached here. All you have to do is to download the .zip containing the two drivers (net-r816.v00 and net-r816.v01) and extract them onto the pen drive, in its root directory.

After that you have to edit the boot.cfg file located on the pen drive, and add those driver to load list. To do so open your favorite editor (be carefull: you should not use NotePad because it lacks support for UNIX style line endings; you’d better use a general purpose editor like PSPad). Locate the row

modules=/b.b00 --- ...

and then add the following in your favorite position (I added them after all of the net_xxx drivers):

--- /net-r816.v00 --- /net-r816.v01

Save the file and you’re done with the patch.

Patch pen drive to work with serial console

In the last preparation step we have to patch configuration files to tell loader and setup to redirect standard output and standard input over a serial console. This step involves editing three configuration files, that must be still edited with the general purpose editor.

syslinux.cfg

Patching this file only requires you to add the following two lines AT TOP of the file:

CONSOLE 0
SERIAL 0 115200

isolinux.cfg

Patching this file requires you to add the following two lines AT TOP of the file:

CONSOLE 0
SERIAL 0 115200

Then you have to locate the line

  APPEND -c boot.cfg

located in the LABEL install section of the configuration. At the end of the line you have to add the extra parameters needed to instruct loader to redirect standard input and output. Line should look like this:

  APPEND -c boot.cfg text gdbPort=none logPort=none tty2Port=com1

boot.cfg

You have again to patch boot.cfg, as done in the step before. Modification involves the kernel option parameter passed; you have to locate the line

kernelopt=runweasel

and add the extra parameters needed to redirect standard input and output. Line should look like this:

kernelopt=runweasel text nofb com1_baud=115200 com1_Port=0x3f8 tty2Port=com1 gdbPort=none logPort=none

Install ESXi

You are now ready to start ESXi setup. Put the pen drive into the USB port, connect your PC to the console port, start PuTTY and power up the board. You should notice BIOS startup in the console port:

PC Engines APU BIOS build date: Apr  5 2014
Reading data from file [bootorder]
SeaBIOS (version ?-20140405_120742-frink)
SeaBIOS (version ?-20140405_120742-frink)
Found coreboot cbmem console @ df150400
Found mainboard PC Engines APU
Relocating init from 0x000e8e71 to 0xdf1065e0 (size 39259)
Found CBFS header at 0xfffffb90
found file "bootorder" in cbmem
CPU Mhz=1000
Found 27 PCI devices (max PCI bus is 05)
Copying PIR from 0xdf160400 to 0x000f27a0
Copying MPTABLE from 0xdf161400/df161410 to 0x000f25b0 with length 1ec
Copying ACPI RSDP from 0xdf162400 to 0x000f2590
Copying SMBIOS entry point from 0xdf16d800 to 0x000f2570
Using pmtimer, ioport 0x808
Scan for VGA option rom
EHCI init on dev 00:12.2 (regs=0xf7f08420)
Found 1 lpt ports
Found 2 serial ports
AHCI controller at 11.0, iobase f7f08000, irq 11
EHCI init on dev 00:13.2 (regs=0xf7f08520)
EHCI init on dev 00:16.2 (regs=0xf7f08620)
Searching bootorder for: /rom@img/setup
Searching bootorder for: /rom@img/memtest
Searching bootorder for: /pci@i0cf8/*@11/drive@0/disk@0
AHCI/0: registering: "AHCI/0: KINGSTON SMS200S330G ATA-8 Hard-Disk (28626 MiBytes)"
OHCI init on dev 00:12.0 (regs=0xf7f04000)
OHCI init on dev 00:13.0 (regs=0xf7f05000)
OHCI init on dev 00:14.5 (regs=0xf7f06000)
OHCI init on dev 00:16.0 (regs=0xf7f07000)
Searching bootorder for: /pci@i0cf8/usb@12,2/storage@1/*@0/*@0,0
Searching bootorder for: /pci@i0cf8/usb@12,2/usb-*@1
Searching bootorder for: /pci@i0cf8/usb@16,2/storage@1/*@0/*@0,0
Searching bootorder for: /pci@i0cf8/usb@16,2/usb-*@1
USB MSC vendor='Multiple' product='Card  Reader' rev='1.00' type=0 removable=1
Device reports MEDIUM NOT PRESENT
scsi_is_ready returned -1
Unable to configure USB MSC drive.
Unable to configure USB MSC device.
USB MSC vendor='' product='' rev='PMAP' type=0 removable=1
USB MSC blksize=512 sectors=1970176
All threads complete.
Scan for option roms
Running option rom at c000:0003


iPXE (http://ipxe.org) 00:00.0 C000 PCI2.10 PnP PMMpmm call arg1=1
pmm call arg1=0
+DF0F04A0pmm call arg1=1
pmm call arg1=0
+DF04C510 C000



Searching bootorder for: /rom@genroms/pxeboot.rom

Build date: Apr  5 2014
System memory size: 4592 MB

Press F12 for boot menu.

Searching bootorder for: HALT
drive 0x000f24d0: PCHS=0/0/0 translation=lba LCHS=977/32/63 s=1970176
drive 0x000f2500: PCHS=16383/16/63 translation=lba LCHS=1024/255/63 s=58626288
Space available for UMB: c1000-ee800, f0000-f24d0
Returned 253952 bytes of ZoneHigh
e820 map has 7 items:
  0: 0000000000000000 - 000000000009fc00 = 1 RAM
  1: 000000000009fc00 - 00000000000a0000 = 2 RESERVED
  2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
  3: 0000000000100000 - 00000000df14e000 = 1 RAM
  4: 00000000df14e000 - 00000000e0000000 = 2 RESERVED
  5: 00000000f8000000 - 00000000f9000000 = 2 RESERVED
  6: 0000000100000000 - 000000011f000000 = 1 RAM
enter handle_19:
  NULL
Booting from Hard Disk...
Booting from 0000:7c00

SYSLINUX 4.07 EDD 2013-07-25 Copyright (C) 1994-2013 H. Peter Anvin et al

SYSLINUX 4.07 2013-07-25 Copyright (C) 1994-2013 H. Peter Anvin et al

and then will appear ESXi installation menu.

Finalizing ??

And here we come. You expect to see ESXi installation menu… but what you see it’s only a “Relocating modules and starting up the kernel…” message. There are several reason that could lead to this error, and the most common is that you’re trying to install ESXi on a machine with a bad BIOS, which is not correctly working with your video card, or a BIOS that it is supposed to talk to a video card that it’s instead missing from a device (like APU) that is headless.

Somewhere (like here) you could be driven to press SHIFT+O on console to enter the high level command line interface, than issue some switches to properly redirect console message (that’s what we were supposing to do when modifying the boot files). I tried all of these workaround but had no success.

If you want to try… let me know your workaround, if you found any.

I’m just tinking to give over ESXi on the APU and to install a plain pfSense firewall (that is my first need on the APU, and I already installed on another APU at work).

Advertisements

12 comments so far

  1. Bronston on

    Boot tinycore
    https://www.pcengines.ch/tinycore.htm
    Find the esxi partition with all the various files
    Edit the boot.cfg just like you did above
    kernelopt=runweasel text nofb com1_baud=115200 com1_Port=0x3f8 tty2Port=com1 gdbPort=none logPort=none
    And enjoy a small chance of it actually booting, perfect.

    • Bronston on

      On second thought don’t set your kernel options to that or you’ll just get an esxi install script again.
      Try this instead:
      kernelopt= text nofb com1_baud=115200 com1_Port=0x3f8 tty2Port=com1 gdbPort=none logPort=none

      • stefanobettega on

        Well, thanks for your comments. Unfortunately I don’t have many time to test it, but if I had it, I would try that just to learn something new. Actually I switched my ESXi works on a Dell PowerEdge 2950, that it’s a little bit more powerful than the APU (even though it requires a lot more power…)

  2. Bronston on

    Ah I see. Well I landed here looking for help with the same problem so I figured I’d post how I got it to work. Here’s a link with a more comprehensive explanation if anyone stumbles onto it here.
    http://pcengines.info/forums/?page=post&fid=&lastp=1&id=511E5F7D-AD74-4041-8C0C-72FBADD95504

    • stefanobettega on

      Well done, It seems not too complicated, so I think I will give it a try. Actually my APU is running a bare pFsense firewall, but it could be great if I could extend it to somewhat more complicated.

  3. mikeboss on

    after installing ESXi successfully, log into the WebGUI. goto /Host/Manage/Advanced settings. find “VMkernel.Boot.tty2Port” and click once to highlight. click on “Edit option” and change “default” to “com1”. save and reboot. now the serial port console will fully work and the management console will be displayed, too.

    • stefanobettega on

      Hi Mike,
      my problem was that ESXi wasn’t booting at all at first trying. But with Bronston help this problem could be fixed. I will give it a try after returning from holidays, then I’ll post my conclusion.

      Cheers,
      Stefano

      • mikeboss on

        here, it also looked like it didn’t boot because on the SerCon it stopped at “Relocating modules and starting up the kernel…” but I could login to the WebGUI (I checked the DHCP leases on my pfSense to identify the IP of ESXi). then I did set the option as described in my former post and all was good.

      • stefanobettega on

        Uhm, didn’t though about it, I was expecting to configure the IP address from console, but this could be a great idea. This way I wouldn’t have to do all of the steps told by Bronston. It worth a try, it shouldn’t be difficult. Which version did you installed?

  4. mikeboss on

    latest release 6.5.0 U1 (2017-07-27) on APU2. this release also doesn’t need modifications because of the vmkusb stopping the installer like it did on 6.5.0. plus, APU2 has intel NICs, so no need to inject realtek drivers.

    • stefanobettega on

      Ok, I’ll have to inject the Realtek drivers again as I only have an APU1, and I’ll try with the latest version too (hoping that driver will be compatible). I’ll get a new mSATA disk to perform these tests without having to worry on old environment, do it will require me some time. I think that till the end of the month I would be ready.

  5. Ioannis S on

    Hello All,
    Thanks for this excellent article. I achieved to install esxi on my APU 1d4. I followed all the steps, including the last step that required to boot with the tinycore and update the boot.cfg.
    I have to note that while the installation was completed, the first time that I booted the device I got a message that it could find any network adapter. So I booted again using the tinycore and apply the patch for the drivers for a second time (the first was for the installation). Now it works perfectly.
    Thanks again.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: