Archive for the ‘Virtualization’ Category

News from “The Lab” – The new server

I decded to switch over to a new server, as the old HP Proliant was unable to handle 64bit virtual machines (due to lack of BEST in its processors).

I found a cheap Dell PowerEdge 2950 with two quad-core processors, 8GB RAM and two 146GB SAS disks.

But…

When I turned on the server after its installation, this message came up:

It seems that the PERC controller is not working… I just tried reseating it and the riser card, but no luck.

I will try to remove the controller to see if the message still persists, just to isolate the problem source.

If anyone has some ideas on how to troubleshoot this problem, please let me know.

Forcibly shutting down a Virtual Machine under ESXi

Even though this could be a last-used-resource, sometimes you need to shut down a unresponsive virtual machine in your ESXi server.

To do so the only way is to connect to the console via SSH (that should have been enabled from ESXi configuration). Then follow these steps:

  • Identify the machine you need to shut down. This is achieved running the following command and looking for the World ID of the machine you need to stop:
esxcli vm process list
  • issue the first kind of shut down option, that is a “try to shut down gracefully” command:
esxcli vm process kill --type=soft --world-id=xxxxx
  • if this not helps, try to improve your order in this way, performing a “shut down and don’t care”:
esxcli vm process kill --type=hard --world-id=xxxxx
  • if also this command doesn’t work… try this last resource, the “I told you to shut down and you do this now”:
esxcli vm process kill --type=force --world-id=xxxxx

Then you can perform all of the operation you need, for instance delete a stuck or corrupt machine from datastore.

If also the force mode doesn’t work, you’ll have to restart your ESXi server. But usually the force option is good enough to make all stucked host to turn off.

 

Installing ESXi 5.5 U2 on PCEngines apu1d4

AlertI started writing this article on 10th January, 2015. But I didn’t have so much time to finish it… Now time has come, so I decided to finish this work and publish it. Many time has passed by, and many things have changed. A new ESXi version has been carried out, and new APU hardware is coming soon. But I think that this article can be a good starting point also for headless installation of new ESXi and other *nix based headless devices.

Unfortunately installation was not successfully, or better speaking installation correctly ended, but device didn’t turned up as I expected it would. I attempted many ways to overcome this problem, but I haven’t found any of them to correctly work. I posted all of this history here, in the hope that someone would find what I missed.

Introduction

In this article I’m introducing a step by step guide on how to install ESXi 5.5 U2 on PCEngines apu1d4 board.
apu1d4 is a small computer board that can be used for routers, firewalls, VOIP, dedicated servers, special purpose network plumbing, education tools… It’s equipped with an AMD G series T40E, 1 GHz dual core with 64 bit support processor, 4GB RAM, three Gigabit Ethernet ports and a serial console. You can read all of its specification here.

This setup guide it’s also suitable to install many other Linux distributions.

Prerequisites

As apu1d4 does not have a video output, setup must be performed via serial console. This is the major issue when installing ESXi, as the plain setup is intended to be used with a video card. But as it is Linux-based, we can turn on the serial console redirection to handle all of the setup process from here.

In this guide I’m assuming to install ESXi on a mSATA SSD installed on the board. Please make sure the SSD is detected by board BIOS before starting the setup process (you should see your mSATA ID into the BIOS boot messages on the serial console).

You need the following hardware:

  • an apu1d4 board already boxed (for passive cooling to work), with an internal mSATA drive installed
  • an empty USB pen drive (at least 1GB)
  • a null modem cable
  • a PC with a serial port or a USB to serial converter (you may omit the null modem cable if you use a USB to DCE converter like this one, that is intended to be used to connect directly to a console port)

and the following software:

  • ESXi 5.5 U2 install ISO (you don’t need to burn it onto a CD)
  • a utility to create a bootable USB key from an ISO. There are many utilities to do that; here I’m using Rufus
  • a serial communication software which allows you to connect to console port; here I’m using PuTTY
  • a utility to open .zip files; here I’m using 7-Zip
  • a general purpose text editor; here I’m using PSPad
  • the two file contained into this ESXi setup patch: net-r816-drivers.zip

A note on the patch: apu1d4 board integrates three Gigabit Ethernet channels based on the Realtek RTL8111E chipset. As RTL8111E support has been removed from ESXi setup since version 5.5, but it was still present in 5.1 setup, we need to extract the drivers from 5.1 installer and integrate them into the 5.5 one. This is needed as without installing and activating the drivers, you will end your setup into a “No network hardware detected” screen. How to integrate this driver in the setup is better explained later.

Step by step guide

Installation process can be resumed in the following steps:

  1. create ESXi 5.5 U2 USB boot pen drive (called pen drive in the following steps)
    1. unpack ISO to pen drive and make it bootable
    2. patch pen drive to load RTL8111 drivers
    3. patch pen drive to work with serial console
  2. install ESXi
  3. finalize setup

Create ESXi 5.5 U2 USB boot pen drive

Any tool you can find to extract an ISO image to a pen drive and make it bootable is suitable to create ESXi setup pen drive. I frufusound Rufus to be easy and clear to use. It’s also OpenSource and sources can be grabbed from it’s home page. From Rufus main window you have to select the USB drive related to the pen drive you want to use, the partitioning method (usually the default value is correct) and obviously the ESXi ISO image to store on key.

If should be warned that the menu.c32 file which comes on the ISO is obsolete and you should substitute it with a freshly downloaded version. Please accept this suggestion and  let Rufus patch this file.

You should end up in few minutes with the key ready to be patched.

Patch pen drive to load RTL8111 drivers

As said ESXi 5.5 U2 setup does not include RTL8111 drivers. If you do not patch the pen drive with those drivers, installation cannot be performed and you’ll receive a “No network card present” message. Usually drivers can be added later installing the so called VIB packages, but as the network card is considered to be part of the core setup, if no network card can be detected during setup phase, process will be stopped.

To avoid this problem a clean and correct way should be to install ESXi 5.1 version and then perform a version upgrade, but this could take long. A faster-and-quite-clean method is to include ESXi 5.1 drivers directly on 5.5 U2 setup, and that’s what we are going to do now.

When googling around this problem you’ll find several .vib files packaged into a .zip file, that should unpack to upload .vibs to your ESXi machine. Basically each .vib file is in turn a .zip package, containing the driver(s) itself and some descriptors. I simply extracted and repacked the two drivers included in the following .vib:

VMware_bootbank_net-r8168_8.013.00-3vmw.510.0.0.799733.vib
VMware_bootbank_net-r8169_6.011.00-2vmw.510.0.0.799733.vib

into a single .zip file that you’ll find attached here. All you have to do is to download the .zip containing the two drivers (net-r816.v00 and net-r816.v01) and extract them onto the pen drive, in its root directory.

After that you have to edit the boot.cfg file located on the pen drive, and add those driver to load list. To do so open your favorite editor (be carefull: you should not use NotePad because it lacks support for UNIX style line endings; you’d better use a general purpose editor like PSPad). Locate the row

modules=/b.b00 --- ...

and then add the following in your favorite position (I added them after all of the net_xxx drivers):

--- /net-r816.v00 --- /net-r816.v01

Save the file and you’re done with the patch.

Patch pen drive to work with serial console

In the last preparation step we have to patch configuration files to tell loader and setup to redirect standard output and standard input over a serial console. This step involves editing three configuration files, that must be still edited with the general purpose editor.

syslinux.cfg

Patching this file only requires you to add the following two lines AT TOP of the file:

CONSOLE 0
SERIAL 0 115200

isolinux.cfg

Patching this file requires you to add the following two lines AT TOP of the file:

CONSOLE 0
SERIAL 0 115200

Then you have to locate the line

  APPEND -c boot.cfg

located in the LABEL install section of the configuration. At the end of the line you have to add the extra parameters needed to instruct loader to redirect standard input and output. Line should look like this:

  APPEND -c boot.cfg text gdbPort=none logPort=none tty2Port=com1

boot.cfg

You have again to patch boot.cfg, as done in the step before. Modification involves the kernel option parameter passed; you have to locate the line

kernelopt=runweasel

and add the extra parameters needed to redirect standard input and output. Line should look like this:

kernelopt=runweasel text nofb com1_baud=115200 com1_Port=0x3f8 tty2Port=com1 gdbPort=none logPort=none

Install ESXi

You are now ready to start ESXi setup. Put the pen drive into the USB port, connect your PC to the console port, start PuTTY and power up the board. You should notice BIOS startup in the console port:

PC Engines APU BIOS build date: Apr  5 2014
Reading data from file [bootorder]
SeaBIOS (version ?-20140405_120742-frink)
SeaBIOS (version ?-20140405_120742-frink)
Found coreboot cbmem console @ df150400
Found mainboard PC Engines APU
Relocating init from 0x000e8e71 to 0xdf1065e0 (size 39259)
Found CBFS header at 0xfffffb90
found file "bootorder" in cbmem
CPU Mhz=1000
Found 27 PCI devices (max PCI bus is 05)
Copying PIR from 0xdf160400 to 0x000f27a0
Copying MPTABLE from 0xdf161400/df161410 to 0x000f25b0 with length 1ec
Copying ACPI RSDP from 0xdf162400 to 0x000f2590
Copying SMBIOS entry point from 0xdf16d800 to 0x000f2570
Using pmtimer, ioport 0x808
Scan for VGA option rom
EHCI init on dev 00:12.2 (regs=0xf7f08420)
Found 1 lpt ports
Found 2 serial ports
AHCI controller at 11.0, iobase f7f08000, irq 11
EHCI init on dev 00:13.2 (regs=0xf7f08520)
EHCI init on dev 00:16.2 (regs=0xf7f08620)
Searching bootorder for: /rom@img/setup
Searching bootorder for: /rom@img/memtest
Searching bootorder for: /pci@i0cf8/*@11/drive@0/disk@0
AHCI/0: registering: "AHCI/0: KINGSTON SMS200S330G ATA-8 Hard-Disk (28626 MiBytes)"
OHCI init on dev 00:12.0 (regs=0xf7f04000)
OHCI init on dev 00:13.0 (regs=0xf7f05000)
OHCI init on dev 00:14.5 (regs=0xf7f06000)
OHCI init on dev 00:16.0 (regs=0xf7f07000)
Searching bootorder for: /pci@i0cf8/usb@12,2/storage@1/*@0/*@0,0
Searching bootorder for: /pci@i0cf8/usb@12,2/usb-*@1
Searching bootorder for: /pci@i0cf8/usb@16,2/storage@1/*@0/*@0,0
Searching bootorder for: /pci@i0cf8/usb@16,2/usb-*@1
USB MSC vendor='Multiple' product='Card  Reader' rev='1.00' type=0 removable=1
Device reports MEDIUM NOT PRESENT
scsi_is_ready returned -1
Unable to configure USB MSC drive.
Unable to configure USB MSC device.
USB MSC vendor='' product='' rev='PMAP' type=0 removable=1
USB MSC blksize=512 sectors=1970176
All threads complete.
Scan for option roms
Running option rom at c000:0003


iPXE (http://ipxe.org) 00:00.0 C000 PCI2.10 PnP PMMpmm call arg1=1
pmm call arg1=0
+DF0F04A0pmm call arg1=1
pmm call arg1=0
+DF04C510 C000



Searching bootorder for: /rom@genroms/pxeboot.rom

Build date: Apr  5 2014
System memory size: 4592 MB

Press F12 for boot menu.

Searching bootorder for: HALT
drive 0x000f24d0: PCHS=0/0/0 translation=lba LCHS=977/32/63 s=1970176
drive 0x000f2500: PCHS=16383/16/63 translation=lba LCHS=1024/255/63 s=58626288
Space available for UMB: c1000-ee800, f0000-f24d0
Returned 253952 bytes of ZoneHigh
e820 map has 7 items:
  0: 0000000000000000 - 000000000009fc00 = 1 RAM
  1: 000000000009fc00 - 00000000000a0000 = 2 RESERVED
  2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
  3: 0000000000100000 - 00000000df14e000 = 1 RAM
  4: 00000000df14e000 - 00000000e0000000 = 2 RESERVED
  5: 00000000f8000000 - 00000000f9000000 = 2 RESERVED
  6: 0000000100000000 - 000000011f000000 = 1 RAM
enter handle_19:
  NULL
Booting from Hard Disk...
Booting from 0000:7c00

SYSLINUX 4.07 EDD 2013-07-25 Copyright (C) 1994-2013 H. Peter Anvin et al

SYSLINUX 4.07 2013-07-25 Copyright (C) 1994-2013 H. Peter Anvin et al

and then will appear ESXi installation menu.

Finalizing ??

And here we come. You expect to see ESXi installation menu… but what you see it’s only a “Relocating modules and starting up the kernel…” message. There are several reason that could lead to this error, and the most common is that you’re trying to install ESXi on a machine with a bad BIOS, which is not correctly working with your video card, or a BIOS that it is supposed to talk to a video card that it’s instead missing from a device (like APU) that is headless.

Somewhere (like here) you could be driven to press SHIFT+O on console to enter the high level command line interface, than issue some switches to properly redirect console message (that’s what we were supposing to do when modifying the boot files). I tried all of these workaround but had no success.

If you want to try… let me know your workaround, if you found any.

I’m just tinking to give over ESXi on the APU and to install a plain pfSense firewall (that is my first need on the APU, and I already installed on another APU at work).