… and it’s finally here

I was waiting my new gadget, and it’s finally here.

A couple of Onion Omega 2 plus board, a couple of prototyping expansion board and a power dock. I also have a Onion Omega (the previous version) with a couple of power dock, so I can power up all the three boards at the same time.

Now it’s time to think about what to do… Ideas are welcome!

Advertisements

Create and test a bootable USB pen drive

A couple of usefull links to remember, with instruction about to create a bootable USB pen drive with Windows 10 (or any other O.S.) and a way to test it without having to reboot your PC:

Waiting for my new gadget…

About a year ago I received my Onion Omega board, and it’still here on my shelf, waiting for something cool to do.

But then Onion released Omega2, a more powerful device, which doubles RAM and ROM available at a little cost. So I backed the project on Kickstarter, and my new Omega2 is on track now. I’m waiting it to be here in a couple of weeks, and in the meantime I’ll think about something to do with this new board and with the old one. I have some expansion boards (relays, OLED, Ethernet) that will be divided between the two boards to build some IoT projects.

Stay tuned!

Forcibly shutting down a Virtual Machine under ESXi

Even though this could be a last-used-resource, sometimes you need to shut down a unresponsive virtual machine in your ESXi server.

To do so the only way is to connect to the console via SSH (that should have been enabled from ESXi configuration). Then follow these steps:

  • Identify the machine you need to shut down. This is achieved running the following command and looking for the World ID of the machine you need to stop:
esxcli vm process list
  • issue the first kind of shut down option, that is a “try to shut down gracefully” command:
esxcli vm process kill --type=soft --world-id=xxxxx
  • if this not helps, try to improve your order in this way, performing a “shut down and don’t care”:
esxcli vm process kill --type=hard --world-id=xxxxx
  • if also this command doesn’t work… try this last resource, the “I told you to shut down and you do this now”:
esxcli vm process kill --type=force --world-id=xxxxx

Then you can perform all of the operation you need, for instance delete a stuck or corrupt machine from datastore.

If also the force mode doesn’t work, you’ll have to restart your ESXi server. But usually the force option is good enough to make all stucked host to turn off.

 

Firsts steps into the IoT world…

Ok, it’s time to enter the IoT world, because IoT is cool.
With my Udoo Neo board I decided to do something usefull  (at least for me): trace down temperature and humidity data from my garage, that is the place in which almost all of my equipment lives (or try to do it).

First of all you have to think how to collect and how to show data. Many options are available, starting from DIY, and finishing to online services. Many of them are available up to now, but one focused my attention: ThingSpeak.

ThingSpeak is a platform for IoT data collection and analysis, driven by Mathworks (yes, that Matworks that publish Matlab). Within certain limits, you can use it for free (see their statements here).

I then opened a channel and wanted to start publishing data. I had to google around a bit to find a way to publish data, just because the most of the work is done for Arduino and other platforms different from Udoo Neo. But basically publishing data is almost a web service connection exercise, so mixing some script (and having to learn some Python coding), let me to create this script:

import httplib, urllib

def getfcontents(filename):
   "retrieve file contents"
   with open(filename, 'r') as myfile:
      data = myfile.read().replace('\n', '')
   return data

temp_raw = getfcontents('/sys/class/i2c-dev/i2c-1/device/1-0060/iio:device0/in_temp_raw')
temp_scale  = getfcontents('/sys/class/i2c-dev/i2c-1/device/1-0060/iio:device0/in_temp_scale')
baro_raw   = getfcontents('/sys/class/i2c-dev/i2c-1/device/1-0060/iio:device0/in_pressure_raw')
baro_scale  = getfcontents('/sys/class/i2c-dev/i2c-1/device/1-0060/iio:device0/in_pressure_scale')

temperature = float(temp_raw) * float(temp_scale)
barometer = float(baro_raw) * float(baro_scale)

params = urllib.urlencode({'field1': temperature, 'field2': barometer,'key':'<strong><em>your_api_key_goes_here</em></strong>'})
headers = {"Content-type": "application/x-www-form-urlencoded","Accept": "text/plain"}
conn = httplib.HTTPConnection("api.thingspeak.com:80")

try:
        conn.request("POST", "/update", params, headers)
        response = conn.getresponse()
        print response.status, response.reason
        data = response.read()
        print data
        conn.close()
except:
        print "connection failed"

On Udoo Neo, temperature can be read from an external device connected to I2C bus, known as “a brick”. For more information about hardware connection you’d better take a look to the official Udoo Neo brick documentation.

My script does the following:

  1. reads from the I2C device the values for temperature and pressure (they are shadowed into four files on a virtual file system)
  2. calculates the temperature and pressure absolute values
  3. connects to ThingSpeak REST API service and post data

and that’s all. To invoke on a timely-based fashion my script, I simply put the invocation into the crontab file, with the following command:

crontab -e

wich will start your favorite editor, allowing to edit the crontab file. I chose to execute my script once per minute, to see what it happened, but I’ll probably switch the time to once every 5 or 10 minutes, just because these data are not suitable to change so fast. My crontab line so will look like this:

*/1 * * * * python /home/udooer/thing.py

just because my script is in /home/udooer/thing.py file.

Feel free to use the script and modify it; you will have to put your private write API key in the correct place (I put your_api_key_goes_here where you have to do it).

There is something more to do… Did you catch it? At the beginning I said I wanted to record humidity of my garage, but in the following I’ve always spoken about pressure. Why? Because the brick sensor provided is an NXP MPL3115A2, that is a barometer with a temperature sensor included. To measure humidity I need a DHT11-like sensor, but it has to be wired and this will require some more time. To check if all it’s working I’ll go with this setup, then I’ll add a DHT11 sensor later.

And again on the MPL3115A2, it’s actually unclear what is actually the pressure readout supposed to be. This device can output an absolute pressure value, but it can also express an elevation measure depending on its configuration.

udoo neo reconfiguration

Just a quick note on how to reconfigure udoo neo keyboard and timezone from console.

To reconfigure keyboard, from the console issue:

sudo dpkg-reconfigure console-data

give sudo password then select the last option ( ‘Select keymap from full list’). From there you can choose you actual keyboard layout and confirm with enter.

To reconfigure timezone, I think that you can manually edit /etc/timezone with an editor, changing the contents to suit your correct time zone. Otherwise you can simply issue from console:

timedatectl list-timezones

to have a list of available time zones, then

sudo timedatectl set-timezone your/timezone

to set up the correct time zone. For instance I had to issue

sudo timedatectl set-timezone Europe/Rome

and that’s all. For keyboard you should have to restart udoo neo; time zone is immediately set up, and information are update also in lxde.

Previous commands are available also for Ubuntu, from wich udoo Linux distribution comes from.

Acronis 2016 with NVMe disks

Today I helped a colleague to restore its Dell laptop from an Acronis image. The process should have been easy to perform, but there was a little problem… Its laptop was using an M.2 Toshiba SSD (specifically the model THNSN51T02DU7). If you use the Acronis launcher even in WinPE mode, the M.2 disk is not detected.

And now? What to do?

Googling around a bit I found this site which explain (in Italian, many thanks to Gianluca Cecca) how to integrate a Samsung 950 Pro M.2 disk into Acronis. The problem was: where to find THNSN51T02DU7 drivers? We had to look around again with Google, and finally we came up with this explanation:

Intel and Samsung offer a proprietary NVMe driver for their NVMe SSDs, but Toshiba does not. The XG3 utilizes the standard Microsoft NVMe driver that comes baked into Windows 8-10. This makes sense because the XG3 is an OEM SSD, and not needing an additional driver for best performance makes setup easier for OEMs.
So all we had to do was:
  1. search for the original Windows 10 drivers
  2. download them
  3. integrate them into the Acronis WinPE image
  4. cross our fingers…

Locating the driver to download wasn’t an easy taks to perform. Fortunately on Dell website there was an explanation of a similar task related to WinPE and storage controller drivers. So we had to download the Intel Rapid Storage drivers from Intel’s web site (following the lin provided into Dell’s site, we were redirected to a driver version that wasn’t the last one available; Intel site then suggested us to download the last version).

We choosed to download the package f6flpy-x64.zip from the site (f6flpy-x32.zip if your sistem is running in 32 bit mode, but it’s really improbable), then followed the instruction from the integration guide. Basically here are the instructions:

  1. Create the Acronis loader in WinPE format from within your Acronis True Image, writing it onto an USB pen drive
  2. Install Windows ADK; you might have it already installed if you are a developer. You can try to open ad administrator command prompt, then issue the following instruction. If you receive any errors, you might have to download Windows ADK
  3. Create three folders in C:\, naming them Temp, Drivers and Mount
    Copy BOOT.WIM file from the Source folder of Acronis WinPE pen drive, into C:\Temp
  4. Extract the drivers from f6flpy-x64.zip into C:\Drivers
  5. Open the Administrator command prompt
  6. Issue the following command to mount the image
    dism /mount-wim /wimfile:c:\temp\boot.wim /index:1 /mountdir:c:\mount
  7. Issue the following command to merge drivers into image
    dism /add-driver /image:c:\mount /driver:c:\drivers /forceunsigned
  8. Issue the following command to integrate the drivers into the driver library
    dism /get-drivers /image:c:\mount
  9. Issue the following command to write changes into the .WIM file and release resources
    dism /unmount-wim /mountdir:c:\mount /commit
  10. Copy the new BOOT.WIM file from C:\Temp into Sources folder of Acronis WinPE pen drive
  11. Eject pen drive and delete the Temp, Drivers and Mount folders created in point 3.

    Your disk should now be visible from Acronis loader and you can proceed to data restoration.

Further steps

If you found that your Acronis loader won’t recognize your M.2 disk, probably you don’t need to read  more, as you already succeded to boot up your PC from the USB key. But what if you don’t?

Modern PC use UEFI boot process with a secure option into BIOS. This is to avoid your PC to be booted from a device you don’t trust. It might be needed to enter the BIOS and add your Acronis WinPE boot pen drive to the trusted boot sources. How to perform this task may vary from BIOS to BIOS, and you should read your PC manual to perform these steps.

An interesting library

Searching for some nuget packet usage, I came across this really interesting and huge library: Humanizer.

From the site:

Humanizer meets all your .NET needs for manipulating and displaying strings, enums, dates, times, timespans, numbers and quantities.

I’ll keep it in my notes…

Installing ESXi 5.5 U2 on PCEngines apu1d4

AlertI started writing this article on 10th January, 2015. But I didn’t have so much time to finish it… Now time has come, so I decided to finish this work and publish it. Many time has passed by, and many things have changed. A new ESXi version has been carried out, and new APU hardware is coming soon. But I think that this article can be a good starting point also for headless installation of new ESXi and other *nix based headless devices.

Unfortunately installation was not successfully, or better speaking installation correctly ended, but device didn’t turned up as I expected it would. I attempted many ways to overcome this problem, but I haven’t found any of them to correctly work. I posted all of this history here, in the hope that someone would find what I missed.

Introduction

In this article I’m introducing a step by step guide on how to install ESXi 5.5 U2 on PCEngines apu1d4 board.
apu1d4 is a small computer board that can be used for routers, firewalls, VOIP, dedicated servers, special purpose network plumbing, education tools… It’s equipped with an AMD G series T40E, 1 GHz dual core with 64 bit support processor, 4GB RAM, three Gigabit Ethernet ports and a serial console. You can read all of its specification here.

This setup guide it’s also suitable to install many other Linux distributions.

Prerequisites

As apu1d4 does not have a video output, setup must be performed via serial console. This is the major issue when installing ESXi, as the plain setup is intended to be used with a video card. But as it is Linux-based, we can turn on the serial console redirection to handle all of the setup process from here.

In this guide I’m assuming to install ESXi on a mSATA SSD installed on the board. Please make sure the SSD is detected by board BIOS before starting the setup process (you should see your mSATA ID into the BIOS boot messages on the serial console).

You need the following hardware:

  • an apu1d4 board already boxed (for passive cooling to work), with an internal mSATA drive installed
  • an empty USB pen drive (at least 1GB)
  • a null modem cable
  • a PC with a serial port or a USB to serial converter (you may omit the null modem cable if you use a USB to DCE converter like this one, that is intended to be used to connect directly to a console port)

and the following software:

  • ESXi 5.5 U2 install ISO (you don’t need to burn it onto a CD)
  • a utility to create a bootable USB key from an ISO. There are many utilities to do that; here I’m using Rufus
  • a serial communication software which allows you to connect to console port; here I’m using PuTTY
  • a utility to open .zip files; here I’m using 7-Zip
  • a general purpose text editor; here I’m using PSPad
  • the two file contained into this ESXi setup patch: net-r816-drivers.zip

A note on the patch: apu1d4 board integrates three Gigabit Ethernet channels based on the Realtek RTL8111E chipset. As RTL8111E support has been removed from ESXi setup since version 5.5, but it was still present in 5.1 setup, we need to extract the drivers from 5.1 installer and integrate them into the 5.5 one. This is needed as without installing and activating the drivers, you will end your setup into a “No network hardware detected” screen. How to integrate this driver in the setup is better explained later.

Step by step guide

Installation process can be resumed in the following steps:

  1. create ESXi 5.5 U2 USB boot pen drive (called pen drive in the following steps)
    1. unpack ISO to pen drive and make it bootable
    2. patch pen drive to load RTL8111 drivers
    3. patch pen drive to work with serial console
  2. install ESXi
  3. finalize setup

Create ESXi 5.5 U2 USB boot pen drive

Any tool you can find to extract an ISO image to a pen drive and make it bootable is suitable to create ESXi setup pen drive. I frufusound Rufus to be easy and clear to use. It’s also OpenSource and sources can be grabbed from it’s home page. From Rufus main window you have to select the USB drive related to the pen drive you want to use, the partitioning method (usually the default value is correct) and obviously the ESXi ISO image to store on key.

If should be warned that the menu.c32 file which comes on the ISO is obsolete and you should substitute it with a freshly downloaded version. Please accept this suggestion and  let Rufus patch this file.

You should end up in few minutes with the key ready to be patched.

Patch pen drive to load RTL8111 drivers

As said ESXi 5.5 U2 setup does not include RTL8111 drivers. If you do not patch the pen drive with those drivers, installation cannot be performed and you’ll receive a “No network card present” message. Usually drivers can be added later installing the so called VIB packages, but as the network card is considered to be part of the core setup, if no network card can be detected during setup phase, process will be stopped.

To avoid this problem a clean and correct way should be to install ESXi 5.1 version and then perform a version upgrade, but this could take long. A faster-and-quite-clean method is to include ESXi 5.1 drivers directly on 5.5 U2 setup, and that’s what we are going to do now.

When googling around this problem you’ll find several .vib files packaged into a .zip file, that should unpack to upload .vibs to your ESXi machine. Basically each .vib file is in turn a .zip package, containing the driver(s) itself and some descriptors. I simply extracted and repacked the two drivers included in the following .vib:

VMware_bootbank_net-r8168_8.013.00-3vmw.510.0.0.799733.vib
VMware_bootbank_net-r8169_6.011.00-2vmw.510.0.0.799733.vib

into a single .zip file that you’ll find attached here. All you have to do is to download the .zip containing the two drivers (net-r816.v00 and net-r816.v01) and extract them onto the pen drive, in its root directory.

After that you have to edit the boot.cfg file located on the pen drive, and add those driver to load list. To do so open your favorite editor (be carefull: you should not use NotePad because it lacks support for UNIX style line endings; you’d better use a general purpose editor like PSPad). Locate the row

modules=/b.b00 --- ...

and then add the following in your favorite position (I added them after all of the net_xxx drivers):

--- /net-r816.v00 --- /net-r816.v01

Save the file and you’re done with the patch.

Patch pen drive to work with serial console

In the last preparation step we have to patch configuration files to tell loader and setup to redirect standard output and standard input over a serial console. This step involves editing three configuration files, that must be still edited with the general purpose editor.

syslinux.cfg

Patching this file only requires you to add the following two lines AT TOP of the file:

CONSOLE 0
SERIAL 0 115200

isolinux.cfg

Patching this file requires you to add the following two lines AT TOP of the file:

CONSOLE 0
SERIAL 0 115200

Then you have to locate the line

  APPEND -c boot.cfg

located in the LABEL install section of the configuration. At the end of the line you have to add the extra parameters needed to instruct loader to redirect standard input and output. Line should look like this:

  APPEND -c boot.cfg text gdbPort=none logPort=none tty2Port=com1

boot.cfg

You have again to patch boot.cfg, as done in the step before. Modification involves the kernel option parameter passed; you have to locate the line

kernelopt=runweasel

and add the extra parameters needed to redirect standard input and output. Line should look like this:

kernelopt=runweasel text nofb com1_baud=115200 com1_Port=0x3f8 tty2Port=com1 gdbPort=none logPort=none

Install ESXi

You are now ready to start ESXi setup. Put the pen drive into the USB port, connect your PC to the console port, start PuTTY and power up the board. You should notice BIOS startup in the console port:

PC Engines APU BIOS build date: Apr  5 2014
Reading data from file [bootorder]
SeaBIOS (version ?-20140405_120742-frink)
SeaBIOS (version ?-20140405_120742-frink)
Found coreboot cbmem console @ df150400
Found mainboard PC Engines APU
Relocating init from 0x000e8e71 to 0xdf1065e0 (size 39259)
Found CBFS header at 0xfffffb90
found file "bootorder" in cbmem
CPU Mhz=1000
Found 27 PCI devices (max PCI bus is 05)
Copying PIR from 0xdf160400 to 0x000f27a0
Copying MPTABLE from 0xdf161400/df161410 to 0x000f25b0 with length 1ec
Copying ACPI RSDP from 0xdf162400 to 0x000f2590
Copying SMBIOS entry point from 0xdf16d800 to 0x000f2570
Using pmtimer, ioport 0x808
Scan for VGA option rom
EHCI init on dev 00:12.2 (regs=0xf7f08420)
Found 1 lpt ports
Found 2 serial ports
AHCI controller at 11.0, iobase f7f08000, irq 11
EHCI init on dev 00:13.2 (regs=0xf7f08520)
EHCI init on dev 00:16.2 (regs=0xf7f08620)
Searching bootorder for: /rom@img/setup
Searching bootorder for: /rom@img/memtest
Searching bootorder for: /pci@i0cf8/*@11/drive@0/disk@0
AHCI/0: registering: "AHCI/0: KINGSTON SMS200S330G ATA-8 Hard-Disk (28626 MiBytes)"
OHCI init on dev 00:12.0 (regs=0xf7f04000)
OHCI init on dev 00:13.0 (regs=0xf7f05000)
OHCI init on dev 00:14.5 (regs=0xf7f06000)
OHCI init on dev 00:16.0 (regs=0xf7f07000)
Searching bootorder for: /pci@i0cf8/usb@12,2/storage@1/*@0/*@0,0
Searching bootorder for: /pci@i0cf8/usb@12,2/usb-*@1
Searching bootorder for: /pci@i0cf8/usb@16,2/storage@1/*@0/*@0,0
Searching bootorder for: /pci@i0cf8/usb@16,2/usb-*@1
USB MSC vendor='Multiple' product='Card  Reader' rev='1.00' type=0 removable=1
Device reports MEDIUM NOT PRESENT
scsi_is_ready returned -1
Unable to configure USB MSC drive.
Unable to configure USB MSC device.
USB MSC vendor='' product='' rev='PMAP' type=0 removable=1
USB MSC blksize=512 sectors=1970176
All threads complete.
Scan for option roms
Running option rom at c000:0003


iPXE (http://ipxe.org) 00:00.0 C000 PCI2.10 PnP PMMpmm call arg1=1
pmm call arg1=0
+DF0F04A0pmm call arg1=1
pmm call arg1=0
+DF04C510 C000



Searching bootorder for: /rom@genroms/pxeboot.rom

Build date: Apr  5 2014
System memory size: 4592 MB

Press F12 for boot menu.

Searching bootorder for: HALT
drive 0x000f24d0: PCHS=0/0/0 translation=lba LCHS=977/32/63 s=1970176
drive 0x000f2500: PCHS=16383/16/63 translation=lba LCHS=1024/255/63 s=58626288
Space available for UMB: c1000-ee800, f0000-f24d0
Returned 253952 bytes of ZoneHigh
e820 map has 7 items:
  0: 0000000000000000 - 000000000009fc00 = 1 RAM
  1: 000000000009fc00 - 00000000000a0000 = 2 RESERVED
  2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
  3: 0000000000100000 - 00000000df14e000 = 1 RAM
  4: 00000000df14e000 - 00000000e0000000 = 2 RESERVED
  5: 00000000f8000000 - 00000000f9000000 = 2 RESERVED
  6: 0000000100000000 - 000000011f000000 = 1 RAM
enter handle_19:
  NULL
Booting from Hard Disk...
Booting from 0000:7c00

SYSLINUX 4.07 EDD 2013-07-25 Copyright (C) 1994-2013 H. Peter Anvin et al

SYSLINUX 4.07 2013-07-25 Copyright (C) 1994-2013 H. Peter Anvin et al

and then will appear ESXi installation menu.

Finalizing ??

And here we come. You expect to see ESXi installation menu… but what you see it’s only a “Relocating modules and starting up the kernel…” message. There are several reason that could lead to this error, and the most common is that you’re trying to install ESXi on a machine with a bad BIOS, which is not correctly working with your video card, or a BIOS that it is supposed to talk to a video card that it’s instead missing from a device (like APU) that is headless.

Somewhere (like here) you could be driven to press SHIFT+O on console to enter the high level command line interface, than issue some switches to properly redirect console message (that’s what we were supposing to do when modifying the boot files). I tried all of these workaround but had no success.

If you want to try… let me know your workaround, if you found any.

I’m just tinking to give over ESXi on the APU and to install a plain pfSense firewall (that is my first need on the APU, and I already installed on another APU at work).

Coming soon…

It’s time to write down something new…

I’m planning to create a map of my home network using Dia, and post it on Rate My Network Diagram web site. I just only have to find the time to do it…