Getting Packer working with CoreOS on Hyper-V

If you try to perform Packer builds of CoreOS on Hyper-V, you’re going to have issues. This post describes a specific network issue and offers a temporary solution which allows Packer builds to complete successfully.

I say temporary because the issue can be resolved by CoreOS fully implementing Hyper-V Integration Services (hopefully soon) or by Packer adding support for the Hyper-V builder without requiring Integration Services (please)

But until there is a real solution this can help get around it.

The Problem

The TL;DR: The Hyper-V Packer builder requires Integration Services in order to SSH into a Linux guest. CoreOS does not fully implement these services. So Packer cannot build Hyper-V images for CoreOS.

To explain a bit more….

Successfully running the provisioner steps in a packer build requires SSH. To use SSH, Packer needs to know the IP address of the VM. The way that Packer finds the IP address of a Hyper-V VM is by getting using the mac address attached to the adapter and then filtering for a VM with has that mac address.

You can see this process by looking at the code here: https://github.com/hashicorp/packer/blob/master/common/powershell/hyperv/hyperv.go#L967

The problem is that this Powershell cmdlet requires Hyper-V Guest Integration Services to be fully implemented. Unfortunately, at the time of this post, CoreOS does not fully implement the guest services.

Specifically, it is missing this: https://www.freebsd.org/cgi/man.cgi?query=hv_kvp_daemon&sektion=8

As explained here

hv_kvp_daemon is the userspace component of the Hyper-V key value pair
functionality, communicating via a netlink socket with the kernel HV-
KVP driver. This pairing allows the Hyper-V host to pass configuration
information (such as IP addresses) to the guest and allows the host to
obtain guest version information.

Even though some of the Hyper-V components are provided within the Linux kernel, some parts need to be acquired by downloading the Linix Cloud Tools package. This is available but not officially for CoreOS.

If you have a CoreOS VM on Hyper-V that has an IP address and is pingable, this is still what you will see in the Hyper-V management console

hv_gui

And when you query the VM in Powershell in a similar way that Packer does

hv_posh

No IP address is found and the Packer build will fail after the ssh_timeout is hit.

The Solution

As mentioned at the very beginning — there is no perfect solution at this time. But there is a workaround which involves temporarily adding the hv_kvp_daemon into CoreOS.

The basic workflow here is:

  1. Get hv_kvp_daemon from some place
  2. Store this file on a FTP or SSH server
  3. Add the daemon file to the image and add the corresponding unit file to the image during the Ignition run
  4. Add a task in Packer to remove this awful workaround

Steps 1 and 2 are just making available the file which needs to be added to CoreOS. Step 3 is a way of inject the file and creating the unit file to make it into a startup service. Running this service allows Packer to detect the IP of the VM and SSH in for the provisioning step.

Step 4 is just removing the service and unit file because I am not comfortable leaving this on the system.

There are numerous ways to implement this workaround, and while I don’t necessarily think I approached it correctly (especially step 1) — the principle remains the same. And most importantly — it works.

Getting the Integration Services Files

Somehow, hv_kvp_daemon needs to to be placed on the system. There are different ways to get this file and there are different ways to get it onto the CoreOS VM. (you could just use gcc and grab this)

Since I was already on a CoreOS host, I figured I would just start an Ubuntu container, grab the file, and throw it on a web server.

You could either map a volume to the Ubuntu container or just scp the file back to the host from the container. I decided to use a named mount. Keep in mind, if you are on an older version of CoreOS with an older version of Docker you will not be able to use named mounts.

I ran the following command

docker run -v shared:/usr/lib -it ubuntu

In the container, I then updated the repos and installed the Linux Cloud Tools

apt-get update
sudo apt-get install --install-recommends linux-tools-virtual-lts-xenial linux-cloud-tools-virtual-lts-xenial

To find the location of the file we need, just run the find command

find

The middle one is the needed file (…-generic/hv_kvp_daemon).

In my approach, I am going to be serving this to Ignition from an HTTP server. This means I needed to get the file out of the container. Since I did the volume mount, I figured I would just grab the file off the host. Another way would have been to port map when running the container and spinup a quick FTP server on it.

I then exited the container and made sure the file was there

is_it_there

There are plenty of ways to get the file off of the VM. Since I had Mobaxterm handy, I just set a password for the core account so that I could SSH into it and grabbed the file with Moba’s file explorer (I did have to copy it to /home/core first).

To change the password

sudo -i
passwd core

And then I connected with Moba and downloaded the hv_kvp_daemon
mobahhhh

I then threw it on an internal web server in a a place which did not require any authentication.

Ignition

The next step is to use Ignition to create a unit file and copy the file from the last step to the system during the install process.

The Ignition file will be served to the CoreOS via the Packer built-in web server. The Ignition file then specifies the location of the needed hv_kvp_daemon file. Additionally, the Ignition file sets the content of the unit file needed to run the daemon at startup.

The Ignition file looks like this (though you should use YAML and the config transpiler:

{
  "ignition": {
    "version": "2.1.0",
    "config": {}
  },
  "storage": {
    "files": [
    {
    "filesystem": "root",
    "path": "/etc/systemd/system/hv_kvp_daemon",
    "mode": 775,
    "contents": {
        "source": "http://downloads.mhickok.me/coreos/hv_kvp_daemon"
        }
    }]
  },
  "systemd": {
    "units": [
    {
        "name": "hv_kvp_daemon.service",
        "enabled": true,
        "contents": "[Unit]\nDescription=Hyper-V KVP daemon\n\n[Service]\nType=simple\nExecStart=/etc/systemd/system/hv_kvp_daemon -n\n\n[Install]\nWantedBy=multi-user.target"
    }]
  },
  "networkd": {
  },
  "passwd": {
    "users": [
      {
        "name": "core",
        "passwordHash": "sdal;skmdaksmd;lkmmoi23jkl",
        "sshAuthorizedKeys": [
          "ssh-rsa somekey"
        ]
      }
    ]
  }
}

This is doing two main things:

  • Downloading the hv_kvp_daemon file from my file server and copying it (with execute permissions) to the systemd folder.
  • Adding a unit file which will enable this binary to run as a service at startup

Keep in mind that the location and permissions may not be perfect here but it doesn’t matter because the changes will immediately be undone by a script which I run with Packer.

Lastly,  I stored this Ignition file in a folder which will be referenced during the packer build later.

Packer

This part is very standard as far as Packer builds go. It has all of the normal Hyper-V builder values but it references the Ignition file and has a final step which removes the workaround.

{
    "variables": {
        "iso_url": "https://stable.release.core-os.net/amd64-usr/current/coreos_production_iso_image.iso",
        "checksum_type": "md5",
        "checksum": "d97ed6c250f2037b6fd92471c2890176",
        "ssh_timeout": "10m",
        "hyperv_switch": "vSwitch"
    },
    "builders": [
        {
            "type": "hyperv-iso",
            "vm_name": "{{user `vm_name`}}",
            "ssh_username": "{{user `core_user`}}",
            "ssh_password": "{{user `core_pass`}}",
            "iso_checksum": "{{user `checksum`}}",
            "iso_checksum_type": "{{user `checksum_type`}}",
            "iso_url": "{{user `iso_url`}}",
            "ssh_timeout": "{{user `ssh_timeout`}}",
            "http_directory": "http",
            "generation": 1,
            "ram_size": 4096,
            "enable_dynamic_memory": false,
            "enable_virtualization_extensions": true,
            "cpu": 2,
            "disk_size": 8192,
            "switch_name":"{{user `hyperv_switch`}}",
            "boot_wait": "2m",
            "shutdown_command": "sudo -S shutdown -P now",
            "boot_command": [
                "sudo -i<enter>",
                "systemctl stop sshd.socket<enter>;",
                "wget http://{{ .HTTPIP }}:{{ .HTTPPort }}/ignition_hyper.json<enter>",
                "coreos-install -d /dev/sda -C stable -i ignition_hyper.json<enter>",
                "eject<enter>",
                "reboot<enter>"
            ]
        }
    ],
    "provisioners": [
        {
          "type": "shell",
          "scripts": [
              "./scripts/cleanup_integration_services.sh"
              ]
          }
      ]
}

Inside the http folder is my Ignition file. And the “./scripts/cleanup_integration_services.sh” is simply removing what was done earlier. It disables the service and deletes the file:


#!/bin/bash

sudo systemctl disable hv_kvp_daemon
sudo rm /etc/systemd/system/hv_kvp_daemon

As a side note, this isn’t a Packer tutorial but I thought I should point out a couple of things of importance in this Packer template:

  •  I chose Gen 1 for the Hyper-V generation because CoreOS will not work on Gen 2. Gen 2 requires the OS to support UEFI boot, which CoreOS doesn’t
  • I added an ‘eject’ command’ before the reboot in the boot command because Hyper-V kept booting back to the ISO regardless of what I was setting as the boot order
  • Any variables being called here which are not being listed in the ‘Variables’ section are being passed at run time via -var or -var-file

End

And that’s it. I don’t think this workaround will be required for too long but in the meantime hopefully this can help some folks out.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s