If you try to perform Packer builds of CoreOS on Hyper-V, you’re going to have issues. This post describes a specific network issue and offers a temporary solution which allows Packer builds to complete successfully.
I say temporary because the issue can be resolved by CoreOS fully implementing Hyper-V Integration Services (hopefully soon) or by Packer adding support for the Hyper-V builder without requiring Integration Services (please)
But until there is a real solution this can help get around it.
The Problem
The TL;DR: The Hyper-V Packer builder requires Integration Services in order to SSH into a Linux guest. CoreOS does not fully implement these services. So Packer cannot build Hyper-V images for CoreOS.
To explain a bit more….
Successfully running the provisioner steps in a packer build requires SSH. To use SSH, Packer needs to know the IP address of the VM. The way that Packer finds the IP address of a Hyper-V VM is by getting using the mac address attached to the adapter and then filtering for a VM with has that mac address.
You can see this process by looking at the code here: https://github.com/hashicorp/packer/blob/master/common/powershell/hyperv/hyperv.go#L967
The problem is that this Powershell cmdlet requires Hyper-V Guest Integration Services to be fully implemented. Unfortunately, at the time of this post, CoreOS does not fully implement the guest services.
Specifically, it is missing this: https://www.freebsd.org/cgi/man.cgi?query=hv_kvp_daemon&sektion=8
As explained here
hv_kvp_daemon is the userspace component of the Hyper-V key value pair
functionality, communicating via a netlink socket with the kernel HV-
KVP driver. This pairing allows the Hyper-V host to pass configuration
information (such as IP addresses) to the guest and allows the host to
obtain guest version information.
Even though some of the Hyper-V components are provided within the Linux kernel, some parts need to be acquired by downloading the Linix Cloud Tools package. This is available but not officially for CoreOS.
If you have a CoreOS VM on Hyper-V that has an IP address and is pingable, this is still what you will see in the Hyper-V management console
And when you query the VM in Powershell in a similar way that Packer does
No IP address is found and the Packer build will fail after the ssh_timeout is hit.
The Solution
As mentioned at the very beginning — there is no perfect solution at this time. But there is a workaround which involves temporarily adding the hv_kvp_daemon into CoreOS.
The basic workflow here is:
- Get hv_kvp_daemon from some place
- Store this file on a FTP or SSH server
- Add the daemon file to the image and add the corresponding unit file to the image during the Ignition run
- Add a task in Packer to remove this awful workaround
Steps 1 and 2 are just making available the file which needs to be added to CoreOS. Step 3 is a way of inject the file and creating the unit file to make it into a startup service. Running this service allows Packer to detect the IP of the VM and SSH in for the provisioning step.
Step 4 is just removing the service and unit file because I am not comfortable leaving this on the system.
There are numerous ways to implement this workaround, and while I don’t necessarily think I approached it correctly (especially step 1) — the principle remains the same. And most importantly — it works.
Getting the Integration Services Files
Somehow, hv_kvp_daemon needs to to be placed on the system. There are different ways to get this file and there are different ways to get it onto the CoreOS VM. (you could just use gcc and grab this)
Since I was already on a CoreOS host, I figured I would just start an Ubuntu container, grab the file, and throw it on a web server.
You could either map a volume to the Ubuntu container or just scp the file back to the host from the container. I decided to use a named mount. Keep in mind, if you are on an older version of CoreOS with an older version of Docker you will not be able to use named mounts.
I ran the following command
docker run -v shared:/usr/lib -it ubuntu
In the container, I then updated the repos and installed the Linux Cloud Tools
apt-get update sudo apt-get install --install-recommends linux-tools-virtual-lts-xenial linux-cloud-tools-virtual-lts-xenial
To find the location of the file we need, just run the find command
The middle one is the needed file (…-generic/hv_kvp_daemon).
In my approach, I am going to be serving this to Ignition from an HTTP server. This means I needed to get the file out of the container. Since I did the volume mount, I figured I would just grab the file off the host. Another way would have been to port map when running the container and spinup a quick FTP server on it.
I then exited the container and made sure the file was there
There are plenty of ways to get the file off of the VM. Since I had Mobaxterm handy, I just set a password for the core account so that I could SSH into it and grabbed the file with Moba’s file explorer (I did have to copy it to /home/core first).
To change the password
sudo -i passwd core
And then I connected with Moba and downloaded the hv_kvp_daemon
I then threw it on an internal web server in a a place which did not require any authentication.
Ignition
The next step is to use Ignition to create a unit file and copy the file from the last step to the system during the install process.
The Ignition file will be served to the CoreOS via the Packer built-in web server. The Ignition file then specifies the location of the needed hv_kvp_daemon file. Additionally, the Ignition file sets the content of the unit file needed to run the daemon at startup.
The Ignition file looks like this (though you should use YAML and the config transpiler:
{
"ignition": {
"version": "2.1.0",
"config": {}
},
"storage": {
"files": [
{
"filesystem": "root",
"path": "/etc/systemd/system/hv_kvp_daemon",
"mode": 775,
"contents": {
"source": "http://downloads.mhickok.me/coreos/hv_kvp_daemon"
}
}]
},
"systemd": {
"units": [
{
"name": "hv_kvp_daemon.service",
"enabled": true,
"contents": "[Unit]\nDescription=Hyper-V KVP daemon\n\n[Service]\nType=simple\nExecStart=/etc/systemd/system/hv_kvp_daemon -n\n\n[Install]\nWantedBy=multi-user.target"
}]
},
"networkd": {
},
"passwd": {
"users": [
{
"name": "core",
"passwordHash": "sdal;skmdaksmd;lkmmoi23jkl",
"sshAuthorizedKeys": [
"ssh-rsa somekey"
]
}
]
}
}
This is doing two main things:
- Downloading the hv_kvp_daemon file from my file server and copying it (with execute permissions) to the systemd folder.
- Adding a unit file which will enable this binary to run as a service at startup
Keep in mind that the location and permissions may not be perfect here but it doesn’t matter because the changes will immediately be undone by a script which I run with Packer.
Lastly, I stored this Ignition file in a folder which will be referenced during the packer build later.
Packer
This part is very standard as far as Packer builds go. It has all of the normal Hyper-V builder values but it references the Ignition file and has a final step which removes the workaround.
{
"variables": {
"iso_url": "https://stable.release.core-os.net/amd64-usr/current/coreos_production_iso_image.iso",
"checksum_type": "md5",
"checksum": "d97ed6c250f2037b6fd92471c2890176",
"ssh_timeout": "10m",
"hyperv_switch": "vSwitch"
},
"builders": [
{
"type": "hyperv-iso",
"vm_name": "{{user `vm_name`}}",
"ssh_username": "{{user `core_user`}}",
"ssh_password": "{{user `core_pass`}}",
"iso_checksum": "{{user `checksum`}}",
"iso_checksum_type": "{{user `checksum_type`}}",
"iso_url": "{{user `iso_url`}}",
"ssh_timeout": "{{user `ssh_timeout`}}",
"http_directory": "http",
"generation": 1,
"ram_size": 4096,
"enable_dynamic_memory": false,
"enable_virtualization_extensions": true,
"cpu": 2,
"disk_size": 8192,
"switch_name":"{{user `hyperv_switch`}}",
"boot_wait": "2m",
"shutdown_command": "sudo -S shutdown -P now",
"boot_command": [
"sudo -i<enter>",
"systemctl stop sshd.socket<enter>;",
"wget http://{{ .HTTPIP }}:{{ .HTTPPort }}/ignition_hyper.json<enter>",
"coreos-install -d /dev/sda -C stable -i ignition_hyper.json<enter>",
"eject<enter>",
"reboot<enter>"
]
}
],
"provisioners": [
{
"type": "shell",
"scripts": [
"./scripts/cleanup_integration_services.sh"
]
}
]
}
Inside the http folder is my Ignition file. And the “./scripts/cleanup_integration_services.sh” is simply removing what was done earlier. It disables the service and deletes the file:
#!/bin/bash sudo systemctl disable hv_kvp_daemon sudo rm /etc/systemd/system/hv_kvp_daemon
As a side note, this isn’t a Packer tutorial but I thought I should point out a couple of things of importance in this Packer template:
- I chose Gen 1 for the Hyper-V generation because CoreOS will not work on Gen 2. Gen 2 requires the OS to support UEFI boot, which CoreOS doesn’t
- I added an ‘eject’ command’ before the reboot in the boot command because Hyper-V kept booting back to the ISO regardless of what I was setting as the boot order
- Any variables being called here which are not being listed in the ‘Variables’ section are being passed at run time via -var or -var-file
End
And that’s it. I don’t think this workaround will be required for too long but in the meantime hopefully this can help some folks out.
