telvm manual

Telnet to these virtual machines like it's 1998

Basics

The overarching goal of telvm is to make it exceptionally easy to login into QEMU virtual machines from the command line, over terminal connections. It is just yet another convenient shim over QEMU invocations.

The data directory

telvm manages a file hierarchy located by default in the $XDG_DATA_DIR/telvm directory.

Most telvm commands that accept file paths can start with an @ to refer to the root of this data directory. The telvm path command outputs paths as resolved by telvm:

telvm path @    # Path to the data directory
telvm path @images/data.img  # images/data.img file in the data directory

telvm has familiar commands ls, cp, mv and rm which interpret these @ path directly, see Working with disk images for more details.

telvm interprets some of the top-level directories of the data directory specially:

Except for the behaviour mentioned above these directories have no more purpose than that. In general feel free to use and organize the data directory the way you see it fit, its yours.

Getting prompts

The telvm run command runs disk images and shares host directories with QEMU. The disk image files you specify on the command line specify the boot order and are attached as drives using the best available device. Directories you specify on the command line get shared with the booted guest OS, see sharing host directories.

Here's a couple examples that should run out of the box without installation process. We copy the image over to @boot for convenience – they get listed on telvm list – but it's not strictly needed. To build and run Windows images see here.

# Debian
# See https://cloud.debian.org/images/cloud/trixie/latest/
curl -L -O …/debian-13-nocloud-arm64.qcow2
telvm cp debian-13-nocloud-arm64.qcow2 @boot/debian-13-arm64.qcow2
telvm run @boot/debian-13-arm64.qcow2  # Login with root, no password

# FreeBSD
# See https://download.freebsd.org/releases/VM-IMAGES/15.0-RELEASE/
curl -L -O …/FreeBSD-15.0-RELEASE-arm64-aarch64-ufs.qcow2.xz
unxz -c FreeBSD-15.0-RELEASE-arm64-aarch64-ufs.qcow2.xz |
     telvm cp - @boot/freebsd-arm64.qcow2
telvm run @boot/freebsd-arm64.qcow2  # Login with root, no password

By default disk images are attached as virtio-blk devices or virtio-scsi-cdrom for those images that end with .iso. For now we have an exception (TODO get rid of it) on the first drive: nvme is used because the Windows bootloader can't see VirtIO devices; use the syntax described below to override this.

If the bootloader or the guest OS have no VirtIO drivers use the option --no-virtio otherwise the drives will not be seen by them. In this case disk images are attached as nvme drives and .iso files as sata-cdrom drives.

Manual overrides are always possible by ending the image with an @DEVICE specification. For example if you rather boot on an USB device or work around the TODO above:

telvm run @boot/debian-13-arm64.qcow2@usb
telvm run @boot/debian-13-arm64.qcow2@virtio-blk # Work around TODO above

See telvm run --help for other types of devices, or try to autocomplete a final @.

If you use telvm run to install an operating system on a blank disk image, specify the target image as the first argument. That way if the operating system reboots it boots on the install. For example:

# See https://alpinelinux.org/downloads/
curl -L -O …/alpine-virt-3.23.2-aarch64.iso
telvm image create @boot/alpine-arm64.img
telvm run @boot/alpine-arm64.img alpine-virt-3.23.2-aarch64.iso

The telvm run command is really just a wrapper for a QEMU invocation, use option --dry-run to see the full QEMU invocation.

The telvm login has the same interface as run but tries to handle the login procedure for you to really get you directly on a prompt.

Sharing host directories

To share a directory of your host directly add it to the run command line. Note however that depending on what you do it may be much faster to make a disk image and run with the disk image.

For now the default mounts it as virtio-9p-pci device, but that needs some operations on the guest and is not supported on Windows, so it may change in the future.

telvm run @boot/debian-13-arm64.qcow2 dir  # login
…
# For the session
mkdir dir
mount -t 9p -o trans=virtio,version=9p2000.L dir dir

# To Persist edit /etc/fstab and
dir /root/dir 9p trans=virtio,version=9p2000.L   0   0
mount -a

For Windows the solution for now is to use an @smb device. You need to make sure that your WinVOS image has the Microsoft-WinVOS-RemoteFS-Package package. It's a bit limited because for now you can share a single directory with it (TODO get rid of this by operating our smbd)

telvm run @boot/winvos-arm64.vhdx dir@smb # login
…
PS C:\Windows\System32> ls \\10.0.2.4\qemu\

Getting graphics

If you still want to access a GUI use the -g and/or the --use-ramfb option. In this case if the guest OS has no USB drivers use the option --no-usb-input.

TODO clarify virtio GPU behaviour.

Random tips

UI tips

Resizing a disk image

On debian.

qemu-img resize bla.qcow2 +2G
telvm run bla.qcow2 # Login
…
apt install cloud-guest-utils
lsblk -p # Spot the block device of the disk $(dev) and partition $(n)
growpart $(dev) $(n)
resize2fs "$(dev)p$(n)"

Working with disk images

The telvm image create command creates disk images. Additionaly telvm provides familiar commands ls, cp, mv and rm that act on the files contained in disk images mostly like their counterpart Unix utilities do. The with-cwd command allows to run an arbitrary tool from your host operating system in the directory of a disk image.

These operations act on the first partition of the disk image which must have an MBR or GPT partition table. The file system of that partition must be moutable by your host operating system. In general using a raw disk image with the exFAT file system is a good way of ensuring this out of the box on at least Linux, MacOS and Windows (FreeBSD needs FUSE). That's what telvm image creates by default. However these disk images may not be bootable if your bootloader can't read exFAT file systems (e.g. the Windows bootloader it seems).

To refer to a path in a disk image just provide the path to the disk image and then continue as if it was a directory. For example to refer to the /usr directory in an image linux.img you write linux.img/usr; shell completion, if enabled, will work too.

telvm's operations do not generally depend on syntactic path directoryness (a.k.a trailing slash). However this is not the case for disk images: linux.img refers to the disk image on the host system while linux.img/ refers to the root of the disk image's file system.

This section ends with a small tutorial on how to use these commands which you can skip if you are not more interested than that – they should also be unsuprising.

# Create a 50 MB raw disk image formatted with exFAT
telvm image create --size 50M @images/data.img
telvm image list                # Spot your image
telvm ls -R @images/data.img    # The disk image file
telvm ls -R @images/data.img/   # The root of the disk image's file system

We created this image in the images directory of the data directory. For this reason it gets listed in the output of telvm image list. Let's add some data in and out from this image:

# Have a file hierarchy
mkdir -p /tmp/hey
echo "Not much" > /tmp/hey/README.md

# Copy the file hierarchy to the disk image
telvm cp -r /tmp/hey @images/data.img/
telvm ls -R @images/data.img/

# Copy the file hierarchy in the disk image back to the host
telvm cp -r @images/data.img/hey /tmp/ho
telvm ls -R /tmp/ho

# The - path can be used for stdin and stdout
telvm cp - @images/data.img/README.md < /tmp/hey/README.md
telvm cp @images/data.img/README.md -

Bear in mind that you can't use globbing on disk image file paths: your shell resolves these on the host file system and doesn't peek into the files. If you want to copy the contents of a directory to another one you can use option -c. This is useful to act on the root of an image to copy it to another one:

telvm image create --size 5M other.img  # in the cwd
telvm cp -r @images/data.img/ other.img/  # Copies to other.img/data.img/
telvm cp -rc @images/data.img/ other.img/ # Copies content to other.img/
telvm ls -R other.img/

To invoke arbitrary tools (e.g. your VCS) in a directory of the disk image use the with-cwd command:

telvm with-cwd @images/data.img/hey -- ls
telvm with-cwd @images/data.img/ -- $SHELL
ls
^D

The with-cwd command behaves differently than telvm's ls, cp, mv and rm commands as far as paths are concerned. Paths that you write after the -- token cannot use @ paths or disk image file completion. They can escape also the mount point so be careful. For example:

telvm cp /tmp/ho @images/data.img/../hey  # Errors

# This writes in the directory above the temporary mount point (if permitted)
telvm with-cwd @images/data.img/ -- cp /tmp/ho ../hey

To cleanup our experiments.

telvm rm -r /tmp/ho              # Delete our data on the host
telvm rm -r @images/data.img/hey # Delete some data from the image
telvm rm @images/data.img        # Delete the image
telvm rm other.img

Windows Validation OS

Basics

telvm has specific support for Windows Validation OS (WinVOS) via the winvos subcommand. In fact this was the purpose of telvm in the first place, to get you a Windows prompt without clicking or administrative fuss.

The support files for WinVOS can be downloaded or checked for freshness by issuing:

telvm winvos update
telvm winvos update --check

The first time you try to create a WinVOS OS image, there will be a bootstrap procedure which is sadly not yet fully unattended (but not too horrible, trust me). The goal is to devise an Windows imager image whose purpose is to create bespoke WinVOS images. It is invoked automatically on create but it can be invoked manually with:

telvm winvos bootstrap

This boots you into a graphical cmd.exe prompt on which you have to find out the right keys on your keyboard to type:

cd /d E:
bootstrap.cmd

After the shutdown you can create new WinVOS images unattended for a given architecture with for example:

telvm winvos create @boot/winvos-arm64.vhdx
telvm winvos create @boot/winvos-x86_64.vhdx

These images can boot into the SAC console with:

telvm run @boot/winvos-arm64.vhdx
telvm run @boot/winvos-x86_64.vhdx

See the next section to login.

If there is any problem you may want to check the log of the last telvm winvos create operation with:

telvm winvos log --last

SAC console login

To login in your WinVOS image via the Windows SAC console:

telvm run @boot/winvos.vhdx

at some point you should get to the SAC console. If the image gets stuck before use the graphical console by invoking with -g and perhaps try these instructions.

In the SAC console wait for the following to happen. Really wait if you are emulating the architecture as it make take a bit of time.

SAC>
EVENT: The CMD command is now available.
SAC>

Follow up with:

SAC> cmd
The Command Prompt session was successfully launched.

Now type ESC-TAB RET and login with admin/1234 or whichever user your build plan configured to create on first boot.

If you succeded up to here, type pwsh from that cmd.exe to get a more (but typed) Unix experience, cp, mv, rm, cd, ls invocations work as expected.

Microsoft Windows [Version 10.0.26100.7309]
Copyright (c) Microsoft Corporation. All rights reserved.

C:\Windows\System32>pwsh
PowerShell 7.5.4
PS C:\Windows\System32>ls
[…]
PS C:\Windows\System32> Invoke-WebRequest -Uri https://example.org
[…]

Notes on enabling SAC from the imager

On Arm64 the SAC console works out of the box once the Microsoft-OneCore-SerialConsole-Package is installed. On x86_64 it needs to be explicitely enabled. We tried to do so by installing the boot files with bcdboot on the .vhdx mounted by QEMU in D: by the imager but it's not picked up by susbequent boots. So we mount the hidden SYSTEM partition of the vhdx by its GUID (TODO brittle currently hardcoded) and enable it on this partition, see the script for details:

telvm winvos plan base -a x86_64 --script

If somehow that plan fails or the hardcoded GUID changed, boot the graphical system with with telvm run -g and invoke in the graphical cmd.exe:

bcdedit /ems on
bcdedit /ems /emssettings emsport:1 emsbaudrate:115200
shutdown /p

Making your own images

To make your own bespoke image you need to write a plan file. For example you can start with the plan file used by default by telvm winvos create:

telvm winvos plan base > myimage.ini

Tweak the plan as you wish, see the available keys and then

telvm winvos create myimage-arm64.vhdx --plan myimage.ini
telvm run my-image-arm64.vhdx

If things seem off see Troubleshooting image creation because for now failures are not reported back to winvos create (TODO).

WinVOS packages

The telvm winvos package helps to inspect WinVOS packages. To list packages simply issue:

telvm winvos package       # List package of your host architecture
telvm winvos package -a x86_64  # Those on x86_64

Use the option --list-contents to see what they contain:

telvm winvos package Microsoft-OneCore-SerialConsole-Package --list-contents

Troubleshooting image creation

The last build log can be consulted with

telvm log --last

To avoid shutdown after imaging and log in into the imager SAC console use:

telvm winvos create myimage-arm64.vhdx --plan myimage.ini -v --plan-no-shutdown
… have fun
C:\Windows\System32> more E:log.txt
C:\Windows\System32> shutdown /p

Bootstrap procedure details

This section documents and reproduces via bare telvm commands the bootstrap procedure performed on telvm winvos boostrap. The goal of the bootstrap procedure is to produce an image that can run DISM for unattended production of bespoke WinVOS images from a non-Windows host.

This is the result of a lot of trials, looking at cryptic errors and hangs. If you know Windows or QEMU any better than we do and see obvious improvements, please get in touch to help the amateur. See also notes and improvements in the TODO.

The imager image needs these WinVOS packages:

We also add to the image the following VirtIO Windows drivers for better virtualisation performance:

The bootstrap procedures runs the X86-64 version of WinVOS regardless of your host architecture because the ValidationOS.vhdx image in the ISO can be booted by QEMU into a graphical cmd.exe with keyboard input via the PS/2 port.

In contrast the ARM ISO cannot be interacted with: ARM systems lack such port and the image has no USB drivers and we did not find any QEMU hardware that could serve for keyboard input. Both images lack direct access to the SAC console.

So first we extract the ValidationOS.vhdx file from the WinVOS X86-64 ISO.

telvm winvos update
telvm cp @telvm/winvos_x86_64.iso/ValidationOS.vhdx ValidationOS_x86_64.vhdx
chmod u+w ValidationOS_x86_64.vhdx

This image is bare bones, it of course lacks VirtIO drivers and USB drivers. But it can be booted with QEMU as an NVMe device into an graphical cmd.exe prompt with keyboard input via PS/2 (the crucial bit that is missing in the ARM64 .vdhx).

If you want to inspect that, execute:

telvm run ValidationOS_x86_64.vhdx -g --no-usb-input --no-virtio

While ISOs can be exposed as SATA optical drives, the OS only sees the the VirtIO ISO which is an ISO 9660 media. It can't mount the WinVOS ISO file because it is an UDF file system and the system lacks a driver for it. It is in the Microsoft-WinVOS-Filesystems-Package.

So we create a temporary exFAT bootsrap.img disk image to which:

So given the host architecture $HOST_ARCH the data image used for bootstraping is created via:

telvm image create --size 2G bootstrap.img
telvm cp -rc @telvm/winvos_$HOST_ARCH.iso/ bootstrap.img/
telvm with-cwd @telvm/winvos_x86_64.iso/cabs/neutral/ -- \
      cabextract -p Microsoft-WinVOS-Provisioning-Package.cab \
      -F '*/bcdboot.exe' | \
      telvm cp - bootstrap.img/bcdboot.exe
telvm winvos plan --script imager | \
      telvm cp - bootstrap.img/bootstrap.cmd

With this bootstrap image we can access the DISM tool distributed in the ISO to build our imager image. Initially we wanted to mount the ValidationOS.vhdx image from the ISO to serve as the base to add packages and drivers but that failed (see the notes). So instead we mount and act on the ValidationOS.wim file and we apply the .wim file on an additional disk image we create before with:

telvm image create --partition-scheme mbr --fs fat32 --size 2G \
      @telvm/winvos-imager-$HOST_ARCH.img

This disk image can't be exFAT as the Windows bootloader doesn't understand it. Since NTFS is not readily available in macOS/Linux we use old Fat32 with an MBR partition scheme (GPT did not seem to sit with the Windows bootloader).

With this in place we boot to get to the graphical cmd.exe

telvm run -g --no-usb-input --no-virtio \
      ValidationOS_x86_64.vhdx \
      @telvm/winvos-imager-$HOST_ARCH.img \
      bootstrap.img \
      @telvm/virtio-win.iso

at which we need to type:

cd /d E:
bootstrap.cmd

When the system shutdowns, the imager image is ready. It can be run to the SAC console with:

telvm run @telvm/winvos-imager-$HOST_ARCH.img   # SAC console only.
telvm run -g @telvm/winvos-imager-$HOST_ARCH.img # Graphical and SAC console

This image is transparently booted and shutdown (via the G:\create-image.cmd script) on telvm winvos create to make WinVOS images.

Plan files

Plan files are .INI files that describe how to build disk images and how to run them. The plan that builds the base WinVOS images can be inspected with

telvm winvos plan base

A plan file is compiled to a target architecture dependent sequence of commands that are invoked by the imager to create the image. You can see the script with:

telvm plan script myplan.ini

and for the base WinVOS image with:

telvm winvos plan base --script
telvm winvos plan base --script -a x86_64

Plan key reference

Note. If an atom has spaces, use double quotes around it.

[]

Top-level keys.

[create]

Section for image creation

[create winvos]

Section dedicated to WinVOS image creation

[create user $NAME]

Section dedicated to create a user named $NAME.

TODO

Next

List

Bootstrap ideas and notes

  1. When we run ValidationOS.vhdx it seems impossible to mount a ValidationOS.vhdx (even from another architecture). We thought it was about the virtdisk.dll and tried to extracted it from Microsoft-WinVOS-DiskTools-Package and add it to the mix but that fails further down the line with a cryptic 0xc03a0014 a virtual disk support provider for the specified file was not found despite that it seems that vhdprovider.dll is around. Suspicion: the disk GUID clashes with the running OS.

  2. Unattended bootstrap (non windows host). Couple of options.

    • Convert ValidationOS.vhdx to a raw image, mount the image (need NTFS support on the host), poke the registry to append the bootstrap script in Userinit of Microsoft\Windows NT\CurrentVersion\Winlogon like we do in the imager. Problem is to find a cross platform tool to do so. On macos nothing seems readily available through brew, hivexsh seems unusable to perform a simple thing like updating a single key.
    • Try an offline install of the Microsoft-OneCore-SerialConsole-Package in order to directly get the SAC console? We'd also need to update the users to be able to log in. Likely not worth pursuing. This remains attended but at least we don't need to go through an emulatinon run
    • Tried to use the various startup folders from a mount of the image to side step but it seems they are not invoked (no explorer.exe shell in winvos ?)
  3. The bootloader(s) need to be able to see the hardware, understand the partitioning scheme and file systems. In particular the Windows bootloader doesn't see virtio based hardware.
  4. Initially rather than create a new admin user to log in to the SAC console we would change the Administrator password. But that would lead to some hangs. The system seems to rely on it being blank

Old notes

Install .msixbundle

dism /online /add-ProvisionedAppxPackage /PackagePath:C:\Path\To\Your\Package\YourAppName.msixbundle /SkipLicense

Add-AppxPackage: -Path pack.msixbundle

Install winget

https://www.codestudy.net/blog/install-winget-by-the-command-line-powershell/

https://gist.github.com/jonahbeckford/1287252fd72d14add44c4f3efb7f9ea9

Install git-for-windows

It would be easier to get in winget to work.

Invoke-WebRequest -Uri https://github.com/git-for-windows/git/releases/download/v2.51.0.windows.2/Git-2.51.0.2-arm64.exe -OutFile GitInstaller.exe
Start-Process -FilePath .\GitInstaller.exe -ArgumentList '/VERYSILENT', '/NORESTART'

Start-Process -FilePath .\GitInstaller.exe -ArgumentList "/SILENT", "/NORESTART", "/DIR=C:\Program Files\Git" -Wait -NoNewWindow
Invoke-WebRequest -Uri https://github.com/git-for-windows/g
                                                                                  git/releases/download/v2.51.0.windows.2/MinGit-2.51.0.2-arm64.zip -OutFile git.zip
Expand-Archive .\git.zip -DestinationPath git

Old bootstrap

The first tentative to bootstrap was using real Windows 11 ISO image with a lot of manual operations based on the procedure in described in this gist and here.

May still be useful if you want a "real" Windows environment. Has not been tested again, the telvm invocations may need adjustments.

Download a Windows 11 ISO image.

If you don't care, match the architecture of your own CPU otherwise it will be excruciately slow. Let WINISO be the path to that ISO file, for example:

WINISO=/tmp/Win11_24H2_EnglishInternational_Arm64.iso
IMG=win11_arm64.qcow2

Continue with:

qemu-img create -f qcow2 $IMG 64G
telvm run $IMG $WINISO @telvm/virtio-win.iso --no-virtio -g

and proceed with these steps. In general if an install reboot gets stuck on the bootloader screen kill the telvm invocation and retry.

  1. You have to hit a key to load the image otherwise you will get into the bootloader shell (you can invoke the bootloader from EFI directory of the ISO in this case).
  2. Proceed until keyboard selection. Press shift-F10 run regedit Navigate to HKEY_LOCAL_MACHINE/SYSTEM/Setup create a key called LabConfig and inside it two DWORDS set to 1: BypassTPMCheck and BypassSecureBootCheck.
  3. Proceed to the drive selection.
  4. Load the virtio-blk driver from E:\viostor\w11\ARM64
  5. The destination disk can now be selected
  6. Continue with installation until it asks for the network
  7. At that point install a couple of divers Virtio drivers, first install the virtio-gpu driver E:\viogpudo\w11\ARM64 also add E:\vioscsi\w11\ARM64 (?) then install the network driver E:\NetKVM\w11\ARM64. Choose the folder with the right architecture.
  8. If at some point it reboots but fails to startup you get stuck on the bootloader screen. At that point kill the vm disable your network and restart with the telvm run command without the --no-virtio flag
  9. When it says 'lost internet connect' use shift-F10 and run oobe\bypasnro. if it gets stuck on reboot again kill the vm and run again
  10. The VM can now be run with telvm run $IMG