telvm manual

Telnet to these virtual machines like it's 1998

Basics

The data directory

telvm manages a file hierarchy in the telvm data-dir directory. It defaults to the $XDG_DATA_DIR/telvm.

Most telvm commands that accept file paths can start with an @ to refer to the root of this data directory. The telvm path command outputs paths as ressolved by telvm:

telvm path @    # Path to the data directory
telvm path @images/data.img  # images/data.img file in the data directory
telvm data-dir  # Output path to the data directory

telvm interprets some of the top-level directories of the data directory specially:

Except for the behaviour mentioned above the @boot and @images directories do not have more purpose than that. In general feel free to use and organize the data directory the way you see it fit.

Working with disk images and files

The telvm image create command allows to create disk images. Additionaly telvm provides familiar commands ls, cp and rm that act on the files contained in disk images like their counterpart unix utilities do. The with-cwd command allows to run an arbitrary tool from your host operating system in the directory of a disk image.

For these operations to work, the disk image and its file system has to be mountable by your host operating system. In general using a raw disk image with the exFAT file system is a good way of ensuring this out of the box on at least Linux, MacOS and Windows. That's what telvm image create does by default. Note however that these disk images may not be bootable if your bootloader can't read exFAT file systems (e.g. the Windows bootloader).

To refer to a path in a disk image just provide the path to the disk image and then continue as if it was a directory for example to refer to the /usr directory in an image linux.img you write linux.img/usr; shell completion, if installed, will work too.

While telvm's operations do not generally depend on syntactic path directoryness (a.k.a trailing slash), this is not the case for disk images: linux.img refers to the disk image on the host system while linux.img/ refers to the root of the disk image's file system.

This section ends with a small tutorial on how to use these commands which you can skip if you are not more interested than that.

# Create a 50 MB raw disk image formatted with exFAT
telvm image create --size 50M @images/data.img
telvm image list                # Spot your image
telvm ls -R @images/data.img    # The disk image file
telvm ls -R @images/data.img/   # The root of the disk image's file system

We created this image in the images directory of the data directory this is the reason why it appears in the output of telvm image list. Let's add some data in and out from this image:

# Have a file hierarchy
mkdir -p /tmp/hey
echo "Not much" > /tmp/hey/README.md

# Copy the file hierarchy to the disk image
telvm cp -r /tmp/hey @images/data.img/
telvm ls -R @images/data.img/

# Copy the file hierarchy in the disk image back to the host
telvm cp -r @images/data.img/hey /tmp/ho
telvm ls -R /tmp/ho

# The - path can be used for stdin and stdout
telvm cp - @images/data.img/README.md < /tmp/hey/README.md
telvm cp @images/data.img/README.md -

Note that you can't use globbing on disk image file paths since your shell resolves these on the host file system which doesn't see the temporary mounts. If you want to copy the contents of a directory to another one you can use option -c. This is useful to act on the root of an image to copy it to another one:

telvm image create --size 5M other.img  # in the cwd
telvm cp -r @images/data.img/ other.img/  # Copies to other.img/data.img/
telvm cp -rc @images/data.img/ other.img/ # Copies content to other.img/
telvm ls -R other.img/

To invoke arbitrary tools (e.g. your VCS) in a directory of the disk image use the with-cwd command:

telvm with-cwd @images/data.img/hey -- ls
telvm with-cwd @images/data.img/ -- $SHELL
ls
^D

The with-cwd command behaves differently than telvm's cp, ls and rm commands as far as paths are concerned. Paths that you write after the -- token cannot use @ paths and disk image file hierarchy completion. Besides they can escape the mount point. For example:

telvm cp /tmp/ho @images/data.img/../hey  # Errors

# This writes in the directory above the temporary mount point (if permitted)
telvm with-cwd @images/data.img/ -- cp /tmp/ho ../hey

To cleanup our experiments.

telvm rm -r /tmp/ho              # Delete our data on the host
telvm rm -r @images/data.img/hey # Delete some data from the image
telvm rm @images/data.img        # Delete the image
telvm rm other.img

Getting prompts

The telvm run command runs disk images with QEMU. The disks you specify on the command line specify the boot order and the command takes care to attach them using the best available interface. Use option --dry-run to see the full QEMU invocation.

# See https://alpinelinux.org/downloads/
curl -L -O …/alpine-virt-3.23.2-aarch64.iso
telvm image create @boot/alpine.img
telvm run @boot/linux.img alpine-virt-3.23.2-aarch64.iso

If you use telvm run to install an operating system on a blank disk image as in the invocation above, specify the target image as the first argument. That way if the operating system reboots it boots on the install.

If the booted OS has no virtio drivers use the option --no-virtio, if it has no USB drivers use the option --no-usb-input. To force drives to be attached in some way or the other use the options --use-nvme-drives or --use-usb-drives, TODO remove these options.

If you still want to access a full GUI use the -g option.

Disks and virtio devices

In general telvm tries to use virtio devices on run. However this means that your bootloader needs to be able to interact with them. This is notably not the case of Windows bootloader so the disk image with Windows should not be specified as a virtio block device.

Random UI tips

Windows Validation OS

Basics

telvm has specific support for Windows Validation OS via the winvos subcommand. In fact this was the purpose of telvm in the first place.

The support files for Windows Validation OS can be downloaded or checked for freshness by issuing:

telvm winvos update
telvm winvos update --check

The first time you try to create a Windows Validation OS image, there will be a bootstrap procedure which is not fully unattended (yet). It is invoked automatically on create but it can be invoked manually with:

telvm winvos bootstrap

This will boot you into a graphical cmd.exe prompt on which you will have to find out the right keyboard keys to write:

cd /d E:
bootstrap.cmd

After the shutdown you can create new Windows validation OS image unattended with for example:

telvm winvos create @boot/winvos.img

This image can boot into the SAC console with:

telvm run @boot/winvos.img

SAC console login

To login in the SAC console boot your winvos image:

telvm run @boot/winvos.img

and wait for the message that says that CMD became available.

Then type cmd followed by ESC-TAB RET and login with admin/1234 or whichever user your build plan configured to create on first boot.

More details about the SAC console can be found in the Microsoft documentation.

Making your own images

To make your own bespoke image you need to write a plan file file. For example you can inspect the plan file used by default on telvm winvos create with.

telvm winvos packages       # On your host architecture
telvm winvos packages -a x86_64

Use the option --list-contents to see what they contain (that needs the cabextract tool in your PATH):

telvm winvos packages Microsoft-OneCore-SerialConsole-Package \
      -a x86_64 --list-contents

Bootstrap procedure details

This section documents and reproduces via bare telvm commands the bootstrap procedure performed on telvm winvos boostrap. The goal of the bootstrap procedure is to produce an image that can run DISM for unattended production of bespoke Windows Validation OS images from a non-Windows host.

This is the result of a lot of trials, (often cryptic) errors and hangs. If you know Windows or QEMU any better than we do and see obvious improvements, please get in touch to help the amateur.

The imager image needs these Validation OS packages:

We also add to the image the following VirtIO Windows drivers for better virtualisation performance:

The bootstrap procedures runs the X86-64 version of Validation OS regardless of your host architecture because the ValidationOS.vhdx image in the ISO can be booted by QEMU into a graphical cmd.exe with keyboard input via the PS/2 port. In contrast the ARM ISO cannot be interacted with: ARM systems lack such port and the image has no USB drivers and we did not find any QEMU hardware that could serve for keyboard input. Both images also lack support for the SAC console which is in the Microsoft-OneCore-SerialConsole package.

So first we extract the ValidationOS.vhdx file from the Validation OS X86-64 ISO.

telvm winvos update
telvm cp @winvos/winvos_x86_64.iso/ValidationOS.vhdx ValidationOS_x86_64.vhdx
chmod u+w ValidationOS_x86_64.vhdx

This image is bare bones, it lacks VirtIO drivers (of course) and USB drivers but it can be booted with QEMU as an NVMe device into an graphical cmd.exe prompt with keyboard input via PS/2 (the crucial bit that is missing in ARM64). If you want to try that execute:

telvm run ValidationOS_x86_64.vhdx -g --no-usb-input --no-virtio

While ISOs can be exposed as SATA optical drives, the OS only sees the the VirtIO ISO which is an ISO 9660 media. It can't mount the Validation OS ISO file because it is an UDF file system and the system lacks a driver for it. It is in the Microsoft-WinVOS-Filesystems-Package.

So we create a temporary exFAT bootsrap.img disk image to which:

So given the host architecture $HOST_ARCH the data image used for bootstraping is created via:

telvm image create --size 2G bootstrap.img
telvm cp -rc @winvos/winvos_$HOST_ARCH.iso/ bootstrap.img/
telvm with-cwd @winvos/winvos_x86_64.iso/cabs/neutral/ -- \
      cabextract -p Microsoft-WinVOS-Provisioning-Package.cab \
      -F '*/bcdboot.exe' | \
      telvm cp - bootstrap_arm.img/bcdboot.exe
telvm winvos bootstrap --show-script | \
      telvm cp - bootstrap.img/bootstrap.cmd

With this we can access the DISM tool distributed in the ISO to build our imager image. Initially we wanted to mount the ValidationOS.vhdx image from the ISO to serve as the base to add packages and drivers but that failed (see the notes). So instead we mount and act on the ValidationOS.wim file and we apply the .wim file on an additional disk image we create before. This disk image can't be ExFAT as the Windows bootloader doesn't understand it, since NTFS is not readily available in macOS/Linux we use old Fat32 with an MBR partition scheme (GTP did not seem to sit with the Windows bootloader).

telvm image create --partition-scheme mbr --fs fat32 --size 2G \
      @winvos/imager_$HOST_ARCH.img

telvm run -g --no-usb-input --no-virtio \
      ValidationOS_x86_64.vhdx \
      @winvos/imager_$HOST_ARCH.img \
      bootstrap.img \
      @winvos/virtio-win.iso

With this in place we boot to get to the graphical cmd.exe at which we type:

cd /d E: && boostrap.cmd

When the system shutdowns. The system should be ready, it can be run to the SAC console with:

telvm run @winvos/imager_$HOST_ARCH.img   # SAC console only.
telvm run -g @winvos/imager_$HOST_ARCH.img # Graphical and SAC console

Ideas and notes

  1. When we run ValidationOS.vhdx it seems impossible to mount a ValidationOS.vhdx (even from another architecture). We thought it was about the virtdisk.dll and tried to extracted it from Microsoft-WinVOS-DiskTools-Package and add it to the mix but that fails further down the line with a cryptic 0xc03a0014 a virtual disk support provider for the specified file was not found despite that it seems that vhdprovider.dll is around. Suspicion: the disk GUID clashes with the running OS.

  2. Unattended bootstrap (non windows host). Couple of options.

    • Convert ValidationOS.vhdx to a raw image, mount the image (need NTFS support on the host), poke the registry to append the bootstrap script in Userinit of Microsoft\Windows NT\CurrentVersion\Winlogon like we do in the imager. Problem is to find a cross platform tool to do so. On macos nothing seems readily available through brew, hivexsh seems unusable to perform a simple thing like updating a single key.
    • Try an offline install of the Microsoft-OneCore-SerialConsole-Package in order to directly get the SAC console? We'd also need to update the users to be able to log in. Likely not worth pursuing.
    • Tried to use the various startup folders to side step but it seems they are not invoked (no explorer.exe shell in winvos ?)
  3. The bootloader(s) need to be able to see the hardware, understand the partitioning scheme and file systems. In particular the Windows bootloader doesn't see virtio based hardware.
  4. Initially rather than create a new admin user to log in to the SAC console we would change the Administrator password. But that would lead to some hangs it seems something relies on it being blank

Old bootstrap

The first tentative to bootstrap was using real Windows 11 ISO image with a lot of manual operations based on the procedure in described in this gist and here.

May still be useful if you want a "real" Windows environment. Has not been tested again, the telvm invocations may need adjustments.

Download a Windows 11 ISO image.

If you don't care, match the architecture of your own CPU otherwise it will be excruciately slow. Let WINISO be the path to that ISO file, for example:

WINISO=/tmp/Win11_24H2_EnglishInternational_Arm64.iso
IMG=win11_arm64.qcow2

Continue with:

qemu-img create -f qcow2 $IMG 64G
telvm run $IMG $WINISO @winvos/virtio-win.iso --no-virtio -g

and proceed with these steps. In general if an install reboot gets stuck on the bootloader screen kill the telvm invocation and retry.

  1. You have to hit a key to load the image otherwise you will get into the bootloader shell (you can invoke the bootloader from EFI directory of the ISO in this case).
  2. Proceed until keyboard selection. Press shift-F10 run regedit Navigate to HKEY_LOCAL_MACHINE/SYSTEM/Setup create a key called LabConfig and inside it two DWORDS set to 1: BypassTPMCheck and BypassSecureBootCheck.
  3. Proceed to the drive selection.
  4. Load the virtio-blk driver from E:\viostor\w11\ARM64
  5. The destination disk can now be selected
  6. Continue with installation until it asks for the network
  7. At that point install a couple of divers Virtio drivers, first install the virtio-gpu driver E:\viogpudo\w11\ARM64 also add E:\vioscsi\w11\ARM64 (?) then install the network driver E:\NetKVM\w11\ARM64. Choose the folder with the right architecture.
  8. If at some point it reboots but fails to startup you get stuck on the bootloader screen. At that point kill the vm disable your network and restart with the telvm run command without the --no-virtio flag
  9. When it says 'lost internet connect' use shift-F10 and run oobe\bypasnro. if it gets stuck on reboot again kill the vm and run again
  10. The VM can now be run with telvm run $IMG

Plan files

Plan files are .INI files that describe how to build disk images and how to run them.

Key reference

Old notes

Run at startup

The image build script distributed with ValidationOS seems to indicate it's not the case.

reg add "HKLM\Software\Windows NT\CurrentVersion\Winlogon" /v Shell /t REG_EXPAND_SZ /d "C:\Windows\System32\cmd.exe /k D:\mk-image.cmd" /f

https://learn.microsoft.com/en-us/windows/win32/setupapi/run-and-runonce-registry-keys

dism.exe /online /add-ProvisionedAppxPackage /PackagePath:C:\Path\To\Your\Package\YourAppName.msixbundle /SkipLicense

Powershell bootstrap

Try to install winget

Note with -Scope AllUsers modules are installed in \Program Files\PowerShell\Modules. Apparently this is not in the path add:

$Env:PSModulePath = $Env:PSModulePath+";C:\Program Files\PowerShell\Modules"

TODO try to find a way to this automatically via the registry

Install-Module -Name Microsoft.WinGet.Client -Force -Repository PSGallery -Scope AllUsers -SkipPublisherCheck
Import-Module Microsoft.WinGet.Client
Repair-WinGetPackageManager -AllUsers

This fails with:

Repair-WinGetPackageManager: The term 'Add-AppxProvisionedPackage' is not recognized as a name of a cmdlet, function, script file, or executable program.
Check the spelling of the name, or if a path was included, verify that the path is correct and try again.

I suspect we are missing software from the `Microsoft-WinVOS-Deployment-Package` however installing it freezes the system (note it's labeled as experimental in the packages).

https://github.com/microsoft/winget-cli/releases/download/v1.11.510/Microsoft.DesktopAppInstaller_8wekyb3d8bbwe.msixbundle

Install winget

https://www.codestudy.net/blog/install-winget-by-the-command-line-powershell/

Install git-for-windows

It would be easier to get in wget to work.

Invoke-WebRequest -Uri https://github.com/git-for-windows/git/releases/download/v2.51.0.windows.2/Git-2.51.0.2-arm64.exe -OutFile GitInstaller.exe
Start-Process -FilePath .\GitInstaller.exe -ArgumentList '/VERYSILENT', '/NORESTART'


Start-Process -FilePath .\GitInstaller.exe -ArgumentList "/SILENT", "/NORESTART", "/DIR=C:\Program Files\Git" -Wait -NoNewWindow
Invoke-WebRequest -Uri https://github.com/git-for-windows/g
                                                                                  git/releases/download/v2.51.0.windows.2/MinGit-2.51.0.2-arm64.zip -OutFile git.zip
Expand-Archive .\git.zip -DestinationPath git

Update PATH

:: Update %PATH% in registry
reg load HKLM\TTYWIN %MOUNT%\Windows\System32\config\SYSTEM
set key="HKLM\TTYWIN\ControlSet001\Control\Session Manager\Environment"
for /f "tokens=2*" %%A in ('reg query %key% /v Path') do set value=%%B
:: Avoid %SystemRoot% expansion
setlocal enabledelayedexpansion
set value=!value!;%%systemroot%%\System32\powershell
reg add !key! /v Path /t REG_EXPAND_SZ /d "!value!" /f
endlocal
reg unload HKLM\TTYWIN