ethonion10/v86
1
0
Fork 0
Code Issues Pull requests Packages Projects Releases Wiki Activity

Update archlinux build docs (#613)

Browse source 
Update archlinux build docs, add throttling note in networking
This commit is contained in:
Taras Vozniuk 2022-02-04 06:14:35 +08:00 committed by GitHub
parent 503eeb0921
commit 8a0e1955c9
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
2 changed files with 204 additions and 74 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

276
docs/archlinux.md
View file

@ -13,11 +13,14 @@ Basic installation using QEMU
Installing Archlinux like this will result in a raw disk image that can be booted by v86.
```sh
# fetch archlinux32 installer
wget https://mirror.archlinux32.org/archisos/archlinux32-2021.12.01-i686.iso
# Create a 10 gigabyte disk image. If you intend to pacstrap only 'base' then 1.5G should be fine also.
qemu-img create arch.img 10G
# Follow the normal installation process (you can add accel=kvm if your system supports it to speed up the installation)
qemu-system-x86_64 -m 256 -drive file=arch.img,format=raw -cdrom archlinux-2017.02.01-dual.iso
qemu-system-x86_64 -m 256 -drive file=arch.img,format=raw -cdrom archlinux32-2021.12.01-i686.iso
```
For keyboard support it is necessary to open /etc/mkinitcpio.conf and edit the following line:
@ -56,35 +59,39 @@ cat > packer/template.json << 'EOF'
],
"builders": [
{
"accelerator": "kvm",
"type": "qemu",
"boot_command": [
"<enter><wait10>",
"dhcpcd<enter><wait10>",
"usermod --password $(echo toor | openssl passwd -1 -stdin) root<enter><wait10>",
"<enter><wait30><enteropenssl passwd help<wait10>",
"dhcpcd<enter><wait5>",
"usermod --password $(echo root | openssl passwd -1 -stdin) root<enter><wait5>",
"echo 'root:magic' | chpasswd<enter><wait5>",
"systemctl start sshd<enter>"
],
"headless": true,
"boot_wait": "10s",
"disk_size": 1500,
"disk_interface": "ide",
"http_directory": "http",
"iso_url": "https://mirror.archlinux32.org/archisos/archlinux-2017.04.01-i686.iso",
"iso_checksum": "aa4718837c95e607233aecca43824c08c798b1a4",
"iso_url": "https://mirror.archlinux32.org/archisos/archlinux32-2021.12.01-i686.iso",
"iso_checksum": "90c6f5aecb095d5578f6c9970539da7c5e9324ec",
"iso_checksum_type": "sha1",
"ssh_wait_timeout": "120s",
"ssh_pty": true,
"ssh_username": "root",
"ssh_password": "toor",
"ssh_password": "magic",
"ssh_port": 22,
"format": "raw",
"vm_name": "Archlinux-v86"
"vm_name": "archlinux",
"disk_detect_zeroes": "unmap",
"memory": 2048,
"vnc_bind_address": "0.0.0.0"
}
]
}
EOF
```
You can tweak the options a bit to match your situation. For debugging you can set `headless` to `false`. That will show you the vnc instead of running the `boot_command` in the background. For a `base` pacstrap using a 1.5G disk should be sufficient. The `raw` disk format is important. v86 does not read qcow2 images, only raw disk images. If your system does not support kvm (the default accelerator), you can add `"accelerator": "none"` to the settings. Other accelerator options can be found [here](https://www.packer.io/docs/builders/qemu.html#accelerator).
You can tweak the options a bit to match your situation. For debugging you can set `headless` to `false`. That will show you the vnc instead of running the `boot_command` in the background. For a `base` pacstrap using a 1.5G disk should be sufficient. The `raw` disk format is important. v86 does not read qcow2 images, only raw disk images. If your system does not support kvm (the default accelerator), you can change `"accelerator": "none"` to the settings, in macos you may use `"accelerator": "hvf"`. Other accelerator options can be found [here](https://www.packer.io/docs/builders/qemu.html#accelerator).
After gaining SSH connectivity to the VM, packer will run the `scripts/provisioning.sh` script in the guest.
@ -120,9 +127,15 @@ mkdir -p /mnt/var/cache/pacman/pkg
echo "Mount the package cache dir in memory so it doesn't fill up the image"
mount -t tmpfs none /mnt/var/cache/pacman/pkg
# Install the Archlinux base system
echo "Updating archlinux-keyring"
pacman -Sy archlinux-keyring --noconfirm
# uncomment to remove signing if unable to resolve signing errors
sed -i 's/SigLevel.*/SigLevel = Never/g' /etc/pacman.conf
# Install the Archlinux base system, feel free to add packages you need here
echo "Performing pacstrap"
pacstrap -i /mnt base --noconfirm
pacstrap -i /mnt base linux dhcpcd curl openssh --noconfirm
echo "Writing fstab"
genfstab -p /mnt >> /mnt/etc/fstab
@ -136,10 +149,10 @@ ExecStart=
ExecStart=-/usr/bin/agetty --autologin root --noclear %I $TERM
EOF
# This is the tricky part. The current root will be mounted on /dev/sda1 but after we reboot
# This is the tricky part. The current root will be mounted on /dev/sda1 but after we reboot
# it will try to mount root during boot using the 9p network filesystem. This means the emulator
# will request all files over the network using XMLHttpRequests from the server. This is great
# because then you only need to provide the client with a saved state (the memory) and the
# will request all files over the network using XMLHttpRequests from the server. This is great
# because then you only need to provide the client with a saved state (the memory) and the
# session will start instantly and load needed files on the fly. This is fast and it saves bandwidth.
echo "Ensuring root is remounted using 9p after reboot"
mkdir -p /mnt/etc/initcpio/hooks
@ -150,7 +163,7 @@ run_hook() {
mount_9p_root() {
msg ":: mounting '$root' on real root (9p)"
# Note the host9p. We won't mount /dev/sda1 on root anymore,
# Note the host9p. We won't mount /dev/sda1 on root anymore,
# instead we mount the network filesystem and the emulator will
# retrieve the files on the fly.
if ! mount -t 9p host9p "$1"; then
@ -175,38 +188,120 @@ EOF
# want to use the network filesystem you only need these. The 9p, 9pnet and 9pnet_virtio
# modules are needed for being able to mount 9p network filesystems using the emulator.
echo "Configure mkinitcpio for 9p"
sed -i 's/MODULES=""/MODULES="atkbd i8042 virtio_pci 9p 9pnet 9pnet_virtio"/g' /mnt/etc/mkinitcpio.conf
# Because we want to mount the root filesystem over the network during boot, we need to
sed -i 's/MODULES=()/MODULES=(atkbd i8042 psmouse virtio_pci 9p 9pnet 9pnet_virtio)/g' /mnt/etc/mkinitcpio.conf
# dump mkinitcpio.conf to check if MODULES get actually set
cat /mnt/etc/mkinitcpio.conf
# Because we want to mount the root filesystem over the network during boot, we need to
# hook into initcpio. If you do not want to mount the root filesystem during boot but
# only want to mount a 9p filesystem later, you can leave this out. Once the system
# has been booted you should be able to mount 9p filesystems with mount -t 9p host9p /blabla
# without this hook.
sed -i 's/fsck"/fsck 9p_root"/g' /mnt/etc/mkinitcpio.conf
# enable ssh password auth and root login
sed -i 's/#PermitRootLogin.*/PermitRootLogin yes/g' /etc/ssh/sshd_config
sed -i 's/#PasswordAuthentication.*/PasswordAuthentication yes/g' /etc/ssh/sshd_config
echo "Writing the installation script"
cat << 'EOF' > /mnt/bootstrap.sh
#!/usr/bin/bash
echo "Re-generate initial ramdisk environment"
mkinitcpio -p linux
echo "Installing the grub package"
pacman -S os-prober grub --noconfirm
# uncomment to remove signing if you are unable to resolve signing errors otherwise
sed -i 's/SigLevel.*/SigLevel = Never/g' /etc/pacman.conf
echo "Setting grub timeout to 0 seconds"
sed -i 's/GRUB_TIMEOUT=5/GRUB_TIMEOUT=0/g' /etc/default/grub
pacman -S --noconfirm syslinux gptfdisk
syslinux-install_update -i -a -m
echo "Installing bootloader"
grub-install --target=i386-pc --recheck /dev/sda --force
# disabling ldconfig to speed up boot (to remove Rebuild dynamic linker cache...)
# you may want to comment this out
echo "Disabling ldconfig service"
systemctl mask ldconfig.service
echo "Writing grub config"
grub-mkconfig -o /boot/grub/grub.cfg
sync
EOF
echo "Chrooting and bootstrapping the installation"
arch-chroot /mnt bash bootstrap.sh
cat << 'EOF' > /mnt/boot/syslinux/syslinux.cfg
# Config file for Syslinux -
# /boot/syslinux/syslinux.cfg
#
# Comboot modules:
# * menu.c32 - provides a text menu
# * vesamenu.c32 - provides a graphical menu
# * chain.c32 - chainload MBRs, partition boot sectors, Windows bootloaders
# * hdt.c32 - hardware detection tool
# * reboot.c32 - reboots the system
#
# To Use: Copy the respective files from /usr/lib/syslinux to /boot/syslinux.
# If /usr and /boot are on the same file system, symlink the files instead
# of copying them.
#
# If you do not use a menu, a 'boot:' prompt will be shown and the system
# will boot automatically after 5 seconds.
#
# Please review the wiki: https://wiki.archlinux.org/index.php/Syslinux
# The wiki provides further configuration examples
DEFAULT arch
PROMPT 0 # Set to 1 if you always want to display the boot: prompt
TIMEOUT 100
# Menu Configuration
# Either menu.c32 or vesamenu32.c32 must be copied to /boot/syslinux
UI menu.c32
#UI vesamenu.c32
# Refer to http://syslinux.zytor.com/wiki/index.php/Doc/menu
MENU TITLE Arch Linux
#MENU BACKGROUND splash.png
MENU COLOR border 30;44 #40ffffff #a0000000 std
MENU COLOR title 1;36;44 #9033ccff #a0000000 std
MENU COLOR sel 7;37;40 #e0ffffff #20ffffff all
MENU COLOR unsel 37;44 #50ffffff #a0000000 std
MENU COLOR help 37;40 #c0ffffff #a0000000 std
MENU COLOR timeout_msg 37;40 #80ffffff #00000000 std
MENU COLOR timeout 1;37;40 #c0ffffff #00000000 std
MENU COLOR msg07 37;40 #90ffffff #a0000000 std
MENU COLOR tabmsg 31;40 #30ffffff #00000000 std
# boot sections follow
#
# TIP: If you want a 1024x768 framebuffer, add "vga=773" to your kernel line.
#
#-*
LABEL arch
MENU LABEL Arch Linux 9p
LINUX ../vmlinuz-linux
APPEND root=/dev/sda1 rw quiet
INITRD ../initramfs-linux.img
LABEL arch2
MENU LABEL Arch Linux Disk
LINUX ../vmlinuz-linux
APPEND root=/dev/sda1 rw quiet disablehooks=9p_root
INITRD ../initramfs-linux.img
LABEL hdt
MENU LABEL HDT (Hardware Detection Tool)
COM32 hdt.c32
LABEL reboot
MENU LABEL Reboot
COM32 reboot.c32
LABEL poweroff
MENU LABEL Poweroff
COM32 poweroff.c32
EOF
umount -R /mnt
```
@ -216,14 +311,7 @@ With the packer template and the script you have enough to create an image that
Now that we have an image that contains a filesystem, we can convert that filesystem into something we can host on the webserver together with the v86 library.
To do so, we need to mount the image once and create a json mapping of the filesystem. This can be done with the [fs2json](https://github.com/copy/fs2json) python script.
Get the script:
```sh
wget https://raw.githubusercontent.com/copy/fs2json/master/fs2json.py
```
The following script shows how to map the filesystem in an automated fashion.
To do so, we need to mount the image once and create a json mapping of the filesystem. The following script shows how to map the filesystem in an automated fashion.
Create a script to builds the image and then creates v86 compatible artifacts:
```sh
@ -235,82 +323,114 @@ Example script:
```sh
#!/bin/sh
SRC=packer
TARGET=output
# build the boxfile from the iso
(cd packer && packer build -force template.json)
(cd $SRC && sudo PACKER_LOG=1 PACKER_LOG_PATH="./packer.log" packer build -force template.json)
# test if there is a boxfile where we expected it
if [ ! -f packer/output-qemu/Archlinux-v86 ]; then
if [ ! -f $SRC/output-qemu/archlinux ]; then
echo "Looks like something went wrong building the image, maybe try again?"
exit 1
fi;
# clean up any previous loops and mounts
echo "Making sure mountpoint is empty"
LOOP_DEV=$(sudo losetup -f)
sudo umount diskmount -f || /bin/true
sudo kpartx -d /dev/loop0 || /bin/true
sudo losetup -d /dev/loop0 || /bin/true
sudo kpartx -d $LOOP_DEV || /bin/true
sudo losetup -d $LOOP_DEV || /bin/true
# mount the generated raw image, we do that so we can create
# a json mapping of it and copy it to host on the webserver
mkdir -p diskmount
echo "Mounting the created image so we can convert it to a p9 image"
sudo losetup /dev/loop0 packer/output-qemu/Archlinux-v86
sudo kpartx -a /dev/loop0
sudo mount /dev/mapper/loop0p1 diskmount
sudo losetup $LOOP_DEV $SRC/output-qemu/archlinux
sudo kpartx -a $LOOP_DEV
sudo mount /dev/mapper/$(basename $LOOP_DEV)p1 diskmount
# make images dir
mkdir -p output/images
mkdir -p $TARGET
mkdir -p $TARGET/images
mkdir -p $TARGET/images/arch
# map the filesystem to json with fs2json
sudo python fs2json.py --exclude /boot/ --out output/images/fs.json diskmount
sudo ./tools/fs2json.py --out $TARGET/images/fs.json diskmount
sudo ./tools/copy-to-sha256.py diskmount $TARGET/images/arch
# copy the filesystem and chown to nonroot user
echo "Copying the filesystem to output/arch"
mkdir output/arch -p
sudo rsync -q -av diskmount/ output/arch
sudo chown -R $(whoami):$(whoami) output/arch
echo "Copying the filesystem to $TARGET/arch"
mkdir $TARGET/arch -p
sudo rsync -q -av diskmount/ $TARGET/arch
sudo chown -R $(whoami):$(whoami) $TARGET/arch
# clean up mount
echo "Cleaning up mounts"
sudo umount diskmount -f
sudo kpartx -d /dev/loop0
sudo losetup -d /dev/loop0
sudo kpartx -d $LOOP_DEV
sudo losetup -d $LOOP_DEV
# Move the image to the images dir
mv packer/output-qemu/Archlinux-v86 output/images/arch.img
sudo mv $SRC/output-qemu/archlinux $TARGET/images/arch.img
```
An example repository with these scripts can be found [here](https://github.com/vdloo/archlinux-v86-builder)
Given that the packer template and provision.sh is rooted at `packer` (adjust the value of `$SRC` otherwise), run the `build.sh` at root of your `v86` repo:
```
chmod +x build.sh
./build.sh
```
Generated artifacts are now available for serving from `output`.
### Using the created artifacts in v86
Now that we have everything we need to host a server that serves an Archlinux environment over the network.
Create a checkout of v86 and run `make build/libv86.js`. Then edit `examples/arch.html` and replace the `hda` and `filesystem` section with something like this:
```sh
hda: {
url: "http://localhost:8000/images/arch.img",
async: true, # For this option we need to run a webserver that supports the Range header
size: 1.5 * 1024 * 1024 * 1024, # This needs to be the size of the raw disk.
# See the `disk_size` item in the packer template.
},
filesystem: {
baseurl: "http://localhost:8000/arch/",
basefs: "http://localhost:8000/images/fs.json",
},
```
Create a checkout of v86 and run `make build/libv86.js`.
We can then edit `examples/arch.html`, we have two options:
Then copy the created image, filesystem and json file to the v86 directory:
```sh
cp -f /your/packer/dir/output/images/arch.img /your/v86/dir/images/arch.img
cp -f /your/packer/dir/output/images/fs.json /your/v86/dir/images/fs.json
cp -r /your/pacher/dir/output/arch /your/v86/dir/
```
1. Boot the arclinux from the 9p filesystem (generated .bin artifacts at `/output/images/arch`):
Next, we need a webserver that supports the Range header. For example [this extension of the SimpleHTTPServer](https://github.com/smgoller/rangehttpserver).
```sh
filesystem: {
baseurl: "../output/images/arch/",
basefs: "../output/images/fs.json",
},
bzimage_initrd_from_filesystem: true,
cmdline: [
"rw",
"root=host9p rootfstype=9p rootflags=trans=virtio,cache=loose",
].join(" "),
acpi: false,
autostart: true,
```
2. Boot the archlinux from the qemu raw disk image:
```sh
hda: {
url: "../output/images/arch.img",
# set to true if you want to load it asynchrously during runtime (for this option we need to run a webserver that supports the Range header)
# NOTE: async: false is slow but proved to be more realiable
async: false,
# This needs to be the size of the raw disk.
size: 1.5 * 1024 * 1024 * 1024,
# See the `disk_size` item in the packer template.
},
acpi: false,
autostart: true,
```
Next, we need a webserver that supports the Range header. For example [this extension of the SimpleHTTPServer](https://github.com/smgoller/rangehttpserver). At your `v86` root, run:
```sh
cd /your/v86/dir
wget https://raw.githubusercontent.com/smgoller/rangehttpserver/master/RangeHTTPServer.py
python2 RangeHTTPServer.py
```
@ -335,8 +455,9 @@ network_relay_url: "ws://localhost:8080/",
This will make the emulator try to connect to a [WebSockets proxy](https://github.com/benjamincburns/websockproxy). Running the proxy is very easy if you use the Docker container.
```sh
sudo docker run --privileged -p 8080:80 --name relay benjamincburns/jor1k-relay:latest
sudo docker run --privileged -p 8080:80 --name relay bennottelling/websockproxy
```
**NOTE:** original `benjamincburns/jor1k-relay:latest` has throttling built-in by default which will degrade the networking. `bennottelling/websockproxy` has this throttling removed via [websockproxy/issues/4#issuecomment-317255890](https://github.com/benjamincburns/websockproxy/issues/4#issuecomment-317255890).
You can check if the relay is running correctly by going to `http://localhost:8080/` in your browser. There you should see a message that reads `Can "Upgrade" only to "Websocket".`.
@ -357,7 +478,14 @@ It might take a while for a carrier to become available on the interface. If the
When the network is up you should be able to curl a website. To check, run `curl icanhazip.com`. There you should see the public IP of the machine running the proxy.
You can't do inbound traffic into the VM with the websockproxy Docker container because it uses a basic NAT. To SSH into the VM running in the browser, you can create a reverse SSH tunnel to expose the SSH port of the sshd in the VM to the outside world.
You can't do inbound traffic into the VM with the websockproxy Docker container because it uses a basic NAT. To SSH into the VM running in the browser, you can create a reverse SSH tunnel to expose the SSH port of the sshd in the VM to the outside world. You may need to start `sshd` first, it may also be reasonable to change root password:
```sh
passwd root
systemctl start sshd
```
then create a reverse SSH tunnel:
```sh
# This will create a port 1122 on the example.com server

2
docs/networking.md
View file

@ -5,3 +5,5 @@ https://github.com/benjamincburns/websockproxy.
The network card could also be controlled programatically, but this is
currently not exposed.
**NOTE:** original `benjamincburns/jor1k-relay:latest` docker image has throttling built-in by default which will degrade the networking. `bennottelling/websockproxy`docker image has this throttling removed via [websockproxy/issues/4#issuecomment-317255890](https://github.com/benjamincburns/websockproxy/issues/4#issuecomment-317255890).