Since I won't get a better chance to find knowledgeable people: How exactly does iPXE work? Lets say I want to boot a random old PC from one of these images - does the bios need to have iPXE support? Or PXE? Or does it need to first boot into some local system first, which then goes out and gets the image to boot the rest of the way from?
Then there's mention of DHCP and self hosting. Do I need to have some service running within my LAN first or can this go right out to some public server and boot from images there? How does DHCP factor in? I am in so far over my head on this but it seems interesting and something fun to try out.
PXE [0] is a standard for netbooting, typically implementated in the system's boot firmware. A PXE-compatible client sends a DHCP request on boot, to which the network's DHCP server responds with the IP address of a server to netboot from. The client will then connect to this server via TFTP, download an executable image file in the PXE format, and boot it. Using PXE on your own computer requires enabling PXE the firmware setup, setting up a TFTP server to serve the PXE image, and configuring your DHCP server to point to your TFTP server.
iPXE [1] is an open-source implementation of PXE, and much more -- it's much more flexible, it supports additional protocols including HTTP(S) and DNS, it has configuration and scripting options, a basic command-line interface, etc. In order to run iPXE, you need to boot an iPXE image somehow -- e.g. from a MBR or EFI image on a disk drive or USB drive (or even over PXE, I guess). But because iPXE supports more protocols and more configuration, you don't need to set up TFTP and DHCP, and it can chainload into e.g. an EFI image or a Linux kernel instead of being limited to booting images in the PXE format.
An example of iPXE in the wild is the Arch Linux netboot image [2]. They provide pre-configured iPXE images that display an interactive menu to select a mirror, download the Arch Linux installer, and boot it. (It's really convenient since you can just drop the UEFI image at "/efi/boot/bootx64.efi" on a FAT32 thumb drive instead of having to download the whole installer image and 'dd' it onto the drive.)
The submitted project, netboot.xyz, is a similar idea: a preconfigured build of iPXE that lets you interactively download and boot installers for many popular operating systems from a single image.
And while the options previously were interpreted for TFTP use, newer PXE software now understands the use of "http[s]://" in the file name and use that instead of TFTP.
This is super common actually! Most built-in PXE only supports TFTP, which is pretty slow compared to TCP-based stuff. It can make sense to use the built-in PXE to grab a (small) iPXE image over TFTP, then have iPXE grab the (big) real image over HTTP(S). This is also useful if you want to store your main image on something like S3 that doesn’t support TFTP.
For a while I had a script that would create iPXE images dynamically on the fly with the correct HTTPS URL and auth information embedded in them.
Off topic, but does anyone know a good resource that explains how to correctly use "e.g."? (I've looked before, but didn't find one)
While I may seem like a grammar pedant, there are many things that have entered common use while arguably incorrect that I'm not so bothered about ("That begs the question", for example). However while this (incorrect) use of "e.g." is a hill I will die on, even so, I struggle to explain why it's just plain wrong. (It can be used to replace "for example" when preceding a list if examples that illustrate a point, but not as a generic replacement as in this case.)
Someone somewhere must have explained it better...
You can use netboot.xyz from a flash drive to boot various operating systems and utilities. Alternatively, PXE (Preboot Execution Environment) has been around for a while and works by allowing a network-capable device to boot from its network interface. A PXE-compatible network card requests a DHCP lease during the boot process, which provides the IP address of a TFTP (Trivial File Transfer Protocol) server and the file that needs to be loaded from the server.
Typically, the network card contains a basic PXE kernel. To enhance this environment, you can chainload iPXE, which offers a broader range of features. iPXE allows for more advanced booting options, such as loading scripts or initiating an unattended installation directly from the network.
Since nobody directly answered some of your specific questions:
> Lets say I want to boot a random old PC from one of these images - does the bios need to have iPXE support? Or PXE?
The most-desirable way to use iPXE is to boot it from a system's built-in PXE, chainloading it from a local DCHP+TFTP server. All routers have a DHCP server, and many enthusiast/professional routers will have a TFTP server available. pfSense/OPNSense for example can both do this.
If you want to go down the rabbit hole and can mod your system BIOS, you can install iPXE into many of them. I wouldn't recommend it, but I did it once and thought it was pretty cool.
> random old PC
If you _literally_ meant like a vintage PC that predates the PXE spec[1], you can boot iPXE from any local media — I architected a project 10 years ago where I actually gave floppy disks to technicians because some of the ancient Dells we were working with required a BIOS update to enable USB booting.
You can also burn iPXE into a network card's ROM chip. There were a lot of PCI/ISA network cards that took an EEPROM and could boot from them if populated. I always thought it would be fun to boot a vintage PC with iPXE straight off of an add-in network card like that. iPXE will hook the BIOS, and you can literally boot MS-DOS via iSCSI.
> Do I need to have some service running within my LAN first or can this go right out to some public server and boot from images there?
Again, typical deployments will have iPXE chainloaded from the system's existing PXE implementation, which means you've got a DHCP server on the local segment, and a TFTP server somewhere on the local network. But you _could_ boot iPXE from any other boot medium, which would mean the DHCP/TFTP setup are not required. iPXE itself, once loaded, can do its work with any network connection.
Generally speaking, if you have iPXE already compiled and flashed onto your NIC, then you can follow these instructions: https://netboot.xyz/docs/booting/ipxe
DHCP is only needed for getting an IP address. You can use the Docker image with the proper DHCP parameters to load it automatically when using PXE/iPXE: https://netboot.xyz/docs/docker
So what I do here is that I have a local PXE server (it's a DHCP option on your router) that serves up iPXE with a preconfigured script to boot via HTTP off of netboot.xyz. So whenever I want to install linux on a new computer, I set the BIOS boot option to boot from PXE next boot and restart. In a few seconds, I am presented with my choice of linux environment.
For my data center servers, I have it booting via PXE to an iPXE with a custom script to take a unique identifier from the host and build the corresponding configuration (NixOS). So essentially for that I define my NixOS configuration in a NixOS flake and plug the new host in and it will boot to the correct configuration. I actually don't have any OS installation on most of the hosts and share the nix store via NFS.
I also keep an iPXE thumb drive around in case I need to do this for something not on my network. In that case, I insert the drive, boot from it, and then ask it to boot from netboot.xyz manually.
There are a bunch of different options to "bootstrap" iPXE in the first place. Least invasive is probably to boot it from a USB drive. You can also set up a traditional PXE boot process using DHCP/TFTP on your network to "chain-load" iPXE (if your client's network card supports PXE), or you can even flash iPXE into the boot ROM on certain network cards.
However you do it, once iPXE starts running, it will take control of the NIC and fetch the actual OS images that it needs from the internet over HTTP.
Some HP server UEFIs also support boot from http. So you can boot the .efi executable directly from the internet.
For https you need to enroll the certificate first and it's a bit more involved.
I am sure there are other implementations I have only seen it on HP servers so far.
Roughly from memory
iPXE is a bootable image that acts similar to a PXE BIOS, so you boot to that image like a regular disk.
It acts like a bios by for example hooking your interrupts for storage reads.
In order to find where the image is, you need some kind of discovery mechanism. This is where DHCP comes in. Remember this is all happening before you have an OS, so it has to be very bare bones
If you want to see how all the magic happens, have a look at the code and docs from Debian’s FAI (Fully Automated Install). Once you have a Debian FAI all setup, you can bootstrap your machine from zero every morning while you have your morning coffee
Why would you ever trust something like this that will helpfully pull images from the internet? I'm sure whomever is running it today is perfectly honest and none of their computers have been compromised, but there's no reason to expect it to stay that way, is there?
Firewalling the application so that only local images are available seems like the only safe way to use this.
> Why would you ever trust something like this that will helpfully pull images from the internet?
How is that different than pulling an ISO image of your favorite distro, or using a package manager like apt?
Yes, I know that Linux ISOs have checksums and apt uses digital signatures, but so does iPXE. The only difference here would be that for some reason you trust the websites of your Linux distro vendor, but not netboot.xyz?
> you trust the websites of your Linux distro vendor, but not netboot.xyz
Well... yeah... that's not that crazy of a position to take.
Not saying there's anything wrong with netboot.xyz, but it's a question of how many cooks to let in the kitchen, and how many public eyes are on each cook.
If I read the the docs correctly, since source locations are printer, it’s about as trustworthy as trusting “wget $ISO_URL” on your installation to not download anything malicious. Unfortunately what seems to be missing is a hash check after the fact - a missed opportunity since images are loaded to RAM anyway.
(The limitation here is that you have to be able to load the installer image into RAM, which does exclude a lot of smaller nettop/thin/SoC clients unfortunately.)
I tried netboot once, and lost interest when it took minutes to download an ISO over cable internet (DSL would be even worse) every singe time I tried booting into a live distro while debugging things. It's faster to download the ISOs I need ahead of time, save them on a Ventoy drive, and boot (and reboot) into each one in seconds.
I’ve found it to be much faster and far more convenient than downloading the image I want and writing it to a usb drive.
I did at one point keep a few stock images on my tftp server but even there, they would go out of date quicker than my need to use them. So I ended up sticking with NetBoot.xyz for convenience
How do I preload or cache ISOs on a flash drive holding the netboot.xyz client? I haven't seen any mention of it in docs. I've seen some people talking about setting up local iPXE servers, but I don't know if my machines always have functional networking to my server laptop during startup (and this fails outside my home network).
I got really into this sort of thing about ten years ago and set up my wireless router to serve up the initial iPXE image and then load various OS's over the internet. I got about ten or so OS's set up to boot this way then I lost interest and never used it again.
I was looking for a alternative to VenToy... so I scanned all the embedded binaries... every last one... and its completely and utterly clean... but does not support windows, which is a minus only in my case, but an absolute godsend in all others. Not a single Linux I have ever used has been passed over.
I've been meaning to give this a shot myself. I do like netboot.xyz's maintained catalog for convenience, but when I self hosted it (to get the caching proxy benefits), I felt that cache cleanup was a little clunky (basically, go into the cache dir and delete the ISO/image). For the few times I need to install an OS on something, it'd be totally fine to grab the image and stage it on the PXE server.
I'm experimenting on and off with PXE network booting.
In my experience, TFTP is really slow (ancient UDP based protocol) and I'd like to do away with it. Unfortunately TFTP is still required to load something like iPXE to load the actual OS Kernel over HTTP, assuming iPXE isn't part of the NIC firmware or can't be configured to do so.
Turns out that downloading over HTTP is just as slow (few MB/s) when using iPXE so I'm not sure what I'm gaining.
Ideally HTTP boot is possible through the UEFI bios but it seems that this is common on servers, but not on clients? My HP 1L PCs don't have an option to boot over HTTP directly.
> Turns out that downloading over HTTP is just as slow (few MB/s) when using iPXE so I'm not sure what I'm gaining.
iPXE should be able to max out any gigabit NIC that has a decent driver. If you're using a BIOS UNDI driver (undionly.kpxe) or EFI SNP (snp.efi/snponly.efi) you may be experiencing the misery of a crummy driver supplied by the built-in PXE ROM.
If your NIC is supported, try using the bin/ipxe.pxe or bin/ipxe.efi build targets[1] and chainload that instead. If they work at all, it'll mean you've got a NIC with a native iPXE driver, and using that driver may speed things up.
Alternatively, the opposite may also be true. Some of the drivers aren't great, and UNDI/SNP may perform better. If you're already using those builds, try it the other way around.
An interesting concept... load an OS from an OS image file over the network...
If that's the case, then OS'es which do not use any local storage devices, and instead mount remote filesystems over the network -- can't be far behind...
And, if that's the case, then the only thing that the local PC is doing is using its CPU and RAM to run the OS, and network port to read/write data to the filesystem...
But the local PC could also simply run a Remote Desktop or VMWare Client or VNC or equivalent to another remote computer somewhere else on the Internet -- in which case it really doesn't even need to run its own Operating System proper, it only needs to run the software which allows network connectivity to the running instance of the remote Operating System somewhere else on the Internet...
Maybe it would be simpler to build a future "computer" which is basically the glorified equivalent of a long-distance KVM switch over the Internet with local video output...
Hey, such a device would never need its own hardware upgrades... just upgrade whatever remote machine it connects to on the other end, and you've effectively upgraded the local one...
Then there's mention of DHCP and self hosting. Do I need to have some service running within my LAN first or can this go right out to some public server and boot from images there? How does DHCP factor in? I am in so far over my head on this but it seems interesting and something fun to try out.
iPXE [1] is an open-source implementation of PXE, and much more -- it's much more flexible, it supports additional protocols including HTTP(S) and DNS, it has configuration and scripting options, a basic command-line interface, etc. In order to run iPXE, you need to boot an iPXE image somehow -- e.g. from a MBR or EFI image on a disk drive or USB drive (or even over PXE, I guess). But because iPXE supports more protocols and more configuration, you don't need to set up TFTP and DHCP, and it can chainload into e.g. an EFI image or a Linux kernel instead of being limited to booting images in the PXE format.
An example of iPXE in the wild is the Arch Linux netboot image [2]. They provide pre-configured iPXE images that display an interactive menu to select a mirror, download the Arch Linux installer, and boot it. (It's really convenient since you can just drop the UEFI image at "/efi/boot/bootx64.efi" on a FAT32 thumb drive instead of having to download the whole installer image and 'dd' it onto the drive.)
The submitted project, netboot.xyz, is a similar idea: a preconfigured build of iPXE that lets you interactively download and boot installers for many popular operating systems from a single image.
[0]: https://en.wikipedia.org/wiki/Preboot_Execution_Environment
[1]: https://ipxe.org/
[2]: https://archlinux.org/releng/netboot/
Specifically there are particular DHCP options (66, 67) that tell the client about this, and the client software (PXE) understands them:
* https://datatracker.ietf.org/doc/html/rfc2132
* https://www.iana.org/assignments/bootp-dhcp-parameters/bootp...
* https://www.iana.org/assignments/dhcpv6-parameters/dhcpv6-pa... (RFC 5970)
And while the options previously were interpreted for TFTP use, newer PXE software now understands the use of "http[s]://" in the file name and use that instead of TFTP.
This is super common actually! Most built-in PXE only supports TFTP, which is pretty slow compared to TCP-based stuff. It can make sense to use the built-in PXE to grab a (small) iPXE image over TFTP, then have iPXE grab the (big) real image over HTTP(S). This is also useful if you want to store your main image on something like S3 that doesn’t support TFTP.
For a while I had a script that would create iPXE images dynamically on the fly with the correct HTTPS URL and auth information embedded in them.
Off topic, but does anyone know a good resource that explains how to correctly use "e.g."? (I've looked before, but didn't find one)
While I may seem like a grammar pedant, there are many things that have entered common use while arguably incorrect that I'm not so bothered about ("That begs the question", for example). However while this (incorrect) use of "e.g." is a hill I will die on, even so, I struggle to explain why it's just plain wrong. (It can be used to replace "for example" when preceding a list if examples that illustrate a point, but not as a generic replacement as in this case.)
Someone somewhere must have explained it better...
Typically, the network card contains a basic PXE kernel. To enhance this environment, you can chainload iPXE, which offers a broader range of features. iPXE allows for more advanced booting options, such as loading scripts or initiating an unattended installation directly from the network.
> Lets say I want to boot a random old PC from one of these images - does the bios need to have iPXE support? Or PXE?
The most-desirable way to use iPXE is to boot it from a system's built-in PXE, chainloading it from a local DCHP+TFTP server. All routers have a DHCP server, and many enthusiast/professional routers will have a TFTP server available. pfSense/OPNSense for example can both do this.
If you want to go down the rabbit hole and can mod your system BIOS, you can install iPXE into many of them. I wouldn't recommend it, but I did it once and thought it was pretty cool.
> random old PC
If you _literally_ meant like a vintage PC that predates the PXE spec[1], you can boot iPXE from any local media — I architected a project 10 years ago where I actually gave floppy disks to technicians because some of the ancient Dells we were working with required a BIOS update to enable USB booting.
You can also burn iPXE into a network card's ROM chip. There were a lot of PCI/ISA network cards that took an EEPROM and could boot from them if populated. I always thought it would be fun to boot a vintage PC with iPXE straight off of an add-in network card like that. iPXE will hook the BIOS, and you can literally boot MS-DOS via iSCSI.
> Do I need to have some service running within my LAN first or can this go right out to some public server and boot from images there?
Again, typical deployments will have iPXE chainloaded from the system's existing PXE implementation, which means you've got a DHCP server on the local segment, and a TFTP server somewhere on the local network. But you _could_ boot iPXE from any other boot medium, which would mean the DHCP/TFTP setup are not required. iPXE itself, once loaded, can do its work with any network connection.
[1]: https://en.wikipedia.org/wiki/Wired_for_Management - this happened in 1998/1999 and implementation was widespread by 2001 or so.
Generally speaking, if you have iPXE already compiled and flashed onto your NIC, then you can follow these instructions: https://netboot.xyz/docs/booting/ipxe
DHCP is only needed for getting an IP address. You can use the Docker image with the proper DHCP parameters to load it automatically when using PXE/iPXE: https://netboot.xyz/docs/docker
For my data center servers, I have it booting via PXE to an iPXE with a custom script to take a unique identifier from the host and build the corresponding configuration (NixOS). So essentially for that I define my NixOS configuration in a NixOS flake and plug the new host in and it will boot to the correct configuration. I actually don't have any OS installation on most of the hosts and share the nix store via NFS.
I also keep an iPXE thumb drive around in case I need to do this for something not on my network. In that case, I insert the drive, boot from it, and then ask it to boot from netboot.xyz manually.
However you do it, once iPXE starts running, it will take control of the NIC and fetch the actual OS images that it needs from the internet over HTTP.
I am sure there are other implementations I have only seen it on HP servers so far.
In order to find where the image is, you need some kind of discovery mechanism. This is where DHCP comes in. Remember this is all happening before you have an OS, so it has to be very bare bones
Edit: Seriously? You click on docs and almost all of those questions are answered. It's 1 click away. Parent didn't even try.
Firewalling the application so that only local images are available seems like the only safe way to use this.
How is that different than pulling an ISO image of your favorite distro, or using a package manager like apt?
Yes, I know that Linux ISOs have checksums and apt uses digital signatures, but so does iPXE. The only difference here would be that for some reason you trust the websites of your Linux distro vendor, but not netboot.xyz?
"Some iPXE builds do not support HTTPS connections. If you get an "Operation not supported" error message, run this instead:
chain --autofree http://boot.netboot.xyz"
Which.. think about that advice for a minute.
"Some" reason? I think I'd have a very good reason to place much more trust in the Debian folks than some guy who runs some random netbooting website.
I'm not going to lie, this made me laugh out loud.
"For some reason, you trust a doctor to perform surgery on you, but not this lovely man that I met on the subway?!"
Well... yeah... that's not that crazy of a position to take.
Not saying there's anything wrong with netboot.xyz, but it's a question of how many cooks to let in the kitchen, and how many public eyes are on each cook.
(The limitation here is that you have to be able to load the installer image into RAM, which does exclude a lot of smaller nettop/thin/SoC clients unfortunately.)
I did at one point keep a few stock images on my tftp server but even there, they would go out of date quicker than my need to use them. So I ended up sticking with NetBoot.xyz for convenience
Brilliant work on a brilliant solution.
https://www.iventoy.com/en/index.html
I haven't tried it but I use its USB cousin Ventoy almost every day. It's fantastic.
In my experience, TFTP is really slow (ancient UDP based protocol) and I'd like to do away with it. Unfortunately TFTP is still required to load something like iPXE to load the actual OS Kernel over HTTP, assuming iPXE isn't part of the NIC firmware or can't be configured to do so.
Turns out that downloading over HTTP is just as slow (few MB/s) when using iPXE so I'm not sure what I'm gaining.
Ideally HTTP boot is possible through the UEFI bios but it seems that this is common on servers, but not on clients? My HP 1L PCs don't have an option to boot over HTTP directly.
iPXE should be able to max out any gigabit NIC that has a decent driver. If you're using a BIOS UNDI driver (undionly.kpxe) or EFI SNP (snp.efi/snponly.efi) you may be experiencing the misery of a crummy driver supplied by the built-in PXE ROM.
If your NIC is supported, try using the bin/ipxe.pxe or bin/ipxe.efi build targets[1] and chainload that instead. If they work at all, it'll mean you've got a NIC with a native iPXE driver, and using that driver may speed things up.
Alternatively, the opposite may also be true. Some of the drivers aren't great, and UNDI/SNP may perform better. If you're already using those builds, try it the other way around.
[1]: https://ipxe.org/appnote/buildtargets
If that's the case, then OS'es which do not use any local storage devices, and instead mount remote filesystems over the network -- can't be far behind...
And, if that's the case, then the only thing that the local PC is doing is using its CPU and RAM to run the OS, and network port to read/write data to the filesystem...
But the local PC could also simply run a Remote Desktop or VMWare Client or VNC or equivalent to another remote computer somewhere else on the Internet -- in which case it really doesn't even need to run its own Operating System proper, it only needs to run the software which allows network connectivity to the running instance of the remote Operating System somewhere else on the Internet...
Maybe it would be simpler to build a future "computer" which is basically the glorified equivalent of a long-distance KVM switch over the Internet with local video output...
Hey, such a device would never need its own hardware upgrades... just upgrade whatever remote machine it connects to on the other end, and you've effectively upgraded the local one...
Anyway, an interesting concept!