I've been told my a good friend that I should write up at least one of my expeditions. Please excuse me if I do not sound like a usual blogger, and I also warn that I do like to ramble on about details, and also like commas with semi-run-on sentences. With that disclaimer out of the way and expectations moderately set, let's pow-wow!
It was not done today, but was a recent undertaking in the past several weeks that I finally looked into crafting th Ultimate Boot USB device I have always (for a while) wanted. I have wanted this ever since I had thought of putting all of my leftover, older desktop machines together, and creating my own private cloud, for fun, which may be the topic of another post. That, and I do tend to reinstall my main Windows desktop machine semi-frequently. I also do regular IT work for friends and family where this would also be useful, including the who-knows-how-old Windows XP PC my wife still has, which incidentally also needs to be update as of right now..
To expound upon the background, I have built my current desktop computer, in 2009, without any type of optical drive, or really anything that would fill a 5.25" drive bay. I have a 500-page DVD booklet I finally bought to store all of my backups in, which is of course a mix with the hodge podge of various external drive I have laying around that are also full.
The point of this story is I wanted to quickly boot off of several choices that were never convenient in the past, and also be able to quickly update the boot library for the future. I wanted something that was versatile and open so I could spend my days configuring and customized the set-up just the way I like. This would call for something that is open and has a large community of people and materials supporting it.
I tried to do some initial research about best options, but one idea kept ringing in the back of my head the entire time. It has been used before, and for a while; it is popular, or at least exists commonly among other software and by itself; is fairly mature for what functionality it has; there are many example of configurations I can find online; and it's open. The choice of course, is Grub2, the more updated version. Other choices I found were burg and lilo, with possibly another that I cannot remember, but I was slightly familiar with Grub(2) already, already knew where to look for reference, and how to interact with it. After this whole time and now just writing this, I remember syslinux, and found this link that actually states it would be a better choice for external, one-time boot compilation media.
It is the popular boot loader of choice among Linux distros once installed to the hard disk with the operating system, and in that state is already versatile for adding new entries, both automatically and manually. I would look to this example later.
I also chose this because of my light research, I try to find an example where another person has made an Ultimate Boot Device. I found this link which explains the beginnings of my process. Also know that at this time in that same research I have also found the names of tools, some more comprehensive than others, that would also help me accomplish this task. I'm sure these would be enough for most people wanting to accomplish a task like mine and be done with it, but like I said, I like to get things just right for myself, and have more granular control over them. The names of some of these tools are here: YUMI, UNetbootin, SARDU, MultiSystem, XBOOT, LiveUSB Install, Grub4Dos, WinGrub, the list goes on... It may still help to inspect the configuration some of these tools produce for reference, especially if they use Grub underneath!
Now that I have somewhere to start, I begin to follow the guide I just linked, while also trying to read more about the boot process. I have always been somewhat familiar with boot records, and fairly knowledgeable about drive partitions, but some points in the process were still a bit cloudy to me. I quickly realize the one part of the boot process I should spend most of my time reading about is the link between the motherboard and the drive. I could not imagine the motherboard would have such specific code instructions as to have the drivers loaded, and to look for specific files inside of a filesystem, a structure for organizing data that is usually different between operating systems, especially for an old boot system such as the 'BIOS' (yay for nomenclature!). I believe this is in fact the case with a more modern boot system known as (U)EFI (UEFI is the newer one), not relying on a boot sector anymore.
Here I have mentioned the black box I would need to investigate: the boot sector. I was always familiar with the MBR, but more as just a partition table. While that is true, I was unfamiliar with that 'link' I mentioned earlier, which goes from the motherboard to which partition to start booting. The trick here is in addition to a table of partitions, there is a block of raw machine code at the beginning that gets loaded into memory by the motherboard, simple as that. So apparently it is this code that must load some basic I/O and other system drivers like a kernel, figures out the particular type of partition table being used (which also has different choices, such as GPT), and then with the information in the table containing boot bits either being set or not set, loads the first bootable medium.
I have also learned that at this stage (and don't quote this part, in case you have been quoting everything up to this point), a VBR can be loaded, which is essentially an MBR, but at the beginning of a single partition, and not a single one for the entire disk. This may be optional, which is why I say don't quote me. Around here should be where Grub2 as a boot manager can take over once loaded. It runs similar code that also gives the user an interface and way to interact with it. Windows also has its bootloader which usually remain hidden with a single installation of Windows, such as NTLDR and WINLOAD with the Windows Boot Manager (bootmgr) (you may see these entries in my configuration later on).
The amazing thing about having Grub here in this space dominated by legacy standard and utilities is that it is extensible. It provides, through configuration files, an easily extensible way to modify menu entries, in its own human-readable language, for booting various artifacts on a device. IT is also versatile in that it can boot various types of artifacts using different modules, offer immediate, in-place editing of menu entries, an interactive console to play around with, and various rescue options and hotkeys as well!
With that selected, I started my process. I bought a 64GB flash drive to use just for this purpose; large enough to support having many OSes available to install from it, and other utilities that would undoubtedly be smaller. I started the tutorial, installed Grub2 into the flash drive, and started to load up the materials. When it was installing Grub2, it mentioned a warning/error that it would be using unreliable or fragile blockchain strategy to bootload into Grub2. I am still unsure what exactly this means, but it must not be as robust as possible. Maybe if the partitions or something else changes with the device, some pointers would not be correct anymore, leading to 'no-boot' situation.
After that I found that the boot sector had been written, most likely to then load the boot files I specified the location of in the mounted filesystem, and I could being working with Grub2 configuration!
Now the hard part here was I Was quite unfamiliar with boot commands and the Grub2 environment. After poking around some basic configuration it almost seemed Bash/Shell-like environment with added boot-based commands or executables. This was nice as the Linux way is to reuse applications and tools for a single layer of functionality, giving the openness to chain them together to create solutions. I was already familiar with Bash/Shell scripting by this point, and brought some of my common techniques and organization strategies when working with it.
This environment even had a different way of accessing a system's drives that were not simply filesystem-mapped device descriptors, but were close enough to them that I figured out how to deal with them well enough.
This would remain a difficult task if I only had my limited knowledge of these commands and the environment, along with only a couple of boot scripts to work off of that certainly did not detail what I wanted to boot, but after a better bit of Google-searching I hit the jackpot. On GitHub there were many files shared where people had also added other entires to their configuration. It seems they did this for one purpose or another that wasn't necessarily making a USB boot device, maybe an advanced recovery console, other helpful utilities, or simply an alternate operating system (dual-boot). These files helped a lot as they provided examples for the first type of boot I wanted to accomplish, and that was loading various Linux operating system distributions from an ISO for to either try or install it.
For reasons I have explained before, this would allow me to quickly try and install various Linux distros all from a single location with the already common simplicity of a file copy over to the USB device, without having to burn DVDs, waiting for that read time, and all of that business. I took some standard distros I found such as Ubuntu, possibly Red Hat, Fedora, Debian, CentOS, Xubuntu, Lubuntu, Mint, elementaryOS. Others are more focused uses or styles such as Kali/Backtrack, Tails, TinyCore, Arch Linux, Finnix, GRML, DSL; it was just so easy to add more!
I say that, but the exciting part would soon end as I would see that these distributions were not necessarily consistent with the way they would boot. It was very beneficial that they all, or mostly, supported loading with a command-line option specifying the ISO file to use as a resource container after starting the boot from Grub2, but they would probably need their other flags to boot properly. This would also be remedied by finding these Grub2 configuration file examples from around the net in GitHub, forums, and otherwise, and I have tried to link my sources in my own file when possible. In the same spirit of finding this helpful information, I put this file in a Gist under my account to help others that would also want to build an Ultimate Boot USB. I would continue to add sections to my file, but would assuredly have to check each entry as a sort of integration test since all of this boot business was still a bit fragile.
With those being relatively in order and figured out, the next order of items to add to my drive were some tools and utilities. I have known some were based on Linux just like the operating systems were to some degree, and should boot similarly. Without going back and looking into more detail, I would say this was fairly correct, and some tools (GParted?) even provided Grub2 and other boot configuration on their website.
Note: If you get an error about having an "invalid magic number" from an entry for booting one of these tools, check that the ISO file you have downloaded is correct. I swear I verified the file size, but I guess some introduced during the download, I still can't believe downloads could still be this inconsistent.
Just to list some more of the tools I was hoping to add to my utility drive are Memtest, SpinRite, GParted / Parted Magic, System Rescue CD, CloneZilla, Ophcrack, DBAN, FreeDOS/MS-DOS, Hiren's and Ultimate Boot CD. The Boot CD compilations were also another special reason why I wanted to make this drive, was to be able to use their full complement of abilities as well, without DVDs. I also may want to add (original) Xbox utilities to my drive as well, as I have had to reformat, backup, and replace Xbox FATX drives and EEPROMS.
Another Note: One strange piece of functionality is that variables are 'reset' and disappear when entering sub-menu groups! I bet this was intended at some point, and I finally found the answer in a mail thread or forum, but it took me a bit of time to figure out what was causing the issue. This would be specific to a Grub2 unique construct (menuentry) that I have now become more familiar with. After finding this out, I put a comment in the configuration file that would help me and anybody else in the future remember this potential error path, and help save time avoiding it again.
I noticed that to try to run some of the tools (like SpinRite) in Grub, people suggested using memdisk as a way to load and boot that tool from a USB device. This was a bit involved and I had to try several configurations, but finally got it booting. I had to realize where to get a memdisk since it didn't seem to come with a Grub installation, or one that I had on hand. After looking for it and syslinux came up enough times, I fetched the binary from that package as I thought it was a standalone tool, or incompatible and I needed another, but it is simply in the syslinux package and is a versatile tool!
After figuring some of those out, or at least pasting them all into the file and organized them for test later on, I move to the next set of items that have a large importance for also needing this device, and is Windows!
I'm sure a lot of people just cheered (sarcasm), but I have been using Windows for many years; it has its purpose in my power-user life (Gaming, friendly IT support for others, running esoteric programs for my many curiosities). Also like I stated before for this purpose, I wanted to be able to 'refresh' the install of Windows for my computers and others, without needing a DVD drive, being more convenient, and I could select from the different versions as needed, but boy would I find this to be a much more complicated requirement.
Very soon into the process, I find out that Windows will only recognize a single partition on a removable device, no matter what. That's it, I can't believe those words go together like that. I almost (that's polite, I definitely) for embarrassed for this (lack of) functionality. I even tested this with (ex)FAT(32) and NTFS partitions; Windows just won't do it. I won't say this took an embarrassingly long time to figure out either, but it took long enough with testing between different systems and careful filesystem and partitioning structures I didn't have complete knowledge over. Was this a problem with physical vs. logical volumes? I may have ended up finding en article that explained the history behind some of this is that there was a bug in the way Windows does FAT filesystem allocation, and would not allow users to partition anything above a certain size (32GB), and that it may be because my drive was larger than that at 64GB, and also disallowed resizing volumes already formatted as such. However true that was, I may have been able to use a command line utility, as directed by some post, instead of the GUI (I love commas..), to format the device under Windows, or at least was able to with relative ease in a Linux environment (mkfs.vfat/mkdosfs?). Any way I tried, it still would not be read under Windows, effectively reducing the management I could do from any computer now (the Windows ones). I originally wanted to create a couple of partitions (not an exact couple) just for organizational purposes, but more importantly as a backup plan in case I could not emulate or boot everything that I needed from Grub2 and self-contained files. The latter was a neat idea I had even before I learned about the 'chainload' command and setting 'root's in the Grub environment.
With that, I finally found I could at least somewhat manage the drives partitions by using a program different from the one in Administrator's utilities on Windows, Computer Management > Storage, and that was MiniTool Partition Wizard. It's a nice little shareware program that at least recognizes and allows manipulation of a removable drive's partitions. I'm sure there are a lot of similar programs for Windows that do the same thing; I just found this one first. Earlier I had looked into get around the Windows limitation. Some had said I could 'flip a bit' on the USB device that would have it be recognized 'more like' a hard drive is instead of a removable device with Windows' imposed limitation on those. I later learned that this process is a bit more destructive and specific than setting some bit in the header on the drive's contents, but instead in the firmware itself. After it didn't work on the couple of modern USB drives that I had I put this aside. I also looked into installing a drive filter that would change this bit on the fly, but this was getting a bit complicated, and I wanted to just be able to insert this drive into any computer and be able to manage it without having to first prepare the computer any further, and it was accessible on different operating systems anyway. The real problem however is if this means a Windows Installation, that I would later hopefully be successfully able to boot, would also not be able to read its contents on the same flash drive, so I would also be limited there.
After those limitations were discovered and exercised, I continued on trying to find a way to boot a self-contained Windows installation, without needing a separate partition for it. I started with about nothing to go on, but later stumbled upon an automatically generated menuentry from a recent installation I had done that pointed to Windows. To clarify, through this process I had a hard drive crash on me, causing me to find another way to work while I tried to get another drive ready for my computer. As this was perfect timing, I actually installed a version of elementaryOS on the very boot stick I was working on. This took some 'bootstrapping' (:P) to get on there a sort of a chicken and egg situation. It was here where I also found a very nice way to perform quick integration testing for my USB device.
As a brief aside: It's too slow to have to restart a bare metal device and select a boot medium every time I want to change a single line in my configuration. I could edit it some in Grub2, but that was limited. I found I could use VirtualBox with a VM created just with a VMDK file as an 'existing disk', which really just acted as a pointer to the actual disks on my system, the one I concerned myself with was the USB device. I had to perform some command-line-fu since this wasn't an option through the VirtualBox GUI, and there are plenty of online articles telling you how to do this. Upgrading VirtualBox to the major 5 version also helped make this quicker because in 4, in order to mount the USB device so the VM would have access to it (you may need some 'USB Extensions' package additionally installed into VirtualBox as well), it would need to unmount from the Host OS, depriving me of read/write access to it until my test was done. VirtualBox 5 allowed both to see it at the same time so I could test, make a change, issue a quick VM restart, test, and repeat, excellent!
To finish my erratic thought pattern through writing this article (it's like a tree), I installed a Linux OS on the USB stick by using VirtualBox. I was able to write to it this way, but I was not able install to itself when booting a live image from it, I think because of mounting reasons. The OS would freeze the second I unmounted it, disallowing me to continue installation. So I used VirtualBox to load the live image as an ISO on my Host OS, and installed it on the mounted USB device my VM detected as storage. Great!
Another random note: Getting to the stage of having to place the Windows installation files on a separate partition seemed to be required at a point in my investigation, and the self-contained file did not work from Grub. Once these files had a new partition made for them, it messed up my blockchain (I think)! I had to reinstall the Grub boot loader on the USB device, and only needed to run that one command again, to get Grub to boot. At that point of course my partition references were incorrect in Grub so I corrected those, but I still found some system not working, like booting into my USB on-board OS. It might need some boot re-installation, but at least my live image works again!
Through this time or later I was doing that Windows installation troubleshooting I sort of mentioned earlier, and on successive reboots to my Linux OS I realized it had detected 'some type of Windows' as a separate partition on the USB device, and generated a Grub menuentry for it. That would be perfect for my uses, as long as it works. I would continue to try this, and slightly modified options, to no avail, and the attempts should be recorded in my file. The same goes for when I was trying to boot from self-contained files. The problem I started running into is even if I found some instructions that would boot the Windows, it had no way of accessing the ISO file I was using, as it was only designed to run off a CD, and later a USB device by itself as the only partition. I would later take advantage of this change by applying a Windows Installer to a blank USB device, and then moving that partition over, verbatim, to one on the ultimate boot device I was building. There was no way this was going to work (because of pointers :) ), but I wanted to try it! This actually copied over fairly well, but the stage I am now left at is trying to access this location and boot from it (VBR anyone), but all I get is a single fixed-width textual character of a double exclamation mark on a green background in the center of the screen, towards the upper right corner a bit; such a strange error display.
So now at this point I realize I am at least bit over my head, and I didn't feel like disassembling boot sector code to find out how that little black box works. Maybe it will hit me later on, or I'll get the willpower back, but in lieu of finding a 'magic' solution of loading a self-contained Windows installer file, I'll probably just have to apply the ISOs to a fresh, blank, separate USB device each time I want to install a different version of Windows.
Also, this calls for a part of my story I looked into at an earlier point and forgot to mention. Maybe I already knew none of this would work, but really I thought it was cleaner. There's this environment called Windows PE that is built specifically for Windows system recovery and installation, with neat tools like help apply WIM files. I spent a good amount of time looking into this, setting up the build environment, applying it to my USB as a separate partition, and booting into it, which does work well, but then I can't quite do what I want with it. Instead of running an installed from a WIM, I found out they are more just like Windows images, which isn't bad, but I kind of felt like I did not want to get into imaging a machine, setting it up the way I like, and then trying to capture the WIM from it and doing it all over again, since I do this relatively infrequently; I'll just re-image a USB stick from time to time I guess. I did find out there is some cool technology around this though, like if you are an administrator that needs to maintain and deploy Windows images in a large network of computers.
So yes, still building the USB device up occasionally, testing, fixing errors, leaving some alone, but I had a fun time learning about all this boot technology with at least one serious effort. I could continue it, and I certainly welcome feedback as I can quickly test it out now with my working knowledge. We'll see where this goes from here, and if you read this larger text, then maybe you're crazier than me!
TL;DR
I wanted to make an Ultimate Boot USB with all the lovely things I wanted on there: Linux installers, tools, and Windows installers. I wanted to do away with managing optical media and drives, as they are going out of style for PCs to some degree, have the convenience of trying and loading many of these different boot options simply by copying files over (and finding the right Grub configuration), and being able to load this up for my purposes or others as IT help by pulling it out of my pocket, all while learning more about the boot process along the way!
End result: I got most of the way, learned some things (darn Windows), and will continue to test each item as needed.
