Copyright © 2010 Erik Nørgaard
This guide describes how to set up FreeBSD for PXE booting network clients for Jumpstart installation and diskless operation. The guide has been updated for FreeBSD 8.0-RELEASE on the i386 platform and some errors corrected.
I wrote this guide while setting up a Jumpstart server to document what I had done so I could get back on the track if necessary. I used other guides as reference, but found that these were either outdated or lacked important details.
The diskless setup is similar in many some respects, with the main difference that the root file system must be nfs mounted. It is possible to do Jumpstart with nfs mount of the installation media. For this reason I decided to do a complete guide for PXE Boot covering both Jumpstart installation and diskless operation.
This guide was first written for FreeBSD release 6.1 but I have now updated it for to FreeBSD release 8.0. Many things have become much easier since previous releases, both for Jumpstart installation and diskless clients. This guide may not be correct for previous versions of FreeBSD.
Warning Following the instructions in this guide may render your system(s) useless and cause data loss. Whatever you are up to, it is on your own risk. I did screw up my own system quite seriously while trying to get things to work with jumpstart installation. I hope I have documented the path that does not screw up (too much) :-)
If you find errors, omissions or ambiguities in this guide feel free to contact me.
This section introduces the general concepts and ideas and the motivation for going into the next sections. You can skip this if you are in a hurry.
Intel's PXE (Preboot eXecution Environment) is a small boot code that lets your client query network configuration using DHCP and fetch a boot loader using TFTP. This has become standard in many computers and has replaced etherboot.
This guide sets out to explore two uses of network booting: Jumpstart installation of FreeBSD and diskless clients:
Jumpstart installation has roots at SUN (at least the name) and can be used to automate installation of a large number of identical machines or simply avoid using install cd's. A homogeneous server farm is easier to maintain, and automated installation lets you quickly setup a new server: plug it in, turn it on, drink coffee.
Diskless clients has a number of advantages: You can easily set up and manage a large number of identical diskless clients. No disk means less noise and longer lifetime of your hardware and a beter working environment for your users. No disk means that files must be fetched from a central server, hence only updating on the server.
In both cases the benefit is ease of maintenance when working with a large number of workstations or servers. If you're only doing it for one machine, then it's mostly for the sport and learning, but those too are valid reasons.
Setting up a server has been documented in numerous documents, however not every step is clear and various documents describe different paths to follow and some parts are obsolete. This document is probably no different.
The hardware must support PXE network booting. Many motherboards with on board Intel based network interface supports PXE. VIA produces mini-ITX format motherboards including some fanless ones, these are ideal for building cheap diskless and completely silent clients.
Any system can function as server as long as they support the services needed to serve the files to the booting client. However, it is easier to setup when using the same platform and version of FreeBSD on the server and client.
When a PXE client boots, the boot process goes through a number of stages, which are roughly as follows:
PXE sends a bootp query on the network and receives information for configuring the network interface: IP address, netmask, default router. Also it receives information about where to fetch a boot loader and the path name of the root device.
PXE fetches the boot loader with tftp (the boot loader is actually the normal FreeBSD loader with some extra code to support tftp or nfs). If the boot loader has been compiled with nfs support, the root file system is mounted and the boot script, kernel and other files are fetched with nfs. If the loader is compiled with tftp support, these files are fetched with tftp.
The kernel is loaded and the root file system mounted. The kernel will need to acquire information about the network configuration, either by using information passed from stage 2 or by sending a new bootp request.
To enter stage 1, PXE must be supported by the hardware. Alternatively one can install a flash boot image or boot on a floppy - this method is not covered in this document.
In stage 2, if pxeboot is compiled with nfs support (default) it will try to nfs mount the root device specified in the root-path option sent by the dhcp server. If pxeboot is compiled with tftp support, it will fetch the files using tftp. In both cases, it will first read the file boot/loader.rc where the path is relative to the nfs root directory or tftp directory.
What happens after stage 3 depends on how we populate the root file system.
The root device can either be a memory disk or an nfs share, both have their advantages and disadvantages:
A memory disk is loaded into RAM making file access fast once loaded, but storage space is limited.
A network mounted device, files are fetched as needed. Loading programs is slower but only programs actually needed are kept in memory and storage space is bigger.
Serving a memory disk is easy as it can be fetched with tftp, and tftp has to be setup anyway to serve the pxeboot loader. But, the memory disk is a bit more cumbersome to populate, customise and update.
The main disadvantage of using tftp to fetch a memory file system is that all clients will fetch the same files, load the same kernel and memory file system, regardless of hardware, version of FreeBSD or the purpose of the client.
With an nfs root device you can specify different root mounts for different clients. This means that it is posible to support multiple hardware platforms and configurations from the same server.
The main reason to avoid nfs is that it is difficult to set up in particular if there is a firewall in the way.
The jumpstart installation section will take the approach of using a memory disk device while for diskless clients we use nfs. So both methods are covered. Setting up pxeboot, you can choose the path you find best and ignore the other.
Keep in mind that all data will be transferred across the network, and at least initially without encryption making your traffic target for eavesdropping.
Do not boot diskless clients on an untrusted network! After boot, connections can be encrypted, but the encryption keys will have been sent in the clear at boot time.
Warning Configuring your server for serving files to network booting clients may render your server vulnerable. You should not use a server directly connected to the internet.
The server configuration is similar for both diskless clients and jumpstart installation, allthough for diskless operation we will use NFS to serve files and for jumpstart we will use FTP.
This section describes the configuration of the required services. The population of the directories with files to be served is covered in the following sections.
We will assume that the server and clients are located on the 192.0.2.0/24 subnet with the server assigned ip 192.0.2.1 and clients assigned ip addresses in the range 192.0.2.128/25. All services may be identified by DNS lookup, such that nfs.example.com is the nfs server and tftp.example.com is the tftp server.
Note: The range 192.0.2.0/24 has been reserved for documentation purposes and should not be deployed on any network, see RFC 3330.
We will be using dhcpd v. 3.0 from the Internet Software Consortium (ISC) which can be installed from ports (net/isc-dhcp3-server). The configuration file, dhcpd.conf(5), should be something like this for the basic configuration:
# Configuration file for ISC's dhcpd v. 3.0
#
# Server is authoritative for the subnets.
authoritative;
# Disable dynamic dns update
ddns-update-style none;
subnet 192.0.2.0 netmask 255.255.255.0 {
range 192.0.2.128 192.0.2.254;
default-lease-time 3600;
max-lease-time 86400;
option domain-name "example.com";
option domain-name-servers ns.example.com;
option routers gateway.example.com;
}
This will assign unknown clients IP in the specified range. For jumpstart installation we add the following section within the subnet block:
group { # jumpstart clients
use-host-decl-names on;
next-server tftp.example.com;
filename "boot/pxeboot";
}
The line next-server tftp.example.com; tells the client where to fetch the bootloader, and filename "boot/pxeboot" tells which file to request. The path to the boot loader is relative to the root directory of the tftp server.
For diskless clients, the following section within the subnet block should replace the section above for jumpstart clients.
group { # diskless clients
use-host-decl-names on;
next-server tftp.example.com;
filename "boot/pxeboot";
option root-path "nfs.example.com:/var/diskless/FreeBSD";
host diskless-1 {
fixed-address 192.0.2.128;
hardware ethernet 00:40:63:d4:89:73;
}
}
For diskless client setup, we also need to specify the root device: option root-path "nfs.example.com:/var/diskless/FreeBSD" tells the client where to get an NFS root mount, if not set it will default to <next-server>:/pxeroot. An alternative is to edit loader.conf(5) to specify a root device (this method is not covered for diskless clients, but is used for jumpstart).
For each client a host declaration is included, this may be omitted. but in particular if the network also hosts non-diskless clients it may be used to tighten control. A fixed ip-address is assigned to the diskless client with the host declaration. This makes it easier to configure NFS shares.
To enable dhcpd, add the following lines to rc.conf:
dhcpd_enable="YES"
Then start dhcpd:
# /usr/local/etc/rc.d/dhcpd.sh start
Note: If you make any changes to dhcpd.conf you need to restart the dhcpd daemon, you cannot just send a SIGHUP.
You must enable tftpd(8) at least to serve the pxeboot loader to the clients. To enable tftp edit the file /etc/inetd.conf to enable the following line:
tftp dgram udp wait root /usr/libexec/tftpd tftpd -l -s /var/tftp
The -l option tells tftpd to log to syslogd(8), the -s /var/tftp instructs tftpd to change its root directory into /var/tftp.
If you have not enabled inetd(8) in your /etc/rc.conf, first add the line:
inetd_enable="YES"
Then start (or restart) inetd:
# /etc/rc.d/inetd start
For jumpstart installation we want to install using ftp rather than nfs. Using a local ftp server speeds up the installation as the local network is faster, saves bandwidth on the external connection and reduces load on the public servers.
You can use the ftpd distributed with FreeBSD and enable it in inetd.conf uncommenting the line:
ftp stream tcp nowait root /usr/libexec/ftpd ftpd -l
Then start (or restart) inetd:
# /etc/rc.d/inetd start
If you decide to use your own FTP server, it is recommended to keep the FreeBSD release files in the same path as that found the FreeBSD ftp mirror sites.
NFS is a protocol for sharing file systems over the local network, it is an RPC service which makes it particularly difficult to handle, in particular across a firewall.
There are some limitations when exporting file systems. You can export an entire partition (disk label) or a specific directory. If you export a partition with the option -alldirs, then any sub-directory can be mounted, but with the permissions set for the partition. This means you can't export the root partition read-only, and other partitions read-write. Alternatively you have to list each directory you wish to be exported and the permissions.
The latter should be preferred, although cumbersome, because it gives you more fine grained access control.
Enable nfs in /etc/rc.conf:
rpcbind_enable="YES" # Run the portmapper service (YES/NO). nfs_server_enable="YES" # This host is an NFS server (or NO). mountd_enable="YES" # Run mountd (or NO). mountd_flags="-r -p 59" # Force mountd to bind on port 59
As a minimum you need to enable rpcbind, nfsserver and mountd as shown above.
By default, when mountd starts it binds to some arbitrary port, and rpc is used to discover which, making it impossible to filter. With option '-p' mountd can be forced to bind to a specific port. Port 59 is assigned to "any private file service", so it sounds reasonable to use this.
You may optionally want to enable Lockd and statd which provides file locking and status monitoring. The problem with these services is that they cannot be forced to bind to specific ports making filtering impossible.
Before starting up nfs, we must export the file systems the diskless clients need to mount. Users wants more than a bare base system, in particular they want access to their files in their home directory.
For the diskless clients, all partitions to be exported will be located in /var/diskless, with the root file system in /var/diskless/FreeBSD and for each diskless node we create the directory /var/diskless/<hostname>. The exports(5) file should be something like this:
/var/diskless/FreeBSD -ro -maproot=root:wheel -network 192.0.2.0 -mask 255.255.255.0 /home -alldirs -network 192.0.2.0 -mask 255.255.255.0 /var/diskless/<node>/var -maproot=root:wheel <node>
Where <node> is the diskless clients ip address. One line must be added for each diskless client.
Note: If /var/diskless is a directory residing in /var then you cannot use the option -alldirs and you cannot mount subdirectories of /var/diskless. Instead, you will have to list each directory which will be mounted as shown above. The positive effect is that this will enforce a more strict access control.
The -maproot option specifies which privileges the root user on the client will have for accessing files on the server via nfs. By default root:wheel is mapped to nobody:nobody.
We will need to create var and (optionally) /tmp directories and a swap file for each diskless client. This is described in the section diskless clients. In the above, it is assumed that the hostname of the diskless clients can be looked up and fixed ip address is assigned. If you control the network, you may choose not to specify the hostname or ip address.
Then start nfsd:
# /etc/rc.d/rpcbind start # /etc/rc.d/nfsd start # /etc/rc.d/mountd start
You should be able to see the exported shares with the command showmount(8). The directories exported should be created if they do not exist. If you update the exports file, you need to reload the exports file with the command
/etc/rc.d/mountd reload
for the changes to take effect.
If you are on a closed and controlled network you need not protect your server by a firewall, although it is recommended always to be cautious. If you have a firewall enabled however, you need to open such that the clients can fetch the files needed.
If you are on a closed and controlled network you need not protect your server by a firewall, although it is recommended always to be cautious. If you have a firewall enabled however, you need to open such that the clients can fetch the files needed. This section lists the firewall rules you need to include, the listings are for Packet Filter.
To enable DHCP, you must allow packets with source port 68 and destination port 67 in, and the reverse out for any ip. The reason is that when the first dhcp request is sent, the client don't know it's ip and source ip port 67 in, and the reverse out for any ip. The reason is that when the first dhcp request is sent, the client don't know it's ip and source ip is set to 0.0.0.0, nor does it know the ip of the server so the destination ip is also set to 0.0.0.0:
pass in quick proto udp from 0/32 port = 68 to 0/32 port = 67 keep state pass in quick proto udp from 192.0.2.0/24 port = 68 to 192.0.2.1/32 port = 67 keep state
for inbound traffic and for outbound:
pass out quick proto udp from 192.0.2.1 port = 67 to 0/32 port = 68 keep state
TFTP receives requests on udp port 69. To allow tftp you must add the following rule:
pass in quick proto udp from 192.0.2.0/24 to 192.0.2.1 port = 69 keep state
FTP is more complicated. The server listens on port 21 for incoming connections, as with any other service. FTP specifies two data transfer modes, active and passive. In active ftp-data, the server connects back to the client from port 20 to some unprivileged port chosen by the client. In passive ftp-data the client connects to some unprivileged port chosen by the server, typically in the dynamic port range.
# ftp pass in quick proto tcp from 192.0.2.0/24 to 192.0.2.1 port 21 flags S keep state # ftp-data passive pass in quick proto tcp from 192.0.2.0/24 to 192.0.2.1 port 49152:65535 flags S keep state
allows ftp-sessions and passive ftp-data, to allow active ftp-data, add the following line
# ftp-data active pass out quick proto tcp from 192.0.2.1 port 20 to 192.0.2.0/24 port > 1023 flags S keep state
We forced mountd to bind to a specific port, 59, this makes firewalling easy:
pass in quick proto tcp from 192.0.2.0/24 to 192.0.2.1 port 59 flags S keep state pass in quick proto tcp from 192.0.2.0/24 to 192.0.2.1 port 111 flags S keep state pass in quick proto tcp from 192.0.2.0/24 to 192.0.2.1 port 2049 flags S keep state pass in quick proto udp from 192.0.2.0/24 to 192.0.2.1 port 59 keep state pass in quick proto udp from 192.0.2.0/24 to 192.0.2.1 port 111 keep state pass in quick proto udp from 192.0.2.0/24 to 192.0.2.1 port 2049 keep state
Note: Originally, nfs was based on udp but FreeBSD's implementation supports tcp as well.
The easiest way to populate the tftp root directory is to copy the boot directory from the instalation media.
The loader.rc works fine, it will show the standard menu and allow the user to select boot mode. But this may not make much sense for diskless or jumpstart operation. Edit the loader.rc to contain only the following lines:
include /boot/loader.4th start check-passwd
For jumpstart installation, the pxeboot loader must be built with support for tftp in order to enable the loader to fetch the kernel an other files using tftp, rather than mounting the root file system with nfs and then load kernel and other modules.
# cd /usr/src/sys/boot # make -DLOADER_TFTP_SUPPORT=YES
Install the pxeboot loader into the tftp directory, /var/tftp:
# cp i386/pxeldr/pxeboot /var/tftp/boot/
In this section covers how to set up a jumpstart installation. By the end of this section, you should have set up your server such that all you need to do to install FreeBSD is to connect your new machine to your network, and it will boot and load the interactive installation program. FreeBSD can then be installed downloading the release using FTP.
If you copied the boot directory from the installation media you have the necesary loader configuration file. This will load the memory file system and start the interactive installation program just as if you're booting off a cd-rom. Otherwise, for Jumpstart installation, create boot/loader.conf containing the following:
mfsroot_load="YES" mfsroot_type="mfs_root" mfsroot_name="/boot/mfsroot" autoboot_delay=0
You may add any kernel modules to be loaded with the usual syntax, in that case these are fetched with tftp and must be in the tftp directory.
Although the path specified in mfsroot_name appears absolute, on the server, this is relative to the tftp directory.
When the system boots, the kernel will look for an exectuable to startup, this can be controled setting the variable init_path. This must be set if you want to start sysinstall to install a new system and both init and sysinstall are on the root file system.
The PXE loader will load the memory file system specified above using tftp. If you copied the boot directory from an installation media you have the needed memory file system installed.
If you plan on interactive installation or only simple scripting, there is no reason to do the trouble of customizing a memory root file system.
Booting with PXE takes more time as all the files needs to be fetched over the network. The pxeboot program will first look for compressed files, then the non-compressed, the only file you cannot compres is pxeboot.
So, to speed up the transfer, you can gzip the kernel, any kernel modules and the memory file system. We may need to edit and customize the configuration files, so we let these be uncompressed.
The server is ready, all you need to do is to boot up the client and it will boot up in interactive installation mode. The FreeBSD installer will by default install the release correresponding to the kernel release.
If you compiled the kernel from the sources, then the loaded kernel may be named X.X-STABLE. There is no FreeBSD release with that name. To change the release, in the interactive menu, enter into Custom then Options and set the release name.
Before you rebooting your new system, make sure it won't boot again using pxeboot, hence repeating the process. My experience with the VIA board was that I had to disable LAN boot completely in the bios, not just have it as secondary boot device.
Warning This section covers advanced topics and should not be assumed to be exhaustive or precise. Be prepared to do some debugging as well as looking into the source code, as well as your own scripting.
If you are going to set up a large number or servers with the same base installation, you'll quickly get tired of navigating the sysinstall menus, after all the whole benefit of Jumpstart is to plug in your new machine and drink coffee. This section is not intended to be exhaustive but to get you going in the right direction.
There are four ways to attack this problem:
Script sysinstall: sysinstall offers limited scripting but this may be enough for your needs. sysinstall allows calling other system commands.
Create a custom memory file system with extra scripting funciontalities and a script that fetches the desired configuration and any other files needed. The idea is that a minimal script is executed on boot, this script loads all the customization, configuration and scripting files from a server.
If the client has plenty of RAM, just create a big memory filesystem and install the base system into that. This is not much different from diskless operation, but it takes much more time to boot as the entire file has to be transfered before it can be mounted.
Let the server the server boot in diskless operation. Then you have the posibiity of installing any tool you might need to script the installation to suit your needs. This method won't be covered here but can be derived from the methods described and the section on diskless operation.
If you only want to script sysinstall then you can use the memory file system distributed with the installation media, and skip to Scripting sysinstall
We need to create and mount the memory device to install the memory root file system. It can be any size as long as the jumpstart client has enough RAM. If the client has sufficient RAM you can create a memory file system big enough to install the entire base system. That will save you the trouble of making a bootcrunch file. To create a 16MB memory disk (this should be sufficient), run the commands:
# dd if=/dev/zero of=/var/tftp/boot/mfsroot bs=1k count=16k # mdconfig -a -t vnode -f /var/tftp/boot/mfsroot -u0 # disklabel -r -w md0 auto # newfs /dev/md0 # mount /dev/md0 /mnt
We now have an empty memory disk which will be served the client. Now we need to populate the file system to start the system installation, first create the base directory layout:
# mtree -U -p /mnt -f /etc/mtree/BSD.root.dist
Essentially, the memory disk device distributed on the installation media contains just a minimal set of configuration files and a "bootcrunch" file. A bootcrunch file is a special compressed binary containing statically linked executables. This format is useful when storage is limited. This section explains how to create a bootcrunch file to suit your needs.
There are two ways of creating a bootcrunch file, you can use crunchgen(1) to generate the make files and build the bootcrunch file, or you can use the rescue make files as a template.
Important: This approach is really not recommendable, I've inclouded it for completness of the guide.
Copy the file release/i386/boot_crunch.conf to a new directory. This file configures the programs and libraries to be included. You can edit this adding programs as needed. Then run the command:
# crunchgen boot_crunch.conf
This generates make file for compiling the bootcrunch file. Without any changes to boot_crunch.conf this will build the bootcrunch file for the memory file system distributed with the installation media. To build the bootcrunch file, run the command:
# make -f boot_crunch.mk
This builds the boot_crunch binary which must then be installed. on the memory file system. But, I have not found make files for installing the boot_crunch, this must be done manually, and for each command you must create a hardlink with the command name refering to the boot_crunch binary.
In this section we will use the makefiles found in the directory resuce as skeleton to create a custom bootcrunch file. It turns out to be easier, because the make file also allows us to easily install the bootcrunch binary on the memory file system.
First create a copy of the rescue directory to leave original unmodified:
# cd /usr/src # cp -R rescue jumpstart # cd jumpstart # mv rescue jumpstart # mv librescue libjumpstart
In this directory there are three make-files, one in the top directory and one in the two sub-directories jumpstart and libjumpstart. Edit all these to replace all occurences of "rescue" with "jumpstart". Do not replace the upper case occurences of "RESCUE".
Edit jumpstart/Makefile to add any additional programs or libraries you may need. For a start add the programs compiled in the standard boot crunch not included, adding the following lines to the appropriate sections of the make file:
CRUNCH_PROGS_usr.bin+= cpio find CRUNCH_PROGS_usr.sbin+= arp ppp sysinstall usbconfig CRUNCH_LIBS+= -lftpio -lnetgraph -lncurses -ldialog -ldisk -ldevinfo -lusb CRUNCH_BUILDOPTS_ppp+= -DRELEASE_CRUNCH CRUNCH_BUILDOPTS_sysinstall+= -DRELEASE_CRUNCH -Dlint
Next can add any additional programs that can be useful for scripting the installation. Whenever you add programs you might need to add libraries as well, check the Makefile of each program as you add them. You can use my customized Makefiles.
Now build and install the bootcrunch file:
# make obj && make # mkdir /mnt/jumpstart # make DESTDIR=/mnt install
The above assumes the memory file system is created and mounted as shown in the previous section. Optionally you can run make with one or more of the following parameters: WITHOUT_ATM=yes, WITHOUT_INET6_SUPPORT=yes, WITHOUT_IPFILTER=yes, WITHOUT_IPX=yes, WITHOUT_OPENSSL=yes, WITHOUT_RCMDS=yes, WITHOUT_TCSH=yes, WITHOUT_ZFS=yes.
And to finish up,
# cd /mnt # rmdir /mnt/sbin /mnt/bin # ln -s jumpstart bin # ln -s jumpstart sbin # ln -s jumpstart usr/bin # ln -s jumpstart usr/sbin # ln -s jumpstart stand
Creating symbolic links ensures that any scripts or programs that assume a particular path will still work. Unmount the file system and run fsck:
# umount /mnt # fsck -t ufs /dev/md0 # mdconfig -d -u 0 # gzip /var/tftp/boot/mfsroot
At this point we have a fully equipped memory file system that can be used for installation or a rescue network boot. You can download my custom mfsroot created with the custom make files above.
Important: If you use a custom memory filesystem such as that created above with both init and sysinstall you must configure the kernel to start sysinstall. This is done editting loader.conf and setting the parameter init_path=/stand/sysinstall (or whichever is the path to sysinstall).
sysinstall(8) will first look for the file install.cfg on the root file system which sets installation parameters. This file can also be used to script sysinstall. When the execution exits, sysinstall will return to the interactive mode.
Hence, to automate the installation we need to create install.cfg and place it in the root of the memory filesystem we created. Remount the memory file system to create and store the installation configuration file.
The syntax, variables and commands are described in sysinstall(8), however not all variables and commands are documented, for theese, we need to take a look at the source code also. Also a sample install.cfg is given in the source code directory of sysinstall.
Note: install.cfg is read and executed strictly top down, variables must be set before the function using them.
If anything fails in the script, sysinstall will abort the rest of the script and go interactive.
We don't want any interactive questioning, the whole point is to power up the machine and drink coffee. But we do like to have some debug information for when we get back and everything has failed misserably.
################################ # install.cfg for jumpstart of FreeBSD # # See sysinstall(8) for details about how to script the process # This file has been edited from /usr/src/usr.sbin/sysinstall/install.cfg # Turn on extra debugging. debug=YES nonInteractive=YES noConfirm=YES noWarn=NO ################################
Next select the disk and create the slices for the FreeBSD install. Multiple disks can be partitioned but you must end each disk defintion with diskPartitionEditor.
################################ # Disk partitioning # # WARNING: This will format the disk and dedicate the entire disk to # FreeBSD disk=ad0 partition=all bootManager=none diskPartitionEditor ################################
Warning The sample install.cfg provided with sysinstall source sets partition to exclusive, this will set your disk in "dangerously dedicated" mode which is not recommended unless you really need it.
For each slice, create the labels and specify the mount points. The line labeling /home is special, this will use any remaining space, so it must be last. This section must end with diskLabelEditor.
################################ # Disk labeling # # All sizes are expressed in 512 byte blocks! # # For example: # / 512MB, swap 512MB, /usr 8192MB, /var 8192MB, /home remaining ad0s1-1=ufs 1048576 / ad0s1-2=swap 1048576 none ad0s1-3=ufs 16777216 /usr ad0s1-4=ufs 16777216 /var ad0s1-5=ufs 1048576 /tmp ad0s1-6=ufs 0 /home 1 diskLabelEditor ################################
Apparently sysinstall suffers from amnesia when it comes to the network configuration. Even though the network was configured on boot, sysinstall will need a network configuration as above unless you are installing from a disk.
This should be a general configuration file and all host configuration should be done with dhcp. For unknown reason, hostname is required even if passed with dhcp.
################################ # Host specific configuration: tryDHCP=YES netDev=vr0 # The following optional if using dhcp to configure the network hostname=jumpstart domainname=example.com ################################
By default sysinstall determines the FreeBSD release to install from the kernel variable kern.release, and will try to fetch installation files from a directory with that name. If you built the kernel from sources, then the kernel release may not corresond to a system release and you must specify the release to install:
################################ # Select release to install # releaseName 8.0-RELEASE ################################
FreeBSD has some predefined distributions you can choose: distSetMinimum, distSetUser, distSetXUser, distSetKernDeveloper, distSetXKernDeveloper, distSetDeveloper, distSetXDeveloper and distSetEverything. You can also customize defining the variable dists and then use distSetCustom.
Note: Some have reported problem using the custom distribution, that the kernel is not correctly installed. A work around is to install the minimal distribution and commit it, then select custom and commit that.
################################ # Set distribution to install, either use distSetCustom and choose # individual components or choose a collection, for example # distSetKernDeveloper which installs base and kernel sources distSetKernDeveloper ################################
Again (!?) we need to specify the interface we want to use and configure it using dhcp. For ftp installation we must specify a server and path to the selected release. You can always check the release announcement for a list of available ftp servers, if you do not set up your own.
################################ # Select installation method # # We want an FTP install, so we also need to specify the ftp server to # fetch from netDev=vr0 tryDHCP=YES _ftpPath=ftp://ftp.example.com/pub/FreeBSD mediaSetFTP ################################
With the above settings, sysinstall will fetch the release from this directory:
ftp://ftp.example.com/pub/FreeBSD/releases/i386/8.0-RELEASE
That is, sysinstall assumes the release files are found in the subdirectory releases/<architecture>/<releaseName> in the ftp-path.
At this point, your old disk is still alive and happy, the next section will commit everything to the disk, and we have set noConfirm!
################################ # # OK, everything is set. Do it! installCommit ################################
You can customize further after install commit setting the variable package to any package you'd like installed and followed by the command packageAdd. At this point, when installation is finnished it will return to the well known sysinstall menu to allow you to add any further customization. To avoid this end the install.cfg with the command shutdown(8). It may be better though, if you compiled your memory disk with halt(8) to terminate the installation with system halt -p.
If everything works well, simply power up your system, it should be busy installing installing and you busy drinking coffee :-)
It is a bit tiresome to change the install.cfg if it is located on the memory disk. If you created a custom memory file system with support for the ftp(1) command, then could create a minimal install.cfg and then fetch one to be included like this:
################################ # Host specific configuration: tryDHCP=YES netDev=vr0 # The following optional if using dhcp to configure the network hostname=jumpstart domainname=example.com ################################ system /bin/ftp ftp://ftp.example.com/pub/FreeBSD/jumpstart.cfg loadConfig jumpstart.cfg
This should configure the network interface and fetch the installation configuration file (altough I have not tested it yet). Furter scripting is posible, and you can also change loader.conf to run init(8) instead and script everything - the limits are given by the commands available on the root file system.
Note: This section is is a work in progress. Things may or may not work following the proceedures described.
You don't need to use sysinstall to install the system, we can start init and let init run a custom installation script. This is the approach for this section. The benefit is more powerful scripting. To do this, the memory file system must provide some additional system tools to configure and partition disks and install the operating system.
Configure the loader by editing the file loader.conf and add the following lines:
init_path=/sbin/init init_script=/etc/rc.jumpstart init_shell=/bin/sh
With these lines, the kernel will run init as the initial process and init will run the script rc.jumpstart. The last parameter is required although the loader(8) writes this is the default value. Building using the rescue Makefiles, this defaults to /rescue/sh.
If you boot at this point, init will complain that no rc.jumpstart is found and let you drop to a shell. From here, all you need to do is to create the script that suits your needs.
This section covers how to set up a diskless environment. By the end of the section you should be able to boot up your diskless client with access to the FreeBSD base system.
If you use the pxeboot loader distributed with the installation media, the loader will nfs mount the root file system and boot in diskless mode, you only need to configure the root file system.
If you built the pxeboot loader with tftp support, the loader will fetch loader configuration, kernel and kernel modules using tftp. Then once the kernel loads, mount the root file system using nfs.
The problem with this is that the kernel will have lost the parameter specifying the root file system to mount, that is if you're using the GENERIC kernel. The reason is that the GENERIC kernel is not built with the options:
options BOOTP # Use BOOTP to obtain IP address/hostname
# Requires NFSCLIENT and NFS_ROOT
options BOOTP_NFSROOT # NFS mount root filesystem using BOOTP info
options BOOTP_NFSV3 # Use NFS v3 to NFS mount root
These options allow the kernel to set certain parameters using dhcp, including the root-path. If you build the kernel with BOOTP_NFSROOT option, the root-path set with dhcp will override any settings in your loader.conf file. You can build the kernel with the above options or use the pxeboot loader with nfs enabled.
Alternatively, the root file system can be configured in the loader configuration file.
In the following we will create a separate root file system for the diskless clients. This has many advantages: You can have different kernel and load different modules on the diskless client than on the server, you can easily move or copy the base system to another server, upgrade or migrate without affecting the server and you don't risk exporting confidential information, you can disable the root account on the diskless clients.
You can have different base systems for different clients, in the dhcp configuration you just need to specify different root paths. If you use the pxeboot loader with nfs support, you can even have different kernels for different clients.
You can make a more lean installation by disabling unneeded features in /etc/make.conf. Build and install of the kernel is optional if the kernel and modules are fetched with tftp from the tftp boot directory, but it allows loading of kernel modules after boot. You must install the kernel if you use pxeboot with NFS.
The following builds the base system and installs into /var/diskless/FreeBSD:
# cd /usr/src # make buildworld # make buildkernel # mkdir /var/diskless/FreeBSD # make DESTDIR=/var/diskless/FreeBSD installworld # make DESTDIR=/var/diskless/FreeBSD installkernel # make DESTDIR=/var/diskless/FreeBSD distribution
At this point you should be able to boot up the diskless clients with the root file system mounted read-only.
For each diskless node we must create separate file systems for /var and swap. Swap over NFS is slow, but may be necesary. For each node do:
# mkdir -p /var/diskless/<node>/var # mtree -U -f /etc/mtree/BSD.var.dist -p /var/diskless/<node>/var # dd if=/dev/zero -of=/var/diskless/<node>/var/swap bs=1m count=64
Where <node> is the node ip address. In this example 64MB was allocated for swap. Rather than creating a separate nfs share for /tmp create a symlink to /var/tmp:
# cd /var/diskless/FreeBSD # rmdir tmp # ln -s var/tmp tmp
The configuration of the client is done on the server. The first to do is do edit /var/diskless/FreeBSD/.cshrc and set set promt = "FreeBSD diskless # ". Now chroot(8) into /var/diskless/FreeBSD. The prompt should now change,
FreeBSD diskless #
indicating that you are in the diskless file system, this helps you avoid messing up your server configuration. In the following, paths are relative to the root in the chrooted environment.
The root file system is mounted at boot, but this is mounted read only. The node specific share /var, and /home must be mounted, and we must enable swap on the swap file created above.
All diskless clients will share the same fstab for the common file systems:
# Common fstab for diskless clients # Device Mount FStype Options Dump Pass nfs.example.com:/home /home nfs rw,late,userquota 0 0
The dump frequency and pass numbers are set to 0 to avoid running fsck(8). fsck should never be run on nfs mounted file systems, if the partition becomes corrupt, run fsck on the server. The late key word means that the home partition will be mounted last by the mountlate rc-script.
The /var Partition is not included in the common fstab above since this is different for each host. To mount /var, create the rc script nfsvar in /etc/rc.d:
#!/bin/sh
#
# PROVIDE: varnfs
# REQUIRE: mountcritremote
# BEFORE: devfs
. /etc/rc.subr
name="varnfs"
stop_cmd=":"
start_cmd="varnfs_start"
start_precmd="varnfs_precmd"
# Mount var NFS filesystem
#
varnfs_precmd()
{
case "`mount -d -a -t nfs 2> /dev/null`" in
*mount_nfs*)
# Handle absent nfs client support
load_kld -m nfs nfsclient || return 1
;;
esac
return 0
}
varnfs_start()
{
# Mount nfs filesystems.
#
echo -n 'Mounting var NFS file system:'
if [ -z "${varnfs_server}" -a \
-n "`/bin/kenv boot.nfsroot.server 2> /dev/null`" ]; then
nfsserver=`/bin/kenv boot.nfsroot.server`
fi
if [ -z "${varnfs_path}" -a \
-n "`/bin/kenv boot.nfsroot.path 2> /dev/null`" ]; then
nfspath=`/bin/kenv boot.nfsroot.path`
fi
if [ -n "`/bin/kenv boot.netif.ip 2> /dev/null`" ]; then
ipaddr=`/bin/kenv boot.netif.ip`
fi
if [ -z "${varnfs_server}" -o -z "${varnfs_path}" -o \
-z "${ipaddr}" ]; then
echo "Failed determining nfs share"
exit 1
fi
/sbin/mount -t nfs "${varnfs_server}:${varnfs_path}/${ipaddr}/var" /var 2> /dev/null
echo '.'
# Make sure we have /var/log/lastlog and /var/log/wtmp files
if [ ! -f /var/log/lastlog ]; then
cp /dev/null /var/log/lastlog
chmod 644 /var/log/lastlog
fi
if [ ! -f /var/log/wtmp ]; then
cp /dev/null /var/log/wtmp
chmod 644 /var/log/wtmp
fi
# Cleanup /var again just in case it's a network mount.
/etc/rc.d/cleanvar quietreload
rm -f /var/run/clean_var /var/spool/lock/clean_var
}
load_rc_config $name
run_rc_command "$1"
The above script will mount the /var from the same server as the root file system unless the parameter nfsserver is specified in the rc.conf. Likewise, it will assume finding the var share in the same path as the nfs root path, unless set with nfspath in rc.conf. For our setup, we set nfspath=/var/diskless.
We setup the /tmp as a symbolic link to /var/tmp. If you want a separate tmp share, you need to create a similar script for mounting this share.
At this point, you should be able to boot up the diskless client but there is still tons of stuff that remains to be tweaked to make the system useful. Among other things, we haven't considered how to manage users. The next section covers management and finetuning.
Warning This section covers advanced topics and should not be assumed to be exhaustive or precise. Be prepared to do some debugging as well as looking into the source code, as well as your own scripting.
While we can boot now, there are still a lot of things that remains to be configured or tweaked to work properly. This section discusses some of these. This section will probably be incomplete forever, some things are not included because they are not any particular to diskless clients.
For a start, you might want to copy the password and group files from the server, however, managing users becomes significantly easier if you use LDAP or NIS to make user account information accessible from all hosts.
NIS is parcially supported in the FreeBSD base system, but LDAP is recommended as it easily integrates with other network services and other systems. This section will document an LDAP solution.
To support LDAP you need to install the following packages:
Obviously you also need to setup an LDAP server. You need to install net/openldap22-server. Setting up the LDAP server is beyond the scope of this document. Creating an LDAP directory and configuring clients to authenticate against this will not be covered at this moment.
Since we installed the full system in a separate directory, maintaining it is as simple as chroot'ing into that directory and update/install as on a normal system, both for installing third party applications from ports and for updating the base. This will not interfere with the server applications.
One thing to be aware of, when we set up the server, we installed kernel and modules in /var/tftp/boot on the server. This path is not within the diskless base directory. To solve this, you can change the configuration of the tftp server such that the kernel is loaded from /var/diskless/FreeBSD/boot or such that kernel is loaded via nfs.
For some ports to build correctly, you may need access to the device file system within the chroot'ed environment. To get this, before chrooting, run the command:
# mount -t devfs devfs /var/diskless/FreeBSD/dev
The first thing you want to do is to update the ports tree and source files.
FreeBSD Jumpstart Guide, Alfred Perlstein
Diskless X Server: a how to guide, Jerry Kendall
Pluggable Authentication Modules, Dag-Erling Smørgrav
MAN-pages: diskless(8) pxeboot(8)