OwnPages Zero-Cost Phone and Mail System

CONTENTS

1. Introduction

2. Audience

3. Install pilot machine

3.1: Setting up an test machine
3.2: Setting up an diskless directory on NFS
3.3: Compile a kernel

4. Configure

4.1: Configure Pxelinux,Tftp,Dhcp
4.2: Editing fstab

5. Test

5.1: Test first diskless desktop

6. Optimise

6.1: Creating swap
6.2: Fs-caching
6.3: Kernel parameters tuning
6.4: Optimizing start-up scripts

7. Replicate the diskless machines

7.1: Replicate onto other directories

8. Benchmarks

8.1: Benchmarks and performance testing:

*************************************************

1. Introduction

Diskless desktops give better manageability and central control. Read this link for some reasons to use diskless servers, arguments equally valid for diskless desktops.

However diskless desktops often suffer from performance issues, and if they number more than about 20, they are also prone to network congestion and NFS server load.

This tutorial will help you setup high performance diskless desktops employing NFS-caching, a feature of kernel 2.6.30 onwards. FS-cache allows NFS resident files to be cached locally on a USB stick or hard disk transparently to the user. For more info on fs caching visit this link

2. Audience

Tech administrators and experienced Linux hands. You will need to compile a custom kernel for this.

3. Install pilot machine

3.1: Setup distribution on test machine with a disk (preferably a virtual one)

I used Slackware - 13, but these instructions should work for almost any distro. Install your favorite distro on a disk partition, or a virtual machine (as I did). Ensure it boots and works and install all applications you are going to need. You can add applications with ease later however.

3.2: Setup pilot diskless desktop

Once the distro is setup on some sort of disk, create a directory on your NFS server:
Copy the root directory of your test machine into this directory. For this you will first have to export / from your test machine using its nfs server and mount it into your organisation's nfs server
.
Assuming your test machine has ip 192.168.1.100, then on your nfs server


Move the /usr and /opt (if you have one) directories to a lower directory and recreate empty directories in their place:
The above is because these two directories consume the bulk of disk space and can be shared read-only amongst the diskless desktops. E.g, in slackware 13, /usr consumes about 5GB fully loaded, out of about 5.8GB for the whole machine. For Ubuntu9 it consumes about 2GB out of 2.5GB overall.

3.3: Compile custom kernel

You will also need to compile a custom 2.6.30 kernel with network, graphics and sound drivers for all diskless machines you will be employing. The alternative to compiled-in drivers is initrd which increases boot time. While compiling the kernel, enable FS-cache in filesystems-->caches. Also enable "Root file system on NFS" and "NFS client caching support" in filesystems-->network filesystems

4. Configure

4.1: Configure PXELINUX, tftp and dhcpd

4.1.1: PXELINUX

We are assuming familiarity with Pxelinux. For more information on pxelinux and pxelinux.cfg please visit this site.
Create the file in pxelinux.cfg directory, so that the diskless machine would get the intended kernel and the root directory with appropriate options. Eg: suppose the IP address of the diskless_machine0 is 192.168.1.200, its hex format would be C0A80168(192=>C0,168=>A8 ...). The file under pxelinux.cfg directory would be named this. On catting C0A80168 file we should get:
The second line states that the machine with IP address 192.168.1.200 will have a hostname set to dl0 and will take its root partition from 192.168.1.11:/someplace/diskless_machine0 directory, where 192.168.1.11 is your nfs server.

4.1.2: tftp

The tftpserver is used to provide the kernel for the diskless machines to boot from.The first line indicates the kernel which the machine named dl0 will load during boot up. This kernel should be placed in the tftp server directory (usually /tftpboot ) and pointed to by the DHCP server.

4.1.3: dhcpd

In dhcpd.conf put:
where dl0 is the name of the first diskless machine as explained earlier in PXELINUX .

4.2: Setup fstab entries

Earlier I advised you to move /usr and /opt directory to a lower parent directory, because we will keep /usr and /opt identical for all diskless desktops. This will reduce disk space used, and also aid in maintainability. Any application added will usually reside in /usr or /opt, hence by adding it once it will be available to all.
Example of fstab:
In above note /dev/sda1, (which has not been explained so far!) and the ro and fsc flags for /usr and /opt. Since these directories will be shared, we prefer to mount them read-only. the fsc flag pertains to fs-caching which we will soon explain.
We have put /tmp on tmpfs, which takes care of emptying temporary files periodically, and also speeds things up. However, the desktop should be rebooted regularly, else /tmp will slowly consume a lot of memory.

5. Test

5.1: Test first diskless desktop

Now boot the pilot diskless machine.The boot steps will be: diskless machine dl0 will issue a dhcp request, it will get an ip along with the tftp server ip. It will ask the tftp server for its pxe configuration, and then request it for the kernel mentioned in the config. It will boot the kernel with the parameters specified in pxelinux, chief of which is the root nfs options.The kernel mounts the nfs server directory /someplace/diskless_machine0 as root. All this should be observable in the log files of the dhcpd, tftp and nfs servers respectively.

6. Optimize diskless desktop

6.1: Setup swap

After booting the pilot diskless machine, we have to setup the swap space on NFS for each diskless-machine. Create a file named swap on diskless_machine0's /var directory which will be used as swap by diskless_machine0. For a 256MB swap file:
Make necessary changes in the startup scripts to enable the swap services during boottime. Do not include it in fstab (the reasons for which are lost in the sands of time). Eg: I included following lines rc.local file of diskless_machine0.

6.2: Setup FS-cache

FS-Cache is a kernel module that provides a caching facility to a network filesystem such that the cache is transparent to the user. The cache resides on local storage; we have tested USB sticks as well as small hard disks. If you choose the latter your machine may not qualify to be a diskless machine any longer (:-)) but the maintainability advantages remain.
For implementing FS-caching for your diskless desktops create a directory in /var (of the diskless desktop) named fscache, fscache directory will act as an external cache. And then mount the external cache device (usually /dev/sda1) on /var/fscache. We used a 2GB USB stick from Transcend, the cheapest we could find. The second line in fstab listed earlier ensures /var/fscache gets mounted during bootup. Note the options in that line. To enable user_xattr in /dev/sda1 use:
FS-caching service is started by cachefilesd daemon which resides in /sbin directory. Make necessary changes in your start-up scripts to run cachefilesd during bootup. Your distribution will probably not include it at the time of this writing, so download and compile cachefilesd from here.
For Slackware I included following lines in my /etc/rc.d/rc.inet2 script file before mounting /usr and /opt (i.e right in the beginning).

Check to see if FS-cache is working by:
Run any application for the first time after setting up FS-cache and check to see if it is writing into its cache by:
The first invocation of any application may be painfully slow as it writes the application into cache. The next time however, it should be be reading from the cache in the sar command above.

6.3: Set kernel parameters

Diskless machines break some of the assumptions behind standard kernel behavior. We found some good tips on optimizing diskless machines here . We put the following options in /etc/sysctl.conf:

6.4: Optimizing start-up scripts

The diskless desktop normally would not be running as a server, so comment out various services that get started during boot time. Also there are various databases which get updated at each boot which are often overkill.

7. Replicate the diskless machines

After optimizing and testing the pilot diskless machine, we replicate it by simply copying the /someplace/diskless_machine0 directory as many times are required. Use cp -dpR. We found it useful to name the directories on the ip addresses. So machine 192.168.1.74 was named dl74 and its root directory on the nfs server is /someplace/diskless_machine74. If you have configured the machines as we have indicated, then no customization of the individual root directories should be needed*. You will have to put the ip and mac address of each desktop in dhcpd.conf and create a file in pxelinux.cfg directory with its hostname and nfs root directory.

8. Benchmarks

8.1: Benchmarks and performance testing:

Here's the shocker: If you do performance testing of fs-cache on a usb stick vs nfs with no caching under IDEAL conditions, there is almost no improvement. Ideal conditions we define as the application to be loaded being warm in the RAM cache of the nfs server, and the nfs server serving just the testing nfs client. However, in real conditions when there are more than 15 - 20 diskless clients, the nfs server and the network becomes a bottleneck. In such situations FS-cache shines, since the files are served locally from the usb stick, allowing excellent scalability. Fs-cache continues to check for version changes of the file against the nfs server, so any changes and updates are propagated to all diskless machines. Also, FS-cache speed depends on speed of local storage used, so a faster usb stick or one based on usb 3 should be better. The USB stick we used (Transcend 2gb) measured max read performance at 20mbytes/sec and about 1200seeks/sec
With these caveats the performance tests are as below:

	100Mbps without fscache	100Mbps with fscache on USB 2.0 stick	1000Mbps without fscache	1000Mbps with fscache on USB 2.0 stick
bootup-time	20	20	x	x
login to desktop	15	15	x	x
Firefox 3.1	7	5-6	2.5	5
Opera 9.64	7-8	6	2.5	5
Open Office 3.1	17	17	7	16

by Shridhar  

Labels: diskless machine, howto, root nfs, slackware 13