IP Addressing
Wired NICs
The testbed is connected by a wired Ethernet segment, which for security reasons is separated from the regular building network. All NICs on the the wired testbed network have IP addresses in the private 10/24 range, i.e. 10.0.0.*. The allocation is as follows.
Dux - main server: 10.0.0.1
Reserved - aux servers: 10.0.0.2-8
Testbed nodes: 10.0.0.9-up
Switches:
10.0.0.252-254
This allocation is permanent and researchers are expected to leave it as it is. Allocation of these addresses is done in the file /etc/bootptab on the main server. This is also the file to edit or add more nodes to the testbed.
We describe how to add nodes in the testbed in the Deployment section.
Wireless NICs
Each testbed node has one wireless interface (for now). In the standard distro, addresses for these are allocated in the private 192.168.1.* subnet, using the same last 8 bits as in the wired address, as follows.
testbed nodes 192.168.1.9-up
However, it is up to each experimenter to allocate whatever wireless addresses he or she wishes. To change the address allocation, please edit the file /etc/init.d/borg in the borg distro.
Below we provide a detailed, easy and to the point tutorial on how to get guns blazing with your code on the testbed. In order to set-up the server for the testbed, you have to perform the following steps. These steps include solutions to many problems that you will come up against, if you try to configure things without help. Our tips will save many weeks of configuration, both for the client and the server set-up. Note that you need to have some basic knowledge about Linux, in order to deal with such a situation.
Server set-up steps
1. Set-up a machine with linux installation, which will be the testbed server. This machine - server must be equipped with 2 ethernet cards. One of them will be used for the connection of the server to the rest of the world, and the other for the communication of the server with the wireless nodes (we will call them clients throughout the end of this document). Set-up the testbed ethernet interface with a dummy IP address, e.g. 10.0.0.1. We choose Debian v3.1 as our server linux distribution.
2. Disable the server's firewall.
3. Install TCPDUMP, TFTPD (Trivial FTP Daemon) and BOOTP (Bootstrap Protocol) to the server, if they are not already installed.
4. Enable the bootp service by running bootpd, by the command:
bootpd -s -d 7 bootptab <file path>
This will initiate bootpd in debug mode, so as for bootp messages to be
displayed and provide information about the communication procedure.
On our version, we also have to add a route for the broadcast address to make bootpd send its stuff to out the right port
route add 255.255.255.255 eth1
5. Enable the server's NFS system, by initiating the nfsd daemon process. Moreover, edit the /etc/exports and create an entry for the broadcast
address, so as for every node to be able to boot the NFS of the server. This action essential for this step, so as to make sure that the server will definitely not block the client from booting - since the client belongs to the same subnet. In particular, the /etc/exports entry should look like this:
tftpshare/borg 10.0.0.0/255.255.255.0(rw,no_root_squash,no_all_squash)
In addition, for now, we do not add any entries to the /etc/hosts.deny file. In the future, and after we make sure that all desired clients can boot, the /etc/hosts.deny file should block all the clients, while the /etc/hosts.allow should have entries for all the testbed clients and only them (unless the testbed server will be used for other networks as well). Specifically, the permanent hosts.deny file should be the following:
portmap:ALL
lockd:ALL
mountd:ALL
rquotad:ALL
statd:ALL
The hosts.allow file should have the above entries, however instead of
"ALL" the clients' IPs must be entered.
6. Enable the portmap (or rpc.portmap). It should be located in /sbin or /etc/sbin. Make sure that portmap is up and running, by typing
ps aux | grep portmap
Furthermore, the following daemons must be enabled (if they are not enabled already): rpc.nfsd, rpc.portmap, rpc.mountd, rpc.statd, rpc.lockd and rpc.rquotad. After future modifications to /etc/exports, the NFS must be restarted. Alternatively, the command:
exportfs -ra
will force nfsd to re-read the exports file.
7. Make sure that the NFS is up and running. One easy way to do this, is to setup an NFS client in the server machine. For this, just type
mount 10.0.0.1:/root /mnt/root
and the directory /root on server should appear as the directory /mnt/root on the client. More details about setting up an NFS server can be found at NFS-Root mini-HOWTO
Client set-up steps
Here we describe in more detail the steps that we follow in order to boot the client
1. Start with trying to connect the server with one client only. For this, a client machine with an ethernet interface is required. Initially we used a
simple AMD Sepron MICROTEL PC as a test client. In such a case, enable the property "boot from LAN", and make this the default action. It would be better (but not necessary) to disable all the other boot options. Save the new bios configuration. Now, every time the client starts, it tries to send a DHCP/BOOTP boot request to anyone that can hear. For some machines (e.g. Soekris boxes), no bios modification is required.
2. Download an appropriate linux distribution in the server's hard disk. This will be the linux that clients will be booting from the server. There are many potential distributions for this purpose. So far we have experimented with a Debian v3.0 and v3.1 Linux (which we call "borg").
Extract the distribution into tftpshare/. Compile the kernel with NFS capability enabled. During the kernel configuration (prior to the compilation) make sure to remove most useless modules (e.g. usb support, parallel port, mouse, cdrom, sound, etc), and add modules-drivers for the wireless cards, as well as the clients' ethernet card drivers. Also, make sure to enable the "IP-Autoconfiguration" property.
3. Create a BOOTP configuration file (bootptab), wherein the network and client properties are stored. This will make the server know which clients are
likely to send bootp requests to which network. In the bootptab file, the clients' IPs and MAC addresses will be stored. A sample initial bootptab configuration file is the following. We assume that the client's IP is 10.0.0.200.
.net1: \
:bf=/borg/boot/pxelinux.0: \
:rp=/borg: \
:gw=138.23.169.11:
test_client:tc=.net1:ht=1:ha=01142A8BDDFA:ip=10.0.0.200:
Further details about the bootptab configuration file can be found in:
* http://www.br.fgov.be/RESEARCH/INFORMATICS/info/bootp.html
* http://man.he.net/man8/bootpd
* http://man.he.net/man5/bootptab
Make sure to type the IP and MAC of the client correctly.
4. Edit the tftpshare/borg/boot/pxelinux.cfg/default. This file contains the borg kernel parameters. Insert the correct parameters and save it. A sample default file (which works for us) is the following:
prompt 0
default linux
timeout 5
label linux
kernel bzImage
append rw ip=dhcp root=/dev/nfs nfsroot=10.0.0.1:/tftpshare/borg/
Note that the /dev/nfs file in tftpshare/borg must already exist. If not, it can be created by the following command:
mknod 0 255 b /dev/nfs
(assuming that we are in folder tftpshare/borg).
Switch set-up
Here we describe in more detail the steps that we follow in order to install a switch. The server is connected to the nodes through a set of switches. These switches also provide power (over Ethernet) to each node. The testbed's switches have been placed in the department's network closets, together with the department's switches.
In order to add a node to the network, the admin has to know the ethernet port number that the node will occupy. Each ethernet port number corresponds to a port in one of the network closets. This port "must" be connected to the testbed's switch. By this way, the node will be able to communicate with the server (dux.cs.ucr.edu), as long as there is an entry in the bootptab file for this node.
Installing a PoE switch is not a big deal; the most difficult part is to get authorization to access the network closet. The switch will be install in the same closet as the department's switches. As soon as the corresponding ports are connected, the installation is finished.
The switches that we have so far are equipped with a web interface. In order to access it, one has to connect a machine to the testbed network, and use a web client. The web interface provides very useful information, such as e.g. which ports are enabled, and how much power each port consumes. Moreover, through the web interface one can set the IP of the switch.
By default the switch's IP is 192.168.0.1. In order to change this, one has to connect a machine to the testbed, and set the IP of this machine to be the latter one. Further one can change the IP of the switch to the preferable one. There may be times where the IP of the switch is automatically set to be the default one - 192.168.0.1 (e.g. perhaps after a power outage). You need to follow the above steps, in order to change it again. This is of course necessary only if you care contacting the switch now and then (e.g. in order to reboot the nodes).
Tools
So far, for measurement tools we have been using ttcp and mostly iperf. We are using ping to find neighbors per node. We provide a set of useful scripts for measurements and experiments, which can be downloaded from the download section.
Drivers
For the EMP 8602 wireless cards we are utilizing the MADWiFi driver for Atheros chipsets. For the Intel cards we are using the popular ipw2200 driver/firmware.
