Leistbares Supercomputing mit 4 AMD64 AM2+ X2 3800

[kikakater]

Microwulf: Breaking the $100/GFLOP Barrier

There are more lower cost clusters that have laid claim at one time or another to being the price/performance king. As we'll see, Microwulf has the lowest price/performance ratio ($94.10/GFLOP in January 2007, $47.84/GFLOP in August 2007). than any of these.

We installed Ubuntu Server on each of the top 3 "compute" nodes, since it did not carry all of the overhead of Ubuntu Desktop.

Before we actually did the NFSroot for the "compute" nodes, we considered the idea of using iSCSI instead of NFS, mainly based on a paper we found. But we decided that it was more important to get the cluster up and running, and Ubuntu makes NFSroot really easy. It's as simple as editing the /etc/initramfs.conf file to generate an initial ramdisk that does NFSroot and then setting up DHCP/TFTP/PXELinux on the head node, as you would for any diskless boot situation.

We did configure the network adaptors differently: we gave each onboard NIC an address on a 192.168.2.x subnet, and gave each PCI-e NIC an address on a 192.168.3.x subnet. Then we routed the NFS traffic over the 192.168.2.x subnet, to try to separate "administrative" traffic from computational traffic. It turns out that OpenMPI will use both network interfaces (see below), so this served to spread communication across both NICs.

We did achieve a peak performance of 26.25 GFLOPS. The theoretical peak performance for Microwulf is 32 GLFOPS. (Eight cores x 2 GHz x 2 double-precision units per core.) This means we have hit about 82% efficiency (which we find remarkable). Note that one of the reasons we asume that we achieved such a high efficiency is due to Open MPI, which will use both GigE interfaces. It will round-robin data transfers over the various interfaces unless you explicitly tell it to just use certain interfaces.

Epilogue: Prices for Microwulf (August 2007)

System component prices are dropping rapidly. CPUs, memory, network, hard drives - all are dropping at a pretty good clip. So how much would we pay if we built Microwulf in August, 2007? Let's find out using Newegg for our prices.

Component Product Unit Price Quantity Total
Motherboard MSI K9N6PGM-F MicroATX $50.32 4 $201.28
CPU AMD Athlon 64 X2 3800+ AM2 CPU $65.00 4 $260.00
Main Memory Corsair DDR2-667 2 x 1GByte RAM $75.99 4 $303.96
Power Supply LOGISYS Computer PS350MA MicroATX 350W Power Supply $24.53 4 $98.12
Network Adaptor Intel PRO/1000 PT PCI-Express NIC (node-to-switch) $34.99 4 $139.96
Network Adaptor Intel PRO/100 S PCI NIC (master-to-world) $15.30 1 $15.30
Switch SMC SMCGS8 10/100/1000Mbps 8-port Unmanaged Gigabit Switch $47.52 1 $47.52
Hard drive Seagate 7200 250GB SATA hard drive $64.99 1 $64.99
DVD/CD drive Liteon SHD-16S1S 16X $23.83 1 $23.83
Cooling Zalman ZM-F3 120mm Case Fans $14.98 4 $59.92
Fan protective grills Generic NET12 Fan Grill (120mm) $6.48 4 $25.92
Support Hardware 36" x 0.25" threaded rods $1.68 3 $5.00
Fastener Hardware Lots of 0.25" nuts and washers $10.00
Case/shell 12" x 11" polycarbonate pieces
(scraps from our Physical Plant) $0.00 4 $0.00
Total $1,255.80