The Bura supercomputer¶
Cluster: The multi-computer system consists of 288 computer nodes with 2 x Xeon E5 CPUs (24 core per node) that make together 6912 processor cores. Each node has 64GB of memory and 320GB of dedicated disk space therefore there is 18TB of memory and 95TB of disk space available on the Bura supercomputer.
GPGPU accelerated nodes: GPGPU accelerated part consists of eight heterogeneous nodes. Each node consists of two Xeon E5 CPU (12 core each) and two NVIDIA Tesla K40 GPGPU units.
Shared memory multiprocessor (SMP): Multiprocessor system with a large amount of shared memory consists of 12 TB memory, 512 processor cores and 245 TB of local storage system.
The computer nodes are water cooled, and they are connected with unblocking high speed low latency InfiniBand interconnections. The central storage system has a capacity of 1 PB, and the data archive system consists of a library with a capacity of 2.5 PB. A high resolution HPC system is used for the simulation results visualization.
The data center spreads across 100 m2 area. The power supply amounts 250 kW. Two 200 kVA continuous power supply (UPS) devices are built-in, providing 20 minutes of supercomputer operation autonomy. The cooling system is made up of two subsystems that are 100% redundant:
Warm water system for direct cooling of computer nodes (4 x DLC cabinets),
Cold water cooling system for other components and four data cabinet enclosures.
The warm water system prepares the cooling water for 4 enclosures in the 35/40 °C range. The system regulates a constant water temperature of 35 °C with a three-way regulating valve and a hydraulic switch that amortizes major changes in the incoming water temperature. The cold water system prepares the cooling water for 4 enclosures in the data center area in the 10/15 °C range. Hot air enters the air-conditioned enclosure that cools it to a set temperature of 24 °C. The cooling system has a double floor used for the UPS cooling and this system is also used to maintain the desired humidity.
The warm water system has two 127 kW water coolers while the cold water system has two 87 kW coolers. The total cooling capacity of the data center is 210 kW and an additional 210 kW in redundancy.
The cooling system works with 10 pumps, 16 motor locks and one three-way valve. The system is fully automated and controlled. The system has 45 relays, over 110 different sensors while all the active equipment is connected via the Modbus protocol.
The power usage effectiveness – PUE factor is 1.24. The PUE factor was measured when testing the supercomputer at 100% load.