For reasons still unknown our home directory server fulen stopped serving any files via NFS at about 5:20 pm. This stopped most active Linux logins. In order to restore functionality we had to reboot the file server. As of 5:35 pm fulen is again functioning.

Unfortunately, due to the necessary reboot we cannot fully assess the reason for the incident. It follows a history of poor performance that we have been investigating intensly for the last couple of weeks. We are still trying to find the ultimate cause.

We apologise for the inconvenience.

This morning around 03:00 a water ingress in our HIT server room shut down most of our essential infrastructure servers. As soon as power was back around 08:00 we started to bring our services online.
Please let us know if you still experience any problems. We apologize for the inconvenience. I guess water and servers just don't mix very well.

Status 12:14 apart from the BackupPC server everything should be working again.

At around 8am we noticed (different) hardware issues with our SMTP server wiggis as well with one of our Xen hosting servers. This also affected the IDL license server. The problems have been temporarily solved and since approx. 10:30am both systems are working again as before. Expect though maintenance downtimes for those services within the next week.

Today at 09:50 a crucial server died in our HIT server room. It took us about 20 min to move the affected services to other machines, during which time most of our machines weren't usable. We're sorry for the inconvenience. We're working hard to get rid of this fault-prone hardware.

Due to a critical condition on our systems we were forced to shut down our backup servers. This means there will be

No backup during this weekend.

Please be aware of this temporary limitation when producing large or important amounts of data. We apologize.

Today at 07:04 a software upgrade of a core router performed by Informatikdienste caused an avalanche of errors in our server cluster, leading to an outage of printing and some other non-critical services. It took us till around 08:00 to get the systems back to normal. We apologize for any inconvenience.

Today around 0715 we experienced a series of power outages in our water-cooled racks in HIT, leading to a shutdown of our most crucial servers. It took us almost two hours to get the systems up and running again, during which time most of our services were not available.

We apologize for any inconvenience.

Today around 14:20 the D-PHYS network started acting up. After about 20 minutes the problem could be tracked down to a machine running amok. Unfortunately, the terminal server plimpy did not survive the downtime and had to be rebooted at 16:10. We apologize for any inconvenience.

We had a partial DHCP outage this morning because the failover system between our two DHCP servers malfunctioned. The problem is currently solved only preliminarily by disabling the failover system and one of the two servers. To get both DHCP servers working again, we scheduled a DHCP server maintenance for today, Friday, 30th of January 2009, starting at 5pm. There may be short DHCP service outages during the maintenance. This means that initially connecting to the network as well as renewing the DHCP lease of your computer may fail for a few minutes. Update 21:45: The DHCP service is working again including the failover system.
(more…)