On Monday the public was made aware of a severe bug in OpenSSL, a cryptography library which is used as the core of many cryptographically secured IT services. Since the bug was in the Heartbeat extension it has been named “Heartbleed”.
This bug allowed attackers to stealthily access parts of the memory used for cryptographic actions, i.e. it may include digital keys in use on servers or passwords transferred over encrypted connections.
If you used any password-protected D-PHYS web services or the D-PHYS mail server between 12th of December 2013 (or used the BackupPC web-interface since end of 2012) and Tuesday, the 8th of April 2014, there is a very small chance that your D-PHYS password and possibly other transmitted data may have been leaked to an attacker. We currently have no indication that this has actually happened on our servers.
To be safe, you might want to change the password of your D-PHYS account and any other account where the same password is used. See this Heise article for a discussion (in German) about whether you should change your password or not.
Read the rest of this entry »
The central network group informed us about a planed network interruption between 6:30 and 7:30 a.m. on the 10th of April 2014 due to maintenance work.
The following rooms are affected by this interruption:
HPT D1 – HPT D20 and HPT E1 – HPT E17.
Due to this interruption it may not be possible to access the D-PHYS services and internet from this rooms.
As announced in an earlier post last year, Microsoft is going to end the support for Windows XP in April 2014.
After this date the central network security group of the ETH will frequently scan our public networks to identify any existing Windows XP machines. Every Windows XP detected by such a scan will be disabled on the network level since it is strictly prohibited to keep this operating system up and running on the public network of ETH.
Since we are aware that there may be Windows XP machines living on after the end-of-life date, we worked out a solution to support these situations and to help you not to get in conflict with the network usage regulations.
We founded a project called eXile which provides very locked down network environments that are monitored by advanced security techniques and provide excessive firewall setups. Furthermore eXile provides easy interfaces for you to manage your computers and overview the security state and network access to your machines in eXile.
You can send your machines to the eXile when they match one of the following scenarios:
- Lab computers (controlling, collecting measure data, or monitoring other systems)
- Industrial computers
- Embedded systems
The following applications are not suitable for eXile and need to be migrated to a supported operating system:
- Office Computers
- Computers on which internet access needs to be available
- Computers on which emails are received and sent
- Computers that provide any services to public computers in the internet
Please note that eXile should not be seen as an excuse not to migrate your Windows XP to a supported operating system as soon as possible. The purpose of eXile is really only to address those few machines that are somehow locked to their operating system.
Nevertheless we invented eXile to address the Windows XP end-of-live problem, it is capable to take up any other computer for which you want to have an extra level of security or on which you run any other outdated or insecure operating system.
If you think your remaining Windows XP computers are candidates to send to eXile, we would be happy if you could send a message to email@example.com and inform us about the number of computers and what application you are using these computers for. Later this month a web interface will be made available on https://exile.phys.ethz.ch/ where you can directly register every machine you want to send to eXile.
After eXile is fully online, another post will be submitted here.
On Monday morning we found out that large incoming mails (1 MBytes or larger) were dropped without leaving any error messages in our log files. These mails were lost between Thursday (Jan 9) evening 18:27 and Monday (Jan 13) morning 11:06. Some indicators (i.e. spam filter rules for this case) lead us to estimate the number of about 560 broken local deliveries to about 300 unique recipients.
If you expected e-mails with attachments close to 1 MB or larger within this time frame there is a high likelihood that they got lost. The only information we still have about these mails are sender, recipient and arrival date and time. If you were one of these recipients, please contact the sender to send it again.
You can check on this web page if mails you should have received were lost. You’ll have to log in with your D-PHYS account and will see sender (or mailing list) of and time when the lost mail arrived. Additionally we’ll inform all affected recipients individually, too.
The problem occured after one of the software updates on Thursday which brought stricter code checking, and is solved since Monday morning 11:06.
The issue was caused by a long standing and subtle programming error in the check which prevents bigger mails from being inspected closely by the main spam filter for performance reasons. It was only triggered upon local mail delivery, so mails sent from D-PHYS to outside D-PHYS were not affected. E-mails to D-PHYS mailing lists (or other mailing lists) with archive should be available in the according mailing list archives.
We’re truly sorry for any inconvenience this may have caused and have already taken measures so that similar issues won’t result in mail loss from now on.
Update: it happens to the best of us: Gmail for iOS bug might cause data loss
On Thursday, the 9th of January 2014, starting in the late afternoon, we will run multiple software updates on the D-PHYS mail server. We do expect multiple downtimes throughout the evening, partially of single mail services, partially of the whole mail server.
This will likely also delay the delivery of incoming mails up to several hours.
Update, 22:30: Everything back to normal.
The ETH Zurich will be officially closed between Tuesday, 24th of December 2013 and Friday, 3rd of January 2014. During this time, we can only provide limited support. Please follow these rules to save us from superfluous work:
- Switch off printers
- Switch off your personal workstation and notebook except for the following:
- Do not switch off our managed Linux workstations.
We will try to follow our e-mail, but you may also have luck and meet some of us in our IRC channel.
This post is meant to give you a short overview of what has been accomplished in D-PHYS IT by ISG this year. We’ve been hard at work to further improve and extend our services for you, our customers. Some highlights of 2013:
- New apprentice: As of August 14, Anastassios has started his apprenticeship with us and is already deeply involved in a complex PHP/Ajax/PostgreSQL project. Keep it up!
- Mailserver: This year saw a massive increase in spam and especially phishing attacks. They’re getting more and more sophisticated and now include valid logos and even personal names. We were forced to tighten email policies and further fortify our mail server in order to battle those waves.
- Backup: For the data on our file servers we provide one month of nightly backups. Now our powerful backup system based on COW BTRFS snapshots allows us to extend this period to up to one year in exponential intervals for most file systems. Note that anything beyond 30 days is best-effort only and we might have to cut back again in single cases. A new web frontend shows the status of all backup runs.
- Windows server: Several Windows server installations have been moved to a new powerful virtualization server and the Active Directory setup has been improved.
- Printer portal: All information regarding our printers can now be found on one website. You might want to check there if you have issues with a particular printer or just to get an idea about printing volume.
- Portal for managed workstations: Our new Chic! frontend shows the software status of our managed Windows and Mac workstations and allows you to request additional software packages. This service will be officially announced in January 2014.
- GitLab: We run a GitLab instance to facilitate collaborative programming projects and sharing of code. Get in touch if you’d like to use it.
- System upgrades: 2013 brought another round of OS upgrades, also for our servers. We updated most servers silently and combined all critical systems into one migration on September 11 in order to minimize downtime for our users.
- Windows XP exile: As reported previously, Windows XP will be end-of-life in April 2014. Since there’s still a substantial number of XP machines out there (most of which cannot be upgraded due to soft- or hardware constraints), we’ll provide a locked-down exile network that will allow a limited and well-controlled survival of those machines under certain conditions. We’ll post an announcement when the system is ready.
- IPv6: This year we laid the groundwork for the slow migration towards IPv6 connectivity in our networks. In particular, we got our monitoring system IPv6-ready and prepared a NFSv4 rollout. We’ll keep you posted about our IPv6 progress.
Apart from these highlights, of course there have been numerous small projects and improvements to our setup, making both your and our life easier.
I would like to take this opportunity to thank my whole team for their hard and dedicated work all year long.
Happy Holidays and see you in 2014!
Starting on Wednesday, the 18th of December 2013 in the late afternoon, we will start upgrading the operating system as well as many web applications on the primary D-PHYS webserver. While we’ll try to keep the downtimes as short as possible, some temporary service interruptions can’t be avoided and are hence expected.
Potential issues with specific websites hosted on the D-PHYS webserver will be tackled in the days after the upgrade.
In the past we allowed sending e-mails over the D-PHYS mailserver from everywhere inside ETH to allow D-PHYS users to send e-mail via VPN or WiFi without the need to enter a password.
However the amount of misuse of this rule in the form of sending out spam from compromised machines inside ETH but outside D-PHYS raised significantly in the last few weeks.
Due to this development, we are forced to restrict password-less sending of e-mail via the D-PHYS mail server to a few D-PHYS networks — and in the future we might tighten this even more.
For now this means that you will need to authenticate yourself with your D-PHYS account, when sending e-mail via the D-PHYS mail server from outside D-PHYS. This includes sending e-mail via the ETH WiFi networks and connections via ETH VPN. This change is effective immediately.
Please see our documentation about how to send e-mails from outside D-PHYS if you need help reconfiguring your e-mail client.
UPDATE Thu 12.09. 07:30 If you’re trying to connect to a SMB share from an unmanaged Windows machine, you have to use “ad\USERNAME” instead of just “USERNAME” from now on.
UPDATE 21:15 apart from the IGP group shares (which will be back in a few hours) all systems are back to normal. Please let us know if you experience any problems.
In order to upgrade the operating system on several core infrastructure servers of the Department, we schedule a general maintenance downtime on
Wednesday September 11, starting at 17:00, lasting for several hours.
Most services will be affected and unavailable during that time, as they require an authentication with your D-PHYS account (email, file server, print server, managed workstations). Note that, even though you will not be able to check your emails or send new ones, all incoming mails will be received and safely delivered to your inbox afterwards.
Please make sure to save all open documents before 17:00 on that day.
Since we will also change the way file server mounts are authenticated, users who haven’t updated their passwords in a very long time might not be able to mount their home directories or group shares after the migration. If you run into this problem on Thursday morning, please first change your password. If the issue persists, contact us.
We will post an update when things are back to normal.