Archive for the ‘Experiences’ Category

Some incoming mails lost between Jan 9, 6pm and Jan 13, 11am

Tuesday, January 14th, 2014

On Monday morning we found out that large incoming mails (1 MBytes or larger) were dropped without leaving any error messages in our log files. These mails were lost between Thursday (Jan 9) evening 18:27 and Monday (Jan 13) morning 11:06. Some indicators (i.e. spam filter rules for this case) lead us to estimate the number of about 560 broken local deliveries to about 300 unique recipients.

If you expected e-mails with attachments close to 1 MB or larger within this time frame there is a high likelihood that they got lost. The only information we still have about these mails are sender, recipient and arrival date and time. If you were one of these recipients, please contact the sender to send it again.

You can check on this web page if mails you should have received were lost. You’ll have to log in with your D-PHYS account and will see sender (or mailing list) of and time when the lost mail arrived. Additionally we’ll inform all affected recipients individually, too.

The problem occured after one of the software updates on Thursday which brought stricter code checking, and is solved since Monday morning 11:06.

The issue was caused by a long standing and subtle programming error in the check which prevents bigger mails from being inspected closely by the main spam filter for performance reasons. It was only triggered upon local mail delivery, so mails sent from D-PHYS to outside D-PHYS were not affected. E-mails to D-PHYS mailing lists (or other mailing lists) with archive should be available in the according mailing list archives.

We’re truly sorry for any inconvenience this may have caused and have already taken measures so that similar issues won’t result in mail loss from now on.

Update: it happens to the best of us: Gmail for iOS bug might cause data loss

Notes on warranty (Garantie vs. Gewährleistung)

Tuesday, February 19th, 2013

This post might help to clarify some questions related to the warranty conditions of new hardware. It is the result of internal inquiries we performed in reply to customer requests. Skip if you’re not interested.

Switzerland, like other European countries, knows two forms of liability a vendor has to/can offer to clients of its hardware products: Gewährleistung and Garantie.

  • Gewährleistung is mandated by law and covers basic liability if a piece of hardware fails. In Switzerland, Gewährleistung was just extended from 12 to 24 months on Jan 1st, 2013. This means that for the first two years, any defect whose cause was already present at the time of purchase has to be covered by the vendor. As you can probably guess, the part in italic can be the crucial one.
  • Garantie is a voluntary service offered by most, but not all vendors. Its conditions can be pretty freely chosen by the vendor, unlike Gewährleistung where the terms are given by law. Garantie can cover a wider range of defects and it can also be a service you have to pay for.

Now how does this matter to you? Let’s take a current real life example: you’d like to buy a new Apple MacBook Pro 13″. Right now, you have a number of interesting options:

  • Neptun: CHF 1305.-, 2 years of Gewährleistung (by law), 3 years of Apple Garantie (price of Apple Care included)
  • Dataquest: CHF 1240.-, 2 years of Gewährleistung (by law), 2 years of Dataquest Garantie. Additionally, you can pay CHF 99.- for a third year of Dataquest Garantie.
  • Apple EDU Store: CHF 1268.-, 2 years of Gewährleistung (by law), 1 year of Apple Garantie. Additionally, you can pay CHF 210.- for another 2 years of Apple Garantie. IDES offers the same to ETH employees for CHF 195.-

It’s hard to tell if the conditions of the additional Garantie are really more accommodating than those of the mandatory Gewährleistung. Wear parts like the battery for example are typically covered by neither. Harddisks on the other hand (most common failing part in a laptop) should be covered by both. In the end the best option will also depend on your usage pattern and the expected life time of your device. Regardless of the type of warranty you have, you should always report any problem you’d like to get fixed as soon as possible.

Sources:
Apple warranty conditions
Computer World article

The Art of Scaling

Thursday, April 19th, 2012

Note: this is a purely anecdotal posting about our struggles with some performance bottlenecks in the last few months. If you’re not interested in such background information, just skip.

You might have noticed that since about January 2012 using our file and mail servers hasn’t been as smooth as usual. This posting will give you some background information concerning the challenges we encountered and why it took so long to fix them. Let’s begin with the file server.

Way back in the days (i.e. 5 years ago), when the total file server data volume at D-PHYS was about 10 TB, we used individual file server to store this data. When one server was full, we got a bigger one, copied all the data and life was good for another year or two. Today, the file server data volume (home and group shares) is above 150 TB and growing fast and this strategy doesn’t work any longer: individual servers don’t scale and copying this amount of data alone takes weeks. That’s why in 2009 we started migrating the ‘many individual servers’ setup to a SAN architecture in which the file servers are just huge hard drives (iSCSI over Infiniband, for the technically inclined) connected to a frontend server that manages space allocation and the file system. The same is true for the backup infrastructure, where the data volume is even bigger.

This new setup had to be developed, tested and put in place as seamlessly and unobtrusively as possible while ensuring data access at all times (apart from single hour-long migrations). The SAN architecture was implemented for Astro in December 2010 and has been running beautifully ever since. In 2011 we laid the groundwork to adopt this system for the rest of D-PHYS’s home and group shares and after a long and thorough testing period the rollout happened on January 5, 2012. Unfortunately, that’s when things got ugly.

At first, we noticed some exotic file access problems on 32bit workstations. It took us some time to understand that the underlying issue was an incompatibility with the new filesystem using 64-bit addresses for the data blocks. As a consequence we had to replace the filesystem of the home shares. Independently we ran into serious I/O issues with the installed operating system, so we had to upgrade the kernel of the frontend server and move the home directories onto a dedicated server. In parallel, we had to incorporate some huge chunks of group data while always making sure that nightly backups were available. All this necessitated a few more migrations until we finally achieved a stable system on March 28.

The upshot: what we had hoped to be a fast and easy migration turned out to cause a lot of problems and take much longer than anticipated, but now we have a stable and solid setup that will scale up to hundreds or even thousands of TB of data.
See live volume management and usage graphs for our file servers.

As for the mail server, matters are to some extent related and partly just coincidental in time. The IMAP server does need access to the home directories and hence also suffered when their performance was impaired. But even after having solved the file server issues, we still saw single load peaks on the IMAP server that prevented our users from working with their email. Again, we put a lot of time and effort into finding the reason. As of April 13, we’re back to good performance and arrive at the following set of conclusions:

Particular issues:

  • a covertly faulty harddisk in the mail server RAID seems to have impaired performance
  • CPU load of the individual virtual machines on the mail server was not distributed across the available CPU cores in an optimal way

General mail server load:

  • while incoming mail volume doesn’t increase much, outgoing mails have grown 50% in the last year alone
  • more and more sophisticated spam requires more thorough virus and spam scanning, increasing the load on the mail server
  • our users have amassed 1.1 TB of mail storage (up from 400 GB in January 2010), which need to be accessed and organized

Bottom line:

We’d like to thank you for your patience during the last 4 months and apologize for any inconvenience you might have had to endure. In all likelihood the systems will be a lot more stable in the future, but of course we’re constantly working to ensure the D-PHYS IT infrastructure is able to keep up with the fast growing demand of disk space (the data volume has tripled in the last year alone). We’ve learned a lot and we’ll put it to good use.

Do not blindly trust mail

Tuesday, November 3rd, 2009

The current wave of password phishing mails seems to provoke an unusually high attention rate.  People seem to think that mail allegedly coming from help@ethz.ch may be genuine.  The german text itself is so bad that its spammy character is obvious to long time mail users.

Remember: any part of a mail can be faked. This is in the design of the mail system and cannot be fixed without making mail usage a lot harder for everybody.  And even if we used a better system (like cryptographic signatures) the rest of the world would not follow.

Therefore, be sceptic about any mail until the complete impression including the writing style fits the picture.  No IT support worth their salt will ask you to reveal your password.  And if they do they deserve to be ignored!

Solution for PDF printing problems on Windows

Friday, December 19th, 2008

We found the solution for our long term PDF printing problems, which occurred mostly on Windows computers.

The latest workaround was to use the option print as image in Adobe Acrobat. This was very slow, and some times even that didn’t work.

Please follow the instructions in this readme, if you have such problems!