Update On Your Email: Monday Juanuary 23, 2012

Greetings All,

It is my pleasure to address you all by email today. I know many of you used printers, fax machines and telephones for the first time in a while to complete your University business.

First thank you for your patience and assistance as we have work together to communicate during the outage. After a hardware failure, we had to restore the exchange system from backup according to our disaster recovery plan. Staff and faculty email was the only system effected. Student email, Banner, Blackboard, Internet…all other systems were available. The recovery was successful. Anyone who sent email to the University during the outage received notification that their emails were being held for delivery, and they received requests to resend to you to ensure delivery.

Details and timelines are below.

What happened:
On Tuesday, January 17th just before 6am, the exchange system, which is comprised of two servers (two post offices that receive and sort the mail) and a very large network storage drive (all of the individual mailboxes that we each see every day) went offline. As the servers were writing (post office was sending) data to the storage drive (our mail boxes) the storage drive had a hardware failure. This failure cause a corruption of the databases that make up our mailboxes.

The ITS staff and Exchange engineer who set up the system for us, were able to replace the hardware that failed once the problem was identified. We keep spare parts for just such an occasion. However, the databases had to 1) recovered from back up that was taken at 1am on Tuesday; 2) re-indexed so they could be accessed; and 3) remounted to the servers. The recovery from back up took about seven hours, we store a very large amount of email. But the backup was successful because of the disaster recovery planning that was done.

The next step of re-indexing the databases took the most time. The databases that comprise much of the STX mailboxes were the first online. These became to users at about 4pm on Wednesday. This was not all of the users on STX, it was all of the users on the server called STX. These folks starting receiving their email and received the stored backlog of email.

The second server contained considerably more data, and the re-indexing did not finish until 7pm on Sunday night. This was the necessary process of touching every email that we saved – a direct correlation to the size of our mailboxes.

The staff and faculty on the second server received access to their email at approximately 8:30 last night. The email that was being held in queue was delivered to email boxes. There were emails that were bounced to senders so that they could resend.

MAIL SENT TO UVI DURING OUTAGE: When mail was received that could not be delivered, senders received updates each day on whether their email was delivered to you. On day one, they were told there was a delay not to resend. On day two, the sender was told there was a further delay, not to resend. For mail over 72 hours old, senders were asked to resend. Regardless of when the email was sent to you, it will have a date and time when it was delivered to you – not sent to you. So there will be a gap in your email box, and there may be email that senders did not resend to you.

WHAT IS NEXT:
The exchange server hardware and storage server is currently at end of life. This fall planning had already started to move UVI to a new system because of aging server hardware, lack of capacity to accommodate mailbox size, and unmet annual costs. This plan was presented to the UVI senate and cabinet already. We will be accelerating that plan to move UVI mail to the CLOUD. You will be hearing more about that in the very near future. We would expect to move all users to the new system within 90 days. But we cannot move you if you have a very large mail box. So please continue to reduce your email box size. The helpdesk can assist you.

Please let me know if you have additional questions, or if you are having difficulties.
Thank you again.

Tina