All IT core services are fully functional

Previous status updates can be found below.

May 23, 9:55 AM - The distribution of IT-Alerts remains constant although the form used by IT staff to submit announcements continues to behave erratically.  IT will continue investigating the submission issue.  A work-around of using docs.rice.edu blog posts will ensure an uninterrupted flow of IT system maintenance and status updates as needed. IT-Alerts is not a core service; all IT core services remain fully functional.

May 21, 5:00 PM - Kennel.rice.edu suffered a severe hardware failure.  The RUF.rice.edu web sites remain active, but content cannot be updated until Kennel.rice.edu is revived. IT continues to work on the failed hardware. RUF.rice.edu is not a core service; all IT core services remain fully funcitonal.

May 21, 9:55 AM - Kennel.rice.edu, the server that manages content access to RUF.rice.edu web sites and other files, stopped working.  All RUF.rice.edu web sites are still live, they just cannot be edited while Kennel is offline.  

May 9, 10:30 AM - Printing.rice.edu was restored to full functionality.

May 9, 9:30AM - IT received reports that printing.rice.edu was offline. Students use a different printing system, owlprinting.rice.edu.

April 29, 2;00pm - The IT-Alerts system was restored to full functionality.

April 18, 2:15PM - A new work around is being used to publish notices on the web sites as well as to the mailing list.  The only part of the application that is not functional is the web form to generate the announcements with appropriate categories and status indicators.  IT will continue working on the issue.  NO critical systems are affected.

April 17, 5:00PM - A work around is being used to send IT-Alerts to the mailing list, but the blog and web sites are still not displaying the messages.

April 16, 6:00PM - Although all critical systems are fully functional, the IT-Alerts system is performing inconsistently.  IT-Alerts is a notification system used to publish updates on IT web sites and to a list of subscribers on the IT-Alerts mailing list.  Currently, IT-Alerts are being distributed to the mailing list but are not displayed on the web sites.  To subscribe to the IT-Alerts mailing list, enter your email address in the subscription form: https://mailman.rice.edu/mailman/listinfo/italerts-l.

December 2, 2:45PM - The network hardware that was affecting various services and web sites has been resolved.  All IT systems are fully functional.

December 2, 1:50PM - IT identified a blade (part of the hardware supporting Rice's network) that is failing, resulting in Exchange mail problems, issues for some web sites that fall under rice.edu, and other service interruptions.  A replacement blade is being installed and services will return to normal soon.

November 21, 9:00AM - IT traced issues with blank forms on https:// Rice web sites to a separate system (Cognos) and interrupted the connection causing the forms to lock up.  The affected web sites returned to normal behavior.

November 20, 9:30AM - IT has a temporary fix in place for a troubled web site server that has been dropping some web sites offline, making them unavailable at random times.  A permanent fix is being tested for implementation on Thursday morning, November 21.

November 19, 2:30PM - IT is researching the root cause that is affecting a Cold Fusion server hosting several secure (https://) Rice web sites.


November 18, 11:18 AM - IT found and resolved a server issue and both web applications, GradApps and TouchNet, are fully functional again. 

November 18, 11:00 AM - IT received reports that two web applications, GradApps and TouchNet were down and began researching cf server issues.


November 4, 5:30 PM - IT resolved the email issues and all core IT services are fully operational again.

November 4, 4:00 PM - IT is researching a hardware failure that may have resulted in email delivery issues. Updates will be posted to the IT web site.


August 21 12:05 PM - DROPPED CONNECTIONS limited to Mudd Lab - Access to Rice email, storage and web sites was momentarily interrupted when a network access switch in Mudd Lab rebooted.  The switch is back up; while it was rebooting, connected systems could not access campus network services like email and storage.  These services were never offline, but some of the employees and students in Mudd could not access the services for about one minute. 


August 20 NO BLACKLISTS - Rice has rolled off the previous blacklists on Monday afternoon, August 19. No further email bounces or returns have been reported. 


August 16 11:00AM, EMAIL DELIVERY OF EXTERNAL MESSAGES WILL IMPROVE BY AUGUST 19 - A limited number of outbound messages are still affected by Rice's August 12 blacklisting for sending significant amounts of spam.  No internal Rice to Rice mail is affected and most of the major email service providers no longer include rice.edu on their blacklist.  However, some email providers block messages for a 7-day period after a sender has been identified as a spam sender.  Rice will roll off the remaining blacklists on Monday afternoon.  If you received a rejection notice for a message you tried to send to an external, non-rice.edu address this week, please try resending the message now.  If you still receive a rejection notice, resend the message Monday afternoon.  For further assistance, contact the Help Desk (helpdesk.rice.edu or helpdesk@ or 713.348.4357).


August 16 10:15AM, EMAIL DELIVERY WORK-AROUND - If your externally addressed message is rejected with a message about "too many hops" you can resend the message and it will go through now.  IT continues to research the root cause of other rejected messages for non-rice.edu destinations.  This is still a limited problem and internal Rice to Rice email is not affected.


August 16 - LIMITED EMAIL DELIVERY ISSUES: Early morning research indicates some external email servers are rejecting messages flowing through one of the Proofpoint servers.  IT is working on the blacklists individually as reports come in.  The Problem Management Arbitrator and management team sent out the first update around 9:15 AM; follow up meetings are scheduled approximately every 60-90 minutes beginning at 10:30 AM.


August 15 - PROOFPOINT RETURNED TO SERVICE- Proofpoint is returned to service at the end of the day so that sensitive data will not leave campus via email addresses to non-rice.edu recipients.  A few Help Desk tickets begin trickling in reporting email rejection.  


August 13 - EMAIL DELIVERY ISSUES RESOLVED - mail flowing well within Rice and to most external addresses.  One of the services Rice depends on for protecting sensitive data, Proofpoint, has not yet been returned to service.  Proofpoint runs automated scripts that search for personally identifiable information (PII) in email content or attachments addresses to recipients outside of Rice.  If a message that is addressed to a non-rice.edu destination contains multiple instances of PII, the message is returned to the Rice sender as undeliverable due to its inclusion of sensitive data.  To safeguard Rice's sensitive data, all outbound non-Rice.edu messages pass through the Proofpoint servers for scanning.  While researching the email delivery issues of August 12, all Proofpoint scanning was turned off.  University outbound messages are not scanned for sensitive data for four days.


August 12 - WIDESPREAD EMAIL DELIVERY ISSUES: two Rice.edu email accounts were compromised and used to send a heavy stream of spam to both rice.edu and external addresses.  The outpouring of junk mail from rice.edu email servers resulted in a disruption of internal (rice.edu) email delivery and blacklisting with external email servers.  Blacklisting means that an email sender is identified as sending a significant amount of junk mail.  The organizations that provide email services (Yahoo!, AOL, etc.) protect their customers by blocking all mail from known blacklisted email senders.  Depending on the email provider, it may take a few hours or a few days to get off their blacklist.  When the email provider removes Rice from their blacklist, email from rice.edu will once again flow to those addresses.  All messages rejected during the blacklist time frame must be resent after the block is removed.