All IT core services are fully functional
Previous status updates can be found below.
May 23, 9:55 AM - The distribution of IT-Alerts remains constant although the form used by IT staff to submit announcements continues to behave erratically. IT will continue investigating the submission issue. A work-around of using docs.rice.edu blog posts will ensure an uninterrupted flow of IT system maintenance and status updates as needed. IT-Alerts is not a core service; all IT core services remain fully functional.
May 21, 5:00 PM - Kennel.rice.edu suffered a severe hardware failure. The RUF.rice.edu web sites remain active, but content cannot be updated until Kennel.rice.edu is revived. IT continues to work on the failed hardware. RUF.rice.edu is not a core service; all IT core services remain fully funcitonal.
May 21, 9:55 AM - Kennel.rice.edu, the server that manages content access to RUF.rice.edu web sites and other files, stopped working. All RUF.rice.edu web sites are still live, they just cannot be edited while Kennel is offline.
May 9, 10:30 AM - Printing.rice.edu was restored to full functionality.
May 9, 9:30AM - IT received reports that printing.rice.edu was offline. Students use a different printing system, owlprinting.rice.edu.
April 29, 2;00pm - The IT-Alerts system was restored to full functionality.
April 18, 2:15PM - A new work around is being used to publish notices on the web sites as well as to the mailing list. The only part of the application that is not functional is the web form to generate the announcements with appropriate categories and status indicators. IT will continue working on the issue. NO critical systems are affected.
April 17, 5:00PM - A work around is being used to send IT-Alerts to the mailing list, but the blog and web sites are still not displaying the messages.
April 16, 6:00PM - Although all critical systems are fully functional, the IT-Alerts system is performing inconsistently. IT-Alerts is a notification system used to publish updates on IT web sites and to a list of subscribers on the IT-Alerts mailing list. Currently, IT-Alerts are being distributed to the mailing list but are not displayed on the web sites. To subscribe to the IT-Alerts mailing list, enter your email address in the subscription form: https://mailman.rice.edu/mailman/listinfo/italerts-l.
December 2, 2:45PM - The network hardware that was affecting various services and web sites has been resolved. All IT systems are fully functional.
December 2, 1:50PM - IT identified a blade (part of the hardware supporting Rice's network) that is failing, resulting in Exchange mail problems, issues for some web sites that fall under rice.edu, and other service interruptions. A replacement blade is being installed and services will return to normal soon.
November 21, 9:00AM - IT traced issues with blank forms on https:// Rice web sites to a separate system (Cognos) and interrupted the connection causing the forms to lock up. The affected web sites returned to normal behavior.
November 20, 9:30AM - IT has a temporary fix in place for a troubled web site server that has been dropping some web sites offline, making them unavailable at random times. A permanent fix is being tested for implementation on Thursday morning, November 21.
November 19, 2:30PM - IT is researching the root cause that is affecting a Cold Fusion server hosting several secure (https://) Rice web sites.
November 18, 11:18 AM - IT found and resolved a server issue and both web applications, GradApps and TouchNet, are fully functional again.
November 18, 11:00 AM - IT received reports that two web applications, GradApps and TouchNet were down and began researching cf server issues.
November 4, 5:30 PM - IT resolved the email issues and all core IT services are fully operational again.
November 4, 4:00 PM - IT is
researching a hardware failure that may have resulted in email delivery
issues. Updates will be posted to the IT web site.
21 12:05 PM - DROPPED CONNECTIONS limited to Mudd Lab - Access to Rice email, storage and web
sites was momentarily interrupted when a network access switch in Mudd Lab
rebooted. The switch is back up; while it was rebooting, connected
systems could not access campus network services like email and storage.
These services were never offline, but some of the employees and students
in Mudd could not access the services for about one minute.
20 NO BLACKLISTS -
Rice has rolled off the previous blacklists on Monday afternoon, August 19. No
further email bounces or returns have been reported.
16 11:00AM, EMAIL DELIVERY OF EXTERNAL MESSAGES WILL IMPROVE BY
AUGUST 19 - A limited
number of outbound messages are still affected by Rice's August 12 blacklisting
for sending significant amounts of spam. No internal Rice to Rice mail is
affected and most of the major email service providers no longer include
rice.edu on their blacklist. However, some email providers block messages
for a 7-day period after a sender has been identified as a spam sender.
Rice will roll off the remaining blacklists on Monday afternoon. If you
received a rejection notice for a message you tried to send to an external,
non-rice.edu address this week, please try resending the message now. If
you still receive a rejection notice, resend the message Monday
afternoon. For further assistance, contact the Help Desk
(helpdesk.rice.edu or helpdesk@ or 713.348.4357).
16 10:15AM, EMAIL DELIVERY WORK-AROUND - If your externally addressed message is
rejected with a message about "too many hops" you can resend the
message and it will go through now. IT continues to research
the root cause of other rejected messages for non-rice.edu destinations.
This is still a limited problem and internal Rice to Rice email is not
16 - LIMITED EMAIL DELIVERY ISSUES: Early morning research indicates some external email servers
are rejecting messages flowing through one of the Proofpoint servers.
IT is working on the blacklists individually as reports come in.
The Problem Management Arbitrator and management team sent out the
first update around 9:15 AM; follow up meetings are
scheduled approximately every 60-90 minutes beginning at 10:30 AM.
15 - PROOFPOINT RETURNED TO SERVICE- Proofpoint is returned to service at the end of the day so
that sensitive data will not leave campus via email addresses to non-rice.edu
recipients. A few Help Desk tickets begin trickling in reporting email
13 - EMAIL DELIVERY ISSUES RESOLVED - mail flowing well within Rice and to most external
addresses. One of the services Rice depends on for protecting sensitive
data, Proofpoint, has not yet been returned to service. Proofpoint
runs automated scripts that search for personally identifiable information
(PII) in email content or attachments addresses to recipients outside of Rice.
If a message that is addressed to a non-rice.edu destination contains
multiple instances of PII, the message is returned to the Rice sender as
undeliverable due to its inclusion of sensitive data. To safeguard
Rice's sensitive data, all outbound non-Rice.edu messages pass through the
Proofpoint servers for scanning. While researching the email delivery
issues of August 12, all Proofpoint scanning was turned off.
University outbound messages are not scanned for sensitive data
for four days.
12 - WIDESPREAD EMAIL DELIVERY ISSUES: two Rice.edu email accounts were compromised and used to
send a heavy stream of spam to both rice.edu and external addresses.
The outpouring of junk mail from rice.edu email servers resulted in a
disruption of internal (rice.edu) email delivery and blacklisting with external
email servers. Blacklisting means that an email sender is identified as
sending a significant amount of junk mail. The organizations
that provide email services (Yahoo!, AOL, etc.) protect their
customers by blocking all mail from known blacklisted email senders.
Depending on the email provider, it may take a few hours or a few days to get
off their blacklist. When the email provider removes Rice from
their blacklist, email from rice.edu will once again flow to those
addresses. All messages rejected during the blacklist time frame must be
resent after the block is removed.