[update 22:16] The new switch is online and systems appear to be recovered. Please contact SDSC operations with any questions or concerns [update 20:32] engineers are on site swapping a chassis. No recovery ETA at this point The top of rack switch in data center rack M47 is offline. Some data center hosted systems are …
Category Archive: Emergency Maintenance
Emergency Maintenance announcements
SDSC Outage Announcement – HPC, Commvault, Cloud resources unavailable – 6 May 2019, 18:00
[Update – 11:18] Networking service has been restored and appears stable. Please contact support if you continue to experience any issues. More updates will be posted once a cause is determined. [Update – 07:53] Commvault services are enabled and working — file restores are successful. [Update – 07:42] The reboot did not fully correct the …
SDSC Emergency Maintenance – Apache HTTPD update / website restart – 15 Apr 2019, 17:00-18:00
[Update – 17:34] All servers are complete. Please contact support with any concerts or questions [Update – 17:21] awesomelabs, artichoke, ciabatta, and nsgr are complete. Engineers are still working on the other hosts. — SDSC will apply critical Apache HTTPD webserver patches to the Linux environment tonight, 15 Apr 2019, starting at 17:00. These updates …
SDSC Emergency Maintenance – Apache HTTPD update / website restart – 11 Apr 2019, 17:00-18:00
[Update – 17:25] All services have been restarted. Please contact support with any concerns or questions —- SDSC will apply critical Apache HTTPD webserver patches to the Linux environment tonight, 11 Apr 2019, starting at 17:00. These updates address recently announced security vulnerabilities related to CVE-2019-0211. The web services will require a restart causing a …
SDSC Emergency Maintenance – Linux patch/reboot – 17:00-23:00, 22 Jan 2019
[Update 20:58] All systems are patched and rebooted. Please contact support with any questions. —- SDSC will be applying critical patches to Centos 7 Linux systems tonight, 22 January 2019, starting at 17:00. This maintenance will require a reboot of all systems listed below. Please contact support@sdsc.edu with any questions or concerns. Updates will be …
SDSC Outage Notification – Firewall Network Issues – 26 Nov 2018, 14:35-14:50
On 26 November 2018 between roughly 14:30 and 14:50, systems behind the SDSC firewalls experienced limited network connectivity. Engineers have corrected the issue and systems are accessible. Please contact SDSC Operations or regular support channels with further questions.
SDSC Outage Notification – Project Storage ps-078:/ps-data/yeolab – 25 Sept 2018, 10:45
[Update – 11:19] The JBOD was brought back online and verified. The JBOD storage for ps-078:/ps-data/yeolab has faulted and the array is offline. Engineers are investigating and working to repair the array.
SDSC East Data Center Cooling – 30 Aug 2018, 11:00
(Update 13:55) All air handlers have restarted and the room temperature is normal. Please contact Operations with any concerns. —— The air handlers in the east data center did not properly restart after the power outage. As a result, there are higher than normal ambient temperatures throughout the room. Engineers are working to restore operation.
SDSC Outage Notification – Power Outage Resolved – 30 Aug 2018, 08:15
Update 8/30/2018 10:00 AM PT: A general power outage occurred this morning, Thursday, 8/30/2018, at SDSC around 8:10 AM PT. Power was restored by campus FM around 9:40 AM PT. Individual systems in the data center may need to be powered back on if they were affected by the outage. Generator-backed systems should have remained powered …
SDSC Outage Notification – multiple networks in building/datacenter – 15 Aug 2018, 13:15
[Update – 22:59] All guests are online and appear to be acting normally. Please contact support with any questions or concerns. [Update – 22:00] Engineers are slowly booting guests and monitoring stability of the VMware cluster. Additional updates will be posted once all guests are on and the environment is operating normally. [Update – 18:42] …