[Update – 11:18] Networking service has been restored and appears stable. Please contact support if you continue to experience any issues. More updates will be posted once a cause is determined. [Update – 07:53] Commvault services are enabled and working — file restores are successful. [Update – 07:42] The reboot did not fully correct the …
Category Archive: Emergency Maintenance
Emergency Maintenance announcements
SDSC Emergency Maintenance – Apache HTTPD update / website restart – 15 Apr 2019, 17:00-18:00
[Update – 17:34] All servers are complete. Please contact support with any concerts or questions [Update – 17:21] awesomelabs, artichoke, ciabatta, and nsgr are complete. Engineers are still working on the other hosts. — SDSC will apply critical Apache HTTPD webserver patches to the Linux environment tonight, 15 Apr 2019, starting at 17:00. These updates …
SDSC Emergency Maintenance – Apache HTTPD update / website restart – 11 Apr 2019, 17:00-18:00
[Update – 17:25] All services have been restarted. Please contact support with any concerns or questions —- SDSC will apply critical Apache HTTPD webserver patches to the Linux environment tonight, 11 Apr 2019, starting at 17:00. These updates address recently announced security vulnerabilities related to CVE-2019-0211. The web services will require a restart causing a …
SDSC Emergency Maintenance – Linux patch/reboot – 17:00-23:00, 22 Jan 2019
[Update 20:58] All systems are patched and rebooted. Please contact support with any questions. —- SDSC will be applying critical patches to Centos 7 Linux systems tonight, 22 January 2019, starting at 17:00. This maintenance will require a reboot of all systems listed below. Please contact support@sdsc.edu with any questions or concerns. Updates will be …
SDSC Outage Notification – Firewall Network Issues – 26 Nov 2018, 14:35-14:50
On 26 November 2018 between roughly 14:30 and 14:50, systems behind the SDSC firewalls experienced limited network connectivity. Engineers have corrected the issue and systems are accessible. Please contact SDSC Operations or regular support channels with further questions.
SDSC Outage Notification – Project Storage ps-078:/ps-data/yeolab – 25 Sept 2018, 10:45
[Update – 11:19] The JBOD was brought back online and verified. The JBOD storage for ps-078:/ps-data/yeolab has faulted and the array is offline. Engineers are investigating and working to repair the array.
SDSC East Data Center Cooling – 30 Aug 2018, 11:00
(Update 13:55) All air handlers have restarted and the room temperature is normal. Please contact Operations with any concerns. —— The air handlers in the east data center did not properly restart after the power outage. As a result, there are higher than normal ambient temperatures throughout the room. Engineers are working to restore operation.
SDSC Outage Notification – Power Outage Resolved – 30 Aug 2018, 08:15
Update 8/30/2018 10:00 AM PT: A general power outage occurred this morning, Thursday, 8/30/2018, at SDSC around 8:10 AM PT. Power was restored by campus FM around 9:40 AM PT. Individual systems in the data center may need to be powered back on if they were affected by the outage. Generator-backed systems should have remained powered …
SDSC Outage Notification – multiple networks in building/datacenter – 15 Aug 2018, 13:15
[Update – 22:59] All guests are online and appear to be acting normally. Please contact support with any questions or concerns. [Update – 22:00] Engineers are slowly booting guests and monitoring stability of the VMware cluster. Additional updates will be posted once all guests are on and the environment is operating normally. [Update – 18:42] …
SDSC Outage Notification – Project Storage Hotel Node – 6 June 2018, 04:00-07:00
At approximately 04:00 on 6 June 2018 a single project storage hotel node lost connectivity with the attached disk. At approximately 07:00 a storage engineer reattached the disk pool and repaired the disk export. The services appear up and functioning. Please contact support if problems continue. The affected exports are listed below. ps-004:/ps-data/roylab ps-013:/ps-data/dsc ps-015:/ps-data/ucm_engineering …