[Update 5 Nov 2025 1948] ENS is now hands off. The update was successful and we are off of the vulnerable code. [Update: 5 Nov 2025, 1713] Tonight, 5 Nov 2025 between 1800 and 2000 PST, SDSC will apply a vendor provided update to address critical vulnerabilities. Due to redundant hardware, we do not predict …
Category Archive: Incident Notification
[Resolved] SDSC Incident Notification – SDSC Cloud Compute – 11 Jul 2025 11:00 AM
[Update 2:12 PM PT] The incident cause is resolved. SDSC Cloud instances that use ephemeral disks should now be functional again. Please report any lingering issues to support. [Update 1:10 PM PT] SDSC engineers continue to troubleshoot the SDSC Cloud Compute issue. We continue to advise only creating new instances with volume storage for root …
[Resolved] SDSC Outage Notification – Datacenter Firewall Networks – 4 Jul 2025
[Update 10:42 PT] The issue is resolved. Please report any lingering connection issues to SDSC support. [Update 10:02 PT] Engineers are still investigating. SDSC network engineers are investigating an outage related to networks behind hardware firewalls.
SDSC Outage Notification – UCSD Owned Networks – 21 Mar 2025, 14:48-15:05
SDSC engineers are working to determine a cause for a brief outage of UCSD owned networks that reside in the SDSC colo that occurred from ~14:48-15:05.
SDSC Outage Notification – VM – 6 Mar 2025, 0500
[Update 1001] Engineers have resolved the latency issue and are continuing to investigate the root causes. [Update 0839] Engineers continue to diagnose the latency issues. Thank you for your patience. Engineers are working to address storage latency issues with the VM systems. This is leading to unresponsive guests.
SDSC Incident Notification – USS Storage Slowness – 27 Jan 2025
[Update 21:16] Vendor provided a fix to address the quorum events. The storage cluster is now stable and performing as expected. Please contact support@sdsc.edu with any questions or concerns. SDSC engineers are investigating quorum events within the Universal Scale Storage service. Updates will be posted here.
SDSC Outage Notification – SDSC Cloud Compute – 30 Sep 2024, 16:12
[Update – 08:14] Issue has been resolved. Please contact support@sdsc.edu with any concerns or questions. [Update – 19:56] Issue is ongoing and engineers are continuing troubleshooting efforts. SDSC Engineers are investigating API and service issues in the SDSC Cloud Compute environment. This may prevent users from deploying new instances or un-shelving. Additional information will be …
SDSC Incident Notification – M52 top-of-rack switch reboot – 14:04, 12 Sept 2024
A top-of-rack switch in the M52 area of the datacenter was inadvertently rebooted. The services are back online at this time. Please contact SDSC Operations with any questions or concerns.
SDSC Outage Notification – Networking issue – 12:25, 6 Sept 2024
[update – 13:58] Engineers have isolated the issue and resolved the immediate problems. They are now working with network vendors to address the underlying cause of the outage – SDSC is investigating network issues affecting systems. Updates will be posted here.
SDSC Network Outage / Emergency Maintenance – 15 Aug 2024, 10:00
[Update 15:20] Network appears to be stable — please contact SDSC Operations with any concerns or questions. Networking staff continues to work with vendor technical support. [Update 14:47] Hardware swaps have caused unexpected impact to other networks. The networking team is still investigating and working with vendor technical support. — SDSC Networking is investigating a …