[Update : 20:56] The cause was found. During routine maintenance an assumed-unused package was removed. Once the package was re-added, Oracle began working again. Services are up — please contact support for any continuing issues. — At 19:50 on 19 Jan 2017 the primary Oracle nodes serving the SDSC Footprints system and SAM/ART accounting started …
Category Archive: Incident Notification
SDSC Outage Notification – limited Linux guest outage – 14 Dec 2016, 19:45
[Update, 00:05, 15 Dec 2016] guest ‘rupee’ is now online. [Update, 23:37] All guests except ‘rupee’ are online. Please contact support with any issues or concerns [Update, 22:05] Engineers replaced the faulty disk controller and verified the guests function. Guests were then shut down again so that engineers can perform a reboot and ensure the …
SDSC Outage Notification – CDS2 / West UPS Power – 3 Nov 2016, 10:40
[Update – 20:14] The UPS has been reactivated and is protecting the systems. —– At approximately 10:40 the CDS2 UPS system lost power due to a short circuit condition. The power was immediately restored with the UPS system being temporarily bypassed. Updates will be forthcoming. The scope of the outage would be computer which were …
SDSC Emergency Maintenance – Linux patch/reboot – 1 Nov 2016, 20:00-23:00
[Update – 20:41] All patches and reboots have been applied. —- SDSC will be applying critical patches to the Linux environment tonight, 1 Nov 2016 starting at 8pm. This maintenance will require a reboot of all systems listed below. Please contact support@sdsc.edu with any questions or concerns. Updates will be posted to http://status.sdsc.edu as the …
SDSC Outage Notification – Project Storage – 11:30-12:00, 19 Aug 2016 [Resolved]
The SDSC Project Storage Service experienced a partial outage from approximately 11:30AM – 12:00PM. A hung process on a single hotel node consumed a critical amount of processor and memory resources, effectively rendering the node unavailable during the outage. At this time the server is back up and functioning as expected. We apologize for any inconvenience caused by …
SDSC Outage Notification – Commvault – 21:00, 29 Feb 2016
[Update: 23:06, 29 Feb 2016] Deduplication stores have been sealed and systems requiring the deduplication are running backups. At approximately 21:00 on 29 Feb 2016 the Commvault media agent named ‘cvma3’ hung and was power cycled. This caused jobs which were running to pause. There were some jobs which required manual restarts. Some jobs utilizing …
SDSC Outage Notification – Cloud Storage & Compute – 4 Feb 2016, 12:39
[Update, 13:19] Service has been restored. A 10Gb cable was inadvertently removed which provided service to the load balancer. That cable has been replaced. The loadbalancer which provides access to both SDSC Cloud Storage and SDSC Cloud Compute has gone offline. Engineers are investigating.
SDSC Outage Notification – Datacenter Partial Power Loss ~16:00, 6 Jan 2016
Key services have been restored. Please contact datacenter operations or your SDSC system administrators if you are experiencing any lingering issues. Update [23:00]: All commvault services have been restored. Update [20:35]: Commvault backups from cvma4 are working. The media agent cvma3 is still down and operations is assisting in rebooting/recovery of the system. Update [19:30]: …
SDSC/UCSD Intermittent Network Interruptions – 23 Aug 2015
Connectivity to UCSD and SDSC networks from the outside internet has been intermittently unavailable today, 23 August 2015. UCSD network engineers are working to resolve the issues. Additional information will also be available at http://status.ucsd.edu.
SDSC Outage Notification – Datacenter and building networking – 13:00-13:01, 18 Aug 2015
During a routine hardware swap on one side of a redundant networking switch pair, the network became unresponsive for approximately 60 seconds. The outage has cleared however some connections into or out of the datacenter may have been affected. If you continue to see any networking issues, please contact Datacenter Operations or your designated service …