2019.03.30 - Major Planned Maintenance

April 2nd, 3:20PM:

We are very happy to announce that all services are now restored to normal operation.  

This was a particularly severe and unexpected service disruption, but CAIT's Network Operations team worked tirelessly to limit the impact and were critical to making sure alternative methods of access were available during the past few days.

Services restoration timeline:

  • 01:07PM - REDCAT.org, Business Objects (Internal)
  • 01:15PM - CalArts.edu (and associated sites)
  • 01:39PM - Accounting Credit Card Services
  • 02:23PM - hub.calarts.edu
  • 02:38PM - hubhelper.calarts.edu
  • 02:41PM - forms.calarts.edu/start
  • 02:47PM - CalArts Credentials Reset Form (cait.calarts.edu)
  • 02:48PM - photo.calarts.edu, dots.calarts.edu, facilities.calarts.edu
  • 02:49PM - printservices.calarts.edu
  • 03:00PM - Colleague UI (Production & Test)

Although we have said it multiple over the course of this disruption it bears repeating: the CAIT team deeply appreciates our communities patience, understanding, and trust during events like these.

Please send us a note at cait@calarts.edu if you have any further issues with these sites and our Customer Care team will be happy to help!

April 2nd, 1:30PM:

CAIT is pleased to announce that as of approximately 1:10PM services are returning to normal operation.  Due to required steps during restoration, sites and services must be restored individually, which extends the overall time involved.  

Given that we are through the restoration process, however, we felt it best to inform the community that we are coming back online:

  • REDCAT.org: Restored and tested
  • CalArts.edu: Restored and tested

We expect the next announcement will be the final resolution and full services list.

April 2nd, 12:30PM:

At approximately 9:30AM this morning CAIT received the replacement equipment and by 10:30AM the new unit was plugged in and ready for restoration.  That process began shortly thereafter and has continues through the early afternoon.  

Further developments will be available as they occur.

April 1st, 4:58PM:

After a significant engagement with senior vendor support we have determined the failed network equipment needs to be replaced.

Given the severity and impact of this failure, the replacement is being shipped overnight, and will work towards full restoration of regular services shortly after receipt.

While our team continues to work diligently addressing these issues, CAIT greatly appreciates the CalArts communities' patience and understanding during this extended service disruption.

April 1st, 12:50PM:

Working closely with progressively senior vendor support specialists, our Network Operations  team has made progress on the issue through the morning of Monday, April 1st, but we remain without a resolution. If this situation continues  to be unresolved further into the afternoon CAIT will be weighing alternative escalation plans.

It is exceptionally abnormal to have a service disruption be as extended, and severe, as this one has been. The CAIT team continues to work diligently towards  resolution and greatly appreciates the communities' patience and understanding.

March 31st, 12:00PM:

Rudimentary access to REDCAT.org has been restored, however due to technical limitations restoring full access will require diverting resources from resolving the primary issue.  CAIT is aware of, and sensitive to, implications on marketing of upcoming performances, but we are prioritizing resources to resolving the overall problem.

March 30th, 9:40PM:

Network Operations has temporarily restored the following high-priority services:

  • calarts.edu: Access to CalArts.edu has been restored.
  • ui.calarts.edu: Access to Colleague UI has been restored, but requires the use of a temporary URL which has been emailed directly to those with accounts.
  • hub.calarts.edu: Access to the Hub has been restored, but requires the use of a temporary URL which has been emailed to all Students, Faculty, and Staff.
  • selfservice.calarts.edu: Self Service is available at its regular URL, selfservice.calarts.edu/student.

While we have no indication of abnormal experiences with any of the services listed above, please contact IT at cait@calarts.edu.

March 30th, 8:04PM:

Due to the severity of this issue requiring technical resources not available until Monday, April 1st, CAIT staff have begun implementation of a plan to temporarily restore service to high-priority servers and services.

CalArts.edu is now accessible from both on-campus and off-campus clients, although some issues may be present if existing content is linked to unavailable resources.

Our Network Operations team is focusing efforts on temporarily restoring functionality to selfservice.calarts.edu, ui.calarts.edu, and hub.calarts.edu.  

March 30th, 6:07PM:

Our Network Operations team has determined that a critical infrastructure component failed unexpectedly during regular maintenance. Working with vendor support we have further clarified the extent of the failure and are working diligently to develop remediation strategies.

This failure has broad impacts on the accessibility of many CalArts services, including but not limited to:

  • CalArts.edu, REDCAT.org, and associated websites.
  • The Hub, Colleague, and related services.

Events like these, while exceptionally rare, are incredibly disruptive, and we deeply appreciate the CalArts communities' patience and understanding.

Sites and services currently unavailable will continue to remain so until a temporary solution can be developed.

Further communication will be sent promptly as developments occur.

March 30th, 2:59PM:

It is unfortunately necessary to extend todays maintenance window. While  the work associated with the internet connection has been completed,  the infrastructure supporting CalArts.edu and associated websites have  experienced unexpected complications and are currently unavailable.

Our Network Operations team is working diligently towards a resolution with vendor support.

We are extending todays maintenance through 6:00PM and will update the CalArts community as substantive developments occur.

On Saturday, March 30th 2019 from 9:30AM-3:00PM PST, CalArts IT will be performing scheduled maintenance on several core network components.

During this maintenance window, internet connectivity should be considered unavailable or unreliable at best. This will also affect core services including: CalArts Sites, Colleague, HUB, HubHelper, PaperCut (printing), VPN, and various network shares. During this maintenance window, these services should also be considered unavailable.

Please send us a note at cait@calarts.edu if you have any further issues with these sites and our Customer Care team will be happy to help!

Have more questions? Submit a request

Comments

Please sign in to leave a comment.