Good Morning, All,
This is a quick report on the recent network outages on campus and current network status.
At about 11:30am on Tuesday, 28 April, campus network clients began to experience disruptions in service, intermittent initially and then sustained. After an initial diagnosis by Telecommunications and Networking Services staff, the vendor's technical support was contacted. Their initial diagnosis was that either the networking system's internal routing for campus had gotten confused or that we had an equipment failure. Attempts to correct the problem as a routing ("spanning tree") problem were only partially successful, and they concluded that we appeared to be having equipment problems in a couple of boards in a core switch. The replacement boards were installed at about 8 pm, but that only resolved the problem for half of the connections on campus. It appears that the disruption in service on the core system caused other switches on campus to become confused, too, and they did not recover when the core system was restored. The TNS staff completed the manual restarting of those other switches on campus at about 11:30 am this morning.
Aside from a brief disruption between 1:00-1:30pm on Tuesday, off-campus access to core UVM servers was not interrupted by these problems.
If you are continuing to experience network problems after 11:30 am Wednesday, please call 6-8888 to report the problem. That line will be staffed throughout the day to take calls, but if it's busy, please leave a message in which you indicate the physical location (building, room) of the network connection you're reporting.
TNS staff and Cisco's technical support have been working to restore service and haven't had time to fully analyze for possible causes. It appears that at least part of the problem is inability of some of our older equipment (8 years or older) to support the increasing complexity and speed of our network. Over the next few weeks, we will be reviewing our network architecture and installed base of equipment to begin the systematic replacement of older devices.
We apologize for the disruption in campus services. We recognize that the campus relies upon network services for all facets of its work and that disruptions in service are frustrating and costly. We will be making appropriate changes in architecture and reinvestments in equipment to reduce the probability of disruptions like this in the future.
David Todd
Associate Vice President and CIO
This is a quick report on the recent network outages on campus and current network status.
At about 11:30am on Tuesday, 28 April, campus network clients began to experience disruptions in service, intermittent initially and then sustained. After an initial diagnosis by Telecommunications and Networking Services staff, the vendor's technical support was contacted. Their initial diagnosis was that either the networking system's internal routing for campus had gotten confused or that we had an equipment failure. Attempts to correct the problem as a routing ("spanning tree") problem were only partially successful, and they concluded that we appeared to be having equipment problems in a couple of boards in a core switch. The replacement boards were installed at about 8 pm, but that only resolved the problem for half of the connections on campus. It appears that the disruption in service on the core system caused other switches on campus to become confused, too, and they did not recover when the core system was restored. The TNS staff completed the manual restarting of those other switches on campus at about 11:30 am this morning.
Aside from a brief disruption between 1:00-1:30pm on Tuesday, off-campus access to core UVM servers was not interrupted by these problems.
If you are continuing to experience network problems after 11:30 am Wednesday, please call 6-8888 to report the problem. That line will be staffed throughout the day to take calls, but if it's busy, please leave a message in which you indicate the physical location (building, room) of the network connection you're reporting.
TNS staff and Cisco's technical support have been working to restore service and haven't had time to fully analyze for possible causes. It appears that at least part of the problem is inability of some of our older equipment (8 years or older) to support the increasing complexity and speed of our network. Over the next few weeks, we will be reviewing our network architecture and installed base of equipment to begin the systematic replacement of older devices.
We apologize for the disruption in campus services. We recognize that the campus relies upon network services for all facets of its work and that disruptions in service are frustrating and costly. We will be making appropriate changes in architecture and reinvestments in equipment to reduce the probability of disruptions like this in the future.
David Todd
Associate Vice President and CIO