Service Availability - 2011/12

Unplanned Downtime

Services Website

Top Priority Services

Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Total
mins
% Annual
Avalability
Central Wiki Service
Central Wiki Service
  28-Feb (5) Loss of service between EaStMAN and JANET
  11-May (90) Widespread IS services disruption
  21-May (20) Wiki unavailable
  18-Jun (20) Wiki unavailable
  18-Jun (60) Wiki unavailable
  18-Jun (180) Wiki unavailable

0 0 0 0 0 0 5 0 0 110 260 0 375 99.94
Electronic Journals 0 590 0 0 0 0 5 0 0 90 0 0 685 99.88
EUCLID web 30 0 0 0 0 0 5 0 0 90 0 0 125 99.98
Kinetix (KX) 0 0 0 50 75 0 5 60 0 60 0 0 250 99.96
Learn
Learn
  19-Jun (15) Learn unavailable

0 0 0 0 0 0 0 0 0 0 15 0 15 100.00
Library Catalogue 360 0 30 0 0 30 5 0 20 90 125 0 660 99.89
Moodle 0 0 0 0 0 0 0 0 0 130 0 0 130 99.98
MyEd portal framework 0 0 0 0 0 0 30 0 0 90 0 0 120 99.98
Office365 0 0 0 0 0 0 0 0 0 0 0 0 0 100.00
Staffmail 0 0 0 0 0 0 60 30 0 90 0 0 180 99.97
University Website 705 0 0 10 0 15 5 0 0 90 0 0 825 99.86

Top Priority Infrastructure

Core Application storage - SAN 0 0 0 0 0 0 0 0 0 0 0 0 0 100.00
Core Network 0 0 0 0 0 0 0 0 0 0 0 0 0 100.00
EASE 0 0 0 0 0 0 0 0 0 90 0 0 90 99.98
Firewall (Checkpoint 1) 0 0 0 0 0 0 0 0 0 0 0 0 0 100.00
Load Balancers 0 0 0 35 0 0 0 0 0 0 0 0 35 99.99
Mail Relay 0 0 0 0 0 0 0 0 0 0 0 0 0 100.00
Virtual Hosting 0 0 0 0 0 0 0 0 0 0 0 0 0 100.00

Medium Priority

E-Financials
E-Financials
  25-Nov (20) eFinancials unavailable
  13-Feb (90) eFinancials Downtime
  29-Feb (120) eFinancials unavailable
  06-Apr (30) eFinancials
  19-Apr (30) eFinancials
  11-May (90) Widespread IS services disruption
  31-May (80) eFinancials unavailable

0 0 0 20 0 0 210 0 60 170 0 0 460 99.92
EBIS
EBIS
  05-Aug (15) Coldfusion appliactions unavailable
  08-Aug (15) Coldfusion applications unavailable
  12-Aug (15) Coldfusion applications unavailable
  22-Aug (5) Coldfusion and Estates database restart
  23-Aug (5) Coldfusion and Estates database restart
  08-Sep (10) Jobs and EBIS Online unavailable
  26-Oct (30) EBIS Online Down
  11-Nov (30) EBIS Online Down
  16-Nov (20) WebCT, PURE and Polopoly unavailable
  16-Nov (5) WebCT, PURE and Polopoly unavailable
  18-Nov (45) EBIS Online unavailable
  21-Nov (15) EBIS Online unavailable
  25-Nov (10) EBIS Online unavailable
  29-Nov (45) EBIS Online unavailable
  30-Nov (30) EBIS Online unavailable
  12-Dec (10) EBIS Online unavailable
  05-Jan (10) EBIS Online unavailable
  05-Jan (15) EBIS Online unavailable
  05-Jan (10) EBIS Online unavailable
  09-Jan (10) EBIS Online unavailable
  12-Jan (25) EBIS Online unavailable
  13-Jan (20) EBIS Online unavailable
  13-Jan (20) EBIS Online unavailable
  16-Jan (360) EBIS Online unavailable
  19-Jan (15) EBIS Online unavailable
  30-Jan (30) EBIS Online unavailable
  08-Feb (30) AT network outage
  17-Feb (30) Ebis Online Unavailable
  17-Feb (50) Ebis Online Unavailable
  22-Feb (30) Ebis Online Unavailable
  07-Mar (60) Ebis Online Unavailable
  12-Apr (10) Ebis Online Performance degraded
  11-May (90) Widespread IS services disruption

55 10 30 200 10 515 140 60 10 90 0 0 1120 99.81
EdWeb 0 0 0 0 0 0 0 0 0 0 0 0 0 100.00
eRecruitment 55 145 0 0 0 0 0 0 0 95 0 1200 1495 99.74
Event Booking 45 0 125 0 0 0 0 0 0 95 0 0 265 99.95
Hosted Websites 10 0 30 0 0 15 0 0 0 1680 0 0 1735 99.70
HR/Payroll
HR/Payroll
  03-Aug (60) HR unavailable
  31-Oct (30) Restart Oracle HR
  25-Jan (205) HR unavailable
  26-Mar (50) HR Application unavailable
  26-Apr (20) HR unavailable
  11-May (90) Widespread IS services disruption

60 0 30 0 0 205 0 50 20 90 0 0 455 99.92
ID Management
ID Management
  02-Aug (765) IDM System problems
  20-Oct (15) VRS / IDMS Outage
  25-Jan (10) IDM SOA restart
  01-Feb (10) IDM SOA restart
  11-May (90) Widespread IS services disruption
  04-Jun (20) IDM SOA restart
  06-Jun (20) IDM processing delay

765 0 15 0 0 10 10 0 0 90 40 0 930 99.84
Lecture Recording 0 0 0 0 0 0 0 0 0 0 0 0 0 100.00
Network Shared Drives 0 0 0 0 105 0 0 0 0 120 0 0 225 99.96
PURE 45 0 0 25 15 180 5 45 0 0 0 0 315 99.95
Reporting + Analytics (BI Suite)
Reporting + Analytics (BI Suite)
  28-Sep (30) BOXI live server restart (bobj-live)
  28-Oct (30) Boxi restart
  21-Nov (90) Boxi Restart
  10-Jan (15) BOXI unavailable
  31-Jan (20) BOXI Unavailable
  09-Feb (20) BOXI unavailable
  14-Mar (10) Boxi Restart
  25-Apr (30) BOXI server down
  11-May (90) Widespread IS services disruption
  14-May (45) BOXI Application unavailable for some universe
  07-Jun (270) BOXI Unavailable
  27-Jun (60) BOXI unavailable
  28-Jun (90) BOXI Unavailable
  05-Jul (945) BOXI unavailable
  09-Jul (30) BOXI unavailable
  09-Jul (15) BOXI unavailable
  10-Jul (30) BOXI unavailable
  12-Jul (20) BOXI unavailable
  13-Jul (20) BOXI unavailable
  13-Jul (20) BOXI unavailable
  16-Jul (15) BOXI unavailable
  19-Jul (20) BOXI unavailable
  19-Jul (10) BOXI unavailable
  20-Jul (20) BOXI unavailable
  20-Jul (20) BOXI unavailable

0 30 30 90 0 35 20 10 30 135 420 1165 1965 99.66
Timetabling 0 0 0 0 0 0 0 0 0 0 0 0 0 100.00
UniDesk 0 30 60 20 15 30 10 0 380 180 0 20 745 99.87
WebCentral 0 0 0 0 0 0 0 0 0 0 0 0 0 100.00
Worktribe 0 0 0 0 0 0 0 0 0 0 0 0 0 100.00
Colour Coding Definitions
% 99.9% up to 100% availability or up to 8.5 hours downtime in a year
% between 99.5% and 99.89% or between 8.5 and 44 hours downtime in a year
% below 99.5 or above 44 hours downtime
0 events occurring either entirely or partially outwith core hours (Mon - Fri, 5pm - 9am and all day Sat & Sun)

 

Notes Cause(s)
-------- ------------
July
01) Student Mail Service (SMS) was unavailable on 1 occasion Due to an unknown cause. No investigation needed as isolated incident.
02) Jobs was unavailable on 1 occasion For restart of the Coldfusion service. No investigation required.
03) UniDesk was unavailable on 1 occasion Once due to issue with logs on database server. Under investigation.
04) BOXI was unavailable on 12 occasions All due to unknown cause. Under investigation.
June
01) Central Wiki Service was unavailable on 3 occasions All due to an unknown cause. Under investigation.
02) Learn was unavailable on 1 occasion Due to critical work as a result of a very heavy load on service. No investigation required.
03) Library Catalogue was unavailable on 2 occasions Once due to unexpected file system failure on database server. Once due to a reboot of the server as part of continued SAN work. No investigation required.
04) ID Management was unavailable on 2 occasions Once due to restart required to re-activate IDM soa processing. Once due to failed restart of IDM service following security patching. No investigation required.
05) BOXI was unavailable on 3 occasions All due to restarts of the BOXI server. No investigation required because issue not occurring regularily.
May
01) Central Wiki Service was unavailable on 2 occasions Once due to a power interruption at JCMB and once due to a wiki crash. No investigation required.
02) Electronic Journals was unavailable on 1 occasion Due to a power interruption at JCMB. No investigation required.
03) EUCLID web was unavailable on 1 occasion Due to a power interruption at JCMB. No investigation required.
04) Exchange was unavailable on 2 occasions Twice due to restarts of Exchange due to a problem with the backup. No investigation required.
05) Kinetix(KX) was unavailable on 1 occasion Due to kinetics upgrading the Accommodation Services database. No investigation required.
06) Library Catalogue was unavailable on 1 occasion Due to a power interruption at JCMB. No investigation required.
07) Moodle was unavailable on 2 occasions Once due to regular patching overtime and once due to a power interruption at JCMB. No investigation required.
08) MyEd was unavailable on 1 occasion Due to a power interruption at JCMB. No investigation required.
09) Staffmail was unavailable on 1 occasion Due to a power interruption at JCMB. No investigation required.
10) University Website was unavailable on 1 occasion Due to a power interruption at JCMB. No investigation required.
11) WebCT was unavailable on 2 occasions Due to a restart to clear problem with powerlinkss and once due to a power interruption at JCMB. No investigation required.
12) Central Authorisation Service was unavailable on 1 occasion Due to a power interruption at JCMB. No investigation required.
13) Ezproxy was unavailable on 1 occasion Due to a power interruption at JCMB. No investigation required.
14) E-Financials was unavailable on 2 occasions Due to a server restart to resolve size limit issues and once due to a power interruption at JCMB. No investigation required.
15) EBIS was unavailable on 1 occasion Due to a power interruption at JCMB. No investigation required.
16) Event Booking was unavailable on 2 occasions Due to a restart to resolve a configuration issue and once due to a power interruption at JCMB. No investigation required.
17) Hosted Websites was unavailable on 1 occasion Due to the Apache service stopping, restart resolved the issue. No investigation required.
18) HR/Payroll was unavailable on 1 occasion Due to a power interruption at JCMB. No investigation required.
19) ID Management was unavailable on 1 occasion Due to a power interruption at JCMB. No investigation required.
20) Jobs was unavailable on 2 occasions Due to a restart to resolve a configuration issue and once due to a power interruption at JCMB. No investigation required.
21) Network Shared Drives was unavailable on 2 occasions Once due to a restart to resolve resource issues with server and once due to electrical work being carried out at Ashworth. No investigation required.
22) Polopoly was unavailable on 1 occasion Due to a power interruption at JCMB. No investigation required.
23) Unidesk was unavailable on 3 occasions Due to a Unidesk outage and twice due to security patching. No investigation required.
24) BOXI was unavailable on 2 occasions Once due to a restart of the application to resolve responce issues and once due to a power interruption at JCMB. No investigation required.
April
01) Exchange was unavailable on 1 occasion Due to a restart following a problem with the memoory after replication. No investigation required.
02) Library Catalogue was unavailable on 1 occasion Due to a restart of SDLC Voyager application after a server disc issue. No investigation required.
03) E-Financials was unavailable on 2 occasions Once due to a restart to clear up the memory and once due to remounting remounting. No investigation required.
04) EBIS was unavailable on 1 occasion Due to a restart to clear performace issues. No investigation required.
05) HR/Payroll was unavailable on 1 occasion Due to a restart of the application and database to resolve performance issues. No investigation required.
06) Unidesk was unavailable on 6 occasions Three times due to server restarts to resolve performance issues, once due to Java issues, once due to a restart of the St. Andrews instance and once due to debugging of TDESKAPP-KB1. No investigation required.
07) BOXI was unavailable on 1 occasion Due to a server restart. No investigation required.
March
01) Kinetix (KX) was unavailable on 1 occasion Due to a restart of the server in order to pick-up new database parameters. No investigation required.
02) Staffmail was unavailable on 1 occasion Due to a user's mail filter creating a mail storm eventually overloading the SMTP mail submission service. No investigation required.
03) EBIS was unavailable on 1 occasion Due to a restart of EBISLIVE to resolve performance issues. No investigation required.
04) HR/Payroll was unavailable on 1 occasion Due to a database and application re-start to resolve performance issues. No investigation required.
05) PURE was unavailable on 2 occasions Once due to an upgrade as part of the project and once for a restart of the PURE server to apply changes to the PURE Portal domain name. No investigation required.
06) BOXI was unavailable on 1 occasion Due to a restart of BOXI server to resolve performance issues. No investigation required.
February
01) Central Wiki Service was unavailable on 1 occasion A configuration change caused a short loss of network service between EaStMAN and JANET. No investigation required.
02) Electronic Journals was unavailable on 1 occasion A configuration change caused a short loss of network service between EaStMAN and JANET. No investigation required.
03) Exchange was unavailable on 2 occasions Once due to a configuration change caused a short loss of network service between EaStMAN and JANET and once due to network issues. No investigation required.
04) Kinetix (KX) was unavailable on 1 occasion A configuration change caused a short loss of network service between EaStMAN and JANET. No investigation required.
05) Library Catalogue was unavailable on 1 occasion A configuration change caused a short loss of network service between EaStMAN and JANET. No investigation required.
06) MyEd was unavailable on 1 occasion Due to an outage at Appleton Tower. Currently under investigation.
07) Online Student Admin was unavailable on 1 occasion A configuration change caused a short loss of network service between EaStMAN and JANET. No investigation required.
08) Staffmail was unavailable on 1 occasion Due to work carried out to increase memory size. No investigation required.
09) Student Mail was unavailable on 1 occasion A configuration change caused a short loss of network service between EaStMAN and JANET. No investigation required.
10) University Website was unavailable on 1 occasion A configuration change caused a short loss of network service between EaStMAN and JANET. No investigation required.
11) WEBCT was unavailable on 3 occasions Once because a configuration change caused a short loss of network service between EaStMAN and JANET and twice for full restarts of WebCT. No investigation required.
12) Ezproxy was unavailable on 1 occasion A configuration change caused a short loss of network service between EaStMAN and JANET. No investigation required.
13) E-Financials was unavailable on 2 occasions Due to patching of the efinancials software. No investigation required.
14) EBIS was unavailable on 4 occasions Once due to an outage at Appleton Tower and three times for a restart of CF9 for Estates instance. AT issue currently under investigation.
15) ID Management was unavailable on 1 occasion Due to a restart of the IDM SOA service following corporate database outage. No investigation required.
16) Polopoly was unavailable on 1 occasion A configuration change caused a short loss of network service between EaStMAN and JANET. No investigation required.
17) PURE was unavailable on 1 occasion A configuration change caused a short loss of network service between EaStMAN and JANET. No investigation required.
18) Unidesk was unavailable on 2 occasions Once duue to a configuration change that caused a short loss of network service between EaStMAN and JANET and once due to a new SSL certificate being put in place. No investigation required.
19) BOXI was unavailable on 1 occasion Due to an unknown outage. No investigation required.
January
01) Library Catalogue was unavailable on 1 occasion Due to a Voyager system reboot to complete Unix Oracle Patching. No investigation required.
02) Exchange was unavailable on 1 occasion Due to the network switch causing Exchange to fail. No investigation required.
03) University Website was unavailable on 1 occasion Due to instability following the production database patching. No investigation required.
04) EBIS was unavailable on 10 occasions Due to intermitant issues. No investigation required.
05) Hosted Websites was unavailable on 1 occasion Due to a critial patch to the Windows ASP.Net framework. No investigation required.
06) HR/Payroll was unavailable on 1 occasion Due to issues encountered after large scale database patching. Currently under investigation.
07) ID Management was unavailable on 1 occasion Due to a restart of the IDM SOA service following corporate database outage. No investigation required.
08) PURE was unavailable on 4 occasions Due to restarts afte rcrashes caused by large reports being run. No investigation required.
09) Unidesk was unavailable on 1 occasion Due to the network switch causing Unidesk to fail. No investigation required.
10) BOXI was unavailable on 2 occasions Due to technical issues. No investigation required.
December
01) EBIS was unavailable on 1 occasion Due to a restart of EBIS Online. No investigation required.
02) Kinetix (KX) was unavailable on 1 occasion Due to Windows critical patching. No investigation required.
03) Network Shared Drives was unavailable on 1 occasion Due to a lack of responce from hsskhyber, Server was switched. IS ITI Architecture are investigating.
04) PURE was unavailable on 1 occasion Due to a restart of the PURE Application. No investigation required.
05) Unidesk was unavailable on 2 occasions Twice due to server restarting to resolve performance issues. No investigation required.
November
01) E-Financials was unavailable on 1 occasion Due to a space issue on the server which required a restart. We are investigationg why we have not received the space reach notifications as they should be in place.
02) EBIS was unavailable on 7 occasions 5 times due to a restart of EBIS Online, Once due to a Load balancer failure and once for a required Could Fusion restart. A problem has been rasised to investigate rge load balancer failure, other cases require no investigation.
03) Kinetix (KX) was unavailable on 1 occasion Due to a loss of connectivity between Witness and DR server. No investigation required.
04) Load Balancers was unavailable on 2 occasions Twice due to a Load balancer failure. ITI Unix have confirmed that this has been fixed. A Problem call has been raised for investigation in the cause.
05) PURE was unavailable on 1 occasion Due to a Load balancer failure. ITI Unix have confirmed that this has been fixed. A Problem call has been raised for investigation in the cause.
06) Polopoly was unavailable on 1 occasion Due to a Load balancer failure. ITI Unix have confirmed that this has been fixed. A Problem call has been raised for investigation in the cause.
07) Unidesk was unavailable on 2 occasions Once due to a Load balancer failure and once due to an overnight job still running which was affecting performace and required a restart to resolve. ITI Unix have confirmed that the load balancer has been fixed. A Problem call has been raised for investigation in the cause.
08) University Website was unavailable on 1 occasion Due to a Load balancer failure. ITI Unix have confirmed that this has been fixed. A Problem call has been raised for investigation in the cause.
09) BOXI was unavailable on 1 occasion A restart of teh server. No investigation required.
10) WebCT was unavailable on 2 occasions Once due to a restart of WebCT and onec due to a Load balancer failure. ITI Unix have confirmed that this has been fixed. A Problem call has been raised for investigation in the cause.
October
01) EBIS was unavailable on 1 occasion Due to the Coldfusion 9 Instance being restarted. No investigation required.
02) Event Booking was unavailable on 2 occasions Both due to a Database issue, that is still being investigated under alert http://reports.is.ed.ac.uk/alerts/index.cfm?fuseaction=view_alert&alert_id=1498.
03) HR/Payroll was unavailable on 1 occasion Due to a restart of the application. No investigation required.
04) Hosted Websites was unavailable on 1 occasion Due to a network switch replacement. No investigation required.
05) ID Management was unavailable on 1 occasion Due to a database restart to fix service issues. No investigation required.
06) Library Catalogue was unavailable on 1 occasion Due to a network switch replacement. No investigation required.
07) Unidesk was unavailable on 1 occasion Due to a restart of the application. No investigation required.
08) BOXI was unavailable on 1 occasion Due to a restart of the server. No investigation required.
09) WebCT was unavailable on 2 occasions Once to move to the new citrix netscaler loadbalancer infrastructure and once due to Blackboard patch VST-5840 to LIVE WebCT to fix high load on the database server. No investigation required.
September
01) EBIS was unavailable on 1 occasion Once for a restart of the ColdFusion server. No further investigation as systems are being moved off this server.
02) Electronic Journals was unavailable on 1 occasion Once due to a failure on local server meaning the planned switch to ExLibris hosted service had to be carried out immediately
03) Jobs was unavailable on 2 occasions Twice due to a restart of the ColdFusion server. No further investigation as systems are being moved off this server
04) Unidesk was unavailable on 1 occasion Due to deployment of new IDM connector to UniDesk. This work crashed UniDesk. Resatrting UniDesk resolved the problem
05) BOXI was unavailable on 1 occasion Due to an unplanned update to max file size, restart needed to take effect. No investigation required.
August
01) EBIS was unavailable on 5 occasions Twice due to fixing IDM issues requiring a restart of the ColdFusion server, Once for patching of the IDM VRS management interface and Twice for a restart of the EBIS database. No investigation required.
02) Event Booking was unavailable on 3 occasions Twice due to fixing IDM issues requiring a restart of the ColdFusion server and Once for patching of the IDM VRS management interface.No investigation required.
03) HR/Payroll was unavailable on 1 occasion Due to an Apache restart. No investigation required.
04) Hosted Websites was unavailable on 1 occasion Due recent patching the server kb-iis-1 was refusing RDP connections. A restart is required to restore access. No investigation required.
05) ID Management was unavailable on 1 occasion Due to an update 12:43 2/8/2011 The issue was raised as a priority issue with Oracle. Following supplied instructions, the service issue has been resolved. A fix is currently being tested in order to prevent repeat occurrence.
06) Jobs was unavailable on 5 occasions Twice due to fixing IDM issues requiring a restart of the ColdFusion server, Once for patching of the IDM VRS management interface and Twice for a restart of the EBIS database. No investigation required.
07) Library Catalogue was unavailable on 1 occasion Due to indexes being rebuilt to recify a problem with ECA bibliographic data searching. No investigation required.
08) Online Student Admin was unavailable on 1 occasion Due to an outage to accommodate a database restart so that a memory parameter change can take effect. No investigation required.
09) PURE was unavailable on 3 occasions Twice due to fixing IDM issues requiring a restart of the ColdFusion server and Once for patching of the IDM VRS management interface.No investigation required.
10) University Website was unavailable on 2 occasions Once due to a restart of both UWS servers. Reason for server failures not known, but one server seemed to have failed some time ago and second server failed today and Once due to part of the Polopoly upgrade the University Website became unavailable. Cause not known at this time. The upgrade has been reversed and restored from a backup (Sat 27-AUG-11 19:29). Investigation ongoing.
11) WebCT was unavailable on 1 occasion Due to a full shut-down of the WebCT application on all nodes followed by restart. No investigation required.