MHC Outage

Unplanned File Storage Outage

Fri, 12 Mar 2010 13:16:00 -0500

9:00am - All file storage on 50thfs01 is temporarily unavailable.

2:13pm - All file storage is now online.

Unplanned voice / phone outage

Mon, 08 Mar 2010 13:09:00 -0500

9:00am - Cisco phones are unable to receive calls from outside numbers. Cisco phones are unable to make calls to outside numbers.

10:34am - issue resolved. Inbound and outbound calling has been restored.

Unplanned network outage

Sun, 07 Mar 2010 12:41:49 -0500

9:00am - one of the network routers was detected as down causing an outage of:

Ballard network (wired, wireless, phones)
West Seattle, Lake City, and Bellevue (wireless)

9:45am - service restored

General Outage

Fri, 05 Mar 2010 19:38:58 -0500

Update (4:38 PM PST): All connectivity restored.

At about 4:30 PST on March 5th, a router was rebooted, impeding various services, including:

- Phones
- 2x

Firewall Restarted

Fri, 19 Feb 2010 19:23:40 -0500

4:22pm - our firewall was reset and is now online.

LambsList Outage

Sun, 07 Feb 2010 14:36:00 -0500

07:56am - Lambslist reported as being extremely slow / down

08:52am - Lambslist issue verified over multiple campuses.

LambsList has been contacted and is working on this issue.

12:11pm - LambsList is reported as being back online.

Colo Server Upgrade

Fri, 22 Jan 2010 00:17:00 -0500

9:16pm - Colo update has been moved to 9 pm because of a financial report that needed to be run.

9:32pm - Applying update to mhcolovm02.

9:45pm - Updated applied to mhcolovm02, beginning mhcolovm01.

11:05pm - Update is complete, however we are still looking into ftp performance issues.

Unplanned Request Tracker Outage

Mon, 18 Jan 2010 13:48:00 -0500

Update (1:54pm PST): This issue has been resolved.

Request Tracker is currently unable to download emails sent to any of its queues. This includes Helpdesk and AVL Support.

We are currently working with our hosting provider to resolve the issue.

Colo Server Upgrade

Tue, 05 Jan 2010 00:02:00 -0500

9:00pm An upgrade is being preformed on the virtual server environment that will cause an outage of the FTP server, Financial Edge, and all Terminal Services ie (Logos, FE, etc.) Will update when outage is complete.

10:00pm Upgrade is complete.

Unexpected Outlook Errors

Mon, 04 Jan 2010 15:11:00 -0500

Some Windows Outlook users may be prompted to enter their login password multiple times when starting Outlook. Mac Entourage users are not effected by this.

11:00AM - issue reported among several users.

12:00PM - working with hosting provider to provide resolution.

1:30PM - unable to RDC, ping, tracert to machine. Also some users reporting limited access to Public shared folders and calendars.

2:00PM - server restarted.

2:30PM - server back online. RDC, ping, and tracert issue resolved to a routing issue. Outlook issue persisting. Continuing to work with service provider.

3:00 - Outlook issue resolved. Public folder information has been corrupted and will need to be recreated.

Unexpected Blackberry Enterprise Server Outage

Tue, 15 Dec 2009 14:38:00 -0500

11:40am (15Dec) - Server not pushing email to all clients and is restarted.

12:15pm (15Dec) - Server not responding to remote management client. Escalated issue to hosting provider.

11:47am (16Dec) - Full service restored. BES server online.

Planned B50 Outage

Thu, 10 Dec 2009 14:43:00 -0500

At 1150PDT, Mars Hill Technology will be restarting a router at our co-location facility. During this period, all network access will be unavailable.

Everything should be back online by 1215PDT.

Unexpected Albuquerque Outage

Thu, 10 Dec 2009 00:00:00 -0500

This morning, all Albuquerque internet services were unexpectedly interrupted. Our service providers inform us that the outage was caused by weather on the west coast, and will be resolved in a few days.

Unexpected Exchange Outage

Wed, 09 Dec 2009 19:28:27 -0500

4:01pm (PST) - Exchange server back online. All service restored.

3:45pm (PST) - Exchange server is not available. All users are unable to send or receive email.

Blog & theresurgence.com outage

Fri, 30 Oct 2009 02:20:06 -0400

We will be doing some maintenance on all the blog sites and theresurgence.com tonight October 29th starting at 11:30pm PST. The outage may last to 1:00am PST October 30th.

Unexpected Financial Edge Outage

Mon, 05 Oct 2009 14:34:00 -0400

Issue: Financial Edges gives some users an error when logging in

Sunday 4 Ocober

9am - Financial Edge allows some users to connect while other users are given an error upon login

9pm - Server is restarted

Monday 5 October

9am - issue is retested with several users and errors persist

Wednesday 7 October

12pm - issue resolved. all services back to normal

Unexpected Ballard Public Datastore Outage

Mon, 10 Aug 2009 20:26:00 -0400

Monday 8/10

5:22pm - server restarted

5:29pm - server online. service restored.

5:40pm - server offline for second restart.

5:47pm - server online, but shared folders are unavailable.

8:00pm - 11:00pm worked to restore service to the storage device. The storage device has a failed disk, and the storage devices interface is unavailable to access the backups we have been doing.

Tuesday - 8/11

9:00am - All Staff Outage notification set out addressing the issues.

12:00pm - Clone of backup volume has started.

Wednesday - 8/12

Failed Harddrives have been sent off for recovery.

Unexpected Finance Server Outage

Wed, 05 Aug 2009 13:58:00 -0400

10:00am - The Finance Server has stop responding. We are working to restore services as quick as possile.

6:29pm - Restoring data from backups - this has taken the majority of the day.

8:50pm - Server has been restore.

Unexpected 2x Outage

Sun, 02 Aug 2009 14:50:37 -0400

9:45am - 2x determined to be down
11:20am - 2x server restarted
11:25am - 2x server fully operational

Scheduled 2X Outage

Thu, 16 Jul 2009 21:49:00 -0400

6:49PM - Starting update process.

7:15PM - Restarting Server

7:40PM - Server is back up and upgrade process is complete.