Dear Colleagues

BB8 has been restored. CIS will continue to monitor the system and will be
meeting with our Vendors tomorrow to discuss the problems.   Please see the
note below for more detailed information on what went into restoring the
system.

Thank you.
Brian (via Blackberry)


    ----- Original Message -----
    From: Christopher Caprioglio
    Sent: 04/27/2009 07:41 PM EDT
    To: CUNY-CMS Admin sublist <CMS-ADMIN@LISTSERV.CUNY.EDU>
    Cc: Brian Cohen
    Subject: Re: Blackboard 8.0 Status
Folks,

After the implementation of numerous changes suggested by Sun Microsystems
technicians, and CIS's Systems staff, Blackboard 8.0 service was restored
at 6:33 P.M.  Over the last hour almost 4,400 users have signed into the
system.  We are continuing to monitor performance, but thus far the system
appears to be stable.

chris.

Dear Colleagues:

I would like to take this time to update you on the current status of the
Blackboard 8.0 outages today.

At about 8am today (April 27, 2009) CIS, while monitoring system
performance observed a high use of the system resources that caused the BB8
system to become unstable and unresponsive. We initially waited a short
time to see if this problem would resolve itself and, when it did not, we
brought the system down.  At that time we also notified our vendor
community (Blackboard, Sun Microsystems, and Oracle) and collected the
necessary log files and data they would need to help trouble shoot the
problem and provide guidance and direction toward a resolution.

In order to restore the system it was agreed that the best approach would
be  to reset the system and all of its components.   By noon the system was
restored and made publicly available to the greater University community.

At about 1pm, the problem we observed in the morning was occurring again
and as a result we immediately disabled the system pending further
assistance from our vendors.   Since this latest down time We have been
applying several vendor provided patches and bringing up each of the
Blackboard 8.0 components.   We hope the guidance we have received from Sun
and Blackboard will allow us to restore the system to a normal operating
state.

Additionally, We are also working with Blackboard to complete the necessary
planning to restore the system at the Blackboard hosted DR site should this
outage continue.  We will be consulting with many of our CUNY Blackboard
advisors/faculty users  as how best to activate the DR site when needed.
While DR will soon be a possibility for the Blackboard application, the
potential for data loss and down time as we move to the DR site and then
return back to CIS is greatly increased and should be avoided if possible.

Over the past several weeks there has been little change to any parts of
the system other then continued efforts to fine tune the application or to
increase the frequency of our backups to insure the least amount of
potential data loss in the event of a catastrophic failure. These fine
tuning activities are reasonable and often necessary in a complex
environment especially when new computer systems are put into production.

As I am sure you already know, the CIS team is committed to restoring the
system as quickly as possible.  We have all key staff working on this
problem and have all our vendors either on the phone and for some, soon to
be on site to assist and conduct a root cause analysis.  I will keep you
informed you of the progress achieved in this area.


Thank you
Brian (via Blackberry)