From: Brian Cohen
    Sent: 04/28/2009 11:48 PM EDT
    To: ..... <CMS-ADMIN@LISTSERV.CUNY.EDU>
    Subject: BB 8.0 Update

Dear Colleagues:

For reasons I am sure you understand, this has been a very frustrating day
for the team at CIS.  Just prior to the sending of this update the CIS team
was able to restore the Blackboard system.

Beginning around 10am today, Blackboard 8.0 began to show the same slow
response time and high system resource needs as it did for the last week.
In spite of the efforts we undertook yesterday and under the advice and
guidance of our vendor advisors, the high demand for CPU resources caused
the system to once again become unresponsive.  While CIS was able to
restore the system quickly, the system failed again at 12:39pm with all the
same traits as the earlier problems in the day.  After several attempts to
restart the system to acceptable levels of service, we were unable to
achieve a level of service or response time needed to support our user
community.

Throughout the day today, the CIS team has been working side-by-side with
senior and highly specialized engineers from Blackboard, Oracle and Sun
Microsystems.  This team witnessed the erratic behavior and captured
additional diagnostic information that will hopefully assist in identifying
the root cause(s) of these latest BB8.0 problems.  At this time the CIS
team continues to work to get the system restored to a normal level of
service and  our vendor team is reviewing the data captured from today and
will be meeting again on Wednesday to work through their findings.

While CIS strives to provide highly reliable Blackboard services, we have
clearly been challenged by the problems that even some of the most
experienced engineers and technicians cannot solve so easily.  As discussed
in previous reports, we have been working with Blackboard to formalize a
contractual relationship for Disaster Recovery (DR) services.  Today,
Michael Chasen, CEO of Blackboard and I both agreed that best course of
action at this time is to move the CUNY Blackboard 8.0 production
environment over to a Blackboard hosted service.  While in a hosted state,
the CIS team will along with resources from Blackboard, Oracle, and Sun
continue to focus on analyzing the data collected to identify the root
cause(s) of these ongoing problems.  This hosted approach will allow our
user community to continue to use the Blackboard system to meet their class
requirements and for CIS to continue with the needed diagnostic work. I
assure you, CIS will not move the system back to our data center until we
are confident that either a root cause has been identified or sufficient
proof is given that the Blackboard system can operate in a reliable and
steady state at 57th Street.

To carry out this hosted system objective, I have identified a team of CIS
staff responsible for achieving this transition to the Blackboard data
center.  Some of the tasks that need to be completed before we can cut over
to a hosted environment include: moving the most current copy of the CUNY
Blackboard 8.0 data/content to the Blackboard data center, establishing a
technical and secure process that will allow our users to authenticate and
login to the system at Blackboard, and modify the current network
architecture to allow for the increased network traffic over the internet
for this Blackboard usage.  Teams from both Blackboard and CUNY have
already taken steps in each of these areas.  Data transfers have been
tested and the actual transfer of data has started.  I have been assured by
the Blackboard CEO that his team will work quickly to accomplish this cut
over and he will be able to provide a better estimate of the time needed to
complete the cut over once the data transfer has been completed.

As always, I will continue to keep you informed of progress made in each of
these areas.

Thank you.
Brian (via Blackberry)