the challenge

After a number of surprise outages in both key and satellite systems the client requested our help to assess the current production environment, we were given the task to identify inherent weaknesses and provide guidance on how to develop and manage the process to perform continual maintenance to core transaction and reporting systems.

our role & process

Our first step was to identify the appropriate people to engage on the project from the client side and to identify clear ownership for both infrastructure and business application areas. The client had recently undertaken an organizational change that separated production operations from development and the organization was in a state of flux. Our consulting team worked closely with the executive personnel the areas to clearly identify the suitable owners and the reporting lines that would allow the project to move forward without delay.

The second step was to identify likely data sources to allow us to capture and then transform raw data into information to assist in decisions at the CIO level, it turned out the client had developed a data center management tool using Java and an underlying Oracle database that held a lot of the data we needed for this engagement. The bad news was that the internal process within the Information Technology group was not mature so some of the data was available but lacked completeness and no formal controls existed to ensure the data was maintained in an ongoing manner.

Given the situation with the state of the raw data we could see that the most expedient manner to get our hands on the data we needed was to employ the internal tool and that our best investment of both time and energy was to get mindshare on the business process to manage the data using this tool. We arranged a number of workshops with the client personnel to identify how we should develop and implement the business process for introductions of new hardware and software as well as retirements and essentially operate in the same way that a telecommunications group would with respect to MAC (moves, adds & changes). A series of regular checkpoint meetings were arranged to manage the data correction as well as evaluate the control of data entry into the internal tool, the client team responded very well to this framework and over the course of a few weeks this powerful tool that had potential became embedded in the day to day operational activity of the department.

In parallel, our team worked closely with the client to identify how we could manage the data to the greatest effect, we opted to align by business system and associate the underlying components (server, associated network switch, application, operating system and database) to key business process activities. The client had stated a program of Business Process Transformation (BPT) where “tracks” were being led to review and improve the enterprise processes. We simply took these “tracks” and associated each application in the data center to a track, for example the Siebel application for quote and forecast management was aligned to the “Quote to Cash” track. This set of associations allowed the client to review our progress using a set of terminology that they were living and breathing on a daily basis, this helped clarify the ownership and areas being assessed.

Our model needed to include each of the models of hardware or versions of software so we had the additional task to gather data about each component. We approached this by vendor to see if the product was obsolete, maintenance was available and any other date related data that could help derive concrete result. The surprising outcome of this was the number of different models of hardware and the antiquity of some components that were supporting key business processes, with help from the vendors we were able to build this into the model to provide a concrete result as well as point to a diverse suite of production items.

Having established the ownership matrix our team developed a model to import the data from the internal custom tool and transform this using various criteria and data attributes that were added to develop a reporting framework based on the detailed data with a consolidated set of summary management reports that would demonstrate priorities to senior IT leadership. The model had to be flexible allowing for prioritization that was not simply data driven, an example of this would be to tag an application and the associated hardware and software elements if the application was due to be retired in the next six months. The timeline appeared to be optimal in that none of the larger initiatives were due to go live in this period and it would present an opportunity for the team to set the priorities and make progress on the top number.

After the data was finalized in each of the BPT areas a series of reviews were conducted with the executives for Infrastructure and Business Application, this was led by the Program Management Office and decisions in terms of upgrades and server replacements were taken. The overall planning was incorporated into the six month horizon for release planning allowing separation of new implementations from projects derived from healthcheck planning.

lessons learned

There are always a lot of competing priorities in IT groups, one common problem we see is that demand on the IT group usually exceeds the resources available. The impact of unplanned outages is significant in terms of cost and morale, there is the additional impact of interruption to projects as resources are diverted to fight fires. The ongoing management of the production environment needs to be an input to the project management process, it is not enough to assume the production support personnel will keep the environment current if they are tasked with managing tickets and incidents.

One of the key things we identify time and time again is that if the process has no single owner it will often fall by the wayside and cease to exist, too often new tools are introduced and do not make it past the launch phase since there is no single owner or sponsor. In this case the data center management tool was very useful but was not actively managed and the data was aging, the IT group had ceased to use it and the potential benefits were not being realized.

after the engagement

Our client now utilizes the internal data center management tool combined with the healthcheck process to incorporate the planning of upgrades to the existing systems infrastructure into the overall project planning for the IT area. The business teams are presented with recommendations they understand from the BPT tracks and have been able to see the benefits in terms of both scalability and the reduction in unplanned outages that impact availability. The ownership of the process and tools has shifted to a newly created position in Information Technology owned by the Availability Manager.