Application Change Management Best Practices

Database and operating system administrators are ultimately responsible for guaranteeing the quality of their organization’s information processing environments. From protecting against unauthorized access to providing 24×7 availability – “the buck stops with the support unit.” Although the database infrastructure (DB binaries, O/S, hardware) doesn’t change much, there is one component that usually changes a lot – the application. This blog post provides readers with helpful hints and tips on application change management best practices.

I started my career working in a mainframe environment. Well-known database guru, Craig Mullins, and I administered DB2 databases on huge, big iron platforms- platforms that supported thousands (upon thousands) of concurrent users. One of the benefits of this background is that it taught us the importance of change management best practices.

Craig and I learned that a key ingredient of a trouble-free mainframe environment was ensuring that there were no “surprises” when a change was implemented in production, changes that can affect the day-to-day business operations of an entire organization. Throughout my career, I have applied these “mainframe style” best practices to all other database ecosystems that I was responsible for supporting.

It works. The first company I applied these best practices to was selected by Oracle as a “Showcase Environment”. This was back in the old days, when Scott’s tiger was just a cub. Oracle identified shops that it thought had rock-solid support environments and asked them to host visitors from other organizations that wanted to create their own high-quality support infrastructures.

My current organization, Remote DBA Experts (RDX), supports over 300 customers. We are responsible for monitoring and administering thousands (and thousands) of servers. Our customers’ environments consist of a myriad of third-party and homegrown applications running on every imaginable database/operating system/hardware combination. We provide support for all major data infrastructure products including SQL Server, Oracle, Oracle EBS, DB2 LUW, DB2 mainframe, MySQL, PostgreSQL, Windows and UNIX/Linux operating systems as well as Hadoop. During our last SSAE16 audit, we were evaluated on 15,000 individual change requests. The changes ranged from “create this user” to “please build this Oracle RAC 8 Node Cluster”. We implement a lot of changes at RDX.

It is something we are very good at. I thought I would provide you with a few helpful hints and tips on our change management best practices. Since few readers will work for a remote database and operating systems services provider, I’ll tailor my recommendations to readers working in smaller and/or newer shops that many not have a complete set of change management processes in place.

Database Design Reviews

One of my earlier blog posts provides information on database design reviews. Database design review meetings foster effective communication between the DBA unit, system support personnel and application developers throughout the entire application design and implementation process. When database design issues are addressed early in the development lifecycle, problems are minimized and the migration from test to production is more easily accomplished.

If you haven’t read the design review blog, you should. I’m intentionally not covering the importance of rigorous testing of any change before it is implemented in production because it is covered in-depth in the design review blog. From simple changes to new application implementations, there is simply no reason not to perform the test, review, change, test, review, change iteration lifecycle. Although the blog post covers new application implementations, the post will show you how important I think it is to follow a rigorous test plan.

Proceduralize the Change Request Process

Database administrators usually support different business units with each unit having their own set of unique procedural requirements. Formalizing and documenting the change request process minimizes the potential for miscommunication between the business units, application development areas and the database administration unit.

The notification lead-time required for the database administration team to perform a requested change should be documented and distributed to business and application development units. This will prevent your team for getting a request to migrate a database from test to production in the morning with a required date for that afternoon. Of course, we all know that never happens. Since our customers at RDX share support personnel with each other, following our SLAs is of utmost importance. It enables RDX to provide high-quality support to all customers. We completely understand that business needs often demand quick implementations, but we make every attempt to work with our customers to help them plan their changes in advance.

We break our response time SLAs into different categories based on the complexity of the change and the amount of work it requires. We have a different lead time for simple database object changes vs. creating that Oracle RAC 8 Node Cluster I was discussing earlier in this blog.

If your organization doesn’t have a formal change request process in place (and many shops don’t), create your own. There are dozens of work request ticketing, change management and source code versioning tools available on the market today. The prices can range from thousands to tens of thousands of dollars.

Although I highly recommend these types of products, I wouldn’t let the lack of having one prevent me from formalizing the change management process. You can do the best with what you have.

OK, so you don’t have the good fortune of having a formal change management process in place. What do you do? You can begin the formalization of the change request process by:

  • Creating standardized change request documents
  • Establishing change management meetings
  • Creating Service Level Agreements (SLAs), which include change request lead and turnaround times.

Standardized Change Request Documents

Standardized request documents help to increase the quality of the change request process. The forms are sent to the support unit by the application owner of the data to be processed. The forms can be created using a variety of different products. The mechanisms can range from sophisticated ticketing systems using web forms to simple paper documents. As I said, use what’s available. It’s not the mechanism; it’s the process that is important.

Any requests not sent or signed off by the application owner should be rejected. Keep copies of all completed work requests for auditing purposes. Application owners can be virtually any member of the organization that is identified as having the authority to sign off on change requests. The most common persons are application development team leaders, section heads, department heads, etc.. At RDX, if you aren’t identified as a “Change Data Agent” for your organization, we won’t process the ticket.

Each request form contains the following common information:

  • Form identifier – naming convention that allows the form to be easily identified
  • Application name
  • Server name (for OS requests)
  • Database name (for DB requests)
  • Name and contact information of the person requesting the change
  • Request date
  • Required date (including specific time change needs to be implemented)
  • Application owner signoff
  • Data security signoff (if required by shop standards)
  • Schema Change Area
    • Schema owner of object to be changed or created
    • Object name to be changed or created
    • Object type (i.e. table, index, view) of the object to be changed or created
    • Detailed description of change requested
    • Data administratiosign off for new data objects
  • A free form request area that further describes the change. Also provides an area for non-schema changes
  • Verification procedures – other units required to verify as well as verification procedures
  • Notification procedures – who to notify when the change is complete
  • An area that the technician will complete when executing the change that contains the following information:
    • Technician executing change
    • Technician contact information
    • Date and time change was processed
    • Verification procedures followed
    • Notification procedures followed

    Here are a few examples of specific forms that will help formalize the change request process:

    Database and OS Authorization Request Form

    This form is used for requesting authorization changes to the database and/or operating system environment.

    The Database and Operating System Authorization Request Form will include all of the requestor information contained in the previous form but will also record information pertinent to authorization requests:

    • Grantee listing for security grant or revoke
    • Type of security granted or revoked

    Production Environment Change Request Form

    This form will be used for requesting the migration of database objects (databases, table spaces, tables, indexes, etc.) from test to production and the alteration of existing production objects. In addition, the form notifies the support team to perform various database, operating system and hardware parameter and environment changes in production environments.

    Each production environment change request form must have an associated test environment change request counterpart. If the change wasn’t made in test, you don’t implement it in production. To facilitate this process, the identifier for the test change request that was used to notify the support team should be provided on the production change request form.

    The production environment change request form contains the following information pertinent to production environments:

    • Test Environment Change Request Identifier- allows technician to determine if the change was implemented in test. If no change request is found, the person tasked with implementing the request needs to determine the reason why.
    • Form identifier – naming convention that allows the form to be easily identified
    • Application name
    • Server name (for OS requests)
    • Database name (for DB requests)
    • Name and contact information of the person requesting the production change
    • Request date
    • Required date (including specific time change needs to be implemented)
    • Application owner signoff
    • Data security signoff (if required by shop standards)
    • Schema Change Area
      • Schema owner of object to be migrated or altered in production
      • Object name to be altered or migrated
      • Object type (i.e. table, index, view) of the object to be altered or migrated
      • Detailed description of change requested
      • Data administration sign off (if required by shop standards)
    • A free form request area that further describes the change. Also provides an area for non-schema changes
    • Verification procedures – Other units required to verify as well as verification procedures
    • Back off procedures – What to do it the change has an adverse effect on the system or does not work “as initially thought”
    • Notification procedures – who to notify when the change is complete
    • An area that the technician will complete when executing the change that contains the following information:
      • Technician executing change
      • Technician contact information
      • Date and time change was processed
      • Verification procedures followed
      • Notification procedures followed

    Change Management Meetings

    If you read my earlier blog post on database design review meetings, you know I’m a proponent of constant communication between all units that are involved in the change management process. How often should you hold these change management meetings? You should hold them as often as you implement objects in production. If your organization makes changes to production environments on a daily basis, the meetings should be held daily. This is not as big of an imposition on your time as you may think. We provide remote database services for several very large organizations that have these change management meetings on a daily basis. The process takes about 15 to 20 minutes, not a lot of time spent to ensure that everyone knows what is happening.

    To shorten the amount of time these meetings consume and to make them as productive as possible, the following discussion items should be a standard part of the meeting’s agenda:

    • Application name being changed
    • Date and time change will be implemented
    • Change description
    • Potential business impact if the changes don’t go as expected (include both units affected and how they will be affected)
    • Verification procedures
    • Back-off procedures
    • Requestor
    • Tested by

    Service Level Agreements

    Identifying support needs and expectations is required to provide high quality support. You probably won’t be meeting all of your customers’ expectations if you don’t know what any of them are. As stated previously, each application has its own unique set of support requirements and expectations. Service Level Agreements (SLAs) help to solidify support requirements and dispel any inflated expectations a business or application development unit may have. They probably won’t be aware of your current workload and resulting work request lead times until you tell them. The support team lead should meet with each unit supported to establish a set of measurable Service Level Agreements that include work request lead times, support activities required and application performance and availability objectives.

    Wrapup

    This is by no means an all-inclusive list of activities you need to perform to standardize the change request process. It is intended to give you a head start in the right direction.

    DBA Best Practices