The database engine is taking on a much more strategic role in most organizations. It provides the mechanism to store physical data along with business rules and executable business logic. The entire application environment including data storage, business rule enforcement, application program storage, communication and system monitoring are now being controlled by the database. Over time, the database engine will store more information related to the understanding of the business, the meaning of the data stored (metadata) and the mechanisms to secure, control and track versions of data objects, access programs and related software.
The database’s area of influence has expanded to a point where it has become the heart of the modern IT infrastructure. Enterprises expect the DBMS to be reliable, secure and available. The responsibility of supporting these critical data stores falls to the Database Administrator or DBA for short.
Database Administrators are responsible for designing, supporting and safeguarding an enterprise’s database data stores. The DBA is the information technician responsible for ensuring the ongoing operational functionality and efficiency of an organization’s databases and the applications that access those databases.
As a result, the DBA plays a pivotal role in the organization. The DBA is often seen as the “GOTO” technician because of their traditionally strong problem solving skills. In addition, the DBA is seen as the IT staff’s technical generalist because of the working knowledge they have in many different facets of information technology. A day in the life of a DBA is usually quite hectic. The DBA maintains production and test environments, monitors active application development projects, attends strategy and design meetings, selects and evaluates new products, and connects legacy systems to the Web.
Because of this, database administrators have become a sought after commodity. This is reflected in the salaries that they command. According to a recent Data Masters Salary Survey, senior DBAs average $85K per year, while managers can command as much as $135K. Junior DBAs average a starting salary of $50K. Meta Group says that the total cost of hiring and maintaining a DBA is between $150,000 and $160,000 per year.
Information technology executives are finding it increasingly difficult to successfully achieve core database management objectives. Today’s database administration units are operating with smaller staffs, tighter budgets and ever-increasing workloads.
The environment a DBA works in is also changing. Multiple database vendor environments, more complex database technologies, increasing uptime requirements and strategic initiatives are combining to challenge traditional database management strategies. In addition, more than ever before, databases are being placed on a wide range of different platforms, from mainframes to PDAs.
All of these trends and new technologies further complicate database management and impact the jobs of database administrators. Because of these issues, many IT executives are evaluating outsourcing and supplemental services as possible solutions to their database management challenges. Outsourcing database administration to a remote database service provider allows companies to receive high quality support for their databases at a reasonably low cost.
Remote DBA service providers specialize in delivering a complete set of database administration services via the Internet. Remote database administrators provide all the functions of an on-site DBA including maintenance, monitoring, trouble-shooting, backup, recovery and database tuning.
The remote database services model allows them to share resources among the different clients they are contracted to support. The vendors are able to gain efficiencies by economies of scale and standardizing and processing the day-to-day administrative functions.
The two main models for remote administration services are:
Complete database services outsourcing – The entire database support function is outsourced to the remote DBA services vendor. The vendor acts as the customer’s database administration team and supplies all of the administrative activities required to maintain existing databases and design and create new database environments.
Supplemental database services – The remote DBA services vendor provides auxiliary support services to the customer’s internal database administration unit. The customer outsources a subset of administration functions to the remote DBA services vendor. The flexibility of this model allows the customer to choose the set of services as well as the databases supported.
The customer, in effect, tailors the support agreement to match their current and future needs. Companies that have hiring freezes, high workloads or inexperienced DBAs find this to be an attractive alternative to hiring additional in-house DBAs. Supplemental services allow the customer to achieve a high-level of support while freeing internal resources for higher priority and higher ROI activities.
Technological advances in security, internal networking and external communications have allowed the remote DBA services model to become a viable alternative to traditional support structures. No longer are remote DBA services vendors hampered by low-speed communications and complex security implementations. Advances in teleconferencing and voice communications allows remote DBA services vendors to participate in strategy and application design meetings enabling them to provide the same level of support as their in-house counterparts.
Corporate information technology executives understand that their success relies upon their ability to cut costs and improve efficiency. Decreasing profit margins and increased competition in their market segment force them to continuously search for creative new solutions to reduce the cost of the service they provide. They also realize that this reduction in cost must not come at the expense of the quality of services their organization delivers.
Rapid advances in database technology combined with relatively high database licensing and support costs compel IT executives to ensure that their support organization fully utilizes the database product’s entire inherent feature set.
The more solutions the database inherently provides, the more cost effective it becomes. These integrated features allow technical personnel to solve business problems without the additional costs of writing custom code and/or integrating multiple vendor solutions. The issue becomes one of database complexity, as database vendors incorporate new features into the database, the more complex it becomes to administer. Modern database administrators require a high-level of training to be able to effectively administer the environments they support. Without adequate training, problems are commonplace and the database’s inherent features are not fully utilized. The IT executive faces a challenging task – increasing the database’s corporate value while decreasing the costs required supporting it.
Remote database vendors usually charge a set monthly service fee that is, on average, 30-50% less than the costs of an internal DBA. Companies employing remote database services vendors do not have to pay the additional overhead for benefits, training, vacation, sick-days and employee management/HR costs that are required to support in-house personnel.
However, companies considering database outsourcing often overlook the reduction in environmental costs that can be realized by outsourced support activities to remote providers. Remote providers do not require floor space, computers, networks, heat, lights, water and the associated support activities of network technicians, PC support personnel, electricians, etc.
Database administrators, by the very essence of their job descriptions, are the protectors of their organization’s core data assets. They are tasked with ensuring key data stores are continuously available. However, ensuring that data is available on a 24 x 7 basis is a wonderfully complex task. Hardware failures, software failures, user errors and disasters all combine to make many administrators lie awake nights thinking about whether their databases will be continuously available. When a mission-critical application becomes unavailable, it can threaten the survivability of the organization. The financial impact of downtime is not the only issue that faces companies that have critical applications that are offline. Loss of customer goodwill, bad press, idle employees and legal penalties (lawsuits, fines, etc.) must also be considered. It is up to the database administrator to recommend and implement technical solutions that deal with these unforeseen “technology disruptions.” Modern database systems are gaining features to help avoid downtime altogether by enabling change, optimization and management tasks to be performed while databases remain online. It is up to the IT support organization to understand and implement these features to reduce the amount of downtime required to perform administrative activities.
Industry analysts estimate that as much as 80 percent of application failures can be attributed to software failures and human error. Downtime causes can range from improperly entered transactions to scheduled outages for required database maintenance and tuning.
Remote DBA services vendors understand that their customers will hold them to a higher standard than they will their own in-house support personnel. Remote database services vendors know that advanced database education, certifications and experience are not competitive advantages; they are competitive requirements for survival. They must be able to provide support to their customers to a higher-quality than if it were provided by in-house personnel.
Daily support requests and problem solving often overwhelm DBA groups. Understaffing, over-commitment to supporting new and legacy application developments, lack of repeatable processes and lack of budget are the most common causes of DBAs being reactive instead of proactive. The reactive DBA can be compared to a firefighter; they resolve problems only after the problem occurs. The biggest problems garner the most attention. The reactive DBAs time is dominated by these by firefighting activities, reducing the amount of time they are able to spend implementing the process and procedures required to switch their mode from reactive firefighter to proactive problem prevention.
RDX’s remote DBA services team has implemented practices and procedures to avoid problems before they occur. The team dedicates personnel who are responsible for creating, implementing and enhancing a strategic blueprint for the proactive monitoring and trouble-shooting required to prevent database problems before they occur. Since the proactive support blueprint is already in place, companies employing RDX to support their databases do not have to spend the additional monies required to create a proactive support environment.
As we learned previously, experienced database administrators are able to identify and prevent potential problems before they occur. To ensure continuous 24x7 availability for all application components, RDX has selected SolarWinds as the foundation of our monitoring architecture. RDX has custom designed a wide array of SolarWinds monitoring components that allow us to monitor every member of the application architecture including middle-tier, application, database and non-database servers.
RDX dedicates an entire team that is responsible for creating, implementing and enhancing a strategic blueprint for the proactive monitoring and trouble-shooting required to address common issues affecting system availability. Since the support blueprint is already in place, companies employing RDX do not have to spend the additional monies required to create and maintain their own in-house monitoring architecture. The DOC team is much more than a group of help desk techs with a limited knowledge of the systems they support. The DOC team consists of dedicated professionals who are trained in monitoring, database and OS problem resolution and alert reduction.
Documenting processes, procedures and best practices is a task that is often considered to be boring and mundane. Most DBAs would rather perform virtually any other activity than sit in front of a screen using a word processor. As a result, creating documentation is often postponed until the DBA has a little free time to kill. Today’s database administration units are operating with smaller staffs, tighter budgets and ever-increasing workloads. The end result is that the documentation is either never created or created and not kept current.
But a robust detailed documentation library creates an environment that is less complex, less error-prone, reduces the amount of time DBAs spend learning new database environments and reduces the overall time spent on day-to-day support activities. DBAs are able to spend more time administering the environment than finding the objects they are trying to support.
Repetition, even though it can be boring, is the foundation for a high quality support environment. If the scripts and administrative process worked correctly the first time, chances are they will continue to work correctly in the future.
RDX has an extensive library of documented processes and procedures for all daily administrative activities. In addition, senior support personnel work with each customer to document customer specific complex administrative activities. Documenting complex administrative processes such as production to decision support database refreshes and application upgrade activities will allow future iterations of these activities to be executed more quickly and with less errors. New personnel are quickly indoctrinated in RDX's structured approach to database support and all administrators are evaluated on how well they adhere to the predefined processes and procedures.
Time is dedicated weekly to discuss support procedures and processes. All support DBAs participate in these discussions ensuring that a continuing stream of measurement data flows in which is immediately used to further improve the existing processes.
In order for a remote database services vendor to successfully integrate itself into the customer’s support structure, it must adhere to all of the customer’s internal support processes, procedures and standards. RDX's technicians are able to use all internal support toolsets that the customer desires including administrative tools, monitoring tools and trouble ticket management systems.
RDX will also adhere to all of the customer’s internal support processes, procedures and standards. RDX support services team member(s) will be appointed to be the customer’s primary DBA. The primary DBA will not be the sole DBA working on the customer’s account. The primary DBA’s responsibility is to learn the customer’s support requirements, organizational support infrastructure and support philosophies. The primary DBA is then responsible for educating fellow team members on the customer’s environment.
Successful database administration units understand that providing better support to their customers not only comes from advances in technology but also from innovations in human behavior. The selection of support-related technologies is important but it is the effective implementation and administration of those technologies that is critical to organizational success.
The database environment has become so complex that it precludes database administrators from becoming experts in all facets of the database’s technology. RDX’s large administrative staff allows it increase efficiency by creating specialists in key database disciplines. In addition to expertise in providing day-to-day support, each of RDX’s support staff members is required to become an expert in one or more database disciplines including backup and recovery, highly available architectures, SQL tuning and database performance, database monitoring, UNIX/Windows scripting and database security.
RDX allocates the support person with the highest-level skill sets in that particular task to provide the service requested by the customer. This methodology ensures that the customer gets the most experienced person available to perform the given task.
This large staff also reduces the amount of time spent on troubleshooting and problem solving. RDX is able to leverage the expertise of a couple of dozen database administrators. The support staff averages 10+ years of experience and consists of certified DBAs and database instructors. RDX is able to leverage the team’s expertise to provide faster resolution to database performance issues and outages.
Since the support staff works with many different companies, they have seen a number of different approaches to most situations. RDX also maintains close relationships with all database vendors and is a certified partner with IBM, Microsoft and Oracle.
Poorly maintained databases mitigate the company’s ability to drive higher revenue because of high maintenance and hardware costs. An improperly configured database is problematic, performs poorly and wastes finite system resources. Because DBA responsibilities are dynamic and varied in nature, an in-depth review of every production database on a regular basis is often given less priority than it deserves. As a result, the databases continue to perform poorly and hardware upgrades are performed before they are truly required.
Performing standardized and detailed database assessments on a regular basis increases database performance, reduces the number of problem occurrences and promotes efficient utilization of hardware resources. The assessment is the key to a high-quality database environment. The general purpose of the assessment is to evaluate the overall health of the database environment. This can only be accomplished by executing an in-depth, critical analysis of the database being analyzed. For any database assessment to be complete, it must focus on four key database disciplines that are critical to the success of any database implementation:.
Backup and recovery – Data can be corrupted in a variety of ways: statement failure, user failure, process failure, media failure, instance failure and application error. To develop an effective backup strategy, administrators must consider the various possibilities of data corruption. Remote DBA Experts’ analysts identify business requirements and compare them to the overall backup and recovery strategy. The physical database architecture is assessed to determine if it facilitates (or hinders) the database backup and recovery process. Backup and recovery processes and procedures are also evaluated. In addition, backup monitoring processes and procedures are also reviewed.
Availability – One of the major responsibilities of a database administrator is to keep the database available for use. It is imperative that the database’s architecture (files, memory structures, system storage objects) and application objects (tables, indexes, stored programs) are designed and implemented correctly. In addition these objects must be administered and monitored to reduce the database’s MTBF (Mean Time between Failures) and MTTR (Mean Time to Recover) cycles.
Performance – One of the benefits (and to some, a drawback) of a database environment is its “tunability.” The database architecture (file layout, memory structures, system storage objects, database engine settings) and application objects are evaluated both individually and how they interact with each other as a group. Monitoring practices of key performance indicators (a must for maintaining optimal performance) are also assessed.
Security – Database security entails allowing or disallowing user actions on the database and the objects within it. Modern databases use schemas and security domains to control access to data and to restrict the use of various database resources.
RDX's security review evaluates user definitions and privilege grants to determine if the data is adequately safeguarded against unauthorized access.
Support technicians perform assessments on all database environments it is contracted to administer. Databases under RDX's control are also assessed on a regular basis to adjust each database environment to changes in application utilization.
A highly competitive market arena forces all database vendors to accelerate the release of new products as well as enhancements to existing products. Database vendors know that new features and functionalities are not a requirement for competitive advantage; they are a requirement for competitive survival. As are result, each new database release contains dozens of new features and functionalities. The fastest database title moves from vendor to vendor due to strong vendor competition and fast paced advancements in database performance “tuning knobs,” hardware and operating systems.
As the number of features contained in the database release grows, so does the importance of taking advantage of those new features. Database product licensing and vendor maintenance contracts command a premium price in the marketplace. To gain the most return on their database investment, companies must ensure that they are completely leveraging the benefits of the particular database being used. Understanding all of the inherent database features is important but selecting the features that bring each, individual application the most benefit is the key to success. RDX’s close ties with the database vendors allow it to have access to information on new database releases before it becomes public. All support technicians spend dedicated time learning the benefits of each new feature announced in all new database releases. Quarterly presentations are provided by senior level instructors to all customers. These presentations highlight future database directions, including new database products, new features being added to existing products and upcoming sales and marketing strategies for each database vendor.
Historically tracking the various performance indicators in the system allows the RDX DBA team to trend database performance. Historical tracking allows the team to track and forecast usage spikes. This allows RDX’s technicians to forecast when performance will be good and provide a possible reason when performance degrades. The IT staff’s knowledge of performance trending information allows them to better understand why the performance degrades during a particular time period and prevents them from trying to solve the same performance problem on a regular basis.
Key hardware usage indicators are also tracked historically. Memory, disk and CPU usage are recorded daily and stored in historical tracking tables. Tracking database resource consumption historically allows RDX to help customers predict and budget for required hardware upgrades months in advance. This provides the customer with the ability to budget for the hardware upgrades instead of having to react when application performance begins to suffer due to a lack of free hardware server resources.
Effective measurements are required to judge the success of any activity. Remote DBA vendors probably won’t meet all their customer’s expectations if they don’t have a firm understanding of what they are. Each application has its own unique set of support requirements and expectations. Service Level Agreements (SLAs) help to solidify support requirements and prevent vendor-customer miscommunications from occurring.
The quality of support the remote DBA service team provides also needs to be reviewed on a regular basis. A services vendor report card allows customer business and application development units to provide feedback on DBA support activities. The report card differs from the Services Level Agreement, which is a set of contractual promises made by the vendor to meet or surpass specific goals and levels of service. The report card is much more granular in nature. The intent of the report card is to identify what each individual team member expects from the remote support provider’s staff. These expectations are then compiled to create the vendor’s report card on how they will measure the success of the relationship. The following tips should help to create the vendor scorecard:
Report cards should be used to gather and report information on a periodic basis. There is a direct correlation between the amount of care and nurture put into a relationship, and the overall success of that relationship. If the customer does not provide feedback, services providers won’t know how well, or how poorly, they are performing their assigned tasks. Timely feedback is required to allow services providers to quickly adapt and adjust.
The intent of this document was to “peel back the veil” of remote DBA services and take an in-depth look at how remote services vendors are able to provide a high level of service at a reasonable cost. Outsourcing databases should not be seen as a last resort to cut costs; it should be viewed as a strategic and viable alternative to supplying these services by in-house personnel.