Automation Framework Architecture for Enterprise Products: Design and Development Strategy

Introduction

This article presents the case for using an automation framework for implementing and testing ERP product suites. The structured design and development methodology described in this paper evolved through the experience of the Oracle Fusion Financials product quality assurance team and has been proven to bear significant benefits over time. The efficiency gains that are realized by using an automation framework translate into enhanced product performance at significantly lower cost. Special attention is required during automation framework design, since any presumptuous move at this stage could lead to dire consequences in terms of escalated costs, lost time, and inferior product quality.

This article provides information about the following tasks of the automation framework development process:

· Provide a strategy for design and development of an efficient automation framework in an iterative incremental development model used for large ERP applications

· Address issues surrounding framework compatibility and productivity with respect to certifications and implementations on multiple technology stacks

· Provide a fit-for-purpose automation solution that is neither too costly nor too scanty in functionality

· Focus on the scalability of the framework, leading to higher efficiency in activities related to enhancing its functional footprint

· Use modularity to assist maintenance and ensure consistent performance in a scenario where the application under test is constantly evolving

· Briefly outline the framework design methodology, customized to needs and maturity of the organization implementing the solution

· Identify key activities that help ensure the project stays on track and minimize the probability of loss on investment

Some of the tenets above were used in the architecture of the PSR3 (portable, scalable, rerunable, retestable, and reliable) automation framework used for quality assurance and implementation of Oracle Fusion Financials Applications. The benefits realized by using this automation framework are described in the Conclusion section.

Automation Frameworks

What Is an Automation Framework?

An automation framework is a platform developed by integrating various hardware and software resources and various tools and services based on a qualified set of set of assumptions. It enables efficient design and development of automated test scripts and reliable analysis of issues for the system under test.

What Is the Purpose of an Automation Framework?

An automation framework is primarily designed to do the following:

1. Enhance efficiency during the design and development of automated test scripts by enabling the reuse of components or code

2. Provide a structured development methodology to ensure uniformity of design across multiple test scripts to reduce dependency on individual test-case developers

3. Provide reliable issue detection and efficient root-cause analysis with minimum human intervention for the system under test

4. Reduce dependence on subject matter experts by automatically selecting the test to execute according to test scenarios and dynamically refining the test scope according to changes in the test strategy or conditions of the system under test

5. Improve the utilization of various resources in each cycle to enable maximum returns on effort and also ensure an uninterrupted automated testing process with minimum human intervention

The Automation Framework Design Challenge: Balance Quality, Time, and Resources

The challenge is to build a fit-for-purpose automation framework that is capable of keeping up with quickly changing automation testing technologies and changes in the system under test. The challenge is accentuated by the various combinations that are possible using the wide gamut of available automation tools. Making the right choices in the preliminary design stage is the most critical step of the process, since this can be the differentiator between a successful framework and failed investment.

As if this were not tough enough, add to this the even more formidable challenge of balancing the quality of the framework against the desired utility and the need to develop the framework within a stipulated timeframe using available resources to ensure the economic viability of the solution. Therefore, it is very important to benchmark the framework, the associated development time, and the required resources to ensure the framework's quality justifies the use of the framework.

Figure 1: Framework Lifespan

Broad Types of Automation Framework

Automation frameworks can be classified according to four broad types, each of which has unique advantages and disadvantages, as shown in Table 1. The choice of a particular type depends upon a wide range of external factors.

Factor	Linear/Structured Test	Data Driven	Keyword Driven	Hybrid
Objective	This consists of procedural code or a set of instructions	Data is persisted outside the test	Code is bound with keywords that are then used to implement the desired tests	This is a combination that chooses the strengths of other framework types and mitigates their weaknesses
Approach	The approach is logical with a view of making the task at hand work	Data is the key and everything is designed to accomplish the desired task with a simple change of data	Most of the common tasks are organized into basic keywords that are bound with associated code; complex tasks are achieved by combining multiple keywords	The approach could depend on what needs to be accomplished and also on the imagination of the architect
People skill requirement	Does not require advanced programming knowledge; simple logical skills are sufficient	Might require intermediate-level programming skills coupled with logical skills	Might require intermediate-level programming skills coupled with logical skills	Might require advanced programming knowledge, which might be used to make the framework more generic and adaptable
Complexity	Low	Medium	Medium	High
Framework development time comparison	Might require less time to develop the framework for individual cases	Requires a moderate amount of time	Requires a moderate amount of time	Might require significantly more time
Individual-test development time	Higher	Lower once the base script is developed	Lower once the base script is developed	Lower once the base script is developed
Individual-test development time	Higher	Lower once the base script is developed	Lower once the base script is developed	Lower once the base script is developed
Flexibility with respect to changes in the system under test and technology	Low	Medium	Medium	High
Advantages	Simplicity and low cost	Moderate complexity and flexibility	Moderate complexity and flexibility	High flexibility with boundless possibilities
Disadvantages	Low flexibility	Flexibility only in terms of data	Flexibility only in terms of components	Might become very complex over time and might require specialized support
Ideal usage scenario	Automation of simple repetitive tasks, which are expected to remain relatively unchanged over time	Automation of repetitive tasks for which only the data is expected to change over time with little or no effect on the technology or the system under test	Automation of repetitive tasks for which significant change is expected in terms of the system under test, but data remains relatively constant	Automation of repetitive tasks for which significant change is expected over time in terms of technology, the system under test, or data; most suitable for ERP applications
Example development tools	Any record-and-play tool, for example, Selenium, Quick Test Professional (QTP), or Oracle Applications Testing Suite (OATS)	Any recording tool that has proficiency in scripting languages such as VBScript or JavaScript and provides external data store integration	Any recording tool that has proficiency in scripting languages such as VBScript or JavaScript	Might require custom component development and programming language support

Figure 2 shows a framework comparison matrix that is based on data from a random sample.

Figure 2: Framework Comparison

Design Strategy for Enterprise Products

Enterprise products are large, complex, and ever-evolving software products that, of late, have resolved to an iterative incremental-growth model. To keep up with the growth rate of an enterprise product, the corresponding testing framework should broadly conform to a design that has the following characteristics:

· Complete: The framework and its assets should allow complete end-to-end testing of the product suite, starting from initial setup and preparation of the environment to various specialized subprocesses and critical one-time tasks.

· Open: As much as possible, the framework should be based on open technologies, such as Java, to limit the development and maintenance costs. Openness might also allow cross-tool execution when a suitable API exits or can be developed.

· Integrated: The framework should allow easy integration with a maximum number of tools and components in the periphery of the product suite.

Figure 3: Optimum Framework Characteristics

Framework Features

Figure 4 and the following feature lists describe the framework features.

Must-have features:

Portability: The framework must be easily deployable in any environment and technology stack for regression testing and certification testing of the product suite.
Scalability: When expansion is required, the framework must allow the addition of multiple scripts and assets, per the testing requirements, through the organized structuring of resources. This need is inevitable in the case of ERP products, which undergo multiple releases for new builds or require different scripts for parallel testing or testing on multiple technology stacks.
Reliability: The framework must ensure that the test results are an accurate depiction of system conditions at the time of testing. It also must ensure that automated testing is carried out by efficiently utilizing system resources.
Retestability: This feature ensures that regression tests can be submitted multiple times without any change to the data or any other component associated with the tests. It also ensures that the functionality being tested is thoroughly verified.
Rerunability: The framework must provide the ability to resubmit the core setup to the same environment, and each time this happens, the framework must ensure that a new setup is created that is used for subsequent testing. All this should happen without any manual change of data in individual scripts.
Remote execution: Provision must be made to create and submit automation jobs, which then must be scheduled automatically according to available resources and preset parameters.
Load balancing: All scheduled jobs must be executed on the first-available machine, thereby making optimum use of the system. The executions should be based on business rules.
Parallel execution in the same environment: Automated scripts conforming to the framework must be able to run simultaneously even though they form part of multiple test cycles in the same environment.
Parallel execution in different environments: Automated scripts conforming to the framework must be able to run simultaneously in different environments.
Script development efficiency: The development time for scripts conforming to the automation framework should not be more than eight man-hours per script for scripts of high complexity.
Script versioning: Versioning is essential to ensure that each modification to a script is accounted for, and versioning must be distinct to enable tracing the evolution of a script to its most recent form.
Addition of functionality: Flexibility should be provided-without limitations posed by the framework-for adding new functionality as requirements evolve.
Maintenance: Changes in functionality should be carried out efficiently and centrally with effects persisting down to lowest level automatically.

Should-have features:

Run-history tracking mechanism: In case of a failure in the prior run, the framework should keep a run history of each test for each run on each instance to ensure that the framework picks up the effort for that instance where it left off. This is an especially important feature for tests with a large set of data, such as "chart of accounts" in financial applications.
Disaster recovery: In event of catastrophic failure, the framework should automatically determine the recovery mechanism and react appropriately.

Could-have features:

· Role-based security: This type of security should allow access to various features and setups of the framework according to the authorization level that is used to log in to the system.

· Multiple Application Under Test (AUT) support: This type of support ensures that the same framework can be reused for testing various applications with minimal changes.

· Multiple tool support: This support must allow the development and execution of automated regression tests on multiple automation tool platforms.

Figure 4: High-Level System Architecture

The Next Steps

Figure 5, Figure 6, and the following steps describe the framework design process and the framework development process.

1. Current State of the Architecture: The focal point of this step is to identify the current business practices with respect to automation in the target organization. Benchmarking the current state is an important first step in the process of clearly identifying the benefits offered by implementing an automation framework. This step demands that adequate attention be paid to various activities, such as the following:

Assessing customer maturity, primarily in terms of people, technology, and processes
Identifying current automation architecture details
Assessing end-to-end techniques with a quantized transaction time matrix
Putting best practices in place to accomplish the planned effort

Future State of the Architecture: This step identifies the automation framework in an "ideal world" scenario. The following tasks are performed in a top-down manner:

Identifying candidates for improvement in the current-state architecture
Identifying additional components to be added by the automation framework
Identifying tasks addressed by each component, the general behavior of the components, and the limitations of each component
Mapping how task achieved in the current-state architecture will be addressed in the new architecture
Identifying and removing redundancies that exist in the current-state architecture

f. At this stage, comparisons among the various available automation tools, programming languages, and technology stacks that will be used to implement the future-state architecture might also be performed, subject to the target organization's needs.

Gap Analysis: This is the process in which the current-state architecture and the automation framework are compared to identify and quantize the advantages offered by using the developed framework. Multiple tools are available to accomplish this task, including the use of a traceability matrix that identifies the vision, the goals that are required to reach the vision, quantifiable objectives that can help measure the performance of the legacy framework, and advantages offered by the developed automation framework for attaining these goals.
Business Case: In this step, a detailed business case is prepared for the use of the developed automation framework over the legacy automation framework. The metrics for the gaps that were identified between the current-state and future-state architectures are captured and compared at a decided frequency. Costs of development of the framework in terms of capital expenditure and operational expenditure might be worked out for each component of the automation framework. Information about the development time and the complexity of components might also be presented.

The focus is on the new capabilities added to the existing infrastructure by using the developed automation framework. Enhancements in efficiency that might be realized over time are projected and they might also be benchmarked against industry standards. Benefits might be categorized mainly as expense savings, labor productivity savings, alleviation of pain areas, and so on.

Figure 5: Framework Design Process

Requirements Analysis: Based on the inputs provided by the gap analysis, custom components to be included in the automation framework are identified and elaborate requirements for each component are defined. A functional prototype of the system can be presented, if required.

6. Work Plan: A detailed work plan needs to be prepared for each component included in the requirements analysis. This can be achieved in multiple ways, but significant common steps are as follows:

Identify the key activities required to build the component and break them down into appropriate tasks and subtasks.
Prepare task-level estimates for each task using any standard available model.
To identify the dependencies, sequence the activities and tasks.
Ensure level resources are assigned to each task.

Risk Management: This is a method for proactively identifying, tracking, and measuring risks and handling them as each situation occurs. The following are the main steps in this process:

Create a risk "watch list" that comprises situations that might cause issues during the development of the automation framework.
Quantify the risks in terms of high, medium, and low risks to assist in prioritizing risk responses.
Quantify the impact of the risks on the development of the automation framework.
Determine the risk response plan that should be executed when each identified risk occurs.

Cost Estimate: This is the estimated cost of building the framework so that it has the features identified in the requirements analysis. This process is initiated with the level of information available at the time of initiation and is refined progressively as more information is made available. This step is performed in conjunction with the output of the risk management step and the work plan step. Multiple models are available for cost estimates including "Expert Judgment."

Balance Time, Resources, and Cost: It is important to fine-tune the mix of time, resources, and cost in just the right proportion depending on the needs of the target organization. Scenarios might exist where time is of the essence and cost is capped. In such cases, requirements and development have to be prioritized. In other cases, cost might be fixed and time and resources can vary to develop the automation framework within stipulated costs. This step is done with inputs from the previous stages and might be updated on demand.

Figure 6: Framework Development Process

Conclusion

The design and development approach presented in this paper provides a consolidated and comprehensive schema for designing and developing automation frameworks. The PSR3 application implementation and testing framework that was designed and developed using this approach resulted in significant gains for various parameters shown in Table 2, which formed the part of certification challenge.

Parameter	PSR3 Automation Framework	Legacy Framework	Manual Certification
Execution of over 1000 regression tests per day	5 hours	8 hours	Not achievable
Creation of new environments from scratch	Completely automated and created in 6 hours	Not achievable	32 man-hours
Average script development and stabilization time	3 hours for simple script 6 hours for medium-complexity script 8 hours for complex script (new component or framework service)	8 hours for simple script 12 hours medium-complexity script 16 hours for complex script	Not applicable
Reconfiguration for setting up a new organization with the same environment	Due to automatic reconfiguration and with parallel execution, any number of environments could be created in same time required for setting up one environment (that is, 6 hours).	24 man-hours	Re-execution of all setups for the creation of a new environment requires 32 man-hours.
Parallel execution in multiple environments with the same technology stack and same application version	Parallel execution for any number of environments can be done in the same time requires for one environment	Parallel execution of any number of environments can be done in the same time required for one environment	Re-execution of all scripts in multiple environments is significantly labor-intensive
Ability to run the same tests on two different application versions	Possible in parallel	Not achievable	Requires re-execution of all scripts in multiple versions
Ability to run the same tests on two different technology stacks	Possible in parallel	Not achievable	Requires re-execution of all scripts in multiple versions
24x7 automated job scheduling	Achievable	Not achievable	Not achievable
Ability to re-execute tests and transactions with changed data	Automatic, though this can also be done manually once the system takes over	Data needs to be modified and synchronized between scripts before each run	Requires re-execution of all scripts with different data
Change in application functionality or objects	Can be handled centrally (2 hour average fix time)	Affected components need to be traced and appropriate changes need to be carried out individually (50 hours approximately)	Not applicable
Quality and quantity of regressions identified	Highest	Less	Least (quality is moderate; quantity much less)
Framework development and stabilization time	Highest	Less	Not applicable
Cost	Highest	Less	Not applicable

Automation framework design and development is a process that requires structured and calibrated attention in various stages to attain the desired benefits. This process is the result of a marriage between cutting-edge technology and business advantage, and it requires constant evolution to maintain significance in the industry.

If properly designed and developed, an automation framework can be a process enabler and an efficiency multiplier for ERP product suites-which are some of the most comprehensive software solutions in the industry-thereby significantly improving the ROI for organizations that implement ERP products. Process and quantization is the key for a successful framework, and their absence can cause a significant loss of time, effort, and investment.