Agile on the Move with APEX in a Strongly Regulated Environment

By Hansjörg Grässlin, Christophe Girardey and Dr. Christian Wattinger

 

Many industries have strict regulations (automotive industry, pharmaceuticals, air traffic control, banks, etc.) regarding the implementation of IT projects, and there is a trend towards more and stricter regulations in many places. At first glance, APEX with its focus on agile and rapid development approaches (e.g. low code, rapid application development) does not appear to be suitable per se for solutions in a rigid, regulated environment. Using the example of an APEX implementation in a strictly validated environment in the pharmaceutical industry in the field of clinical development, we show how these inherent problems can be successfully overcome.

 

In this article, we would like to use the example of a successful project that we carried out in a large corporation in the pharmaceutical industry to show how the concepts of "agile" and "strictly regulated", which seem contradictory at first glance, can be brought together with APEX. To do this, we will first describe the two opposing concepts before presenting the solution approach.

 

Validation is...

...according to the American Food and Drug Administration (FDA):

"The establishment of documented evidence that provides a high level of assurance that a particular process produces a product that meets its predetermined specifications and quality characteristics."

In the pharmaceutical industry, the regulatory requirements are referred to as validation and are specified by the relevant authorities in the various countries.

 

 

In Table 1 lists some relevant examples: In this context, a distinction is made between system and software. This is important because a validation always refers to the entire system, in which the software is only one - albeit important - part. However, the system also includes the procedures and people involved (see Figure 1).

 

 

The central guideline for validation is GAMP 5 [2] (Good Automated Manufacturing Practice), which describes a risk-based approach to validating systems. GAMP divides software into the categories shown in Table 2 categories listed in Table 2, with the validation effort increasing with the category. In the pharmaceutical sector, it is indicated and mandatory to validate an IT system in the following cases if the system concerns one of the following topics:

• Safety of persons, especially patients
• Product quality
• Data integrity and thus ultimately trust in the data (important, for example, in clinical studies for the approval of a drug)

 

 

In all of these cases, validation should ensure that sources of error are minimized and the reliability of the processes is maximized. Furthermore, any process deviations that occur must be clearly recognizable and traceable. In the validated environment of the pharmaceutical industry, software developments are usually carried out according to the V-model. This is shown in Figure 2 illustrated.

 

 

Starting at the top left, the following sequential process is run through:

• Creation and acceptance of a validation plan (VP) that describes how the system is to be validated.
• Creation and approval of the User Requirements Specification (URS), which describes the user requirements for the system.
• Creation and approval of the Functional Specification (FS), which explains how the URS is functionally implemented.
• Creation and approval of the Design Specification (DS). This shows exactly how the functionalities from the FS are implemented technically.
• Then - at the very bottom of the V-model - the software is programmed. At the end of implementation, a code review is carried out to subject the programming to a quality check using the dual control principle. From this, an installation guide (IQ) to be tested is created and approved.
• On the right-hand side, one step up, the unit tests are then executed to check the accurate implementation in accordance with the DS. Test deviations must be dealt with formally and all test results must be approved at the end.
• Only then does it go one step up again, where the functional tests (OQ) check the correct implementation of the functionality in accordance with FS. Again, the same applies as above: Test deviations must be dealt with formally and all test results must be approved at the end.
• Only then is the acceptance test (PQ) carried out by the user!
• At the very end, a validation report (VR) is created, which shows how all planned test and acceptance steps have been successfully completed; it thus proves the traceable quality of the result - i.e. the finished software. The formal release of the validation report leads to the release of the system for productive use.

 

The central tool of the entire process is the functional risk analysis (FRA). This involves analyzing and evaluating the various possible risks. Based on these assessments, measures are defined that serve to minimize the risk. This can lead to additional test activities, procedural solutions, additional training or even design changes.

It is immediately clear that this is a very documentation-heavy, inflexible process. The validation effort is often much higher than the actual programming and adjustments can involve a disproportionate amount of effort, especially if they are detected late (e.g. only in the acceptance test by the end user representatives - a very typical case). The large number of signatures required for acceptance tests must also be taken into account; collecting all the necessary signatures can often take a long time if those responsible are not available. This is particularly critical because in the V-model you are not allowed to start the next step until all previous steps have been approved by signatures.

In many ways, this contradicts the modern agile approach - for example Scrum - which is becoming increasingly popular in software development, and is therefore almost the exact opposite of the flexible and iterative approach that we are more used to in modern software development.

 

Scrum

Scrum is now the methodology of choice for agile software development in many organizations. We will therefore only briefly outline this methodology here in order to highlight the contrast with the concept of validation. The elements of the Scrum framework are shown in Figure 3 illustrated:

 

 

Scrum is an iterative process in which the team works through a so-called backlog in work blocks of equal length - "sprints". More emphasis is placed on functioning software than on extensive documentation.

Regular retrospectives and short "Daily Scrum Meetings", in which all problems are addressed directly and promptly, serve to improve teamwork and thus increase the team's implementation speed ("velocity"), i.e. the capacity to successfully complete more work during a sprint.

Three cornerstones of Scrum are therefore:

1. Addressing hurdles openly and directly - "Transparency"
2. Self-review of the team and processes - "Inspection"
3. Adaptation and improvement of the team - "Adaptation"

 

The challenges of validation with Oracle APEX

Changing requirements are a major challenge in validation. This is where the agile methodology collides with the V-model. In an agile environment, requirements are not fixed and can change constantly during the project. This is a strength of the agile approach. Conversely, the waterfall method (V-model) is based on rigid requirements.

So how can the regulated VModel be implemented with an agile methodology?

A second challenge in connection with APEX is the strong emphasis on the traceability of adjustments in the validated environment. APEX does not provide a built-in audit trail that shows directly who changed what, when and for what reason. Version control - for example with Git or Subversion - is not part of the APEX core philosophy.

How can this situation be addressed efficiently so that the traceability that is critical for validation can be guaranteed?

We illustrate these points below using a customer project.

 

A validated customer project with Oracle APEX

The customer is one of the ten largest pharmaceutical companies in the world. The project was in the clinical development phase.

The supported processes are highly regulated, as the software was used within the approval process. The confidence gained in the study data collected has a direct impact on the approval of the drug. In addition, a software error during a trial could put patients at risk - with all the resulting consequences.

The validation services to be provided by the customer included creating the documents prescribed by the V-model in HP ALM (an application lifecycle management system), carrying out code reviews (dual control principle), performing the test cycles (unit tests - with utPLSQL, OQ) and installation (in accordance with IQ).

 

The pragmatic approach

In this constellation, we opted for a pragmatic approach and continuously refined it during the project. We used Microsoft collaboration tools (MS Teams) to work together on documents and Atlassian Jira for agile tracking of the work blocks and for planning the sprints.

We divided the project into time intervals, similar to sprints, and worked agilely in each of these intervals. At the end of an interval, we defined a baseline and applied the waterfall-style validation according to the V-model (FS-DS-UT-IQ-OQ-PQ) in a validation sprint to achieve the formal release (see Figure 4).

 

 

We then recorded the deviations and adjustments identified from unit tests, system tests (OQ) and user acceptance tests (PQ) in a change management system and subsequently assessed the associated risks using the Functional Risk Assessment (FRA).

During implementation, we attached great importance to keeping the APEX part simple. This means, among other things:

1. to keep the source code as small as possible by utilizing the low-code approach of APEX,
2. PL/SQL code in APEX has been consistently outsourced to DB packages (e.g. pipelined functions in APEX Interactive Reports (IR), database function calls in APEX processes, authorization control in database function calls),
3. all database objects (packages, functions, views, etc.) were consistently subjected to version control in Git. Git was defined as a single source of truth.

 

Coding guidelines are very important, right from the start of the project. We had not rigidly fixed these guidelines, but continuously developed and refined them in the sprint retrospectives. These guidelines then served as the basis for the formal code review process.

In the course of the project, we also had the opportunity to learn from mistakes: error handling turned out to be very important. The persistence of errors, warnings and information turned out to be indispensable for debugging. From our point of view, the open source software Logger from OraOpenSource [3] is highly recommended here, as it standardizes the entire error handling. Furthermore, it has proven to be very useful to incorporate an APEX page for the information and error log, which is visible with admin authorization in order to increase transparency.

We use the so-called "Build Options" of APEX in order to be able to flexibly integrate functionalities (see Figure 5). With a simple configuration change, these "hidden" functionalities could be easily switched on and off for the end user (pages, menu items, etc.). This served to verify the new functionalities without making them available to the end users; in this way, the introduction of functionality could be decoupled from the rigid framework of the V-model.

 

 

Conclusion

These pragmatic measures enabled us to successfully use the agile principles to the benefit of the project and at the same time meet the strict regulatory requirements.

This made a decisive contribution to the successful implementation of our project to the satisfaction of the customer.

The lessons learned can also be helpful for similar APEX projects in strictly regulated environments.

 

Sources

[1] Components of a validated computerized system, PIC/S Quelle: PIC/S (2007)
[2] GAMP5, A Risk based Approach to Compliant GxP Computerized Systems, ISPE (2008). Tampa, Fla. ISPE Headquarters [u.a].
[3] Logger, OraOpenSource, https://github.com/OraOpenSource/Logger