Automation MOC – securing the softer side of your plant
Author : Chris Lyden, President - PAS
20 November 2011
Management of Change (MOC) is a subject that process industry manufacturers know well. It is a core process for managing modifications to such things as plant infrastructure and equipment, and although it can be a significant consumer of time, most acknowledge that it improves safety.
Most countries have established regulations governing the MOC process, however this guidance does not extend to automation changes - and as a result most industrial plants operate with significant vulnerabilities associated with poor automation change management.
Undocumented and unapproved changes to automation systems have been identified as contributing factors to a number of process industry incidents and accidents around the world. To learn more about this, PAS recently conducted a survey of its petroleum, power generation, and chemical industry clients asking the question: “Have modifications to automation systems ever led to an incident or process disturbance at your site?” Shockingly, nearly sixty-five percent of the respondents said that they had! While not definitive, this data indicates that proper management and documentation of automation configuration changes, while an essential element of control system security and plant safety is not being practiced as it should be.
Automation systems in process industry plants contain an astronomical amount of data from disparate systems, measurements and applications. Because these systems are the main vehicles for continuous improvement of the process, they are also continually undergoing change. While automation systems are modified often, an MOC process for those changes is only sometimes required, and our customer survey revealed that automation modifications have led to incidents and/or lost production at nearly two-thirds of the sites polled.
Why was automation missed out of the MOC process?
MOC has long been deemed best practice by many companies, but only became law in the United States in 1992 when Congress passed the 29 CFR 1910.119 Process Safety Management (PSM) regulation. This regulation, which is aimed at preventing or minimising the consequences of catastrophic releases of toxic, reactive, flammable or explosive chemicals, mandated that all manufacturers of these chemicals have a compliant MOC process in place by 1997. Other governments around the world quickly saw the benefits of this regulation and enacted similar laws within their jurisdictions.
The other reason why MOC is sporadically applied to automation is the relative ease with which configuration changes can be made to automation systems. Generally, it is possible to make modifications to automation systems that have significant consequences by simply typing in database parameters – for example changes to critical alarm limits or re-ranging an instrument are accomplished in this way. Because the traditional MOC process takes significantly more time and energy to implement than the actual changes, it can be seen as excessive and as such MOC for these types of changes is often excluded from the work process.
Ignoring automation changes in the MOC process opens the door for unapproved and undocumented changes to be introduced into the configuration, which can carry with them the potential for disastrous results. It should be noted that a robust MOC process is almost always followed if an automation configuration modification affects a drawing, procedure or any other entity currently under change control – although the parametric changes that are currently overlooked could have equally serious consequences.
Undocumented change caused flaring event
One such incident, caused by undocumented changes to an automation system, occurred at a petrochemical plant operated by one of our clients. Plant personnel were conducting a routine Safety-Integrity Level (SIL) test of an interlock in a Safety Instrumented System (SIS). As per their test procedure, they bypassed the output of the SIS so that when they exercised the logic to test the system, the valve would not actually trip. However, when the logic was actuated, to their astonishment, the plant tripped.
Subsequent investigations determined that a well-intentioned individual had made a perfectly logical automation configuration change, but had not documented or communicated it. He had added some start-up assistance logic in the Distributed Control System (DCS) that sensed when that specific interlock had tripped in the SIS and then placed all the DCS controllers on the unit in manual and their outputs at 0% to facilitate restart of the unit. His assumption was that if that particular signal in the SIS were true, that the SIS would have already shut the unit down.
Because this start-up assistance logic was undocumented, and hence not reflected in the SIL test procedure, it was not disabled prior to the test. The result: the logic shut the unit down and a significant flaring event occurred. The plant not only lost production during the shutdown, but incurred stiff environmental fines from the flaring as well. There is little doubt that proper automation system MOC would have prevented this incident by requiring peer reviews and approvals of the modifications before implementation, proper documentation of the new logic, modification of the SIL test procedure and a pre-start-up safety review.
Detecting automation configuration changes
In the day-to-day operation of a plant, the problem is that automation changes occur so frequently that the sheer volume of automation changes can quickly become overwhelming. This problem has been addressed by using sophisticated Integrity software and more than fifty vendor asset models to interface with and import the entire automation configuration database for all systems at regular intervals, then compare it to the database image taken at the last import. The software then tracks and reports on all changes made between import intervals, both within a specific system and also among interoperable systems. While most automation system manufacturers are able to track changes within their own systems, Integrity does so across system boundaries, and is therefore able to track a signal’s genealogy throughout integrated systems. A common example of a signal crossing the boundaries of interoperable systems is a safety transmitter that is connected to an SIS, which passes the measured value to a DCS workstation for display, which in turn passes it to a historian for archiving.
In order to track configuration changes across the plant, Windows-based automation systems must also be taken into consideration. Managing modifications to this underlying automation system infrastructure is as important to safety and security as managing modifications to the actual configuration – in fact it is commonly understood that the Stuxnet virus was initially introduced through a removable “thumb drive”.
Process manufacturers almost always standardise their underlying Microsoft Windows-based automation systems infrastructure by implementing a uniform configuration of hardware and software referred to as a Common Operating Environment (COE). Because most sites have a multitude of disparate automation systems, administrators accommodate the differences among them by defining multiple COEs within the facility and auditing them for compliance at regular intervals. For example, Stuxnet highlights the importance of considering whether to disable the USB ports into which thumb drives may be inserted in a COE. Modifications to such things as the USB port enable/disable status or Windows active directory security settings represent a significant security concern and must therefore be under MOC control.
Software tools available make this achievable by tracking compliance to COE specifications for the servers, workstations and desktop computers on process automation networks.
Having considered how to efficiently detect these changes, the remaining hurdle is to bring them into an MOC process. To ensure that no unauthorised or malicious changes are lurking within an automation system, each change to the actual configuration or the underlying infrastructure must be under strict MOC control. Furthermore, it is essential that automation databases be continually checked for changes and compared to approved MOC cases to determine if those changes were actually authorised. Any changes that do not reconcile would therefore be unapproved and require investigation.
Streamlining the automation MOC process
We have worked with our customers to develop an intelligent management of change application that specifically addresses the needs of automation. It works with both Integrity software and Integrity Recon to reconcile the changes detected by those applications with the MOC cases that authorised them and ensure that only approved and managed changes occur. Reports of unreconciled changes are automatically generated, exposing the vulnerabilities associated with improper automation change management.
It is clear that if the MOC process is to be applied fully to automation, it must be both greatly simplified and made very easy to use. For most automation system configuration changes there is no need to change drawings or procedures, and thus the traditional MOC process would be overkill. However, to ensure that only safe and appropriate modifications are made to configurations, each change must be evaluated, approved and tracked. A shorter, more streamlined, MOC process that does not entail more effort than the modification itself is clearly needed for most parameter-only automation configuration modifications.
In these cases, it is suggested that a special classification of the MOC process is initiated, approved and closed out by the automation professional that actually makes the change. This process would employ a simple electronic checklist that would be completed after the modification is made and electronically signed by the person making the change.
It is assigned a tracking number and saved as an MOC case by the software, and then on the next import of the configuration database the modifications are reconciled with the MOC case that authorised them (Figure1). A report of all unreconciled modifications is generated so that unauthorised changes are immediately identified and communicated to the responsible parties. Those unreconciled changes may then be either accepted as they are or a new MOC case may then be created to authorise changing them back to an acceptable state. With minimal effort, this process ensures that no unauthorised changes remain in the automation systems.
However, this simplified MOC process is not appropriate for all MOC classifications, so new workflow templates can be created when a more complex sequence is required. For example, there is a class of automation modifications that affect drawings, procedures and other items, and therefore requires a longer, more complicated workflow. Figure 2 is an example of one such workflow template.
Another area where the process can be streamlined is the approval process. Engineers spend a substantial amount of time acquiring approvals in order to initiate cases, move them to the next phase, or close them out. The new software reduces time spent gathering approvals by using Web 2.0 technology to “push” them to the approvers. User-defined approval logic may be applied so that for example, if a primary approver is unavailable, a secondary approver would automatically receive the approval request. Furthermore, if the approvers don’t respond in a timely manner, it reminds them that they have an outstanding approval task.
These features provide a significant reduction in time spent on non-value-added activities associated with the MOC process.
This next generation MOC software also helps reduce engineering time spent on preparing initial Requests For Change (RFC) by automatically identifying and cross-referencing all configuration linkages and places where an automation parameter is used. This feature significantly accelerates the preparation of RFCs by reducing the time spent researching the scope of the change and also ensures that no critical use of the parameter under change is omitted from the process.
While MOC is a core process in process industry plants around the world, the amount of effort required to implement it for typical automation changes is so disproportionate to the changes themselves that is not always applied. This practice leaves plants vulnerable to unapproved, undocumented and potentially malicious changes made to their automation systems. Only through a program of rigorous MOC, that reconciles automation changes to the MOC cases that authorised them, can unapproved or undocumented changes be eliminated as a cause of incidents.
By embracing advances in software and web 2.0 technologies it has become both feasible and manageable to provide MOC for automation systems in a busy plant, taking control of all plant-wide system configurations to significantly improve safety and security.