Changing an unknown known to a known known in Functional Safety
Author : Andrew Derbyshire & Chris Bell, DNV
08 March 2022
The introduction of edition 2 IEC 61511 has brought with it a set of new challenges to process industries. The standard now calls for mandatory stage 4 functional safety assessments (FSA) to be performed periodically in-service and the behaviour of the safety system needs to be monitored.
Representative image: Shutterstock
(Click here to view article in digital edition)
In this article, DNV presents a computational method for analysing a large number of maintenance records using a variety of data mining techniques. Currently, many technical authorities (TAs) and asset operators are aware of problems on their plant but struggle to demonstrate the cause, for example “repeat offenders”, or even that a problem exists at all, due to the way data is recorded, often in unstructured text.
The methods discussed here use data mining and machine learning algorithms to analyse an entire set of maintenance records for a Safety Instrumented System (SIS). The algorithm can detect anomalies in the data and the way that maintenance records have been recorded, allowing for more targeted assessments and better understanding of the actual behaviour of the SIS related devices by plant owners. The subjectivity of the assessor is removed, and the assessment can focus more on records which have anomalies, as opposed to a random sampling method which makes identifying anomalies more difficult. For asset owners this allows more understanding of actual behaviours which can then contribute to a more accurate demand, failure and spurious trip rate associated with the Safety Integrity Level (SIL) level for each Safety Instrumented Function (SIF).
The algorithm automatically highlights areas for investigation in a matter of seconds which crucially reduces the amount of time spent reviewing non-erroneous records. This increases efficiency and supports prioritisation of budgets based on the findings and recommendations. The algorithm also supports automating high-cost, error-prone tasks in which the cumulative effects of inconsistencies and errors in the analysis can adversely impact safety.
Functional safety assessments introduction
The increase in digitalisation across the oil and gas sector offers offshore operators the chance to automate high-cost, error-prone tasks in which the cumulative effects of inconsistency and analytical error can adversely impact safety. For instance, as part of any asset’s assurance process, it can be instructive to review maintenance records for insights, particularly trending issues and identifying potential improvements. The goal is to ensure the asset is performing safely and effectively, with high reliability while adopting the most cost-effective strategies for all maintenance work.
In addition, the second edition of IEC 61511, published in 2016, has placed greater emphasis in the operational phase on measuring the performance of a Safety Instrumented System to ensure continued safe operations. Coupled with these additional requirements is the expectation that duty holders will now perform periodic Stage 4 Functional Safety Assessments (FSA-4) in the operational phase as an additional mandatory requirement of IEC 61511. Part of the scope of the FSA-4, as defined in industry guidance such as a CDOIF Guide on Installed SIS, is to ensure performance is being adequately measured. These new requirements only reinforce the push in an increasing digital landscape to measure performance so that cumulative effects of inconsistency and analytical error do not adversely impact safety.
Figure 1 – CRISP-DM key cycle steps
DNV utilises a CRISP-DM (cross-industry standard process for data mining) sprint methodology. CRISP-DM is an iterative approach whereby DNV works closely with the data owner or operator in short “sprints” to explore the data sources, determine potential outcomes and develop models for data analytics. The key cycle steps as shown in Figure 1 are:
1. Business understanding: develop and understand the business/user/FSA requirements.
2. Data understanding: review and understand the data; what anomalies are present in the data, and can it be used to provide insights?
3. Data preparation: prepare the data to be analysed; remove anomalies, reshape as required, merge with other sources etc.
4. Model preparation: develop algorithms and models to read the prepared data, and provide statistics, plots or insights to be evaluated.
5. Evaluation: evaluate the model’s effectiveness and ability to provide the answers required. Does it raise more questions that should be answered before proceeding further?
Figure 2 – Work Order counts in dataset by Test Result
After each evaluation step, the team reviews the business requirements. Hence, the requirements are subject to change during the FSA-4 review life cycle. This is a challenge for a traditional project setup, which may suffer from a hierarchical structure and an inability to control scope creep. Also, most engineering organizations are not structured to facilitate such approaches; they are not agile. However, by moving to this approach the project team can shortcut management debate cycles and reduce requirements gathering and solution design into a single week.
This sprint approach requires close working and regular feedback between the data analytic teams, the FSA review engineers, and the operator to ensure they derive maximum benefit from the process.
Data mining
An example of a study considered the SIL 1 inspection and maintenance dataset from an offshore operator. This dataset was chosen because SIL 1 items are common elements offshore – hence there exist a lot of maintenance records – which are prone to failure or mis-calibration, so there should be a suitable number of FAIL records against which to predict. Both preventative or planned maintenance (PM) and corrective maintenance (CM) records were provided. From this, the Damage Code and Cause Code details were assessed using text recognition to produce a Test Result field marked as PASS, FAIL or FAILFIX indicating whether the SIL 1 equipment item functioned as expected. Depending on the configuration of the maintenance system this Test Result field may not have to be artificially created as many operators do have this field as part of their standard reporting.
The location and date fields in the dataset were useful for secondary calculations, such as reliability and availability of instrumentation types over time. Reliability being whether the instrument functioned on demand, and availability being a measure of downtime. Comparison of target and actual finish dates gives a measure of the work deferral rate. Similarly, identification of which locations had the most corrective actions raised can pinpoint weak signals of bigger asset problems. This article does not consider such calculations further as they are relatively simple to implement once the Test Results are known.
Instead, this article will focus on the long test description field: a free-text field in which the test and the result are typically described. Using a combination of natural language processing and machine learning (support vector machine) algorithms, we explore the question: can a computer predict the test result by analysing the text?
We note some important features of the dataset at this stage. As shown in Figure 2 approximately 20% of the records are unclassed or NULL, that is, they do not a have recorded test results. We cannot use those records to train a machine learning model, hence our available training data reduces to 1,409 records, which is still a sufficient number when considering two classes only (PASS and FAIL). There could be several reasons why the Test Results are not recorded, for example a change in management system could mean the previous results were not carried forward into the new system. In such cases, the method presented here can assist operators reconstruct the missing data. This re-construction of data will be essential if we are to accurately determine the reliability/availability of instrumentation.
Figure 3 – ML cleaning and processing steps
Using a support vector machine (SVM) machine learning (ML) predictive algorithm developed with several cleaning and processing steps, the entire maintenance dataset could be reviewed autonomously and re-classified in minutes as shown in Figure 3 where the y axis of the confusion matrix displays the Test Result as reported and the x axis shows the Test Result re-classified using the ML algorithm. A job which would take an engineer several months of time-consuming work that is prone to error.
Now the job of the FSA-4 reviewer can start by posing the following questions:
- Why are 10% of the pass records being flagged as fail by the algorithm?
- Why are 9% of the fail records being flagged as pass?
- Why are 39% of the unmarked records being flagged as fails?
* Is this failure captured at a different system level?
* Are poor procedures and processes in place to capture failure information?
* Are technicians mis-reporting to safe-tome and paper work?
* Does the engineer see value in reporting failures?
Andrew Derbyshire & Chris Bell, DNV
Conclusion
This work demonstrates that a trained SVM algorithm can rapidly identify records with potential anomalies. The methods employed gives FS teams a more focussed approach to check records by specifically targeting the anomalous maintenance records. Crucially, by reducing the amount of time spent reviewing non-erroneous maintenance records, this approach increases project efficiency and allows more useful findings and recommendations to be made. The processes detailed in this article can easily be applied in other systems and assessment criteria’s such as IECEx, ATEX where classifying data into distinct categories for trending and analysis purposes can be deemed useful.
DNV also carried out datamining on the demand rates of safety instrumented functions (SIFs) as part of the FSA-4 review. The demand rates were determined by taking downloads from the distributed control system (DCS) or instrumented control and safety system (ICSS). When these demand rates were reviewed it was clear that many of the demand rates were way above the designed demand rates, this data mining technique was able to identify where SIFs were being called upon routinely to control normal process flows. Had this detail not been picked up via the data mining process, and the SIF had failed to act, the outcome could have been catastrophic.
About the authors:
Andrew Derbyshire is a Principal Safety Engineer at DNV specialising in functional safety consultancy and independent conformity assessment activities throughout the lifecycle. He is also a member of the IEC 61508 Association management committee and the current chair and a director of the CASS Scheme which is a not-for-profit organisation aimed at promoting the correct use of the IEC 61508 group of standards.
Chris Bell is a Digital Innovation Consultant at DNV based in Aberdeen where he works on the development of new technologies and digital tools for use in the oil & gas sector with a focus on technical safety and asset integrity. Chris has been involved with many projects utilising data mining and machine learning to improve current engineering practices with regards to FSAs, EX management and Asset Integrity.
Contact Details and Archive...