Maximizing Your SMS: Differences Between Audits, Evaluations and Investigations

Audits, evaluations and investigations are three distinctively different and powerful tools for supporting decision-making within any organization’s proactive safety management system (SMS). Understanding the differences of the SMS “big three” safety assurance (SA) elements can help us utilize each tool at the right time with minimal resources and result in the quality decision-making required within the SMS risk management framework.

Defining each of the three SA elements can help us explore their mechanics, goals, applications and objectives. As an aid to learning the true purpose of detailed safety assurance practices, we might want to begin by blowing up the paradigm that an audit and an internal evaluation program (IEP) always default to pre-baked regulatory data collection tools or standards and recommended practices.

Audits, at best, are a snapshot in time and begin aging immediately after their formal finding report and corrective action requests (CAR) are issued. A fully completed internal audit will assure minimum standards of care but has no means to adapt. Then, up your proactive SA management approach and use the following discussion and tables to plan and treat a much different series of ongoing business management analyses and evaluations.

Be mindful that there may be some IEP/IAP terms that fall outside our too-familiar SMS lexicon.

Audits (Internal/External)

The audit has been present in business for thousands of years. There are different types (according to their objectives), but in a generic way, audits are objective and systematic assessments of how well an organization (or certificate holder) documents, implements and executes (complies with) industry external regulations, industry standards and recommended practices.

In the ISO 19011:2018 world, this is defined as an “independent” process for obtaining evidence of existence—records, statements of fact or other information—that are relevant and verifiable toward predetermined audit criteria.

Audits are scheduled activities carried out by external, third-party teams of auditors, specialty-trained industry auditors or government inspectors. Conversely, internal audits are conducted by company employees using first-party direct department employees or second-party indirect department employees working on the company’s behalf.

Evaluations (Internal/External)

Evaluations involve assessing the strengths and weaknesses of programs, policies, personnel, products, organizational design, behavior, and culture, to improve their resiliency and effectiveness. Their purpose is to make judgements based on selected corporate performance criteria while providing near real-time status of program success.

Analyzing interprets data as it deals with meanings and implications, while evaluating assesses something’s worth. Hence, results are more compulsory for the evaluation process. Analyzing precedes evaluating. Analysis largely involves a longer thinking process as compared to evaluation.

Based on the results of external evaluations, organizations can focus their efforts and resources on those actions that proved most useful and effective to achieve the maximum possible impact. Evaluation allows us to:

• Learn from experience and identify best practices to continually improve the impact of projects.
• Deepen the history behind project results and explain why an intervention was successful or failed.
• Observe the results of the project from multiple angles and consider all categories of beneficiaries and stakeholders involved.
• Increase the transparency of an organization and demonstrate the relevance, effectiveness and efficiency of its work in the eyes of financiers, stakeholders and process owners.
• Easily communicate the value of an intervention, including quantifying its results and social and economic impact.

Investigations (Internal/External)

Investigations are usually undertaken in response to reports of deviation from protocol or procedure Or they involve potential violations of operating regulations and specifications and focus on persons or teams instead of policies, processes or procedures.

Investigation reports may be prepared to refer to matters of personal licensing and certification or correct serious deficiencies. They may also inform other government or local agencies of the need for action, discipline, letters of warning or formal prosecution.

The following list maps the control elements of effective SMS safety assurance tools:

Key purpose?

Internal Audit:
Objective and systematic assessments of how well the program is documented, and effectively implemented.

Internal Evaluation:
Front-line assessment of strengths and weaknesses programs, policies, personnel, practices, products and organizational structure to improve their effectiveness.

Investigations:
Impartial review usually.

Key objectives?

Internal Audit
Standards audit (compliance to external regulations and company-adopted standards) of WHAT we should be doing.

Internal Evaluation

• Measure effectiveness of HOW we should be doing something. (risk assess degree of conformance to internal company policy).
• Business case analysis supporting WHY we should be doing something. (ID gaps in conformity and errors in current policy design).

Investigations
Determine causes of events requiring correction of serious errors or deficiencies, certificate action, potential discipline, or inform other departments/agencies of the need for action within their oversight and jurisdiction.

Who performs it?

Internal Audit
Second- or third-party trained auditors.

Internal Evaluation
Company department process owners, section managers, front-line employees or first-party auditors who work within the scope of the procedural area.

Investigations
The investigator should possess:

• An ability to investigate objectively without bias. No stake in the outcome.
• No personal relationship with the involved parties.
• No concerns that the outcome will directly affect their position within the organization.
• Skills that include prior investigative knowledge and working knowledge of employment laws.
• Strong interpersonal skills to build a rapport with the parties involved and to be perceived as neutral and fair.
• Attention to detail.
• The right temperament to conduct interviews.

Who oversees its results?

Internal Audit
Key program or process owners, department managers.

Internal Evaluation
Key program or process owners, department managers.

Investigations

• Key program or process owners, department managers.
• Independent regulators, legally appointed investigation agencies.

When is it done?

Internal Audit
Audits should usually be scheduled at least once per year and should cover all the activities undertaken, especially if they are relevant to SMS.

Internal Evaluation
Once a performance scoring/monitoring baseline measurement is set, ongoing IEP samplings and analysis events are periodically scheduled by the safety manager (SM).

• The SM, with full support of their safety & quality committee members, determines future evaluation criteria plus their sampling frequency, against task design, work tempo or any new change management drivers.

Investigations
Upon occurrence and/or legal appointment.

Why is it done?

Internal Audit
Auditing grew out of the accounting discipline, while evaluation grew out of the social sciences. An auditor looks for instances of things going wrong. They place great weight on the need for accuracy of administrative records and are likely to be suspicious of statistical inferences based on aggregate data if they cannot find actual cases confirming the general conclusion.

Internal Evaluation

• As with any dynamic management system, a proactive SMS requires continual performance monitoring to assure that it remains within control. These controls become the SPIs and KPIs demanded of the company’s strategic objectives.
• The social science evaluator mistrusts administrative records and is predisposed to seek their own direct observations and a much more generalized view of what happened. They prefer large volumes of data and mistrust small numbers of specific events because they have little confidence that what was found in one case is typical.

Investigations
Regarding any event or occurrence reported, if the alleged facts can be substantiated, check what supporting documentation or other materials can be found, preserve and secure basic evidence and determine whether or not an investigation is justified.

How is it carried out?

Internal Audit
Internal audits are carried out independently from program management.

Internal Evaluation
Internal evaluations are carried out dependent on the expertise of front-line workers with the collaboration of program management.

Investigations
Investigations are carried out both internally and externally. Many times, internal and external investigations will occur simultaneously but will maintain their independent sources to assure objectivity.

How are policies, programs, procedures involved?

Internal Audit
Auditors must answer the “standard DIE” audit review questions for all pertinent policies, programs, procedures:

• Is it Documented?
• Is it Implemented?
• Is there Evidence it is completely effective?

Internal Evaluation

• Evaluations are conducted in a neutral manner with integrity in relationships between evaluators and stakeholders.
• SMs and front-line employees collaborate with department heads and managers in the conducting of evaluations and evaluation planning exercises.
• Process owners are responsible for developing and implementing performance measurement in consultation with the SM.
• Department heads ensure that valid, reliable, useful performance data is continually available to meet the needs of an evaluation of the program.

Investigations

• Proper historical revisions of policies, programs, procedures are recalled for evidence of being in place during the event being investigated.

What are the key oversight requirements for internal departments or external agencies?

Internal Audit
Accept findings from a certified audit.

Internal Evaluation
Accept documented and implemented findings in support of the company’s planned policy conformity and strategies.

How is it continually monitored?

Internal Audit
Once you complete an internal audit, you should remediate any gaps identified during the process. In addition, conducting a follow-up audit will increase the likelihood that an external audit will go well.

Internal Evaluation

• Design and integrate multi-faceted IEP plans that meet the company’s unique operating context.
• Use project management tools to scope and target key areas of operational risk management where employees are at highest risk.
• Publish a formal review frequency that assures key processes and procedures are continually maintained in a risk-based ALAP state.

Investigations
Even after a written report is submitted, the SM should take additional steps:

• Submit the findings to the decision-maker (typically not the investigator), who will determine what disciplinary action to take.
• The decision-maker, either a high-level HR professional or a business leader, should be high enough in the organization to determine how people in similar situations have been treated.
• Notify the employee who made the complaint that action was taken—even if details can’t be shared for privacy reasons.
• Reintegrate the employees involved back into the workplace, shifting focus from the complaint to the changes the investigation has brought about.

Remind managers that retaliation won’t be tolerated and check back within 6 months to ensure that there has been none.

Review the investigation to determine what could be done better next time.

Look for patterns in complaints that might suggest more training is needed to avoid similar problems in the future.

How are key indicators status and trends reported?

Internal Audit
There are numerous risks that your organization may identify during an internal audit, including:

• Reputation risk
• Operational risk
• Transactional risk
• Credit risk
• Compliance risk
• Strategic risk
• Country risk
• Legal risk
• Vendor-concentration risk
• IT/cybersecurity risk

Identifying these risks during an internal audit is the first step.

Creating a plan to remediate any risks will assure that your organization is ready for an external audit.

Internal Evaluation

• Risk assessment and ranking of hazards or threats associated with procedural drift, anomalies and/or frequency of irregularities.
• Cultural outliers that remain in a semi-permanent drift state.

Measures of prevention effectiveness and success

Internal Audit

• CAR /CAPA risk ranking, scores.
• Aging open issues and related escalation.

Internal Evaluation

• Key SMS promotion efforts (lessons learned).
• Continuing observation tool results validating initial findings.