Your Investigation Data Can Only Tell You What You Thought to Look For

What new research on AI tells us about human thinking, investigation quality and ICAM

Artificial intelligence can process an extraordinary amount of safety data. It can classify reports, identify patterns and help organisations make sense of information that would take people months to review manually.

But there is a catch.

AI can only analyse what people recorded in the first place. If an investigation captures the immediate event but misses the conditions that shaped it, those conditions effectively disappear from the dataset. That makes a recent Safety Science study on the use of machine learning in occupational safety particularly relevant to incident investigation.

The technology is interesting. The bigger question is what the research tells us about the way humans investigate incidents.

What the researchers found

The study, Using AI to support safety management: Analysing occupational safety data using machine learning within the framework of human factors, examined whether large language models could help classify human factors within substantial collections of occupational safety information.

The researchers worked with accident and near-miss reports, investigation material and more than 300,000 accident descriptions. Their classification framework considered factors across individual actions, characteristics of the work, group and team factors, and organisational factors.

The results showed real potential for machine learning. A fine-tuned language model could classify factors within large volumes of unstructured safety text with useful accuracy.

There was another finding that matters just as much. The information available in routine safety reports was often heavily weighted towards the immediate characteristics of the work. Group, team and organisational factors appeared far less frequently in some datasets.

That does not automatically mean those factors contributed to every event, or that every report represented a full investigation. It raises a more useful question.

Were those factors absent from the event, or absent from the investigation and reporting process?

That distinction matters.

We do not investigate with a blank brain

Every investigator brings human thinking into an investigation. We observe an incident, hear an initial account and begin constructing an explanation of what happened.

That is normal. It is also where investigation discipline becomes important.

The person closest to the event is usually visible. Their actions can often be described quickly. Equipment damage can be photographed. A procedure can be checked against what occurred. These pieces of information are relatively easy to find.

The organisational decisions that shaped the work may be much less visible. They might have been made months earlier and involve planning, supervision, resourcing, change management or the way controls were designed and maintained.

If an investigator settles on an explanation too early, the investigation can become a search for evidence supporting that explanation. Information that does not fit the emerging story may receive less attention.

We specifically warn against premature conclusions. Investigators are expected to work with information validated as fact, remain neutral, ask why and continue beyond the obvious explanation.

That thinking is fundamental to ICAM investigations.

ICAM gives investigators a structure for looking deeper

ICAM was developed by Gerry Gibb at BHP to help investigators understand incidents through a systems-based approach.

The analysis considers Absent or Failed Defences, Individual or Team Actions, Task or Environmental Conditions and Organisational Factors. These areas are connected rather than treated as separate boxes to complete.

An action by a person may be important. ICAM does not require investigators to ignore it. The investigation needs to understand what influenced that action, what conditions surrounded the work, which controls were absent or ineffective and whether organisational processes created or allowed those conditions to exist.

The analysis therefore works backwards from the incident.

What defences were supposed to prevent or reduce the event? What happened at the point of work? What conditions influenced those actions? What organisational factors were sitting behind those conditions?

This structure helps counter a very human tendency to stop when we find an explanation that appears reasonable.

PEEPO helps broaden what enters the investigation

The same issue appears much earlier, during data gathering.

We use PEEPO to structure evidence gathering across People, Environment, Equipment, Procedures and Organisation. The purpose is to create a broad picture of the event before deciding what the evidence means.

That sequence is important.

Imagine an investigator arrives after a mobile equipment collision and immediately believes the operator failed to maintain adequate separation. If data gathering becomes centred on the operator, the investigation may collect training records, statements and the relevant procedure.

All of that could be relevant. It is still only part of the picture.

A structured PEEPO process may also examine visibility, traffic design, equipment configuration, communication arrangements, workload, supervision, previous events, maintenance history and the organisational decisions behind the controls.

The quality of the eventual ICAM analysis depends heavily on the quality of this evidence gathering. You cannot meaningfully analyse a condition that was never examined.

Human memory creates another filter

Witness information adds another layer.

People do not experience an incident like a video recorder. Perception can be influenced by where someone was standing, how long the event lasted, what they were focused on and the level of stress involved.

Memory can then change with time. Conversations with other people, later information and the way questions are asked can influence what someone recalls.

The investigator becomes another part of that process. 

The questions asked determine which areas are explored. The questions not asked can be just as important.

By the time information reaches an incident database, it has already passed through several human filters. Someone noticed something. Someone remembered it. Someone asked about it. Someone decided whether it was relevant. Someone recorded it.

AI arrives after all of that has happened.

Missing organisational factors should create curiosity

This is where the Safety Science research becomes particularly useful for investigation leaders.

If a large safety dataset contains thousands of references to individual actions and very few organisational factors, it is tempting to treat that distribution as a picture of how incidents occur.

It may instead be a picture of how the organisation investigates and records incidents.

That does not mean investigators should manufacture organisational factors to make an analysis appear more systemic. Findings must be supported by evidence. If the evidence does not establish an Organisational Factor, it should not appear in the ICAM analysis.

The point is different.

Before concluding that organisational factors are rare, we should be confident investigators were equipped to look for them.

Were they examining decisions and processes that existed before the incident? Were they asking what produced the task and environmental conditions? Were they exploring how controls were designed, implemented and monitored?

The absence of a category from a database does not necessarily establish the absence of that factor from the workplace.

Recommendations can reveal how we are thinking

The research also identified an interesting difference between factors identified in investigation material and the corrective measures that followed.

In the research dataset, individual factors represented a much larger proportion of identified causal material than they did corrective measures. Organisational factors showed the opposite pattern, appearing more frequently in corrective measures than in the factors identified during analysis.

The terminology used by the researchers differs from ICAM, where Safety Wise uses Contributing Factors. Even so, the finding raises an important investigation question.

Are our recommendations actually connected to what our analysis established? A strong ICAM investigation creates a visible line from evidence to analysis, from analysis to Contributing Factors and from those findings to recommendations.

When that line breaks, recommendations can become generic. Training is added because training seems sensible.

A procedure is changed because changing a procedure feels tangible. A broader organisational action may be proposed without the investigation clearly establishing the condition it is intended to address.

We teach investigators to focus recommendations on the design of the game rather than the player. That requires the investigation to understand the design first.

AI can help, but investigation quality still comes first

There is considerable potential for AI in safety investigation and organisational learning.

An organisation with thousands of incident records could use language models to identify recurring conditions, classify factors and find patterns that are difficult for people to see across large datasets.

The value of that analysis will still depend on the information entering the system.

If investigations repeatedly stop at the immediate action, AI can become extremely efficient at analysing immediate actions. If organisational conditions are rarely explored or poorly recorded, a language model cannot reliably reconstruct information that was never captured.

The opportunity, then, is bigger than using AI to analyse incidents.

AI could also help organisations examine the quality of their investigation process.

An organisation could ask which types of factors investigators identify most frequently, which ones rarely appear and whether particular investigation teams consistently stop at Individual or Team Actions. It could examine whether Task or Environmental Conditions are being explored properly, whether Organisational Factors are supported by evidence and whether recommendations can be traced back to established findings.

Those patterns could tell us something important about the investigation system itself.

Better data starts with better human thinking

The most useful connection between this research and ICAM is not really about artificial intelligence.

It is about the quality of human thinking before the technology becomes involved.

Good investigations require investigators to resist premature conclusions, separate evidence from assumption and reconstruct what actually happened. They need to understand the controls surrounding the work and explore the conditions that influenced people's actions.

PEEPO helps broaden the evidence search. Timeline reconstruction helps establish sequence. ICAM provides the structure for examining failed or absent controls, actions, workplace conditions and organisational influences.

AI may eventually help us analyse that information at a scale humans cannot match.

But the first job remains ours.

Because you cannot learn from a Contributing Factor that nobody thought to investigate.

Reference: Tiikkaja, M., Kannisto, H., Nurmi, A., Puro, V., Heikkilä, T., Kivimäki, I., Asikainen, I. & Teperi, A-M., Using AI to support safety management: Analysing occupational safety data using machine learning within the framework of human factors, Safety Science, article 107434. 

READ THE FULL SAFETY SCIENCE ARTICLE AT- CLICK HERE