Measuring Safety Observer Performance

In our last blog article by Dr. Timothy Ludwig, “The Anatomy of Pencil Whipping”, the idea of safety observer quality was introduced. This article, originally published in the VPPPA Leader magazine in 2012, further expands the subject by introducing quantitative measures of observer quality that can be used to both evaluate and improve the data collection process.

Employee involvement is a key component of most safety management systems, such as OSHA’s Voluntary Protection Program. However, many companies struggle to find meaningful ways to get every employee engaged. During my site visits as an OSHA Special Government Employee (SGE) I would often find employee involvement was constrained to small, fringe groups and typically comprised of only a fraction of the entire workforce. Over the years, I became a firm believer that the best and most meaningful way for employees to get involved is through participation in the site or company Hazard Recognition and Control program. Specifically this can be done by making workplace observations related to the health and safety program as it relates to behaviors, conditions, and/or safety activities. Our studies have shown this to be beneficial from an accident prevention perspective. However, once you begin to involve many people in the safety observation program, new issues arise such as observation quality. As such, I want to identify the benefits of adopting this program as well as introduce an effective way to both measure and improve observation performance.

Recently our company has done a study that found the following correlations related to inspections – a collection of workplace observations – and reduced incident rates. We call these our “safety truths”:

1. More inspections and observations result in a safer worksite
2. More inspectors, specifically more inspectors outside the safety function, result in a safer worksite
3. Too many “100% safe” inspections results in an unsafe worksite
4. Persistently high levels of unsafe observations results in an unsafe worksite

Clearly involving more observers increases the chances of identifying at-risk or unsafe behaviors or conditions so that they can be addressed proactively before an incident occurs. Remember that an observation is only a snapshot in time – no one person can be everywhere at all times to capture this. In addition, if only those with a safety function collect the information, then ownership of the safety process shifts to that of the safety department and not with the workers and the front-line supervisors where it needs to reside.

One of the reasons most companies are reluctant to engage workers and managers in the observation process is primarily due to perceived gaps in skill and/or knowledge that could lead to poor results. This is understandable; however, it is patently false. Every worker exposed to risk in the workplace should have a voice to speak the truth about how safe or unsafe things are. That being said, the more observers collecting data, the more ways biases can be introduced. Just like training workers on how to do their jobs or write and follow safety procedures, involvement in the observation process requires feedback. Engaging an observer in feedback allows those in the safety function to transition from the traditional safety “cop” role to that of being a coach. Through their experience, the safety team can express what is safe and at-risk, what is important in the worksite to focus safety observations on, or even share risk assessments so that everyone understands the impact of working safely. By measuring the strengths and weaknesses of the observer, individualized improvement plans can be developed and monitored over time such that everyone continually improves at their own pace.

Here is a great example: You have an observer who is highly engaged in the observation collection process. This person submits twice as many observation cards or inspection forms as the average worker at your worksite. However, this person rarely, if ever, finds anything at risk, despite the fact that the area this observer works averages 2-3 at-risk observations per inspection. Some would roll their eyes at the submitted data. Others would go so far as to label the observer as a “pencil-whipper”. How then can you improve this observer’s observation performance?

Individuals are evaluated in your workplace every day. In fact, I would be willing to bet that each of you reading this has been involved in an annual evaluation process with your manager. What did this this process look like? Typically it consists of these steps:

1. Expectations are communicated. What are your goals?
2. Constructive feedback is provided. Preferably this is done frequently throughout the year and not just at the evaluation!
3. Measure performance and progress against expectations
4. Communicate to the individual what is important to the organization. This traditionally involves a discussion around strengths to sustain and weaknesses to improve

The same can be done for the observation process provided there are expectations established and communicated. Here are some examples on establishing quality criteria around measurable metrics:

1. Participation – count of submitted cards/forms/inspections (average over a given time period)
2. Average count of safe and at-risk observations per inspection
3. Identified severity of at-risk observations
4. Comments included with observations
5. Frequency of 100% safe inspections
6. Focus on pertinent hazards/tasks
7. Closure of unresolved issues

If the only metric you track is the count of how many cards or forms are submitted, then your ability to provide feedback is tremendously diminished. However, as you develop your inspection strategy and plan around meaningful objectives, then you can quickly and easily identify areas of improvement and measure that progress over time. This is due to the fact that as the data set grows the metrics become normalized – averaged out over a large data set. As such, it becomes easier to spot outliers from the “norm” who can then be engaged for improvement.

With the example of the “pencil-whipper” above, many would point to the poor data collected as proof only safety folks should do safety inspections. What is not clear is why this person is doing what they are doing. By soliciting feedback and engaging in a respectful manner you may discover perhaps the person believed that is what was expected of them. Perhaps their manager scolded them for “making them look bad”. Even worse, perhaps workers had submitted at-risk findings that were ignored by management and now have a “why bother” attitude.

As this process is adopted, it changes the conversation of “did you do an inspection” to “let’s talk about your data and how we can improve what we see”. As this process continues, engagement becomes more meaningful, the data is trusted more, and there is less reluctance to act on the findings. As the results improve, more observations are collected which provides greater insight and visibility into the safety of the organization which leads to a reduction in incidents. Notice how the incident rate was never a focus but becomes a benefit to an improved observation process.