When research gets it wrong
Have you ever read an article that claimed something about “evangelicals” that didn’t match the description of any Evangelical you’ve ever met?
The Centre for Research on Church and Faith is the ministry research arm of The Evangelical Fellowship of Canada (EFC), and we monitor trends among Canadian Evangelicals. We recently researched the popular ways that North American pollsters find Evangelicals on their surveys so we could understand who is being represented when researchers make their claims. We were disappointed with what we found. And it should disappoint you too.
One research organization excluded entire people groups based on race from their “evangelical” sample. Another popular polling group counted Catholics and Mormons, groups that are historically and doctrinally at odds with evangelicalism, in their “evangelical” sample.
Research shapes what people think. And if we get that wrong, people will get wrong information, which can lead to misrepresentation, mischaracterization and misunderstanding. Understanding and representing Evangelicals well helps us present an accurate and true picture in our research, but also to outgroup onlookers. But how do we even start to find true Evangelicals?
There are three main ways that researchers try to identify Evangelicals through survey research:
- They ask them if they’re Evangelical.
- They ask them if they’re a member of a denomination that is Evangelical (Evangelical affiliate).
- They look for certain markers of belief and behavior that are associated with Evangelicals (evangelically aligned).
Each of these strategies on their own presents various strengths and weaknesses.
The first strategy is appealing because it’s simple. The question doesn’t take up a lot of space on a survey or mental effort on the part of the respondent. But the apparent strength of imprecise questioning means equally imprecise results. In fact, one researcher in the United States found that Muslims, Buddhists, and Hindus were increasingly identifying as Evangelical. Individuals clearly have very different ideas about what “evangelical” means, and that’s not helpful for an organization that claims to operate on their behalf. We’re also aware of trends within Canada where individuals who affirm historically evangelical beliefs and exhibit typical evangelical behaviours are hesitant to use the term “evangelical” to describe themselves. So, we are wary of claims about Evangelicals that use a self-disclosure method alone to identify their sample. We need some sort of objective standard.
The second way to identify Evangelicals is to see if respondents affiliate with denominations that trace their histories back to the eighteenth-century Great Awakenings, which spurred the Evangelical movement. We count the number of people that associate with Evangelical denominations and call them Evangelicals. Easy enough, right?
This approach works well when a given denomination proudly and publicly claims its evangelical history and identity. But many Evangelicals belong to independent congregations with no formal denominational affiliation. And Evangelicals can often be found in denominations considered to be outside of the evangelical movement. Finally, recent research with British and Canadian Evangelicals is showing that it cannot be taken for granted that individuals who associate with a denomination adhere to its doctrine or conform to its expected codes of behavior. Affiliating with a denomination alone does not give us a full picture.
The final method is to poll for various beliefs and behaviours that are unique to Evangelicals (what the figure below will refer to as “evangelically aligned”). This solves many of the problems that the previous two methods presented, mostly because it’s rooted more objectively in how people think and act, rather than how closely they identify with a label or denomination. These characteristically evangelical beliefs and behaviours are usually developed in consultation with historians and theologians who study evangelical doctrines and priorities.
Let’s take a look at these methods in action. The chart below is from a 2023 survey from Angus Reid Group. It shows the intersection of “evangelical” populations using each of the methods described above:
Although each method finds 7%–9% of the population, less than half of these three populations intersect. There are those who self-identify as Evangelical (in red) who do not affiliate with an evangelical denomination, nor do they share the beliefs and behaviors of Evangelicals. Asking about beliefs and behaviours (in blue) finds many Canadians who are not affiliated with an Evangelical denomination (usually Catholic, Mainline Protestant, Latter-Day Saints or Jehovah’s Witnesses). And to the extent that these questions are meant to measure evangelical beliefs and behaviours, there are many evangelical affiliates (in green) that don’t share the beliefs and behaviours!
This chart shows that there is substantial difference in the “evangelical” populations found by different methods. This means that comparative representations of Evangelicals are often misleading, and potentially false. But the methodology demonstrated in the chart above is still largely in use today to find Canadian evangelicals.
Survey methodology might not sound like a riveting or pressing issue, but there are important implications.
First, research shapes public perceptions about who Evangelicals are and what they think. We should be concerned when these perceptions are being shaped by inaccurate data.
Secondly, you might not be the one who is taking the survey, but the people in your constituencies and congregations are. The results speak to a crisis of identity for Evangelicals and those who are evangelically aligned. We’ve lost a sense of unity in collective belief, practice and history. This makes it difficult for us as researchers to collect accurate data. And it makes it difficult for you, as ministry leaders, to make data-informed decisions.
Our methods for finding Evangelicals are only as good as our ability to identify the population we want to find. We think we need to keep refining our tools and keep learning from our population so that we can understand what is changing and why. Our hope is that in 2025, we will be able to convene a consultation with historians, theologians, sociologists, and survey researchers to improve our methods. Research in its current state presents a tension for evangelical Christians that researchers should feel no less keenly: If Evangelicals as a research population do not recognize themselves in research findings, have we really found Evangelical Christians?
Lindsay Callaway
Researcher for the Centre for Research on Church and Faith, the ministry research arm of the The Evangelical Fellowship of Canada.
