Surveys are poor tools for detecting religious extremists—and much else
A new Christian menace has emerged. In a recent article for The Atlantic, Stephanie McCrummen profiled the New Apostolic Reformation (NAR), a Christian movement purportedly bent on seizing the reins of power in American society, aiming to dismantle the country’s democratic institutions, and establishing a new era of Christian supremacy. The most alarming claim in the article is that “about 40 percent of American Christians . . . are embracing [this] alluring, charismatic movement that has little use for religious pluralism, individual rights, or constitutional democracy.” The number comes from a Denison University survey showing that 41% of Christians agree with the following statement: “God wants Christians to stand atop the ‘7 Mountains of Society’ (government, education, business, etc.).”
McCrummen’s claim provoked immediate skepticism from religious scholars Philip Jenkins and John Inazu, who argued the prospect that such a large swath of American Christians are committed religious extremists is highly implausible, and warned that we shouldn’t read too much into this agreement with the “7 Mountains” statement. Denison’s Paul Djupe shortly responded with a defense of the alarmist interpretation, showing that agreement with the “7 Mountains” statement overlaps with significant affirmation of a number of other statements that (purportedly) entail a dominionist worldview.
This kind of uncertainty is an inherent feature of survey research. By offering hard numbers, surveys provide a sense of precision in measuring various beliefs embedded in the American population. But survey methodologists have long recognized that this precision is an illusion—survey results carry a great deal of ambiguity, much of it hidden even from the researchers themselves. Below, I describe a few methodological challenges in survey research and explain why these should lead us to share Jenkins’ and Inazu’s skepticism of the Atlantic claim.
Challenge One: Fact or Artifact?
A major goal of survey research is to figure out what the people in a given population believe about some matter of interest to the researcher. The trouble: Most people don’t have well-defined opinions about most matters. People tend to be poorly informed, spend little time reflecting on “the issues,” and hold genuinely ambivalent views. And yet, most people will still offer an opinion when asked for one in a survey—sometimes even about entirely fictitious matters. They may want to be sporting, or they do not want to own up to their own ignorance. As a result, survey responses do not reflect salient and well-considered views for most of those surveyed. This is especially true when it comes to topics that are vague, complex, or of limited public interest.
So what goes into uninformed respondents’ decisions to answer survey questions? Some of this relates to features of the given survey. All else being equal, people will tend to agree with any proposition for the sake of being agreeable—a phenomenon known as “acquiescence bias” or “yes-saying.” This by itself inflates levels of affirmation for survey statements. People are also influenced by cues embedded in the wording of questions. For instance, they may be more likely to express support for a policy aimed at “helping families in need” than one “providing welfare assistance,” even if these phrases describe the exact same policy.
Survey responses are further influenced by the range of options provided for a question. To take a relevant example, consider survey research on the topic of “Christian nationalism.” In the Baylor Religion Survey, respondents were presented with the proposition, “The federal government should advocate Christian values,” a statement that 46% affirmed. But in another survey by Pew Research Center, respondents were presented with two statements to choose from: “The federal government should advocate Christian religious values,” vs. “The federal government should advocate for moral values that are shared by people of many faiths.” Given these options, only 13% chose the statement favoring Christianity. This is quite a dramatic difference, which shows how easily the appearance of support for Christian supremacy can be undermined.
These concerns fall into the category of survey “artifacts”: results that are produced by features of the survey itself rather than the public opinion it was intended to capture. Insofar as survey findings result from artifacts, they can tell us nothing about what goes on in the heads of our respondents. Unfortunately, there is no way to remove the influence of survey artifacts entirely, nor to determine the nature and extent of that influence with any certainty. This “unknown unknown” aspect of survey research should always lead us to be skeptical of strong claims based on patterns of survey responses.
Challenge Two: Responding from the Gut
But even after accounting for artifacts, survey responses surely tell us something about the beliefs of the American public. People do not always affirm the statements presented, and they cannot be responding totally at random. So what exactly do surveys reflect? When people do not have well-developed views on any given matter (which, again, is most of the time), their responses will reflect how much a survey item resonates with them.
To use a highly technical scientific term, they answer based on vibes—or, to be precise, sets of mental associations, broad cultural orientations, group memberships, and so on. For instance, if asked to rate Joe Biden’s handling of the Israel-Gaza conflict, a Republican respondent uninformed about the issue is likely to report disapproval. Such a response has nothing to do with Israel or Gaza. Rather, it is simply a way to signal “Boo Joe Biden!” Thus, survey responses typically should not be understood to represent deep and coherent convictions. They are better viewed as gut reactions that provide researchers with blurry impressions of people’s attitudes.
Given the ambiguity of people’s views and the powerful but unseen influence of survey artifacts, some social scientists have gone so far as to claim that there is no such thing as “public opinion.” What passes for it is manufactured by interested parties in order to tell a story or advance a social agenda. My view is that surveys can reveal something, but we should err heavily on the side of caution when speculating about just what that something is. The influence of the aims and biases of the people collecting and reporting on the data should also loom large in our minds.
With all this in mind, how should we evaluate claims about support for the New Apostolic Reformation based on the Denison survey? Given what we have seen about acquiescence bias and the influence of response options, at least some of the 41% support is presumably due to survey artifacts (though there is no way to know how much). Beyond this, it is difficult to guess what survey respondents might have made of the “7 Mountains” statement, which on the surface appears rather bizarre.
A handful of respondents may have had direct knowledge of the NAR connection and were expressing their approval of its extremist tenets, as McCrummen assumes. But given the public’s ignorance of most issues and the relative obscurity of NAR, our default assumption should be that respondents have no such knowledge and are answering based on vibes. Since the statement contains both a biblical reference and mention of Christians wielding influence in society, the most straightforward interpretation is that it signals something like “Yay Bible!” or “Yay Christians!” This is far more plausible than supposing that it represents an affirmative commitment to a thoroughgoing religious extremist political agenda.
In response to this sort of objection, Djupe argues that we can have some sense of what people “really meant” when affirming the “7 Mountains” statement, because we can look at whether they affirm other related statements to gain a sense of their broader worldview. Specifically, he shows that those who affirm the “7 Mountains” statement also tend to agree with the following: “The offenses listed in the Bible should be punishable by governments today through the courts,” and “There are demonic ‘principalities’ and ‘powers’ who control physical territory.” He takes this as an indication of more general support for dominionism, and thus as evidence for the alarmist interpretation in the Atlantic article.
Is this conclusion reasonable? The prospect of the Bible wielding influence in our laws is not much of a departure from the earlier proposition of Christians wielding influence in society, so the overlap between agreement with each statement is not surprising. The Bible statement also does not obviously connote Christian supremacy—after all, the Bible and the American legal code alike prohibit theft and murder. The “powers and principalities” statement is particularly difficult to interpret but is likely to resonate most strongly with respondents who know the New Testament.
Though the wording is evocative enough to excite the worst fears of the sorts of people whose knowledge of American religion comes from survey reports, it is unclear what the statement has to do with American democracy. Ultimately, these additional statements may also signal little more than a full-throated “Yay Bible!” and “Yay Christians!” There is a chasm between what we can plausibly conclude from affirmation of these statements and McCrummen’s claims of support for a religious extremist movement bent on conquering American society and tearing down its democratic institutions.
I want to be clear that in raising these concerns, I am not suggesting the Denison survey was anything other than a good-faith and methodologically competent effort to make sense of an important religious phenomenon. These challenges of interpretation are not the result of poor execution, but simply inherent in the survey method itself and the tradeoffs it entails.
The alarmist claims in the Atlantic follow a long-standing trend of using surveys to paint large swaths of the population as dangerous extremists, from the “authoritarian personality” research of the 1950’s through to “Christian nationalism” today. Surveys provide an aura of precision and scientific objectivity, easily manipulated in reporting to validate the fears and prejudices we already hold. The use of extreme or outlandish statements in surveys appears to confirm that our rivals really are the stuff of our worst nightmares. The presentation of numbers, decimals, and percentage signs lends an alluring sense of scientific authority to the perceived threat.
But survey responses do not usually mean what we think they mean. Surveys are generally incapable of detecting people’s deepest convictions or operational political goals. Most people simply do not have the kind of deeply held, coherent, and motivational beliefs we want to attribute to them on any given matter. Our survey findings are largely a function not of what is going on in people’s heads, but rather of the ways we steer respondents with our questionnaires. Surveys are helpful for providing impressionistic sketches of attitude patterns in the population, and this is a great deal better than nothing. They are poor tools for tallying committed religious extremists.
When McCrummen declares 40% of Christians to be “embracing an alluring, charismatic movement that has little use for religious pluralism, individual rights, or constitutional democracy,” then—and this is the crucial point—she is not so much reporting on the threat of the New Apostolic Reformation as she is aiding in its manufacture. The blueprint for the construction of this Christian menace is ready-made—it comes from decades of progressive fearmongering about the threat Christians pose to liberal democracy, readily adapted from the alarms over fundamentalism, theocracy, and Christian nationalism that came before. The repeated success of such constructions tells us more about the preoccupations of the people doing the counting than the ones being counted.
Jesse Smith is Assistant Professor of Sociology at Benedictine College. His research is focused on the intersection of family, religion, and politics in the modern United States.
Very useful, clearly written. Thanks.