The figures below show how Democratic and Republican presidents use moral language when speaking about different topics. For example, the first figure shows the proportion of statements that use one of the “Harm/Care” words (see my earlier post for more on the data and methods used here) for each statement. It is no big surprise that both parties are use these words very often when speaking about health issues. Moving down the figure, we can see that Democrats are much more likely to draw on “Harm/Care” language when speaking about the environment than are Republicans. Neither party uses “Harm/Care” rhetoric often when speaking about education.

Harm/Care Foundation

Fairness/Reciprocity Foundation

Ingroup/Loyalty Foundation

Authority/Respect Foundation

Purity/Sanctity Foundation

Several interesting differences between the parties appear when we break out the data by issue. In my next post, I will look more closely at the substance of the differences between the parties.

*[UPDATE] I neglected to credit James Keirstead whose code I liberally borrowed from in constructing the figures above. See this post at his blog for more.

Why focus on SotU speeches? First, the SotU provides modern presidents with an opportunity to lay out their legislative priorities. While political scientists have reached different conclusions as to the actual impact of the speech, several studies have found substantive effects. Hoffman and Howard’s Addressing the State of the Union (2006) finds that presidents achieve about 40 percent of the policy goals they outline in the SotU. The speech serves as a signal as to the priorities of the administration, but more importantly for my purposes, it gives the president the opportunity to frame the debate in favorable terms. This framing is often done by appealing to basic moral values.

A second and by no means secondary consideration for focusing on this particular speech deals with the ever pressing concern for data availability. The good people at the Policy Agendas Project (http://policyagendas.org/) have generously made their comprehensive datasets available. On the State of the Union addresses, they have coded each statement in the speech as belonging to one of about twenty different policy areas. Combined with the Moral Foundations dictionary available on Jon Haidt’s website (here), moving forward into analysis is a relatively painless process.

One of my key expectations going into this data exercise is that Republicans and Democrats will emphasize different moral foundations. A portion of this variance will be due to their focus on different policies. Political scientists have long known that each of the major parties is seen to “own” a particular set issues of issues in the mind of the voter (e.g., Democrats are trusted more with relation to social welfare programs and Republicans have traditionally been perceived to be better at handling foreign policy issues).* It is also probably true that certain moral appeals are just harder to make (for example, it might be difficult to credibly frame an appeal to increase spending on transportation infrastructure in terms of the authority foundation). To the extent that partisans gravitate to the issues that their parties own and these issues lend themselves to a certain kind of framing, we would expect to see differences in the moral appeals of Republicans and Democrats as a function of the subjects that they talk about. But, I would also expect Republicans and Democrats to differ in terms of their emphasis of moral foundations even after controlling in some sense for the particular policy they choose to focus on.

In future posts, I will look more directly at the way in which the different parties talk about different policy arenas. For this post, I want to just give the broad outlines of the data.

Using the Moral Foundations Dictionary (referenced above), I coded (or rather I had the computer code) each statement for whether or not it included one or more morally charged words. Of the 18,854 statements listed in the Policy Agendas dataset (which includes SotU speeches from 1948 to 2005), 3,378 (just under 18 percent) included one or more of the words associated with the moral foundations.

The table below breaks out the data by issue area. The cell entries are rankings (1-20) for the proportion of statements in that particular issue area that refer to one of the moral foundations. For example, Law/Crime ranks 3rd in the Harm/Care foundation. Statements made concerning law and order were much more likely to use language drawing on concerns for harm and care than those dealing with science and technology (which ranked 19th overall in the Harm/Care foundation). The last two columns present the proportion of statements using any of the words from the moral foundations dictionray and the total number of statements included in the dataset on each topic.

The table is sorted on proportion of statements using moral language. This gives a (very) rough sense for the degree to which presidents choose morally charged rhetoric when speaking on each topic. Health, Civil Rights, Law/Crime, and Labor/Employment issues are much more likely to be spoken about in moral terms than Transportation, Energy, Education, and Science/Technology.

Another way to look at these data is to examine the trends over time.This first figure shows the overall use of moral foundations words (don’t make too much of the exact divisions between the presidents as these were added by hand — in the figures that follow the divisions between the presidents are more precisely delimited).

The figures below show the proportion of statements that included words found in the moral foundations dictionary broken out for each of the five moral foundations separately between the period from 1948 to 2005.

One of the most striking things about these figures, from my point of view, is the lack of clear patterns based on partisanship. For several of the foundations, the secular trend seems to be more significant than the partisan differences (for example, the general increasing use of Ingroup language from the 1960s to the mid-1990s or the rapid decrease in Fairness language from Carter through Clinton).

There are several things that these simple trend lines miss, and in the coming posts I will drill down deeper into the data in an effort to better understand how American presidents use moral rhetoric in pursuit of their policy goals.

* For more on the theory of issue ownership, see John Petrocik’s work: http://www.jstor.org/stable/2111797

Source: U.S. Census Bureau, Historical Income Tables—Households, Table H-3, Mean Household Income Received by Each Fifth and Top 5 Percent All Races: 1967 to 2006 (2006 Dollars)

Fortunately, the Knowledge Networks panel study (referred to elsewhere) included an item asking individuals what they felt should be done about the gap. After describing the size of the difference between the top earners and those on the bottom, respondents were asked, “Should this difference be smaller, bigger, or about what it is now?” For the purposes of the analysis that follows, I combined the few respondents that indicated the gap should be bigger (only about 5 percent of the sample) with those who said it should remain the same (about 30 percent).

I ran a statistical model* that predicts the probability that an individual would say that the gap should be smaller (without any specifics about how this would be accomplished, but more on that later). Even after controlling for ideology and party identification, three of the moral foundations are statistically significant and substantively important to the probability of acknowledging the gap as a problem. Increasing the two liberal foundations (Harm/Care and Fairness/Reciprocity) increases the probability of wanting to narrow the gap. Increasing the Ingroup/Loyalty foundation decreases the probability. The effects are shown in the Figure below.

In each panel, I’ve graphed the effect of moving across the range of each foundation on the likelihood of saying that the gap between the rich and poor should be smaller for a hypothetical individual who is a moderate Democrat (in Blue) or Republican (in Red) with income in the $50,000-$85,000 range who has average scores on all of the other moral foundations scores. Within each panel, the individuals are similar in every regard except for their party identification. The figure reveals a persistent partisan gap even after controlling for the moral foundations and ideology, but the gap between partisans with the same scores on the moral foundations is nowhere so large as the gap within each party across the ranges of the foundations listed above. The Authority and Purity foundations were not significantly related to attitudes about the income gap.

We know, however, that the foundations tend to move together (see this discussion for an example). Individuals who score high on Harm also tend to score high on Fairness. The figures above are interesting, but in some ways the “all else equal” assumption that they impose on the relationship between attitudes and the moral foundations is not as straightforward as the clean looking lines suggest. In the table below, I show some more probable combinations of scores. The entries in the table show the predicted change in probability from the baseline case described above. The changes in the foundations are modest (a one point increase or decrease from the baseline case described above for the “high” and “low” figures respectively).

So far, we have seen how increases in the Harm/Care and Fairness/Reciprocity foundations serve to increase concern about income inequality, while the Ingroup/Loyalty foundation decreases concern. That the liberal foundations should increase the likelihood of considering large disparities in income is not especially surprising in itself. However, I was surprised that the effects of the moral foundation scores are substantially larger than partisanship and ideology (the prime movers in most political science literature). Earlier work done by Felicia Pratto and her colleagues on the relationship between social dominance orientation and merit-based versus needs-based allocation of resources (see this JSTOR link for more) suggests why these particular foundations might be important (maybe the psychologists can back me up on this…).

Understanding the factors that lead to one acknowledging that income inequality is a problem that should be solved is only part of the bigger question. A much stickier issue is determining a politically feasible way of narrowing that gap. The recent debate over extending the Bush tax cuts illustrates the powerful emotions and interests that are mobilized when real money is on the table. Both sides, it seems to me, attempt to frame the issue as one of harm and fairness. The right argues that tax raises on the wealthy unjustly punish success. The left argues that it is only fair that those who have benefited so much from the system established by government should pay a little more to support it and those who are hurt by it.

The same Knowledge Networks data included an attitude item asking whether the respondent would support raising taxes on those who make more than $200,000 a year. About half of the sample indicated that they would support raising taxes on the wealthy.

The most powerful relationship that emerged between attitudes about taxes and the moral foundations (indeed the only significant relationship) was found in the Harm/Care foundation. The figure below shows this relationship over the range of the Harm foundation. Even after controlling for party identification and ideological self-placement, income, and the other foundations, the tax issue emerged as an issue of caring rather than equity or fairness.

The figure below shows a partisan differential that persists even after controlling for all of the above factors. However the difference between partisans is nowhere as large as the difference between individuals who score highly on the Harm/Care foundation and those who have low score on that foundation.

The Harm/Care foundation appears to be a more important factor in determining one’s support for raising taxes on the wealthy than party identification or ideological self-placement. Indeed, as the figure shows, a Republican who scores highly on the Harm foundation has a higher probability of supporting taxes on the wealthy than a similarly situated Democrat with a low score.

Several interesting questions are suggested by this brief exploration of the relatively limited selection of items touching on income inequality available to us in this dataset. First, what role does issue framing play in activating certain moral considerations over others? Would the conservative frame described briefly above change the relationship between the Harm foundation and attitudes about taxes? What about the liberal frame? This should be easy enough to test once we identify the relevant frames.

Second, how do the moral foundations relate to other potential remedies for economic inequality. The range of policy options is wide, and, depending on the moral prism through which one looks at them, reactions are sure to vary. Estate taxes, minimum wage laws, maximum wage laws, changes to the tax code, and repealing the sales tax on food and other necessities all might be met with different reactions from individuals with who emphasize different moral foundations. This would be a little trickier to test as it would require coming up with neutral descriptions of fairly complex and unfamiliar policies.

Finally, how much does where you stand on the issues of economic inequality depend on where you sit in the relative distribution of wealth? Psychologists don’t seem to talk much about social class and other kinds of vulgar economic considerations, but they surely play a role. The poor and the rich probably diverge in their attitudes about redistributory policies for reasons quite apart from their morality. This might be the most difficult problem to address from the researcher’s standpoint, as it would require collecting data from a broad enough cross-section of the income distribution. We survey researchers generally have the most success in the middle of the distribution with response rates falling off rapidly toward either extreme.

*I ran a logit regression with controls for Democratic party affiliation, Liberal political identification, income terciles, and the moral foundations scores.

One way to check the validity of these measures is to apply them to real world data. The recent 2010 midterm elections provide an excellent opportunity to do this.

The Guardian generously posted a downloadable version of the 2010 midterm election results on their website. For the following analyses, I will use the estimated moral foundation score for each congressional district derived from the YourMorals data.

The first thing to examine is the simple bivariate relationship between the Republican share of the vote in 2010 and each of the moral foundations.* The figure below displays these relationships. On the y-axis we have the percent of the vote won by the Republican candidate, and on the x-axis we have the estimated moral foundation score of the congressional district. In the upper-right corner, I’ve displayed the simple correlation coefficient between the two variables, and the thick red line shows the best linear fit.

This figure looks very similar to one I posted earlier detailing the relationship between the vote for Obama in the 2008 elections and the moral foundations. Indeed given the strong autocorrelation from year to year in election results, this is no real surprise.

The most important changes in election results from year to year tend to be the across-the-board shifts toward one party or the other (for more on this general trend, see Eric McGhee’s cogent analysis at the Monkey Cage).

To what extent can the moral foundations data explain this shift? After some exploration of the raw data, it became clear that the area to focus on was the seats held by Democrats. For the remaining analysis, I’ll restrict the data to the 257 seats that the Democrats held prior to the 2010 elections. For the dependent variable, I will use the difference between the vote for John McCain in 2008 in a particular district and the vote for the Republican House candidate.

Overall, among seats held by the Democratic Party, there was a two-point shift in the vote toward the Republicans. The figure below shows the distribution of this shift over the partisan voting index (PVI) of the district (the PVI is the average deviation of the district from the national presidential vote in the last three elections—for example, a district with a PVI of -10 votes ten points more Democratic on average than the average vote in the country).

Districts that fell on the left side of the figure (those with large negative values of the PVI), were generally safe Democratic seats. Those that are located toward the center and the right, were in more danger. We can see that the shifts were greatest near in these most marginal seats.

To examine the possible explanations of this shift, I looked at how the relationship between the PVI and the shift in the Republican vote in 2010 changed based the moral foundations. I divided all of the districts into two categories for each of the moral foundations. Those that were above the median score on a foundation were labeled as “high” on that particular foundation and the remaining were labeled “low.” The figure below shows the how the relationship between a district’s PVI score and the shift toward the Republican party changed when we break the data down further into these different foundations. The dashed lines show the relationship in districts that scored above the median on a particular foundations, and the solid lines show those that scored below the median.

There are several interesting patterns to note here. First, there did not seem to be any differential effects in the safest Democratic districts. At a PVI of -15, the data show the general shift toward the Republican Party discussed above (about 2 points more Republican). However, as we move toward the right-hand side of the PVI scale, toward the marginal seats and those that actually go for Republicans in their presidential voting, the gap between the high and low scoring foundations is largest. Most interestingly, it is those districts that scored highest on the liberal foundations (Harm and Fairness) that showed the biggest shift toward the Republicans in 2010. None of these relationships hold up in districts held by the Republican Party. It is something unique to the campaign against incumbent Democrats.

Finally, no discussion of the 2010 midterms would be complete without some mention of the Tea Party. The New York Times compiled a list of all of the Tea Party candidates (see here) in the 2010 elections. Of the 129 candidates that the Times identified, 120 of them ran in districts that the Democrats held in 2008. This represents almost 47% of all the races in districts where the Democrats had control in 2008.

What explains the emergence of Tea Party candidates in general elections?** The figure below shows how the probability of seeing a Tea Party challenger increases over the PVI (again, analysis was restricted to only those 257 seats that were held by Democrats in 2008). As we would expect, the probability increases as we move toward the more competitive districts.***

The most powerful relationship was seen in the districts that scored above the median on the Fairness foundation. Districts that scored high on the Fairness foundation and were competitive had a much greater probability of seeing a Tea Party challenger than those others. Smaller relationships were seen for the other foundations. Interestingly, the foundations did not seem to affect how well the Tea Party candidates did once in the race, only their probability of entering.

* In all of these analyses it is important to keep a few things in mind. First, the district level foundation scores were computed based on the entire voting age population of each congressional district. A big part of the story in 2010 was the “enthusiasm gap” between Democrats and Republicans. These analyses do not make adjustments for the differential turnout. Second (and this is somewhat related to the first point), the district level estimates for the moral foundations are aggregate measures. It would be inappropriate to infer individual level behavior from these district level statistics (this is known as the “ecological inference fallacy”).

** It would probably be most appropriate to examine the emergence and success of Tea Party candidates in Republican primaries. If we had a good measure of the moral foundations of Republican primary voters in 2010, this would make a fascinating analysis. However, we only have a measure of the district’s score on the moral foundations as a whole. At present, it is not possible to break this out by party.

*** The plots show predicted probabilities from similarly specified logistic regressions. The regression equation took the form,

Logit(P(Tea Party Challenger) )= a + b(1)*PVI + b(2)*High Foundation + b(3)*PVI*Foundation

I’ve recently been spending a lot of time thinking about the idea that we can locate individuals in a moral “space.” If you’ve read much about moral foundations theory, you are probably already familiar with the equalizer metaphor. The basic idea is that individual differences in moral considerations can be thought of as different settings on a metaphorical moral equalizer. Moral foundations theory is based on five dimensions of morality: Harm, Fairness, In-group, Authority, and Purity.

A fully specified spatial model of morality would describe individuals as occupying a particular point in a five-dimensional space. Some of the regions of this five-dimensional space would be sparsely populated. Indeed, research on the moral foundations has found that some of the foundations seem to “go together.” For example, individuals who are high on the Harm foundation also tend to be high on Fairness.

Ravi’s post made me wonder if moderates are more likely to live in these lower density areas of the moral space. Maybe they register as moderates because they have several conflicting considerations. Political scientists long recognized that individuals hold many different opinions in their heads at a single time and have shown that this attitudinal ambivalence has measurable consequences for political activity.*

If we think that political elites in the United States are forced to create a one-dimensional (“left-right”) policy space out of the diversity of value structures that exist in the public, it seems natural that they would, by sheer trial and error, identify the most populous regions of the value space. Due to institutional forces in the United States, there is not always going to be a clear home for individuals who live in the moral hinterlands.

I have a really difficult time thinking in five dimensions, and the dataset I am working with doesn’t have enough cases to support much beyond three dimensions. In the graphs that follow, I’ve collapsed the Harm and Fairness foundations into a single category (labeled HF). Similarly, the In-group and Authority foundations form the IA dimension. I’ve left purity separate.

The figure above shows this hypothetical space. The lower right-hand corner would be the place where individuals who have low scores on all three of my dimensions. As one moves to the right, scores on the HF dimension increase (the place for individuals who value Harm/Fairness). Moving up the vertical axis corresponds with higher purity scores. Moving along the diagonal axis varies the In-group/Authority dimension.

Using fancy statistical techniques** and data from Knowledge Networks (which I’ve described a little elsewhere), I estimated a model of political moderation as a function of these three dimensions. We can then consult the model and see where in the space individuals have the highest probability of identifying as a political moderate. Think of the figures below as cross sections of the 3-D moral space displayed in the figure above. First, let’s take a slice from the middle of the cube. Holding the Purity dimension constant at its mean value, what does the landscape look like?

The shadings on the graph represent the predicted probability of identifying as a moderate given an individual’s coordinates in the moral space. Notice the two peaks in the graph (the darkest shaded regions). These occur in the regions where we would imagine the most conflicted individuals reside. Individuals who are high on the HF dimension and high on the IA dimension (as well as those who are low on both dimensions) are most likely to identify as moderate. When attitudes come into the “right” alignment (high on HF, low on IA or vice versa) individuals are least likely to identify as moderate.

Here is a similar picture looking at the interrelationship between HF and Purity (this time holding IA constant at its mean).

And finally, IA and Purity (holding HF constant):

In each case we see a ridge running through the middle of the space where individuals who don’t “fit” well into the existing value coalitions of American liberals and conservatives reside.

*see, for example, Jennifer Hochschild’s work on ambivalence or Diana Mutz’s work on deliberative versus participatory citizens. Most recently, Shawn Treier and Sunshine Hillygus have shown that ideology falls along two dimensions (social and economic) and conflicted individuals are most likely to identify as “moderate” or give a “don’t know” response to the ideology question.

**I’m using a generalized additive model (GAM) to model moderate ideology as a completely non-parametric function of the scores along the three dimensions. As they are non-parametric, GAMs don’t impose any functional form onto the data. In a normal regression context, the effects of each of the dimensions are constrained to be linear. This is problematic for this kind of analysis.

In my previous posts, I’ve  discussed the many potential difficulties in using an entirely self-selected internet sample for inferences about general population parameters (whether or not a particular state or congressional district scores higher than another in terms of its moral foundations) as opposed to the intra-individual comparisons that are the bread and butter of psychologists (like how the foundations tend to correlate with ideology). I think that I have shown that the raw data are unsuitable for talking about the general population. The sample is demographically unrepresentative (see here) and somewhat attitudinally unrepresentative (see here).

What we need is a method to correct for the biases in the sample. Enter Multilevel Regression with Poststratification (or Mr. P as he is affectionately known to statisticians).*

[For those uninterested in the technical aspects of modeling, skip down to the maps below]

Multilevel Regression with Poststratification proceeds in (basically) three steps. First ,we construct a model to obtain the expected values of the variable of interest as a function of variables that we know the underlying population values for (typically this means only items that show up in the Census – geography, age, education, income, race, gender, maybe a few others).

Second, we use the model to predict the expected value of the variable of interest for each combination of variables (or cells) in the model. For example, if the model used four regions (Northeast, South, Midwest, West), three categories of age (18-30, 31-60, 61+), three categories of education (HS, College, Graduate), two categories of income (Less than $50k, More than $50k), two categories of race (white, non-white), and gender as predictors, there would be a total of 4*3*3*2*2*2 = 288 cells. From a 18 year old male with a HS degree or less who makes less than $50k is white and lives in Maine to a 75 year old female with a PhD making more than $50k is Asian and lives in New Mexico.  All individuals in the sample fit into one (and only one) of the cells defined by the combinations of predictors in the model. Many of the cells will be empty. In cells where there is no data, the model borrows statistical power from the other cells to come up with an expected value for every cell.

Finally, we weight each of the estimated cell values by the proportion of individuals in the population to come up with predictions for the geographic regions of interest. In this case, we would have predictions for each region.**

The map above (click on the image for a better view) plots the predictions from the MRP estimates of the difference between the liberal and conservative foundations for each congressional district. Districts in which YourMorals users valued the two “liberal” foundations (H and F) more than the three conservative foundations (I,A,P) are shown in dark green, and those districts cluster in the regions that we know to be the most liberal parts of the country: the North East, and the West Coast (excluding the agricultural parts of California). The districts within which YourMorals users gave the most conservative pattern (IAP > HF) are shown in red, and these districts fall overwhelmingly within the South.

One way to test the validity of these measures of district level foundations is to compare them to observable characteristics of the districts. One ready comparison we can make is to the district’s share of the vote for Obama in the 2008 presidential election.

The figure below shows the simple bivariate relationship between each foundation and vote for Obama. In every case, we see the expected relationships. Districts that scored more highly on the Harm and Fairness foundations were more likely to go for Obama in the election. On the other hand, there is a strong negative relationship between a district’s score on the Purity foundation and its vote for Obama.

Interestingly in many cases (especially the harm and authority foundations), there appears to be a significant “kink” in the fitted line at the midpoint of some of the foundations. Districts that score highly on the harm foundation (or those with low scores on the authority foundation) are associated with increased showings for Obama, but that relationship dissipates after the mid point of the foundation is crossed (all foundation scores are measured in standard deviation units). This is an aspect of the data that deserves further attention, but I’ve run out of time and space to do so here.

Multiple regression estimates confirm the overall story we see here in the bivariate plots. The estimated foundation scores seem to meaningfully correlate with real world phenomena. This bodes well for the validity of the measure and method.***

So, with a little work, it appears as if we can have our cake and eat it too when it comes to the YourMorals data. Scores on the foundations (after adjusting for the biases in the sample) are significantly related to district voting behavior.

*For a more detailed explanation of the methods involved see here. Also, Andrew Gelman and Jennifer Hill’s excellent book, Data Analysis using Regression and Multilevel/Hierarchical Models.

**I made an editorial decision not to include the details of the model and etc. as it didn’t seem of general interest. I’m more than happy to talk about it, but the post was getting wordy as it was.

*** The careful reader might well protest that the relationship we see in the figures presented is merely the product of the correlations with demography picked up from the MRP method. Since I used district demographics to adjust the scores obtained from the convenience sample, is it possible that the positive findings are simply a reflection of secondary correlations between the moral foundation scores and demographics? One easy way to test for this is to include demographic controls in the model. I was happy to see that none of the findings are substantially changed by including district demographics on the right hand side of the regression.

Last time, I gave a broad overview of the descriptive representation of the YourMorals dataset. In a nutshell, we discovered that the YourMorals respondents were much more educated, more likely to self-identify as liberal, and more likely to be white than the population.

In this post, I will explore the question of whether the YourMorals respondents are representative of the population after we condition on observable characteristics. Put another way, would we expect two individuals, one randomly chosen from the population and one drawn from the YourMorals data, who share all the same demographic characteristics (age, race, education, political ideology, place of residence) to look the same in terms of their scores on the Moral Foundations Questionnaire?

To conduct this kind of analysis, first we need a benchmark against which to compare the YourMorals data. As I mentioned in my previous post, the gold standard is a randomly drawn sample from the population. Luckily, we have just such a survey. Prior to the 2008 election, Knowledge Networks* fielded a version of the Moral Foundations Questionnaire to a representative sample of the U.S. population. This provides a good point of comparison for our (much larger) convenience sample.

The first task is to process the YourMorals data so that it looks more like the general population. I used a basic sample matching technique to match individuals from the YourMorals data and the Knowledge Networks data. This is a crude technique, but effective. Basically for each individual in the Knowledge Networks sample (the “match target”), I found an individual (or individuals) in the YourMorals data that matched the demographic information for the “match target.” These cases then become the comparison group. After the samples have been balanced in terms of observable characteristics, any differences we observe between the two can be ascribed to the compounding factors that we cannot observe.**

The following figures show how the distributions of the matched YourMorals data compares with the distributions in the sample from Knowledge Networks. The dashed lines show the distribution for Knowledge Networks, the solid lines represent the YourMorals data.

The distributions of the foundations in the two data sources look very similar for the Fairness/Reciprocity foundation, but for all of the others, there are significant differences between the YourMorals and the Knowledge Networks respondents.

A little more digging reveals some interesting patterns. Splitting up the sample by ideology yields:

Liberals:

Conservatives:

Two of the foundations seem to stand out in these comparisons. Liberals in the YourMorals data are particularly low on the Purity foundation (when compared against liberals in the Knowledge Networks data), and conservatives from the YourMorals sample seem to score lower on the Harm foundation. In both cases, YourMorals liberals seem more like population liberals on the first two foundations (Harm and Fairness), and the conservatives in the sample seem more like population conservatives on the last two foundations (Authority and Purity). No matter how the data is cut, the YourMorals sample seems to score lower on the Ingroup foundation.

The comparisons between the general population sample and the convenience sample in this post raise some significant questions about the possibility of using the self-selected respondents in the YourMorals sample to make inferences about the population. These problems in the data are particularly evident in the Ingroup foundation, the purity foundation (for liberals), and the harm foundation (for conservatives).

As was the case with demographics, all is not lost. One last look at the data shows that again the foundations are more or less proportionally correct. Liberals score higher in on the Harm and Fairness foundations in relation to their scores on the other three, and conservatives show more or less equal scores across each of the foundations. The bar chart below shows the average scores of the foundations broken out by survey source (KN and YM for Knowledge Networks and YourMorals respectively) and ideology:

Next time, I’ll discuss how we might correct for some of these demographic and attitudinal biases in the data.

*For the uninitiated, Knowledge Networks is a survey research firm that has gone to great lengths to put together a panel of internet users that is nationally representative. They have recruited a large panel of individuals to take internet surveys. These individuals were generally contacted by telephone, and in cases where the respondent did not have internet access, Knowledge Networks provided access. See this link for more information.

**For a quick primer on the theory behind sample matching see this Wikipedia entry.  I am using exact matching on categories of age, race, education, ideology, and state of residence.

Earlier Jon Haidt discussed the “problem” of representativeness of the YourMorals data and concluded that it wasn’t such a problem after all. Convenience samples drawn from the internet can produce reliable data. This is particularly true when we are more interested in taking valid measurements than in painting a representative picture of some underlying population.

But what if we would also like to know something about the underlying population? If we had data that were representative of the country as a whole, we would be able to ask a new set of questions. Does knowing where the states fall in terms of their Moral Foundations tell us anything about voting behavior? We might expect scores on the purity foundation to explain state-level attitudes about gay marriage or the fairness foundation to explain attitudes about tax policy. To answer these kinds of questions, we need representative samples (also see Jesse Graham’s comment in the above link).

In sampling theory, the gold standard is the probability sample. When all individuals in the population have a known (but not necessarily equal) probability of being included in the sampling frame, we can construct reliable estimates of the population parameters and, given sufficient sample size, be confident that these estimates are within some distance of the true values in the population. However, the central assumptions of sampling theory are violated in convenience sampling (but see this discussion of the representation problems in traditional “random” sample polls).

First, we would like to get a sense of how the YourMorals data stacks up against other population measures. We collected data on several demographic characteristics of individuals in the YourMorals dataset. We can easily compare these against population values collected from the census or other representative samples.

One area where we can clearly see the representation problems in the YourMorals data is self-reported ideology. Considering only U.S. respondents for the time being (as all of the following analyses do), recent national samples put the proportion of people who consider themselves “liberal” at between 18 and 22 per cent. In the YourMorals data, this figure is nearly 65 percent.* Given this skew in the data, we might be hesitant in trying to make inferences about the general population from a sample that looks so much different.

The figures below show how the YourMorals data compares with the population values across a handful of demographic and attitudinal variables.

Source: Pew Center for the People and the Press, 2001-2008

This figure shows how even with a significant intercept shift (almost 50 points), the rank ordering of the states stays pretty close to the same. This is encouraging as it means we are not drawing the same type of individual from each state. Put differently, knowing the state that an individual resides in tells us something about the probability that he or she identifies as a liberal. What we would not want to see here would be a horizontal line (indicating no relationship).

Source: American Community Survey, 2006-2008

Source: American Community Survey, 2006-2008

With race it is much the same story as ideology. For whites, there is a substantial intercept shift (almost 70 points), but states with larger white populations also are proportionally more white in the YourMorals data. The data for African Americans is noisier (there were fewer than 900 in the sample of over 60,000), but shows the same pattern. Here there is not a large intercept shift (as we have reached the floor of the data), but we see the same kind of increasing pattern.

Source: American Community Survey, 2006-2008

With respect to education, the data are further afield. The figure shows that the YourMorals sample is significantly more educated than the general population, but it becomes more difficult to draw a convincing trend line through the data. Individuals who came from states with higher levels of education were only marginally more likely to be highly educated themselves.

So where does all of this leave us? It is obvious from the plots that the individuals who self-selected into the YourMorals data look very different than the general population. It would clearly be inappropriate to use the raw data in trying to make inferences about the general population parameters (average levels of a particular foundation in a particular state, for example). The sample is much more liberal, highly educated, and white than the general population. But it is not as bad as it could be. The worst-case scenario would show uniformly weird sample across the states. Instead, what we saw in the figures above is a picture that is more-or-less proportionally correct. It is encouraging that the general relationships hold up.

All of this is not to say that we should throw out the analyses presented elsewhere in this blog and in publications based on the YourMorals data. If we condition on ideology (which we saw was particularly skewed) and make statements like “Liberals generally score higher than conservatives on the Harm/Care and Fairness/Reciprocity foundations,” we are probably treading on safe ground.

In the next few posts, I will be revisiting the question of how to construct a representative picture from a convenience sample.

*Beyond the obvious sampling issues, there are a few other problems with directly comparing the measure of ideology in YourMorals with that in nationally representative samples. First, there is a mode difference that could account for some of the discrepancy (although certainly not all or even a very significant portion of it). Another (and more serious) difference between nationally representative samples and the YourMorals data is the choice of a seven point scale rather than a five point scale. Five point scales are used more regularly in telephone samples with the options being “Very Conservative,” “Conservative,” “Moderate,” “Liberal,” and “Very Liberal.” The YourMorals data includes options for “Slightly liberal” and “Slightly Conservative” as well as “Libertarian” and “other” categories. The 65 percent figure lumps all of the “liberals” together. If you believe that the “slightly liberal” respondents might have self-identified as “Moderate” given fewer options, the proportion turns out to be just over 50.