1) Create a Group. Sign in to YourMorals.org and then go to our “Group Creation” page, which you can also get to from near the top of the Explore page. From that page, create a group name. You’ll need to make it something unique, e.g., “Robertsons UVA ethics class” rather than “ethics class.” Spaces are allowed.

2) Send the link. You’ll get a unique URL – a link that you can forward to your group, to invite them to take surveys at YourMorals.org. (See below for an example of an email which you might modify for your purposes and send out.)

3) Pick a few studies. Anyone who creates an account using your group’s link will automatically see a special box at the top of the “explore” page, which says which surveys are the most popular for members of your group. At present, the only way you can get studies into that box is to take those studies yourself. (If you’re the person who created the group, then as soon as you complete a study, it becomes one of the most popular ones for your group).

4) Lead a discussion about the results. We’ll  soon post some ideas or lesson plans to help you or your group interpret and discuss your results.

—————————————————————–

[Sample email to send out to your group]

Dear class [or congregation, or team, or whatever]:

For our next meeting, I thought it would be interesting for us to take a few surveys online and then discuss the results. We’ll use the site www.YourMorals.org, which is a non-profit academic research site run by a group of social psychologists who study morality. Please click on this link:

[paste your group-link here]

That will take you to the site. You should then register to create an account. After you’ve filled out the demographic questions, the site will take you to the “explore” page. At the top of the page you’ll see a table with a few studies listed for our group. The studies I think would be most interesting for us are:

1) Moral Foundations Questionnaire

2) Sacredness Survey

3) Business Ethics Questionnaire

[These three are listed as examples. Most groups will want to suggest the Moral Foundations Questionnaire or else the Sacredness Survey, because those are the two studies that give you scores on the five basic foundations of morality. We will soon offer suggestions for sets of studies that may be of interest to various groups]

After you complete each study, you’ll get an explanation of the study which will include a graph showing your score, the scores of everyone in our group averaged together, and the scores of other people who have taken that study. Please be sure to print this page out and bring it to our meeting. Rest assured that your responses are anonymous. Nobody other than you will see your scores. The researchers at YourMorals.org will only use your data for scientific research, and will never be able to link your responses to your name or identity.

If you have already registered at YourMorals.org, before receiving this email, then you can add your existing account to our group by clicking on this link:

[paste in your add-link here, which was at the bottom of the page when you created your group]

We all witnessed the Tea Party’s hard-line position on the standoff leading up to the debt ceiling crisis — 68% of Tea Partiers wanted lawmakers to stand firm on their principles, even at the risk of government shut-down.  Some have argued that, along with their fiscally conservative values, their willingness to take such a stand, and their unwillingness to compromise, have become the defining features of the entire movement.  However, it is not entirely clear why Tea Partiers might be predisposed to these attitudes about compromise.  Although it is impossible to say that any one of the following variables caused or even contributed to any specific political behavior, it is nonetheless compelling to examine how a number of psychological variables might be related to Tea Partiers’ hard-line stance on compromise.

Most notably, Tea Party supporters are highly reactant, as measured on the Hong Reactance Scale. Reactance is an emotional resistance to the influence of others, and often manifests as defiance to attempted persuasion.  Our data show that Tea Party supporters express consistently high levels of this trait, much like libertarians.  They also show low levels of empathy, or the ability to share the feelings of others (much like conservatives and libertarians, as measured by the Interpersonal Reactivity Index).  Taken together, these two traits may preclude one from a willingness to compromise: a reactive person is highly motivated to disagree with threatening others, and a person who lacks empathy is unlikely to fully consider or appreciate his opponents’ point-of-view.

These findings could also be said of libertarians, but unlike libertarians, Tea Party supporters score low on the Need for Cognition Scale.  This scale measures the extent to which people engage in and enjoy effortful thinking.  Low levels of Need for Cognition are associated with heuristic thinking styles and a lower likelihood of discounting erroneous intuitions and judgments.  As a result, groups that rely less on deliberative thinking styles (i.e., groups with lower Need for Cognition) may be more steadfast in their intuitive convictions, and less receptive to reconsideration.

Another interesting finding is that Tea Party supporters are very sensitive to social desirability concerns, as measured by the Marlowe-Crowne Social Desirability Scale.  In other words, when presented with true-false questions about oneself that were either socially acceptable but unlikely, or socially unacceptable but likely, Tea Party supporters responded in the most self-promoting fashion.  Their results on the Balanced Inventory of Desirable Responding also showed relatively high scores on a related measure of self-deceptive enhancement, suggesting that these responses reflected internal beliefs, rather than intentionally over-reported ones.

Along the same lines, Tea Party supporters were also most likely to demonstrate the better-than-average effect.  That is, more than other groups, they reported possessing positive traits more than the average person, and negative traits less than the average person.  Although this effect is pervasive (e.g., over 93% of people report being above-average drivers), Tea Party supporters demonstrated the highest level of this bias compared to other political groups.

So what does this tell us about Tea Party psychology?  Tea Party supporters have a reactant and intuitive reasoning style, low levels of empathy, and they display a self-enhancing/over-confident style of evaluating themselves.  Could these psychological predispositions play an important role in Tea Partiers’ political behavior, particularly in their principled stands/resistance to compromise on their core values?

Although certainly possible, it would obviously be unwise and premature to claim a causal connection between these factors and any specific political behaviors.  Keep in mind that the analyses reported above were conducted with Tea Party supporters, rather than Tea Party participants, who show a slightly different pattern of results (not reported here).  In my next blog post, I’ll go into more detail about a number of other key predictors of Tea Party support that I believe can help inform our understanding of Tea Party psychology.  Stay tuned.

Why focus on SotU speeches? First, the SotU provides modern presidents with an opportunity to lay out their legislative priorities. While political scientists have reached different conclusions as to the actual impact of the speech, several studies have found substantive effects. Hoffman and Howard’s Addressing the State of the Union (2006) finds that presidents achieve about 40 percent of the policy goals they outline in the SotU. The speech serves as a signal as to the priorities of the administration, but more importantly for my purposes, it gives the president the opportunity to frame the debate in favorable terms. This framing is often done by appealing to basic moral values.

A second and by no means secondary consideration for focusing on this particular speech deals with the ever pressing concern for data availability. The good people at the Policy Agendas Project (http://policyagendas.org/) have generously made their comprehensive datasets available. On the State of the Union addresses, they have coded each statement in the speech as belonging to one of about twenty different policy areas. Combined with the Moral Foundations dictionary available on Jon Haidt’s website (here), moving forward into analysis is a relatively painless process.

One of my key expectations going into this data exercise is that Republicans and Democrats will emphasize different moral foundations. A portion of this variance will be due to their focus on different policies. Political scientists have long known that each of the major parties is seen to “own” a particular set issues of issues in the mind of the voter (e.g., Democrats are trusted more with relation to social welfare programs and Republicans have traditionally been perceived to be better at handling foreign policy issues).* It is also probably true that certain moral appeals are just harder to make (for example, it might be difficult to credibly frame an appeal to increase spending on transportation infrastructure in terms of the authority foundation). To the extent that partisans gravitate to the issues that their parties own and these issues lend themselves to a certain kind of framing, we would expect to see differences in the moral appeals of Republicans and Democrats as a function of the subjects that they talk about. But, I would also expect Republicans and Democrats to differ in terms of their emphasis of moral foundations even after controlling in some sense for the particular policy they choose to focus on.

In future posts, I will look more directly at the way in which the different parties talk about different policy arenas. For this post, I want to just give the broad outlines of the data.

Using the Moral Foundations Dictionary (referenced above), I coded (or rather I had the computer code) each statement for whether or not it included one or more morally charged words. Of the 18,854 statements listed in the Policy Agendas dataset (which includes SotU speeches from 1948 to 2005), 3,378 (just under 18 percent) included one or more of the words associated with the moral foundations.

The table below breaks out the data by issue area. The cell entries are rankings (1-20) for the proportion of statements in that particular issue area that refer to one of the moral foundations. For example, Law/Crime ranks 3rd in the Harm/Care foundation. Statements made concerning law and order were much more likely to use language drawing on concerns for harm and care than those dealing with science and technology (which ranked 19th overall in the Harm/Care foundation). The last two columns present the proportion of statements using any of the words from the moral foundations dictionray and the total number of statements included in the dataset on each topic.

The table is sorted on proportion of statements using moral language. This gives a (very) rough sense for the degree to which presidents choose morally charged rhetoric when speaking on each topic. Health, Civil Rights, Law/Crime, and Labor/Employment issues are much more likely to be spoken about in moral terms than Transportation, Energy, Education, and Science/Technology.

Another way to look at these data is to examine the trends over time.This first figure shows the overall use of moral foundations words (don’t make too much of the exact divisions between the presidents as these were added by hand — in the figures that follow the divisions between the presidents are more precisely delimited).

The figures below show the proportion of statements that included words found in the moral foundations dictionary broken out for each of the five moral foundations separately between the period from 1948 to 2005.

One of the most striking things about these figures, from my point of view, is the lack of clear patterns based on partisanship. For several of the foundations, the secular trend seems to be more significant than the partisan differences (for example, the general increasing use of Ingroup language from the 1960s to the mid-1990s or the rapid decrease in Fairness language from Carter through Clinton).

There are several things that these simple trend lines miss, and in the coming posts I will drill down deeper into the data in an effort to better understand how American presidents use moral rhetoric in pursuit of their policy goals.

* For more on the theory of issue ownership, see John Petrocik’s work: http://www.jstor.org/stable/2111797

When I challenged you to either 1) retract and apologize or 2) affirm your original post, I expected you to invent your own option 3, retracting or qualifying a few claims but not apologizing. Thank you for choosing option 2 so decisively!

I have three points to make in response, and then I will be done with our exchange.

1) When Bill Clinton said “there is no improper relationship” with Lewinsky, he spoke truthfully but deceptively. What the interviewer wanted to know was whether there HAD BEEN a relationship. You tell us that I was wrong to assume that you had not watched my talk before dismissing my argument. You say that in fact “I have indeed looked at it.” Did you mean to say “have” or “had?” If you meant “had,” then readers can better judge the quality of the reasoning in your initial post. Even though you knew my full argument, you dismissed it by focusing on one claim — that underrepresentation is not evidence of bias – which is a claim that I myself made in the talk. But if you really meant “have,” as in “I have now watched the talk, after writing my initial post but before writing this response,” then please explain how your statement differs from Clinton’s.

2) You take issue with my use of Google, and I thank your reader for pointing out to us the difference between the number of hits Google says it found vs. the number of pages it actually delivers. But my basic point still stands: Google finds many cases of “liberal social psychologist” which refer to actual psychologists, but just three hits for “conservative social psychologist,” none of which point to an actual psychologist. You say you found 10 cases of the latter phrase, but did you notice that seven of them referred to my talk?

Your more revealing error is your claim about the role that the Google example played in my argument. You said that my conclusion from it was “Voilà, case closed, bias demonstrated!” Yet anyone who watched the talk knows that this was just my opening example, intended to be humorous, but also intended as one of three demonstrations that it’s very hard to identify any conservative social psychologists, and that point itself was just one of three arguments that my field has become a tribal moral community. I’m no philosopher, but I’m pretty sure that “case opened” is not the same thing as “case closed.”

3) You deny that you have accused me of academic misconduct, and you warn me that I myself may have done something ethically wrong in “throwing those words around.” You say you were just raising a theoretical possibility of motivated cognition. I repeat your original words here:

I suspect that Haidt is either an incompetent psychologist (not likely) or is disingenuously saying the sort of things controversial enough to get him in the New York Times (more likely).

You also called me “silly” and you called my talk “garbage.” Readers will judge for themselves which of us writes recklessly and unprofessionally.

* * *

I have greatly enjoyed our debate. You assert that I “got upset” by your initial post, and you refer to my “outrage” at your accusations. But in fact I was delighted by them. I am an intuitionist. I am building an extended argument that reasoning, when not informed by broad understanding and cultivated intuitions, is an unreliable tool for finding truth. There is mounting evidence in psychology that the evolved function of reasoning is not discovery but social justification and manipulation. We humans use reasoning skillfully to find arguments in support of our intuitively held positions, but we are hobbled by the confirmation bias; we are unable to find evidence or arguments that contradict our favored positions. I believe this is the most serious defect in the writings of the “new atheists” and many other self-proclaimed rationalists: because they are so good at finding reasons to support their views about science and religion, they develop an extraordinary confidence that they are right, which makes them prone to arrogant dismissals of all who disagree with them.

When I issued my challenge to you, I knew that I would soon obtain either an apology or a classroom-worthy demonstration of rationalism in action.

Jonathan Haidt

—————————————————

p.s., If readers would like to see experimental evidence of bias against non-liberals in the academy, or studies of the relative intelligence of liberals and conservatives, they are posted here.

———————————————–

Update: Pigliucci’s response is this comment, posted on his last blog entry:

Hmm, I really don’t think I’m being Clintonesque here. And the slides in the talk make it clear that the google example was central to the argument, especially as data are concerned. But this exchange has gone on long enough, considering that it’s not even about a peer reviewed paper.

In other words: he still won’t say whether he HAD watched my talk before writing his initial post.

His blog describes itself as reflecting “the Enlightenment figure Marquis de Condorcet’s idea of what a public intellectual… ought to be: someone who devotes himself to ‘the tracking down of prejudices in the hiding places where priests, the schools, the government, and all long-established institutions had gathered and protected them.‘”

Below is my response to Pigliucci, which I posted as a comment on his blog.

—————————————————-

Dear Prof. Pigliucci:

Let me be certain that I have understood you. You did not watch my talk, even though a link to it was embedded in the Tierney article. Instead, you picked out one piece of my argument (that the near-total absence of conservatives in social psychology is evidence of discrimination) and you made the standard response, the one that most bloggers have made: underrepresentation of any group is not, by itself, evidence of discrimination. That’s a good point; I made it myself quite explicitly in my talk:

Of course there are many reasons why conservatives would be underrepresented in social psychology, and most of them have nothing to do with discrimination or hostile climate. Research on personality consistently shows that liberals are higher on openness to experience. They’re more interested in novel ideas, and in trying to use science to improve society. So of course our field is and always will be mostly liberal. I don’t think we should ever strive for exact proportional representation.

In my talk I made it clear that I’m not concerned about simple underrepresentation. I did not even make the moral argument that we need ideological diversity to right an injustice. Rather, I focused on what happens when a scientific community shares sacred values. A tribal moral community arises, one that actively suppresses ideas that are sacrilegious, and that discourages non-believers from entering. I argued that my field has become a tribal moral community, and the absence of conservatives (not just their underrepresentation) has serious consequences for the quality of our science. We rely on our peers to find flaws in our arguments, but when there is essentially nobody out there to challenge liberal assumptions and interpretations of experimental findings, the peer review process breaks down, at least for work that is related to those sacred values. (The great majority of work in social psychology is excellent, and is unaffected by these problems).

The fact that you criticized me without making an effort to understand me is not surprising. That is common in the blogosphere (although I rarely see it among philosophers). Rather, what sets you apart from all other bloggers who are members of the academy is what you did next. You accused me of professional misconduct—lying, essentially–and you speculated as to my true motive:

I suspect that Haidt is either an incompetent psychologist (not likely) or is disingenuously saying the sort of things controversial enough to get him in the New York Times (more likely).

As far as I can tell your evidence for these accusations is that my argument was so bad that I couldn’t have believed it myself. Here is how you justified your accusations:

A serious social scientist doesn’t go around crying out discrimination just on the basis of unequal numbers. If that were the case, the NBA would be sued for discriminating against short people, dance companies against people without spatial coordination, and newspapers against dyslexics. Claims of discrimination are sensibly made only if one has a reasonable and detailed understanding of the causal factors behind the numbers. We claim that women and minorities are discriminated against in their access to certain jobs because we can investigate and demonstrate the discriminating practices that result in those numbers. Haidt hasn’t done any such thing. He simply got numbers and then ran wild with speculation about closeted libertarians. It was pretty silly of him, and down right irresponsible of Tierney to republish that garbage without critical comment.

I have two responses to you.

First, please tell me if you would agree or disagree with this claim about when a member of the academy should accuse a peer of professional misconduct:

Accusations of professional misconduct are sensibly made only if one has a reasonable and detailed understanding of the facts of the case, and can bring forth evidence of misconduct. Pigliucci has made no effort to acquire such an understanding, nor has he presented any evidence to support his accusation. He simply took one claim from the Tierney article and then ran wild with speculation about Haidt’s motives. It was pretty silly of him, and down right irresponsible of Pigliucci to publish that garbage without even knowing what Haidt said.

Second, I challenge you to watch the video of my talk (click here) and then either

1) Retract your blog post and apologize publicly for calling me a liar

or

2) State on your blog that you stand by your original post.

If you do stand by your post, even after hearing my argument, then the world can decide for itself which of us is right, and which of us best models the ideals of science, philosophy, and the Enlightenment which you claim for yourself in the header of your blog, “Rationally Speaking.”

Jonathan Haidt

[Note: Pigliucci has now responded to me. He offers no apology, no retreat, no qualification of any of his original claims. He stands fully behind his original post.  My final response to him is here.]

I posted a response on his blog, but because my response was #287 out of 310, nobody is likely to see it. I reprint it here:

——————————

Dear Mr. Krugman:
I urge you to view the video and/or transcript of my talk at www.JonathanHaidt.com.
I very deliberately did NOT make the moral argument that ideological divides are like racial divides. I agree with you that there are many relevant differences. Also, I directly stated that personality differences will always guarantee that academe is mostly liberal, just as you noted that the military is mostly conservative. That’s all fine by me.
Rather, I focused my appeal to my colleagues one one point: that when conservatives are entirely absent (as opposed to simply underrepresented), then there is NOBODY to speak up, nobody to challenge predominant ideas, and our science suffers. I gave examples of several scientific mistakes that my fellow social psychologists make because our shared values make it difficult for us to entertain alternative hypotheses. People who think of such alternatives dare not speak up.
My research, like so much research in social psychology, demonstrates that we humans are experts at using reasoning to find evidence for whatever conclusions we want to reach. We are terrible at searching for contradictory evidence. Science works because our peers are so darn good at finding that contradictory evidence for us. Social science — at least my corner of it — is broken because there is nobody to look for contradictory evidence regarding sacralized issues, particularly those related to race, gender, and class. I urged my colleagues to increase our ideological diversity not for any moral reason, but because it will make us better scientists. You do not have that problem in economics where the majority is liberal but there is a substantial and vocal minority of libertarians and conservatives. Your field is healthy, mine is not.
Do you think I was wrong to call for my professional organization to seek out a modicum of ideological diversity?

Jonathan Haidt

One way to check the validity of these measures is to apply them to real world data. The recent 2010 midterm elections provide an excellent opportunity to do this.

The Guardian generously posted a downloadable version of the 2010 midterm election results on their website. For the following analyses, I will use the estimated moral foundation score for each congressional district derived from the YourMorals data.

The first thing to examine is the simple bivariate relationship between the Republican share of the vote in 2010 and each of the moral foundations.* The figure below displays these relationships. On the y-axis we have the percent of the vote won by the Republican candidate, and on the x-axis we have the estimated moral foundation score of the congressional district. In the upper-right corner, I’ve displayed the simple correlation coefficient between the two variables, and the thick red line shows the best linear fit.

This figure looks very similar to one I posted earlier detailing the relationship between the vote for Obama in the 2008 elections and the moral foundations. Indeed given the strong autocorrelation from year to year in election results, this is no real surprise.

The most important changes in election results from year to year tend to be the across-the-board shifts toward one party or the other (for more on this general trend, see Eric McGhee’s cogent analysis at the Monkey Cage).

To what extent can the moral foundations data explain this shift? After some exploration of the raw data, it became clear that the area to focus on was the seats held by Democrats. For the remaining analysis, I’ll restrict the data to the 257 seats that the Democrats held prior to the 2010 elections. For the dependent variable, I will use the difference between the vote for John McCain in 2008 in a particular district and the vote for the Republican House candidate.

Overall, among seats held by the Democratic Party, there was a two-point shift in the vote toward the Republicans. The figure below shows the distribution of this shift over the partisan voting index (PVI) of the district (the PVI is the average deviation of the district from the national presidential vote in the last three elections—for example, a district with a PVI of -10 votes ten points more Democratic on average than the average vote in the country).

Districts that fell on the left side of the figure (those with large negative values of the PVI), were generally safe Democratic seats. Those that are located toward the center and the right, were in more danger. We can see that the shifts were greatest near in these most marginal seats.

To examine the possible explanations of this shift, I looked at how the relationship between the PVI and the shift in the Republican vote in 2010 changed based the moral foundations. I divided all of the districts into two categories for each of the moral foundations. Those that were above the median score on a foundation were labeled as “high” on that particular foundation and the remaining were labeled “low.” The figure below shows the how the relationship between a district’s PVI score and the shift toward the Republican party changed when we break the data down further into these different foundations. The dashed lines show the relationship in districts that scored above the median on a particular foundations, and the solid lines show those that scored below the median.

There are several interesting patterns to note here. First, there did not seem to be any differential effects in the safest Democratic districts. At a PVI of -15, the data show the general shift toward the Republican Party discussed above (about 2 points more Republican). However, as we move toward the right-hand side of the PVI scale, toward the marginal seats and those that actually go for Republicans in their presidential voting, the gap between the high and low scoring foundations is largest. Most interestingly, it is those districts that scored highest on the liberal foundations (Harm and Fairness) that showed the biggest shift toward the Republicans in 2010. None of these relationships hold up in districts held by the Republican Party. It is something unique to the campaign against incumbent Democrats.

Finally, no discussion of the 2010 midterms would be complete without some mention of the Tea Party. The New York Times compiled a list of all of the Tea Party candidates (see here) in the 2010 elections. Of the 129 candidates that the Times identified, 120 of them ran in districts that the Democrats held in 2008. This represents almost 47% of all the races in districts where the Democrats had control in 2008.

What explains the emergence of Tea Party candidates in general elections?** The figure below shows how the probability of seeing a Tea Party challenger increases over the PVI (again, analysis was restricted to only those 257 seats that were held by Democrats in 2008). As we would expect, the probability increases as we move toward the more competitive districts.***

The most powerful relationship was seen in the districts that scored above the median on the Fairness foundation. Districts that scored high on the Fairness foundation and were competitive had a much greater probability of seeing a Tea Party challenger than those others. Smaller relationships were seen for the other foundations. Interestingly, the foundations did not seem to affect how well the Tea Party candidates did once in the race, only their probability of entering.

* In all of these analyses it is important to keep a few things in mind. First, the district level foundation scores were computed based on the entire voting age population of each congressional district. A big part of the story in 2010 was the “enthusiasm gap” between Democrats and Republicans. These analyses do not make adjustments for the differential turnout. Second (and this is somewhat related to the first point), the district level estimates for the moral foundations are aggregate measures. It would be inappropriate to infer individual level behavior from these district level statistics (this is known as the “ecological inference fallacy”).

** It would probably be most appropriate to examine the emergence and success of Tea Party candidates in Republican primaries. If we had a good measure of the moral foundations of Republican primary voters in 2010, this would make a fascinating analysis. However, we only have a measure of the district’s score on the moral foundations as a whole. At present, it is not possible to break this out by party.

*** The plots show predicted probabilities from similarly specified logistic regressions. The regression equation took the form,

Logit(P(Tea Party Challenger) )= a + b(1)*PVI + b(2)*High Foundation + b(3)*PVI*Foundation

In my previous posts, I’ve  discussed the many potential difficulties in using an entirely self-selected internet sample for inferences about general population parameters (whether or not a particular state or congressional district scores higher than another in terms of its moral foundations) as opposed to the intra-individual comparisons that are the bread and butter of psychologists (like how the foundations tend to correlate with ideology). I think that I have shown that the raw data are unsuitable for talking about the general population. The sample is demographically unrepresentative (see here) and somewhat attitudinally unrepresentative (see here).

What we need is a method to correct for the biases in the sample. Enter Multilevel Regression with Poststratification (or Mr. P as he is affectionately known to statisticians).*

[For those uninterested in the technical aspects of modeling, skip down to the maps below]

Multilevel Regression with Poststratification proceeds in (basically) three steps. First ,we construct a model to obtain the expected values of the variable of interest as a function of variables that we know the underlying population values for (typically this means only items that show up in the Census – geography, age, education, income, race, gender, maybe a few others).

Second, we use the model to predict the expected value of the variable of interest for each combination of variables (or cells) in the model. For example, if the model used four regions (Northeast, South, Midwest, West), three categories of age (18-30, 31-60, 61+), three categories of education (HS, College, Graduate), two categories of income (Less than $50k, More than $50k), two categories of race (white, non-white), and gender as predictors, there would be a total of 4*3*3*2*2*2 = 288 cells. From a 18 year old male with a HS degree or less who makes less than $50k is white and lives in Maine to a 75 year old female with a PhD making more than $50k is Asian and lives in New Mexico.  All individuals in the sample fit into one (and only one) of the cells defined by the combinations of predictors in the model. Many of the cells will be empty. In cells where there is no data, the model borrows statistical power from the other cells to come up with an expected value for every cell.

Finally, we weight each of the estimated cell values by the proportion of individuals in the population to come up with predictions for the geographic regions of interest. In this case, we would have predictions for each region.**

The map above (click on the image for a better view) plots the predictions from the MRP estimates of the difference between the liberal and conservative foundations for each congressional district. Districts in which YourMorals users valued the two “liberal” foundations (H and F) more than the three conservative foundations (I,A,P) are shown in dark green, and those districts cluster in the regions that we know to be the most liberal parts of the country: the North East, and the West Coast (excluding the agricultural parts of California). The districts within which YourMorals users gave the most conservative pattern (IAP > HF) are shown in red, and these districts fall overwhelmingly within the South.

One way to test the validity of these measures of district level foundations is to compare them to observable characteristics of the districts. One ready comparison we can make is to the district’s share of the vote for Obama in the 2008 presidential election.

The figure below shows the simple bivariate relationship between each foundation and vote for Obama. In every case, we see the expected relationships. Districts that scored more highly on the Harm and Fairness foundations were more likely to go for Obama in the election. On the other hand, there is a strong negative relationship between a district’s score on the Purity foundation and its vote for Obama.

Interestingly in many cases (especially the harm and authority foundations), there appears to be a significant “kink” in the fitted line at the midpoint of some of the foundations. Districts that score highly on the harm foundation (or those with low scores on the authority foundation) are associated with increased showings for Obama, but that relationship dissipates after the mid point of the foundation is crossed (all foundation scores are measured in standard deviation units). This is an aspect of the data that deserves further attention, but I’ve run out of time and space to do so here.

Multiple regression estimates confirm the overall story we see here in the bivariate plots. The estimated foundation scores seem to meaningfully correlate with real world phenomena. This bodes well for the validity of the measure and method.***

So, with a little work, it appears as if we can have our cake and eat it too when it comes to the YourMorals data. Scores on the foundations (after adjusting for the biases in the sample) are significantly related to district voting behavior.

*For a more detailed explanation of the methods involved see here. Also, Andrew Gelman and Jennifer Hill’s excellent book, Data Analysis using Regression and Multilevel/Hierarchical Models.

**I made an editorial decision not to include the details of the model and etc. as it didn’t seem of general interest. I’m more than happy to talk about it, but the post was getting wordy as it was.

*** The careful reader might well protest that the relationship we see in the figures presented is merely the product of the correlations with demography picked up from the MRP method. Since I used district demographics to adjust the scores obtained from the convenience sample, is it possible that the positive findings are simply a reflection of secondary correlations between the moral foundation scores and demographics? One easy way to test for this is to include demographic controls in the model. I was happy to see that none of the findings are substantially changed by including district demographics on the right hand side of the regression.

Last time, I gave a broad overview of the descriptive representation of the YourMorals dataset. In a nutshell, we discovered that the YourMorals respondents were much more educated, more likely to self-identify as liberal, and more likely to be white than the population.

In this post, I will explore the question of whether the YourMorals respondents are representative of the population after we condition on observable characteristics. Put another way, would we expect two individuals, one randomly chosen from the population and one drawn from the YourMorals data, who share all the same demographic characteristics (age, race, education, political ideology, place of residence) to look the same in terms of their scores on the Moral Foundations Questionnaire?

To conduct this kind of analysis, first we need a benchmark against which to compare the YourMorals data. As I mentioned in my previous post, the gold standard is a randomly drawn sample from the population. Luckily, we have just such a survey. Prior to the 2008 election, Knowledge Networks* fielded a version of the Moral Foundations Questionnaire to a representative sample of the U.S. population. This provides a good point of comparison for our (much larger) convenience sample.

The first task is to process the YourMorals data so that it looks more like the general population. I used a basic sample matching technique to match individuals from the YourMorals data and the Knowledge Networks data. This is a crude technique, but effective. Basically for each individual in the Knowledge Networks sample (the “match target”), I found an individual (or individuals) in the YourMorals data that matched the demographic information for the “match target.” These cases then become the comparison group. After the samples have been balanced in terms of observable characteristics, any differences we observe between the two can be ascribed to the compounding factors that we cannot observe.**

The following figures show how the distributions of the matched YourMorals data compares with the distributions in the sample from Knowledge Networks. The dashed lines show the distribution for Knowledge Networks, the solid lines represent the YourMorals data.

The distributions of the foundations in the two data sources look very similar for the Fairness/Reciprocity foundation, but for all of the others, there are significant differences between the YourMorals and the Knowledge Networks respondents.

A little more digging reveals some interesting patterns. Splitting up the sample by ideology yields:

Liberals:

Conservatives:

Two of the foundations seem to stand out in these comparisons. Liberals in the YourMorals data are particularly low on the Purity foundation (when compared against liberals in the Knowledge Networks data), and conservatives from the YourMorals sample seem to score lower on the Harm foundation. In both cases, YourMorals liberals seem more like population liberals on the first two foundations (Harm and Fairness), and the conservatives in the sample seem more like population conservatives on the last two foundations (Authority and Purity). No matter how the data is cut, the YourMorals sample seems to score lower on the Ingroup foundation.

The comparisons between the general population sample and the convenience sample in this post raise some significant questions about the possibility of using the self-selected respondents in the YourMorals sample to make inferences about the population. These problems in the data are particularly evident in the Ingroup foundation, the purity foundation (for liberals), and the harm foundation (for conservatives).

As was the case with demographics, all is not lost. One last look at the data shows that again the foundations are more or less proportionally correct. Liberals score higher in on the Harm and Fairness foundations in relation to their scores on the other three, and conservatives show more or less equal scores across each of the foundations. The bar chart below shows the average scores of the foundations broken out by survey source (KN and YM for Knowledge Networks and YourMorals respectively) and ideology:

Next time, I’ll discuss how we might correct for some of these demographic and attitudinal biases in the data.

*For the uninitiated, Knowledge Networks is a survey research firm that has gone to great lengths to put together a panel of internet users that is nationally representative. They have recruited a large panel of individuals to take internet surveys. These individuals were generally contacted by telephone, and in cases where the respondent did not have internet access, Knowledge Networks provided access. See this link for more information.

**For a quick primer on the theory behind sample matching see this Wikipedia entry.  I am using exact matching on categories of age, race, education, ideology, and state of residence.

Earlier Jon Haidt discussed the “problem” of representativeness of the YourMorals data and concluded that it wasn’t such a problem after all. Convenience samples drawn from the internet can produce reliable data. This is particularly true when we are more interested in taking valid measurements than in painting a representative picture of some underlying population.

But what if we would also like to know something about the underlying population? If we had data that were representative of the country as a whole, we would be able to ask a new set of questions. Does knowing where the states fall in terms of their Moral Foundations tell us anything about voting behavior? We might expect scores on the purity foundation to explain state-level attitudes about gay marriage or the fairness foundation to explain attitudes about tax policy. To answer these kinds of questions, we need representative samples (also see Jesse Graham’s comment in the above link).

In sampling theory, the gold standard is the probability sample. When all individuals in the population have a known (but not necessarily equal) probability of being included in the sampling frame, we can construct reliable estimates of the population parameters and, given sufficient sample size, be confident that these estimates are within some distance of the true values in the population. However, the central assumptions of sampling theory are violated in convenience sampling (but see this discussion of the representation problems in traditional “random” sample polls).

First, we would like to get a sense of how the YourMorals data stacks up against other population measures. We collected data on several demographic characteristics of individuals in the YourMorals dataset. We can easily compare these against population values collected from the census or other representative samples.

One area where we can clearly see the representation problems in the YourMorals data is self-reported ideology. Considering only U.S. respondents for the time being (as all of the following analyses do), recent national samples put the proportion of people who consider themselves “liberal” at between 18 and 22 per cent. In the YourMorals data, this figure is nearly 65 percent.* Given this skew in the data, we might be hesitant in trying to make inferences about the general population from a sample that looks so much different.

The figures below show how the YourMorals data compares with the population values across a handful of demographic and attitudinal variables.

Source: Pew Center for the People and the Press, 2001-2008

This figure shows how even with a significant intercept shift (almost 50 points), the rank ordering of the states stays pretty close to the same. This is encouraging as it means we are not drawing the same type of individual from each state. Put differently, knowing the state that an individual resides in tells us something about the probability that he or she identifies as a liberal. What we would not want to see here would be a horizontal line (indicating no relationship).

Source: American Community Survey, 2006-2008

Source: American Community Survey, 2006-2008

With race it is much the same story as ideology. For whites, there is a substantial intercept shift (almost 70 points), but states with larger white populations also are proportionally more white in the YourMorals data. The data for African Americans is noisier (there were fewer than 900 in the sample of over 60,000), but shows the same pattern. Here there is not a large intercept shift (as we have reached the floor of the data), but we see the same kind of increasing pattern.

Source: American Community Survey, 2006-2008

With respect to education, the data are further afield. The figure shows that the YourMorals sample is significantly more educated than the general population, but it becomes more difficult to draw a convincing trend line through the data. Individuals who came from states with higher levels of education were only marginally more likely to be highly educated themselves.

So where does all of this leave us? It is obvious from the plots that the individuals who self-selected into the YourMorals data look very different than the general population. It would clearly be inappropriate to use the raw data in trying to make inferences about the general population parameters (average levels of a particular foundation in a particular state, for example). The sample is much more liberal, highly educated, and white than the general population. But it is not as bad as it could be. The worst-case scenario would show uniformly weird sample across the states. Instead, what we saw in the figures above is a picture that is more-or-less proportionally correct. It is encouraging that the general relationships hold up.

All of this is not to say that we should throw out the analyses presented elsewhere in this blog and in publications based on the YourMorals data. If we condition on ideology (which we saw was particularly skewed) and make statements like “Liberals generally score higher than conservatives on the Harm/Care and Fairness/Reciprocity foundations,” we are probably treading on safe ground.

In the next few posts, I will be revisiting the question of how to construct a representative picture from a convenience sample.

*Beyond the obvious sampling issues, there are a few other problems with directly comparing the measure of ideology in YourMorals with that in nationally representative samples. First, there is a mode difference that could account for some of the discrepancy (although certainly not all or even a very significant portion of it). Another (and more serious) difference between nationally representative samples and the YourMorals data is the choice of a seven point scale rather than a five point scale. Five point scales are used more regularly in telephone samples with the options being “Very Conservative,” “Conservative,” “Moderate,” “Liberal,” and “Very Liberal.” The YourMorals data includes options for “Slightly liberal” and “Slightly Conservative” as well as “Libertarian” and “other” categories. The 65 percent figure lumps all of the “liberals” together. If you believe that the “slightly liberal” respondents might have self-identified as “Moderate” given fewer options, the proportion turns out to be just over 50.