Grading Presidential Rhetoric using Readability Software

If you were a junior high or high school teacher interested in assigning presidential speeches for your American Government class, which presidential speeches would you assign and to whom?  My sense is that most would assign speeches by George W. Bush to the junior high students and Barack Obama’s speeches to the seniors.  If so, you’d be wrong.  Let me explain.

A popular notion is that Obama’s biggest problem is his complex, “professorial” rhetoric.  Where Bush controlled the political agenda because of his easy-to-understand prose, Obama has struggled because he uses polysyllabic words and compound sentences.  I agree with the larger point: the causal mechanism, not so much.  I agree that the president and congressional Democrats could do be doing a better job selling their accomplishments and framing national debates.  For example, there seems to be an incorrect belief that Democrats have not passed very much landmark legislation while in office.  What is at issue in this post is the contention that this framing problem stems from Obama’s use of overly complex rhetoric.  While his prose may be more abstract and philosophical, I do not think it has to do with his sentence structure or use of “big words.”  Empirics, anyone?

Here’s what I did.  I downloaded all State of the Union speeches for presidents Obama, George W. Bush, Clinton, Reagan and Kennedy.  This data can be found on the Miller Center’s Presidential Speech Archive.  I selected State of the Union speeches because it provides a level of comparability across presidents (i.e. they are the same “kinds” of speeches).  However, since Obama only has one State of the Union thus far, I also downloaded his two joint-congressional session speeches.  The first is his address on the economy in 2009 and the second is his address in September concerning health care.  My hope is that these speeches are similar in nature to Sate of the Union speeches and can be combined in a meaningful way.

Next, I analyzed each president’s slate of speeches using Fryinator 1.1.  This freeware provides readability statistics intended for teachers who need to determine the appropriate grade level for a document or passage.  There are a number of nice statistics computed.  The first is the Flesch-Kincaid Grade Level.  The relevant info is in the name—the measure reports the estimated grade level of a particular passage.  Under the hood, this measure relies on the average sentence length and average syllables per word.  The second measure is the Flesch Reading Ease.  Using the number of words per sentence and syllabus per word, higher values on this metric indicate easily understandable text while lower value indicate less easily understandable text.  Additional details are, of course, available on Wikipedia.

Here are the results:

Obama’s first state of the union speech ranks as the most readable, coming in at an estimated 8.8 grade level.  If we include the second pair of observations (using Obama’s joint-congressional session speeches, denoted with the star), Obama grades at a 9.4.  Clinton falls between these two estimates, with a 9.27 estiamted grade level.  Bush, on the other hand, comes in at a whopping 10.2.  Surprisingly, Bush’s speeches are more (!) complex than Obama’s.  Reagan, the “Great Communicator,” grades with Bush (a 10.2 grade level).  The least readable president?  Kennedy, with a grade level of 12.3.

As I tried to qualify in the beginning, this says nothing about the actual words used nor the nature of those words.  That kind of analysis would require a bit more work.   Moreoever, I do think that Obama and congressional dems are losing the message war (and losing badly for that matter).  Still, there seems to be the argument that Obama uses “big words” and “compound sentences.”  I think that notion is dispelled here.  As an aside, I did not expect Bush to score a much higher grade level than Obama.  Rather, I expected the difference to be small.