Caveat: this is a skill that I am working to develop over the next few years, not one that I have mastered.
Reading in graduate school is different from that required for undergraduate coursework. This is true not only of the sheer quantity (it has been likened to drinking from a firehose) but also the types of readings assigned. As Thomas Kuhn has noted, most of the readings assigned to undergraduates are in textbook form. The advantage to this approach is that the reading is comprehensive, or at least provides most of the requisite information for the course.
But there is also a key disadvantage: the textbook is given as ‘received wisdom’ from sages of ages past without any indication that those findings were not uncontroversial at the time, or indeed even presently. This is like a movie: we see the final product, but we don’t know which scenes ended up on the cutting room floor (or at least are being saved for the DVD), which changes were made to the script, and so on.* These differences are apparent sometimes in movies that are adapted from books, but often they are invisible to the major audience. (Have you heard many favorable comparisons between movie adaptations and the original book? I haven’t.) The movie analogy shows that while the final product is often perfectly fine in its own right, it is usually lacking the substance or nuance of the original.
This difference between watching the movie and reading the script is similar to the change from undergrad to graduate course readings.** Rather than having a nice, clean package of information in the form of a textbook, you spend much more time reading journal articles and short papers. Often you will read opposing viewpoints on the same issue/question, either in the same week or over the course of this semester. This type of reading has the impact, on me at least, of showing that science*** is a fluid process. It is not a collection of right answers, it is a resource of ideas that seem to fit with certain facts when they are viewed in a certain way.
Jeff Ely put it very well recently:
My tests don’t contain any information in them that isn’t in the raw data. My tests are just a super sophisticated way to summarize the data. If I just showed you the tables it would be too much information. So really, my tests do nothing more than save you the work of doing the tests yourself.
But I pick the tests. You might have picked different tests. And even if you like my tests you might disagree with the conclusion I draw from them. I say “because of these tests you should conclude that H is very likely false.” But that’s a conclusion that follows not just from the data, but also from my prior which you may not share.
What if instead of giving you the raw data and instead of giving you my test results I did something like the following. I give you a piece of software which allows you to enter your prior and then it tells you what, based on the data and your prior, your posterior should be? Note that such a function completely summarizes what is in the data. And it avoids the most common knee-jerk criticism of Bayesian statistics, namely that it depends on an arbitrary choice of prior. You tell me what your prior is, I will tell you (what the data says is) your posterior.
Pause and notice that this function is exactly what applied statistics aims to be, and think about why, in practice, it doesn’t seem to be moving in this direction.
First of all, as simple as it sounds, it would be impossible to compute this function in all practical situations. But still, an approach to statistics based on such an objective, and subject to the technical constraints would look very different than what is done in practice.
A big part of the explanation is that statistics is a rhetorical practice. The goal is not just to convey information but rather to change minds. In an imaginary perfect world there is no distinction between these goals. If I have data that proves H is false I can just distribute that data, everyone will analyze it in their own favorite way, everyone will come to the same conclusion, and that will be enough.
Like reading the script of a movie and seeing how ideas change, graduate school offers the chance to peek behind the curtain of the scientific process. We can discover many things, some of them profound and some of them fundamental. But hopefully through it all we can remember something that we should not have forgotten in the first place: we are only human.
*Another way that this is sometimes apparent is in closed captioning. When a movie’s subtitles don’t match up precisely with what’s being said on screen it is often because the CC is based on a version of the script rather than someone actual viewing the movie and captioning it.
*** By “science” here I mean simply the organized, falsifiable pursuit of human knowledge.