Wednesday, December 10, 2008

Thoughts on Comprehensive Exams, Part 1

Here at CU it's that special time of year where snow is falling (6" yesterday), the semester is ending, and those of us lucky enough to be finishing out third semester are stressed out of our minds studying for the comprehensive exam. Today, as I was going over my notes on selection rules for photon atomic and molecular emission/absorption, I decided that I would share some of my thoughts on the whole concept of a departmental exam. Since my experience is specific to my department, I would be interested in hearing your experiences as well. I will make some general statements, so please correct me if these turn out not be generally true. Today I'm going to talk about the two versions of departmental exams. In the next week or so I'm going to have additional posts on the mechanics of the exam itself and the need to have any exam at all.

As I see it, there are two main purposes for a departmental exam. I will refer to them as the qualifier model and the comprehensive model. In the qualifier model the purpose of the exam is to make sure all incoming students have a basic level of background so that they are ready for a Ph.D. level program. These tests tend to be given to incoming students and/or those finishing their first year of graduate study. They are essentially testing content at a level such that a very well prepared incoming graduate student can pass them before taking any graduate level classes. My only experience with this model is from the talking to graduate students at Ohio State when I was there for a summer. In Ohio State's physics department, the qualifying exam (as they call it) is offered every August to incoming grad students. If they do not pass they get a second chance to take it again a year later. Two fails means that you are kicked out of the Ph.D. program and can, at most, earn a Master's degree.

In the qualifier model, the exam is really a test of your undergraduate education. If you fail, then your first year of graduate study can include some coursework designed to fill in any deficiencies. This model works well when the department is admitting primarily students with the same undergraduate major - in this case physics.

The second model is what I will call the comprehensive exam model. In this model, the purpose of the exam is to check the level of your graduate classes. This is the model we use here at CU. The comprehensive exam is usually taken later in one's graduate career after the majority of the coursework has been completed. The exam is designed to test what has been learned in graduate level classes. Here in the Astrophysical and Planetary Sciences department at CU, we take the comprehensive exam after our third semester while the required courses are taken in the first two semesters. If a student fails, the exam can be retaken the next year. In the case of two failures, a student generally is given an oral exam and a master's degree after 2 1/2 years. The comprehensive model works well for programs (like mine) that take students from a variety of undergraduate backgrounds (physics is the most common but we have people with Bachelor's degrees in math, astronomy, geology, aerospace engineering, and possibly more).

Both models have advantages relative to each other. In the qualifier model there is an immediate and quantitative measure of incoming students' backgrounds. This allows the department to start the graduate level classes at a higher level with everyone on the same page. It also allows students that fail the exam twice to do so after only 1 year so that they can make other career plans without wasting too much of their time. However, the qualifier does not check to see if a student has gained the required graduate education needed to be a successful researcher in their discipline.

The comprehensive model, on the other hand, has the advantage that those passing it have certified that they know the things that everyone in their field should know. For example, I do not intend to spend my career analysing spectra, but after passing comps I will have certified that I know how to preform basic analysis of spectral lines so that I can at least understand what can and cannot be done with them. This model also allows the exam to happen when classwork is nearly complete (I will have only two classes and a seminar left after this semester), which provides a nice transition between a the course-focused portion of a Ph.D. program and the research-focus portion. The comprehensive model, however, means that some students will devote 2 1/2 years to a Ph.D. program before finding out they will not be earning a Ph.D.. It also does not provide a baseline measure of what students should know when they come into a program so graduate level classes must often start off at a lower level.

So philosophically, the model used by your department is largely determined by whether the bigger concern is the level of the incoming students or the level of the students leaving the coursework section of their educations.


  1. 1st part of a two part comment: (And may I remind everyone I passed the qual on my first try so don't accuse me of having sour grapes).

    This is the most accurate description of the qual (By Sean Carroll): "The candidacy exam, [is a] hazing ritual by which a young student proves that they are ready to take on research."

    Nick, and everybody else, find me peer reviewed research that shows the quality of science research is better off because people had to pass this exam!!!

    The fact is in every textbook on standardised testing will tell you that standardised tests are a poor indicator of weather someone will be successful.

    *And isn't it so ironic* that here we are trying to get students to think like scientists when we go about judging them in such an unscientific way.

  2. The UC Irvine Way (A model for everyone?)

    Turns out, at Irvine you don't pass based souly on your exam score. (They won't even tell you what it is.)

    Two quotes from the same blog post in the comments section:

    1. "An odd example I’ve seen is UC Irvine- they have a quite draconian and extensive qual policy, more so than several top-tier schools."

    2. "p.s. I can attest to the draconian UC Irvine qual policy."

    At Irvine, all the faculty come into a room, and the are shown your grades, qual score, other information I don't know about and they debate weather or not you will make a successful PhD candiatie. You have three attempts, one every six months.

    It can be brutal. The pass rate is about 50%. But at least your passing or not passing is not based on an exam score alone, but on a variety of criteria that combined in my opinion make the process better here.

  3. CU's method is somewhat akin to Irvine's. At CU, once the exams have been graded, the faculty meet and decide who passed. There is no minimum score for passing and whether one passes or not depends primarily on the exam score but also on course grades and "other factors" which can include whether your score is the result of simply blowing one or two questions or preforming poorly on many questions.

    In practice, however, whether one passes or not is largely determined by exam grade. There have been a couple cases though where a student with a lower grade has passed the comps and a student with a higher grade has not, but such events are rare at CU. I know of only one in the past 4 years. In that case one individual did not pass despite scoring higher than two people who did. The faculty's rationale was that the failing student had done poorly on two core class subjects on the exam and had correspondingly low grades in both those classes, while the two passing students both had high grades in the classes that they had trouble with on the exam.

  4. Here at UNC-CH we have a slightly different approach. I guess you might say we have a mixture of both the qualifier and comprehensive approach. The way it works here we are required to take a total of 9 classes (in reality we take more but the bare minimum is 9), 6 of which come in the first year. After the first year of classes (immediately after, the tests come about a week after classes and finals end) we take the qualifier, which consists of 6 tests (written, no oral tests) over two days.

    Typically 80-90% pass the qualifier the first time around. This may seem odd to some to have such a high pass rate, but that is just the test, the actual qualifier starts in the application process. The philosophy here is that if someone is admitted to the program here then it is fully expected that they will go on and get a PhD. In other words they try not to admit people who will not eventually go on to get a PhD. Usually if people do not get their PhD it is for personal reasons.

    So after passing the first round of cuts (by being admitted) they then brutalize you for a year with hard classes and making you work as a TA in the most remedial and demeaning classes (such as grading and teaching the lowest level lab classes, 105-108, 121,123 and 220 equivalent). In this regard I'm ahead of the game because I have been a TA for 4 years already at BYU so they stuck me in the higher level classes to start out.

    So after that they make us take the tests immediately after we take the main core classes, and the tests are usually (emphasis on the usually) geared towards what was covered in the classes. So the quals are also a measure of how the classes are doing in addition to how well we know the material. Once we pass the tests we are allowed to go on and get a PhD, but as the chair of the department told us, she could only think of one person who actually failed the tests both times and was not allowed to continue. Most people who fail decide that they really don't want to get a PhD, though that is not the purpose of the tests.

    Thus after all that the emphasis shifts form classes and general studies to specific research. At some point we have oral examinations about our research and I hear they really put you through the ringer on that one, but the idea is that you will get you PhD even if they have to put your through the process several times, you will eventually pass (but it sounds like an experience that you only want to go through once). Thus they aim to turn you into a top class researcher, no matter what, but only if you really want to, if you don't want to you will drop out at some point for "personal reasons".

  5. In response to Ryan's comment, at CU our comprehensive exam actually comes in two parts. The first is the written exam that I have been discussing and then there is a second part which requires a publication quality research project which is then presented in the same fashion as a thesis presentation after which there is an oral grilling. Like a thesis defense, however, it is extremely rare for someone not to pass because an advisor generally doesn't let a student present and be examined until they are ready to pass. Most students do this in September of their third year but some lag as far as December. If a student doesn't pass the written exam portion of comps for the second time, the research portion of comps can then be used as a Master's thesis. I personally do not know of anyone that has not left the program with at least a Master's degree and then only 1 person in the past 3 years has left with a Master's degree after not passing the written exam twice. Several others have left with Master's degrees after passing the written exam or failing it once for personal reasons or because they simply decided they wanted to do something else.


To add a link to text:
<a href="URL">Text</a>