ACTUARIAL TESTING By Jerry Tuttle, FCAS, FCIA, CPCU I asked everyone but Woody to stand as we raised our beer mugs. “Congratulations to our newest Fellow, Woody Fishman! People who aren’t actuaries have no idea how hard our tests are.” As the senior actuary in the company, I had gathered all the company’s actuaries for a celebratory beer fest at the nearby pub. “So what will you be doing with all your free time?” asked Wendy Chang, one of the younger actuarial students. “I know one thing he will be doing,” Jay Thompson chimed in. “A little mandatory new Fellow volunteer work.” Everyone broke out in hysterical laughter – except Woody. “I don’t get it. What’s so funny?” asked Woody. “You become a new Fellow, and then how soon you forget. Oh man, I would hate to have to do that. How did that tradition ever get started?” asked Wendy. “Well, funny you should ask,” I replied, stroking my beard thoughtfully. “We’re out of the office, right? Whatever is said out of the office stays out of the office?” Everyone nodded in agreement. “I guess it’s OK to tell you. I have to admit that this quaint little custom is really attributable to something I discovered about ten years ago.” * * * In 1982, I was working for a large insurance company with a big actuarial department. I was reporting to the chief actuary, Vince Lentini. I doubt if any of you remember those days, but that’s when there were three big insurance companies in town, and we were all sort of friendly rivals. The three actuarial departments were about the same size, and we generally knew our peers from actuarial meetings and softball games and such. One day Vince pulled me aside and asked me if I was aware of the chief actuary’s bet. It turns out that he and the other two chief actuaries had a standing bet every six months on the total actuarial exam pass ratios of their respective departments. Each of them would put $333.33 into a pool, with the winner taking the pot. It turns out all three of them decided the total exam pass ratio formula was too simple and not necessarily equitable. In fact Vince seemed to lose considerably more often than two-thirds of the time. The three chief actuaries agreed to nominate one actuary per company to a little task force, with the goal of deriving a more equitable pass ratio formula – solely for this chief actuary bet. Vince nominated me. Our task force met. We had experts in automobile merit rating, workers compensation experience rating – and baseball. Individual historical experience is used in all three of these areas to forecast future experience, and we thought we could do the same with people’s actuarial exam passing statistics. We found people’s age and years of experience had some nonlinear relationships with exam passing. We studied the pass ratios by exam – which are much different for the lower exams than the higher ones, by the way – and we investigated the probabilities of passing each exam at the first attempt, the second attempt, and so on, and how these probabilities change when someone enrolls in one of the actuarial cram courses or uses one of the supplementary study manuals. We put all of this into some regression software, and we came up with a preliminary version of an experience rating actuarial exam pass formula. We tested it retrospectively on past exam sittings for each of the three companies, we calibrated it, and came up with a final version that we considered equitable for each company. Now for each upcoming exam sitting, each company provided its own exam history for each student, and we produced a predicted pass ratio from the formula for each company. The winner of the chief actuary bet was the company that performed best, relative to its own predicted pass ratio. This formula worked fine for a good ten years. Each of the three actuaries was winning the bet about one-third of the time because the formula did a pretty good job of equalizing the three companies. It was as if we had created a betting point spread that had equalized the odds among the companies. Suddenly in 1994 the formula started to weaken. We began to seriously underestimate pass ratios, and in fact the company with the lowest expected pass score was outperforming the two stronger companies in the formula. This happened for four consecutive exam sittings, so we wondered what had changed. We reassembled the original task force to delve into the problem. There were numerous possibilities. Maybe the exams had gotten easier. Maybe the students were smarter or better prepared. Maybe something had changed affecting the mix of students or the mix of exams. We knew we had some investigating to do. We used our contacts on the Examination Committee, and they insisted that the exams and the grading were as difficult as ever – no surprise there. There had been no major change in syllabus readings. None of the three companies had increased the study hours. We spoke to the people running the actuarial cram courses and writing the study manuals, and none of them seemed to think anything had changed. We studied the mix of students and exams – always a good answer when there are unexplainable actuarial results is that the mix has changed - had there been a shift so that for example, 60% of the students were taking the first exam for the third time, and was that causing the formula to crumble? We could find no explanation, and we were running out of ideas. It occurred to me that since our three companies were all in the same town, most of the students would take their exams at the same exam site. Maybe the exam proctors had some insight. We interviewed the proctors for all of the exams from the last sitting. Only one of them had any useful thoughts. “To tell you the truth, I have been noticing something over the past few sittings,” began James Leech, a fellow from one of the other companies. “Now, you won’t quote me on this, right? Generally I assign the seating somewhat randomly, to reduce the chance of friends sitting together and cheating. But, my seating is not entirely random. I assign the best looking women to the front row. Hey, they have to sit somewhere, right?” “Other than watching for cheating, I really don’t pay much attention to any particular person, except I might sort of casually gaze at the people in the front row from time to time.” “What I have observed is after the exam is over, a number of students, both females and males, have been much more animated and chatty than they have been in previous years. People have had sort of a wild look in their eyes. If I didn’t know better, I would have thought some of the students might have been on drugs,” continued James. “Now that I think about it, something else has increased in recent years. During exams, people have become more compulsive than ever. They constantly straighten their pile of scrap paper, they arrange their pencils in a precise order – I find it odd, but I guess it doesn’t bother anyone,” concluded James. I’m no expert, but this smacked of drugs. I discussed this possibility with my task force, but they thought the idea was ridiculous. I have a colleague in another company, and if anyone would know about actuaries and drugs, he would. I called Sammy Finch and invited him to lunch. Sammy was not at all shocked at my suggestion. “There is a guy,” he began quietly, looking around to make sure nobody was listening. “I understand he was a bio-chemistry major who took a few actuarial exams and then decided the exams weren’t for him. He then started selling some kind of product online that claims to help people with the exams. I’m not saying it really works. But I have heard that some actuaries swear by it.” Sammy wrote down the address for a web page. The address looked like a random sequence of letters and numbers; it was not a web address you would find by yourself. “That’s all I know about it. And don’t use my name, OK?” I thanked Sammy for his information, picked up the check, and returned to the office. I carefully typed in the web address he had given me. The website that came up was full of new-age gobbledygook about spiritualism, the connection between metaphysical and medication, and other topics that I considered nonsense. I followed the alternative health remedies link and found a statement that “many of these remedies have not been formally validated by scientific research.” What a surprise! But there it was buried several pages into the website – a web page devoted to the actuarial exams. That web page had a good understanding of the actuarial exams – their mathematical nature, the enormous quantity of material that needs to be mastered, the need to both memorize numerous formulas and lists and then apply them creatively. The page went on to discuss the relationship between nutritional deficiencies and disorders such as hyperactivity and attention deficit disorder. Dietary supplements that treat those disorders could also be used in small doses for actuarial students. The supplements contained nutrients that worked with brain chemistry that improve focus, specifically appropriate to actuarial exam study. Finally the web page got to the real ad.