SEG Measurement

Here is this week's math story. To help solve a range of problems during WWII, the government assembled an extraordinary group of statisticians known as the Statistical Research Group (SRG). The SRG included Frederick Mosteller, who would later found Harvard’s statistics department, Leonard Jimmie Savage, the pioneer of decision theory and Bayesian statistics, Norbert Wiener, the MIT mathematician who created cybernetics, and Milton Friedman, future Nobelist in economics. The group also included Abraham Wald, who was arguably the smartest person in the room. He was a master of turning a vague idea into solid mathematics. The military came to the SRG with a problem. The military wanted to minimize the number of planes getting shot down by enemy fighters and wanted to figure out how to best armor them. But armor makes the plane heavier, less maneuverable and greater consumers of fuel. Armoring the planes too much is a problem as is armoring the planes too little; they were looking for the optimal level of armor. The military provided the SRG with data indicating the number and location of bullet holes for planes returning from engagements over Europe.  The damage wasn’t uniformly distributed across the aircraft; they observed more bullet holes in the fuselage, with very few in the engines. Prevailing thought was that the most efficient protection could be achieved if the armor was concentrated where the planes were getting hit the most. But they were unsure how much more armor belonged on those parts of the plane and posed the question to Wald. Wald surprised everyone with his response indicating that the armor should not go where the bullet holes are. Rather, the armor should go where the bullet holes aren’t! (on the engines).  Wald simply asked: where are the missing holes? The ones that would have been all over the engine casing, if the damage had been spread equally all over the plane? Wald reasoned that the missing bullet holes were on the missing planes. The reason planes were coming back with fewer hits to the engine is that planes that got hit in the engine weren’t coming back at all. The large number of planes returning to base with a Swiss-cheesed fuselage was pretty strong evidence that hits to the fuselage can be tolerated. This math is hardly the stuff war movies are made of, but it’s how wars are won. This points out the importance of asking, “What assumptions am I making? And are they justified?” In this case, the team was unintentionally making a false assumption, that the planes that came back were a random sample of all the planes. That assumption was dead wrong; there’s no reason at all to expect the planes to have an equal likelihood of survival no matter where they get hit. The principle underlying this bullet hole problem is a phenomenon called survivorship bias.

Here is an interesting perspective on the release of OpenAI's new AI Model. (called o1) This is the most significant advance in LLM AI models to date. Various AI models were administered an IQ test. The highest IQ score for past models was under 95. The new o1 model came in with an IQ of 120. While this is hardly a complete picture, it is pretty impressive. I have only had a limited chance to play with o1, and so far have found it an improvement--I will have a better idea by the end of the week. It is increasingly clear that this will change education in many ways--some we cannot even envision.

I have worked for nearly 50 years in assessment. I have seen some pretty awful multiple choice questions during that time. But nothing compares with what I've seen on LinkedIn. In recognition of this, I offer the following:

Interesting Development in AI (Excerpted from AI Insights newsletter) Elon Musk, the visionary behind Tesla and SpaceX, has launched an ambitious new project with his company xAI. They’re building what they call the “Memphis Supercluster” in Memphis, Tennessee, aiming to revolutionize AI training with unprecedented power and scale. The Details: § Memphis Supercluster: xAI’s new project involves a powerful AI training cluster equipped with 100,000 Nvidia H100 GPUs, aiming to set a new standard in AI capabilities. § Strategic Development: The cluster, housed in Memphis, represents a significant investment and is poised to become a major technological hub in the region. § Ambitious Goals: Musk aims for xAI to develop the most potent AI by all standards by December 2024, using this state-of-the-art infrastructure. § Competitive Landscape: The initiative places xAI in direct competition with giants like Google, Microsoft, and OpenAI, especially as these companies also accelerate their AI development efforts. Elon Musk’s involvement in the Memphis Supercluster could significantly influence the future of AI. This venture isn’t just about advancing technology—it’s about setting new standards and exploring new possibilities that could one day impact various sectors from healthcare to transportation. It’s a bold step that might change how we interact with technology, making it a development worth watching. #AI #Edtech #educationalpublishing #educationalresearch #educationalassessment