Testing AI: 1918 Influenza vs COVID-19
In the weeds look at Bing's answer to a simple question.
[AI generated image of an AI entity answering a question I asked it.]
I asked Microsoft’s Bing AI this question: “Compare the 1918 influenza epidemic with the recent COVID-19 epidemic in terms of mortality, accounting for differences in population size.”
I already know that the estimate for Influenza deaths ranges from 50 - 100 million, with 50 million being the number most people settle on. I know there were about 1.8 billion people in 1918.
I already know that about 7 million people died of COVID-19, in a world populated by about 8 billion people.
So the rates of population loss were: 2.7% of the population in 1918 and 0.09% of the population in the COVID-19 pandemic to date.
What does AI say?
Bing says…
Sure, let’s compare the 1918 influenza pandemic (often referred to as the “Spanish flu”) with the recent COVID-19 pandemic in terms of mortality rates and population size.
1918 Influenza Pandemic:
The world population in 1918 was estimated to be around 1.8 billion1.
The Spanish flu is estimated to have killed almost 1% of the world population2. This implies that the number of deaths was around 17.4 million2.
In the above, note that Bing has failed to correctly distinguish between the percent killed and the rate of mortality for a disease. Bing took the lowest generally available estimate of disease caused death, which happens to be the first estimate offered in the cited source. The source moves on from that low estimate, suggests the much higher 50 million estimate, then considers the 100 million estimate. Thereafter this source uses 50 million as the lowball and 100 million as the highball, completely moving on from the lowest number, the number Bing chose to use. Lesson learned: Bing found the source, used it wrong because it missed the nuance of the text (which was not too nuanced). But, Bing provided us with the link so we could go to check! OK, what else did Bing say….
Bing continues…
COVID-19 Pandemic:
As of 2022, COVID-19 was the underlying or contributing cause of approximately 244,986 deaths in the United States4. The estimated age-adjusted COVID-19–associated death rate decreased 47%, from 115.6 to 61.3 per 100,000 persons4.
As of 2023, over 760 million cases and 6.9 million deaths have been recorded worldwide since December 20195.
In this case, Bing has confused the disease’s mortality rate (your chance of dying if you get sick with the disease) with the mortality rate. Also, notice that Bing says the world population is 7.9 in 2022, and I said 8. Rounding difference? The correct rounding to the nearest tenth would be 8.0.
In the end, Bing does not actually answer the question; the answer is, as stated, 2.7% of the population in 1918 and 0.09% of the population in the COVID-19 pandemic to date.
Back to bing….
It’s important to note that these numbers are estimates and the actual numbers could be higher. Also, the mortality rate for both pandemics can vary greatly depending on the region and the population’s age, health status, and access to medical care. The COVID-19 pandemic is still ongoing, and the final impact in terms of mortality is yet to be determined. It’s also worth noting that the world population has grown significantly from 1918 to 2022, which can affect the comparison of absolute numbers of deaths. However, the mortality rate gives a more accurate comparison as it accounts for the population size.
Again, Bing is conflating mortality rate of a population with a disease’s mortality. Mortality rate is a simple number. If you die, you are part of numerator, everybody (including those who died) are in the denominator. A disease’s mortality is the chance will be dead instead of better after a disease you as an individual have. What is the chance a disease kills somebody in the population vs. what is the chance that a disease kills a person who caught it. Bing gets mortality rate (see last sentence for example) but confused some of the numbers that are disease mortality, and notes factors that affect mortality as affecting the simple count of who died.
Take-away: Maybe AI is at the same state Wikipedia is. Don’t use it to draw conclusions but it is a heck of a search engine. Another take-away: you can tell when you are reading AI generated text. Right? I think so, but I’m not 100% sure. In other words, the Turing Test might have shades of gray.
I did ask for a followup: “In your previous analysis of influenza vs covid mortality, it might have been good to make a clearer distinction between a disease's mortality rate (death rate) and the overall mortality rate in a population (the number who died divided by the total number of people).”
The response:
From Bing…
You’re absolutely right, and I appreciate your feedback. Let’s clarify the distinction:
Disease-specific mortality rate (death rate): This is the proportion of deaths within a designated population of “cases” (people with a specific disease), over the course of the disease. A “case” may be a person who has been diagnosed with the disease, or a person who is infected with the disease-causing agent. It’s usually expressed as a percentage. For example, if 100 people have a disease, and 5 of them die of it, the disease-specific mortality rate would be 5%.
Overall (or crude) mortality rate: This is the total number of deaths in a population due to any cause (not just a specific disease) during a specific period of time, divided by the total population at risk of dying during the same time period. It’s usually expressed per 1,000 or 100,000 individuals per year. For example, if there are 1,000 deaths in a population of 100,000 people during a year, the overall mortality rate would be 1,000 per 100,000, or 1%.
In the context of the 1918 influenza pandemic and the COVID-19 pandemic, the disease-specific mortality rate would refer to the proportion of people with each disease who died from it, while the overall mortality rate would refer to the proportion of the entire population (infected and uninfected) who died from each disease.
I hope this helps clarify the difference between these two important epidemiological measures. Let me know if you have any other questions!
Better (but not perfect) on the definitions, but Bing did not re-state the answer to the original question. I assume that Bing subscribes to my Substack, and will make necessary adjustments in the future.