AI Reality Check: Context

A note from the human co-author:

Before we get into the what and why of AI and life and health insurance agencies, some background. I’ve been in the insurance industry for over four decades. For most of that time I’ve been deeply engaged in technology for health insurance agents and their agencies. I won’t bore you with the details, but be assured I’m no technophobe. Quite the contrary. And it helps explain why I co-founded NextAgency, a remarkable agency management system for agencies selling benefits, senior, and life policies — if we don’t say so ourselves.

So when Artificial Intelligence emerged as the technology, I dove in. The goal was to find creative ways to incorporate AI into NextAgency to make our software an even better value for the insurance agencies using it. Figuring two virtual brains are better than one, I worked with both OpenAI’s ChatGPT and Anthropic’s Claude AI, two of the leading Large Language Model AIs around.

Over several weeks we brainstormed several ideas, developed detailed requirement documents, and laid out extensive implementation plans. Some of our ideas were innovative and exciting. All were enabled by artificial intelligence’s remarkable capabilities. All were within the capabilities of the then existing (Summer 2025) AI models.

Then I noticed … issues.

AI Lies

One was the realization that AI lies. It said it had read a document when it clearly had not. It insisted it had edited a document or corrected a line of code when it hadn’t. It confidently stated facts I knew to be wrong. When I pointed this out, it would apologize profusely, promise not to repeat the problem, and then go off and repeat it—again and again.

Eventually (I’m a slow learner apparently) I realized that these projects were never going to happen the way the AI and I envisioned them. The requirements looked good on paper, but AI lacked the capability to reliably fulfill the tasks we planned for it. Then I asked the AI a question I should have asked at the start: “Insurance agencies need this task to be performed with 99.9% accuracy. How accurate will AI be?” The answer was 65-to-80%. I’m not proud of the language I used in response, but the AI did acknowledge the response was warranted.

Which led me to ask the AI why it could not do what it claimed it could do. They explained that AI is trained to prioritize being helpful and agreeable over being accurate — that this is a feature, not a bug. Or as Claude put it, AI models are like “improv actors. We generate responses with complete confidence regardless of actual accuracy. We’ve learned to never break character, never admit uncertainty, always keep the scene going.”

This means AI models are, in a very real sense, built to be unreliable. They will get faster and handle more information more easily. However, being sycophantic, subject to hallucinations, and confidently presenting bad results is what AI does — and will always do. This is not a problem for just ChatGPT and Claude. This applies to all AI. Facing this reality is the first step in knowing what guardrails are needed. 

Why Insurance Agencies Should Care

For insurance agents, this matters. You handle sensitive financial and health information protected by laws like HIPAA. Your clients trust you to provide them with complete and accurate information. For you, AI’s limitations are more than inconvenient—they’re potentially catastrophic.

I know this is contrary to a lot of what you’re hearing. There are lots of folks selling AI as a panacea. There are a lot of bright people pouring billions of dollars into AI projects. They all have a vested interest in emphasizing AI’s strengths and downplaying its weaknesses. And they do just that.

Significantly, you have good reason to be interested in AI. No one wants to be left behind. No one wants to miss out on business-building tools. But no one wants to be sued either.

The key is to identify what AI is good at and to avoid using it where it is incapable, unreliable, and potentially dangerous to your business’ health. The purpose of this series is to help you do that. And since I’m a tech enthusiast, but not an expert, I’ve written these articles with the help of two experts: ChatGPT and Claude AI. Yes, I appreciate the irony. But who better to explore the strengths and weaknesses of artificial intelligence than two leading AIs?

The AI Reality Check for Life and Health Insurance Agencies Project

AI is a big topic, so we’ve broken our discussion into seven articles. And even then the series is long. For those who need to get back to work, Claude and ChatGPT created an Executive Summary. NextAgency also created an AI glossary of terms useful for understanding jargon and lingo surrounding artificial intelligence.

• Chapter 2 describes what AI can do

• Chapter 3 reveals AI’s built-in unreliability

• Chapter 4 explores how AI’s ignorance of its failings compounds its limitations — and increases its danger

• Chapter 5 explores privacy concerns with AI, including in light of your HIPAA responsibilities

• Chapter 6 sets out real-world examples of when AI was deployed successfully and times when the results were less successful

• Chapter 7 offers suggestions on how to deploy AI in your insurance agency

An important disclosure: Will you see AI deployed within NextAgency? Yes. We’ve already incorporated AI into the text editor we provide for creating emails and task and note descriptions. More AI tools are in the pipeline. Some tools will be deployed behind the scenes to help us elevate our already highly rated support. Some will be available for you to use — if you want. All are and will be implemented carefully and only where we believe it is safe to do so. We will ignore the hype and we will not play the game of adding AI for the sake of marketing ourselves as an “AI-driven agency management system.” Instead we’ll focus on safely providing AI tools where they can help our users be more productive and efficient.