Aligning Amazon Bedrock with NAIC AI Principles and Model Bulletin

The rapid advancement of generative AI promises new strategies for innovation in the insurance industry, and insurance carriers should implement it in compliance with regulatory requirements. The National Association of Insurance Commissioners (NAIC) has adopted Principles on Artificial Intelligence and released a Model Bulletin on the use of artificial intelligence systems by insurers to provide guidance on the responsible development and use of AI in insurance. As insurance companies increasingly leverage Amazon Bedrock for AI applications, it’s helpful that they understand how to align these implementations with NAIC guidelines.

In this article, we’ll explore key considerations for addressing NAIC AI Principles and Model Bulletin requirements when using Amazon Bedrock. We’ll cover governance, risk management controls, and third-party considerations, which are the three pillars of an AIS (AI System) Program as defined by NAIC.

Implementing an AIS Program

Infographic showing the components of an AI Systems Program

An AIS, as defined by NAIC, is a machine-based system that can generate outputs such as predictions, recommendations, content, or other output influencing decisions made in real or virtual environments. The NAIC Model Bulletin recommends insurers develop and maintain a written program for the responsible use of AI Systems, referred to as an AIS Program. The AIS Program should address governance, risk management, and oversight across the AIS lifecycle. The NAIC’s requirements closely align with the responsible AI dimensions: fairness, transparency, privacy and security, safety, controllability, veracity and robustness, explainability, and governance. While there is different terminology used in the NAIC Model Bulletin, its focus on risk management, data protection, model oversight, and consumer protection, reflects similar principles, and therefore, AWS provides resources and service features which support the implementation of an AIS Program.

Here is how insurers can implement an AIS program with Amazon Bedrock which covers all the NAIC Model Bulletin guidelines:

Governance

AI governance that satisfies Model AI Bulletin guidance prioritizes transparency, fairness, and accountability in AIS design and implementation. It encompasses frameworks, policies, and rules that guide AI development and use, enabling stakeholders to make informed decisions. Key aspects include transparency in data usage, AI decision-making processes, and potential user impacts. AWS offers services, tools, and a comprehensive AI governance framework to help establish and operationalize these practices, covering areas including data and model governance, application monitoring, auditing, and risk management.

Risk Management and Internal Controls

The AIS Program should address the insurer’s risk identification, mitigation, management framework, and internal controls for AI Systems generally and at each stage of the AIS life cycle (see Section 1.7). We will look at these features in terms of AWS’s core dimensions of responsible AI, which we consider to be the most effective way to manage risk.

The following diagram shows how you can use Amazon Bedrock features in the full AI system life cycle to implement responsible AI in risk management and controls.

GenAI example workflow with Amazon Bedrock aligned with the core dimensions of responsible AI (Source)

Fairness
Fairness evaluates the impact of AI Systems on various stakeholder groups. Insurers can utilize Amazon Bedrock’s integrated model evaluation features to conduct thorough assessments of AI models across diverse demographic factors.

Amazon Bedrock offers automated evaluations using curated datasets such as Bias in Open-ended Language Generation Dataset (BOLD) for evaluating text generation models across profession, gender, race, religious ideologies, and political ideologies. This helps them identify potential biases that may lead to unfair treatment of consumers.

Insurers can also create custom datasets within Amazon Bedrock to evaluate fairness in line with industry-specific regulatory requirements. By using automated evaluations with a custom prompt dataset for model evaluation, insurers can thoroughly assess model performance across relevant demographic groups and characteristics unique to insurance operations. Insurers can use both the AWS and custom-provided datasets with Amazon Bedrock Evaluations to evaluate the generated content of their end-to-end Retrieval Augmented Generation (RAG) workflow.

Insurers should use these evaluation tools throughout the AIS lifecycle: from development to deployment and ongoing operation.

Transparency
AWS helps provide transparency to customers in artificial intelligence by introducing AWS AI Service Cards. These cards serve as a dedicated resource designed to deepen customers’ comprehension of AWS AI services and models. They offer a wealth of information, providing comprehensive insights into the intended use cases and limitations of each service and model. Moreover, these cards detail the responsible AI design principles underpinning the models and outline best practices for their deployment and performance optimization.

For Amazon Bedrock models not developed by AWS, model providers publish their own service cards (for example, Anthropic’s Transparency Hub: Model Reports and Meta Llama Models). Each of these cards offers a detailed overview of the specific use cases for which the model is designed and limitations to consider. They also delve into the machine learning techniques employed in the service’s development. Crucially, these cards highlight important considerations for the responsible use of the model in an AIS, ensuring that customers have all the necessary information to utilize these tools ethically and effectively.

Explainability
Explainability focuses on understanding and evaluating system outputs. By using an explainable AI framework, humans can examine the models to better understand how the models produce their outputs. To enhance the output of a generative AI model with information supporting explainability, you use techniques like training data attribution, ReAct prompting, and Chain of Thought (CoT) prompting.

For insurers that need to include audit or attribution information with generative AI model responses, we recommend using RAG with an Amazon Bedrock Knowledge Base. Information retrieved from the knowledge base comes with source attribution and provide a simple way to verify responses and makes it easier to minimize hallucinations. Amazon Bedrock Knowledge Bases manages the end-to-end RAG workflow for you. When using the RetrieveAndGenerate API, the output includes the generated response, the source attribution, and the retrieved text chunks.

Privacy and Security
NAIC emphasizes the importance of protecting non-public information and maintaining data security in AI Systems. Amazon Bedrock addresses these concerns through comprehensive security and privacy features.

Data Privacy and Security
Amazon Bedrock does not store or log your prompts and completions; Amazon Bedrock does not use your prompts and completions to train any AWS models and does not distribute them to third parties.

The service maintains all data within the customer’s chosen AWS Region, which helps customers to meet data localization requirements. Amazon Bedrock provides this functionality regardless of how an insurer implements it in their AIS.

Encryption and Access Control
Amazon Bedrock uses encryption for data in transit and at rest. Insurers can use AWS Key Management Service (KMS) for encrypting certain Amazon Bedrock resources, with the option to use customer-managed keys for full control of the keys. A full list of resources is available in the Amazon Bedrock User Guide. AWS Identity and Access Management (IAM) provides granular access control to Amazon Bedrock resources.

Network Security
Amazon Bedrock offers secure network configurations through Amazon Virtual Private Cloud (Amazon VPC) and AWS PrivateLink. These features provide insurers the ability to establish private connections within their AIS and mitigate exposure to internet traffic.

Model Customization Security
Amazon Bedrock’s architecture prevents model providers from accessing customer data used in fine-tuning and distillation processes. Additionally, AWS Key Management Service (KMS) encryption is applied to all training data and customized models, safeguarding against unauthorized access by third parties. This multi-layered approach maintains customer data privacy and security throughout the model customization workflow. Amazon Bedrock uses Model Deployment Accounts to which model providers do not have access.

Safety
Insurers can leverage Amazon Bedrock Guardrails to implement robust governance controls directly within their AI applications. This powerful feature helps enforce safety measures and compliance standards across all model-access through Amazon Bedrock (AWS and 3rd party).

One key capability is the use of content filters to detect and filter harmful or toxic content in both user inputs and model outputs. This helps to confirm that the AIS maintains appropriate interactions and produces safe content.

Insurers can also define denied topics to prevent discussions in their AIS around sensitive areas, including medical advice and specific financial recommendations, which helps them maintain compliance with applicable insurance laws and regulations.

In the below screenshot of the Amazon Bedrock console, you can see an example denied topic which prevents a claim agent from providing medical advice to claimants.

Example Amazon Bedrock content filter configuration for an insurance claim agent that blocks giving medical advice to a claimant.

The next screenshot shows how Amazon Bedrock Guardrails enforces the denied topic by matching a response and suppressing it with a configured guardrail response.

Amazon Bedrock Guardrails test of content filter showing a filtered denied topic.

Furthermore, Amazon Bedrock Guardrails also provide word filters, which insurers can configure to block specific terms, phrases, or proprietary information in both input prompts and model responses. This granular control helps protect sensitive business information and maintain the integrity of the AIS’s outputs.

Particularly relevant to the insurance industry’s data protection requirements, Amazon Bedrock Guardrails offer sensitive information filters to protect personally identifiable information (PII) and other sensitive data.

Amazon Bedrock Guardrails with Automated Reasoning Checks provides an additional layer of security and accuracy. This feature gives domain experts the capability to build deterministic rules that define policy administration and claim workflows and policies. AIS administrators can validate generated content against an Automated Reasoning Policy to identify inaccuracies and unstated assumptions and to explain why statements are accurate in a verifiable way. This capability is valuable for insurers who need to confirm the precision and reliability of AI-generated content in complex policy and claims scenarios.

The following screenshot shows an Automated Reasoning Policy and rules for a life insurance claim support agent.

Amazon Bedrock Automated Reasoning Policy example for a claims agent.

By implementing these safeguards, insurers can demonstrate to regulators their commitment to responsible AI use, risk management, and consumer protection.

Controllability
The NAIC Model Bulletin emphasizes the importance of maintaining control over AI Systems to confirm they operate within regulatory boundaries and insurance industry standards. Amazon Bedrock offers features that support this requirement through its Amazon Bedrock Guardrails capabilities, which we discussed under the Safety section. These tools provide insurers with direct control over AI application outputs, giving them the ability to steer AI behavior to comply with regulatory standards and company policies.

Insurers can utilize content filters to set sensitivity levels for detecting inappropriate content, helping to avoid outputs that could violate unfair trade practices guidelines. By defining and managing denied topics, they can confirm AI Systems remain within permissible operational boundaries. Amazon Bedrock Guardrails also supports compliance with content policies and privacy standards through custom word filters, denied topics, and content filters, helping insurers to control AI language use and handle sensitive data in accordance with data protection and privacy requirements.

To meet expectations for ongoing oversight, insurers will leverage Amazon Bedrock’s model evaluation capabilities, which include both automatic and human-in-the-loop evaluations. These features facilitate continuous monitoring and adjustment of AI performance to meet regulatory and ethical standards. Insurers that integrate scheduled assessments and custom testing into their AI development pipeline will be better positioned to demonstrate their commitment to maintaining control over AI Systems.

Veracity and Robustness
Veracity and robustness in AI focus on achieving correct system outputs, even when faced with unexpected or adversarial inputs. Insurers should implement testing and controls for model hallucinations, where AI Systems might generate false or misleading information that appears plausible. In the insurance context, such inaccuracies could lead to unfair practices or misleading information provided to consumers.

Amazon Bedrock offers tools for evaluating AI models in terms of toxicity, robustness, and accuracy. These evaluations help verify that models don’t produce harmful, offensive, or inappropriate content and help maintain consistent performance across diverse and challenging conditions. This is essential for insurers to meet state and federal regulatory expectations regarding the reliability and fairness of their AI Systems.

Accuracy evaluation in Amazon Bedrock’s Model Evaluation measures how well a model’s predictions match actual results in a test dataset. An insurer can use a pre-built dataset like TREX or their own custom prompt dataset. Model Evaluation calculates a real-world knowledge score representing the model’s ability to encode factual knowledge. This evaluation helps insurers understand how to use their AI System and maintain integrity and credibility.

Insurers should consider employing the following techniques to enhance veracity and robustness in insurance AI Systems:

Prompt engineering can instruct the model to only discuss known information, reducing the risk of generating inaccurate or speculative content.
Chain of Thought (CoT) prompting techniques improve the model’s problem-solving ability by making its reasoning process transparent, which is valuable for explaining AI decisions in insurance contexts.
ReAct prompting is a technique that lets language models generate both reasoning traces and task-specific actions in an interleaved manner, allowing them to dynamically reason, plan, and interact with external sources to solve complex problems more effectively.
Retrieval Augmented Generation provides context, augmenting outputs with internal data, ensuring responses are grounded in accurate and company-specific information.
Fine-tuning and model distillation help improve model accuracy for insurance-specific contexts without large volumes of annotated data.
Adjusting inference parameters control the model’s creativity, which is important for maintaining factual accuracy in insurance-related responses while balancing the ability of the model to respond to unique customer experiences.
Contextual grounding checks in Amazon Bedrock Guardrails detect and filter responses that deviate from source information or are irrelevant to user queries.

These methods help insurers develop AI Systems that produce reliable, accurate, consistent and verifiable outputs, even when dealing with complex or unexpected inputs. This reliability is key for maintaining consumer trust and meeting regulatory requirements in the insurance industry.

It’s important to note that while these techniques significantly improve AIS performance, insurers should still consider implementing disclaimers about potential AI inaccuracies, and human-in-the-loop workflows.

Monitoring
NAIC emphasizes the importance of robust monitoring and auditing practices for AI Systems used in insurance operations. Insurers can leverage Amazon Bedrock’s built-in integrations with Amazon CloudWatch and AWS CloudTrail for comprehensive monitoring and auditing capabilities.

Amazon CloudWatch collects and processes raw data from Amazon Bedrock into readable near real-time metrics which insurers can use to track critical usage metrics such as model invocations and token count. For example, insurers can build customized dashboards for audit purposes, which cover one or multiple foundation models across single or multiple AWS accounts.

AWS CloudTrail complements these monitoring capabilities by providing a centralized logging service that records user and API activities in Amazon Bedrock. By creating a trail within CloudTrail, insurers collect API data in log files delivered to an Amazon Simple Storage Service (Amazon S3) bucket. This detailed activity logging is crucial for maintaining the audit trails necessary to demonstrate compliance.

Additionally, Amazon Bedrock offers model invocation logging, which collects model input data, prompts, model responses, and request IDs for all invocations in the insurer’s AWS account. This feature provides valuable insights into how models are performing and being used for insurers to make data-driven and responsible decisions about their AI applications. Importantly, these logs also offer traceability data, that insurers can use to track and audit model interactions over time. Insurers can choose to store these logs in either an S3 Bucket or CloudWatch logs with flexible data retention strategies that support to various regulatory and internal compliance needs. This traceability through comprehensive logging is important for maintaining transparency, conducting audits, and demonstrating regulatory compliance in AI-driven insurance operations.

Third-Party Considerations

When incorporating AI Systems or data from third parties, insurers should establish comprehensive due diligence, contractual requirements, and oversight processes as part of their AI governance program.

Insurers should thoroughly review AWS AI Service Cards for AWS models, and external model cards for other providers. These documents provide essential information about intended use cases, limitations, responsible AI design principles, and best practices for deployment and performance optimization. This review process helps insurers understand the capabilities and potential risks associated with third-party AI Systems.

To validate third-party model performance, insurers should implement custom evaluation datasets tailored to their specific use cases and regulatory requirements, so that they have a more accurate assessment of how the model will perform in scenarios specific to the insurer.

When conducting vendor assessments, insurers can leverage Amazon Bedrock’s built-in security and compliance certifications. Amazon Bedrock is in scope for many compliance standards including ISO, SOC, FedRAMP moderate, PCI, ISMAP, and CSA STAR Level 2. Amazon Bedrock has ISO/IEC 42001 accredited certification, which outlines requirements and controls for organizations to promote the responsible development and use of AI Systems. It is also eligible for Health Insurance Portability and Accountability Act (HIPAA) compliance and insurers can use it in compliance with the General Data Protection Regulation (GDPR). The inclusion of Amazon Bedrock in the Cloud Infrastructure Service Providers in Europe Data Protection Code of Conduct (CISPE CODE) Public Register provides independent verification of its GDPR compliance capabilities. For the most up-to-date information about whether Amazon Bedrock is within the scope of specific compliance programs, see AWS services in Scope by Compliance Program and choose the compliance program you’re interested in.

Conclusion

Developing AI Systems that comply with NAIC Model Bulletin guidelines requires a structured approach with formal governance, risk management, and controls. Amazon Bedrock offers built-in capabilities that support the development and deployment of AI Systems meeting regulatory standards for transparency, fairness, and accountability in insurance operations.

Amazon Bedrock’s customizable features and ability to integrate insurer-specific datasets help insurers design AI Systems tailored to unique insurance requirements. Insurers can also use a variety of foundational models effectively while adhering to industry standards and ethical practices.

Insurers should prioritize responsible AI practices throughout the entire lifecycle of their AI Systems, from initial development to ongoing operation. This proactive approach helps verify continuous compliance with NAIC requirements and mitigates risks associated with AI use in insurance. By embedding these principles at every stage, insurers can create AI Systems that are both innovative and compliant with industry regulations, fostering trust with policyholders and regulators alike.

AWS for Industries

Aligning Amazon Bedrock with NAIC AI Principles and Model Bulletin

Implementing an AIS Program

Governance

Risk Management and Internal Controls

Third-Party Considerations

Conclusion

Additional Resources

Resources

Follow