Risk Assessment Workshop

Apply your AI risk assessment knowledge through practical exercises. This workshop provides hands-on experience identifying, analyzing, and treating AI risks.

Workshop Overview

Objectives:

Apply risk assessment methodology to real scenarios
Practice identifying AI-specific risks
Develop risk treatment plans
Experience stakeholder collaboration

Duration: 60-90 minutes

Materials Needed:

Risk register template (Lesson 2.6)
Scenario descriptions (below)
Stakeholder perspective cards
Risk assessment tools

Workshop Scenario 1: Healthcare Diagnostic AI

Scenario Description

System: MedAI - AI-powered diagnostic assistant for detecting pneumonia from chest X-rays

Context:

Used by physicians in emergency departments
Processes X-ray images, provides diagnostic suggestions with confidence scores
Deployed across 50+ hospitals
Processes 1000+ images per day
Critical for triage decisions in busy ERs

Technical Details:

Deep learning model (convolutional neural network)
Trained on 100,000 chest X-rays from 5 major teaching hospitals
92% accuracy on validation set
Provides heatmap showing areas of concern
Integrates with hospital electronic health record (EHR)

Stakeholders:

Emergency physicians (primary users)
Radiologists (secondary review)
ER patients (affected by diagnoses)
Hospital administration
Regulatory bodies (FDA, state medical boards)
Insurance companies
Medical device regulators

Part 1: Risk Identification (15 minutes)

Task: Identify at least 10 distinct risks across multiple categories.

Guiding Questions:

What if the model makes incorrect diagnoses?
Could the training data be biased?
How might physicians over-rely on AI?
What privacy concerns exist?
Could the system be attacked or manipulated?
What if performance degrades over time?
Are there equity issues in performance across groups?
What regulatory compliance is required?

Categories to Consider:

Bias and fairness
Safety and accuracy
Transparency and explainability
Data quality
Security
Privacy
Regulatory compliance
Human oversight
Technical reliability

Sample Risks (for reference):

R001: False negative risk - AI fails to detect pneumonia, leading to delayed treatment R002: Bias risk - Lower accuracy for certain demographics (pediatric, elderly, or specific ethnicities) R003: Automation bias - Physicians over-rely on AI, missing findings AI doesn't detect R004: Privacy breach - Patient X-ray data exposed or misused R005: Adversarial attack - Manipulated images fool the AI R006: Data drift - Model performance degrades as X-ray equipment or techniques evolve R007: Transparency gap - Cannot explain why AI flagged certain regions R008: Regulatory non-compliance - System used without proper FDA clearance R009: Integration failure - Wrong patient data linked to X-ray R010: Access inequality - Only available at well-resourced hospitals

Your Turn: List additional risks you identify.

Part 2: Risk Analysis (20 minutes)

Task: For 3 priority risks, assess likelihood and impact.

Select 3 risks from your list that seem most critical.

For Each Risk, Complete:

Risk 1: __________________________________

Likelihood (Very Unlikely / Unlikely / Possible / Likely / Very Likely):

Consider: training data quality, model complexity, testing rigor, deployment context
Justification: __________________________________

Impact - Individual (Negligible / Minor / Moderate / Major / Severe):

Consider: patient health, missed diagnoses, delayed treatment, harm caused
Description: __________________________________

Impact - Organizational (Negligible / Minor / Moderate / Major / Severe):

Consider: liability, reputation, regulatory action, financial loss
Description: __________________________________

Impact - Legal/Regulatory (Negligible / Minor / Moderate / Major / Severe):

Consider: FDA regulations, medical liability, compliance violations
Description: __________________________________

Overall Risk Level: ____________________

Priority: P1 (Critical) / P2 (High) / P3 (Medium) / P4 (Low)

Sample Analysis (for reference):

Risk: False Negative (AI fails to detect pneumonia)

Likelihood: Possible (30%)

High-quality training data reduces risk
But edge cases (atypical presentations) likely missed
Validation accuracy 92% suggests ~8% miss rate on known data
Real-world performance may be worse

Impact - Individual: Severe

Delayed or missed pneumonia diagnosis
Can lead to sepsis, respiratory failure, death
Particularly dangerous for elderly, immunocompromised

Impact - Organizational: Major

Medical malpractice liability
Reputational damage if publicized
Loss of trust in AI system
Potential product recall

Impact - Legal/Regulatory: Major

Medical malpractice lawsuits
FDA adverse event reporting
Possible consent decree or restrictions
Regulatory scrutiny of all AI-assisted diagnoses

Overall Risk Level: HIGH (Possible + Severe/Major)

Priority: P2 (High - must address before broader deployment)

Part 3: Stakeholder Perspectives (15 minutes)

Task: Consider how different stakeholders view risks differently.

Stakeholder Role Play:

Emergency Physician:

Primary concern: __________________________________
Risk tolerance: __________________________________
Top priority risks: __________________________________

Patient:

Primary concern: __________________________________
Risk tolerance: __________________________________
Top priority risks: __________________________________

Hospital Administrator:

Primary concern: __________________________________
Risk tolerance: __________________________________
Top priority risks: __________________________________

Regulator (FDA):

Primary concern: __________________________________
Risk tolerance: __________________________________
Top priority risks: __________________________________

Discussion Questions:

Which risks do stakeholders agree on as highest priority?
Where do stakeholders disagree about risk importance?
How should conflicting priorities be balanced?
Whose perspective should carry most weight for which risks?

Part 4: Risk Treatment Planning (20 minutes)

Task: Develop treatment plan for one high-priority risk.

Select One High-Priority Risk: ____________________________

Treatment Strategy (Avoid / Reduce / Transfer / Accept): _______________

Justification: _______________________________________

Specific Controls:

Control 1:

Description: __________________________________
Type (Preventive/Detective/Corrective): ____________________
Responsible Party: _______________________________
Timeline: __________________________________
Resources Needed: _______________________________

Control 2:

Description: __________________________________
Type: __________________________________
Responsible Party: _______________________________
Timeline: __________________________________
Resources Needed: _______________________________

Control 3:

Description: __________________________________
Type: __________________________________
Responsible Party: _______________________________
Timeline: __________________________________
Resources Needed: _______________________________

Success Criteria: __________________________________

Residual Risk Level (after controls): ________________________

Monitoring Plan:

Metrics: __________________________________
Frequency: __________________________________
Responsible Party: _______________________________
Alert Thresholds: _______________________________

Sample Treatment Plan (for reference):

Risk: False Negative (missed pneumonia)

Treatment Strategy: Reduce

Justification: Cannot avoid using AI (provides value), cannot fully eliminate risk (AI not 100% accurate), reduction to acceptable level is feasible.

Control 1 - Comprehensive Testing:

Description: Test across diverse patient populations, X-ray equipment types, and pneumonia presentations including rare cases
Type: Preventive
Responsible: QA Team + Clinical advisors
Timeline: 3 months before deployment expansion
Resources: $50K for diverse test dataset curation

Control 2 - Physician Override and Review:

Description: Physician must review all AI outputs, can override, AI is decision support not decision-maker
Type: Preventive + Corrective
Responsible: Medical Director
Timeline: Built into clinical workflow, ongoing
Resources: Training materials, workflow design

Control 3 - Confidence Thresholding:

Description: Cases with low AI confidence flagged for senior physician or radiologist review
Type: Detective + Corrective
Responsible: Engineering team + Clinical team
Timeline: 2 weeks to implement
Resources: 1 engineer, clinical input on threshold

Control 4 - Continuous Monitoring:

Description: Track diagnostic accuracy, missed cases, adverse outcomes. Monthly review by clinical committee.
Type: Detective
Responsible: Clinical Quality Officer
Timeline: Ongoing from deployment
Resources: Monitoring dashboard, monthly meeting

Control 5 - Incident Reporting:

Description: Clear process for reporting suspected AI errors, rapid investigation and learning
Type: Detective + Corrective
Responsible: Patient Safety Officer
Timeline: Immediate, from day 1
Resources: Reporting system, investigation protocol

Success Criteria:

False negative rate < 5% on diverse test set
No preventable pneumonia deaths attributed to AI miss
Physician trust and satisfaction >85%
Incident investigation within 48 hours
No regulatory actions

Residual Risk Level: MEDIUM (Unlikely + Moderate)

Likelihood reduced to Unlikely with comprehensive testing and human oversight
Impact reduced to Moderate because physician can catch errors

Monitoring Plan:

Metrics: False negative rate, physician override rate, adverse events, incident reports
Frequency: Daily dashboard, weekly team review, monthly clinical committee
Responsible: Clinical Quality Officer
Alerts: False negative rate >6%, adverse event, incident report

Workshop Scenario 2: Financial Credit Scoring AI

Scenario Description

System: CreditSmart - AI for automated credit decisions

Context:

Online lending platform for personal loans
Fully automated approval/denial for loans up to $25,000
Processes 10,000+ applications per month
Targets underserved populations without traditional credit history
Uses alternative data (rent, utilities, employment)

Technical Details:

Gradient boosted decision tree ensemble
Trained on 500,000 historical loan applications and outcomes
Accuracy 78% in predicting loan repayment
Uses 150+ features including alternative credit data
Integrates with credit bureaus and bank accounts

Stakeholders:

Loan applicants (affected by decisions)
Underwriting team (reviews denials)
Compliance team (ensures fair lending)
Regulators (CFPB, state banking authorities)
Investors (depend on accurate risk assessment)
Consumer advocacy groups

Exercise Tasks

Part 1 - Risk Identification (10 minutes): List at least 8 risks. Consider bias, fairness, compliance, explainability, privacy, and data quality.

Part 2 - Fairness Analysis (15 minutes): This system claims to help underserved populations, but what fairness risks exist?

Could it discriminate against protected groups?
What if "alternative data" is biased?
How would you test for disparate impact?
What fairness definition is appropriate?

Part 3 - Transparency Requirements (10 minutes): Under fair lending laws, applicants must receive adverse action notices explaining denials.

What information must be provided?
How can a complex model be explained?
What challenges exist?
Design an explanation for a denied applicant.

Part 4 - Risk Treatment (15 minutes): Develop a treatment plan for bias/discrimination risk.

What controls prevent bias?
How would you test for fairness?
What monitoring is needed?
What's the approval process?

Workshop Scenario 3: Social Media Content Moderation AI

Scenario Description

System: SafeFeed - AI for detecting and removing harmful content

Context:

Social media platform with 100M users
AI reviews posts, images, videos for policy violations
Removes content or flags for human review
Processes millions of posts per day
Content categories: hate speech, violence, misinformation, spam

Technical Details:

Ensemble of models (text, image, video analysis)
NLP for text, computer vision for images
Trained on millions of labeled examples
95% accuracy on test set
Some categories (misinformation) harder than others

Stakeholders:

Platform users (posting content)
Users exposed to content
Content moderators (human reviewers)
Civil liberties groups
Regulators
Advertisers
Vulnerable populations

Exercise Tasks

Part 1 - Risk Identification (10 minutes): Identify risks in multiple categories. Consider false positives, false negatives, bias, free speech, psychological harm to moderators.

Part 2 - Ethical Dilemmas (15 minutes): Explore ethical tensions:

Free speech vs. harm prevention
Over-moderation vs. under-moderation
Consistency vs. context
Transparency vs. gaming
Scale vs. accuracy How would you balance these?

Part 3 - Stakeholder Impact (10 minutes): Different stakeholders have different views:

Free speech advocates worry about censorship
Safety advocates worry about harm
Marginalized groups worry about biased enforcement
How do you balance competing interests?

Part 4 - Risk Treatment (15 minutes): Design a governance framework:

What policies guide AI decisions?
What human oversight is needed?
How transparent should you be?
What appeals process?
How do you handle edge cases?

Group Discussion Questions

After completing scenarios, discuss:

1. Common Patterns:

What risks appeared across multiple scenarios?
What risk categories are most challenging?
What patterns in treatment strategies emerged?

2. Contextual Factors:

How did context (healthcare vs. finance vs. social media) change risk priorities?
What stakeholder concerns varied by domain?
How did regulatory landscape differ?

3. Challenges:

What was hardest about risk identification?
What trade-offs were most difficult?
Where did you lack information?
What would you need in real situations?

4. Best Practices:

What worked well in your approach?
What would you do differently?
What tools or frameworks would help?
How would you improve the process?

5. Real-World Application:

How does this apply to your organization's AI?
What risks does your organization face?
What controls are already in place?
What gaps exist?

Key Takeaways

Risk Assessment is Multidimensional:

Technical, ethical, legal, social factors
Multiple stakeholder perspectives
Context-dependent priorities

No Perfect Solutions:

Trade-offs inevitable
Residual risk always remains
Continuous monitoring essential

Stakeholder Engagement Critical:

Different perspectives reveal different risks
Collaboration improves solutions
Transparency builds trust

Controls Must Be Comprehensive:

Multiple layers of defense
Preventive, detective, corrective
Technical and organizational measures

Documentation Matters:

Risk register is living document
Evidence for compliance and audit
Organizational learning

Continuous Process:

Risks evolve over time
Regular review and updates
Adapt to new information

Next Steps After Workshop

1. Apply to Your AI Systems:

Conduct risk assessments for your organization's AI
Use risk register template
Engage diverse stakeholders
Document thoroughly

2. Develop Risk Management Capability:

Train team on AI risk assessment
Establish governance processes
Build risk management culture
Integrate into development lifecycle

3. Monitor and Learn:

Track risks continuously
Learn from incidents
Update risk register
Share lessons across organization

4. Seek External Input:

Consult ethics and domain experts
Engage affected communities
Consider third-party audits
Benchmark against industry

5. Advance to Module 3:

You've now mastered AI risk identification and assessment
Ready to implement controls (Module 3)
Apply learnings to practical control implementation
Build comprehensive AI governance

Workshop Evaluation

Self-Assessment:

I can confidently:

Identify AI risks across multiple categories
Assess likelihood and impact of AI risks
Consider multiple stakeholder perspectives
Develop risk treatment plans
Apply risk assessment methodology
Use risk register template effectively

Areas for Further Development:

Questions or Unclear Topics:

How I Will Apply This:

Module 2 Complete!

You've now mastered:

AI risk assessment process (Lesson 2.1)
Bias and fairness risks (Lesson 2.2)
Transparency and explainability (Lesson 2.3)
Data quality risks (Lesson 2.4)
Security and adversarial risks (Lesson 2.5)
Risk register template (Lesson 2.6)
Practical risk assessment (Lesson 2.7)

Achievement Unlocked: Risk Navigator Badge 🎖️

Ready for Module 3: AI Controls Implementation - Turning risk mitigation plans into action!