Patch Management Challenges

Patch management for operational technology is fundamentally different from IT patching. ISO 27019 recognizes these unique challenges and provides risk-based guidance for keeping energy systems secure while maintaining reliability.

Why OT Patching is Different

Operational Constraints

No Maintenance Windows

• 24/7 operations cannot stop for patching
• Annual or semi-annual outages only opportunity
• Emergency outages too risky for non-critical patches
• Patching happens on operational schedule, not security schedule

High Availability Requirements

• Systems must maintain 99.99%+ uptime
• Redundancy reduces but doesn't eliminate patching challenges
• Patches may require full system restarts
• Testing redundancy failover is itself risky

Change Control Rigor

• Weeks or months of planning for changes
• Extensive testing required
• Multiple approval levels
• Documentation requirements
• Rollback plans mandatory

Technical Constraints

Vendor Support Limitations

• Legacy systems out of support
• Vendors may not provide patches for older versions
• Patches may require expensive hardware upgrades
• Some vendors release patches infrequently

Testing Challenges

• Limited or no test environments
• Test environments don't match production
• Difficult to simulate all operational scenarios
• Risk of production-only issues

System Interdependencies

• Patches may affect interconnected systems
• Protocol compatibility concerns
• Timing and latency sensitive operations
• Third-party integration issues

ISO 27019 Patching Approach

Risk-Based Patching

Not all vulnerabilities require immediate patching:

Compensating Controls

When patching is delayed or not possible:

Network-Based Protections

• Additional network segmentation
• Firewall rules blocking exploit paths
• Intrusion prevention systems (IPS)
• Network monitoring for exploit attempts

Access Controls

• Restrict access to vulnerable systems
• Remove internet connectivity
• Disable vulnerable services
• Increase authentication requirements

Monitoring and Detection

• Enhanced logging of vulnerable systems
• Signature-based detection for known exploits
• Behavioral monitoring for exploitation attempts
• Rapid incident response capability

Patch Management Process

1. Vulnerability Assessment

Information Sources:

• Vendor security advisories
• ICS-CERT/CISA advisories
• Security research publications
• SCADA/ICS vulnerability databases

Assess Applicability:

• Is the vulnerable software present?
• Is the vulnerable feature/service in use?
• Is the system exposed to potential exploits?
• What is the potential impact?

2. Risk Analysis

Evaluate Patching Risk:

• Vendor guidance on patch application
• Known issues or bugs in patch
• Testing requirements and constraints
• Potential operational impact
• Rollback complexity

Evaluate Not-Patching Risk:

• Likelihood of exploitation
• Impact of successful exploit
• Availability of compensating controls
• Threat intelligence (active exploitation?)

3. Compensating Controls (If Not Patching Immediately)

Immediate Actions:

• Implement network-level protections
• Enhance monitoring of affected systems
• Restrict access to vulnerable components
• Update incident response procedures

Document Decision:

• Reason for delayed patching
• Risk acceptance authority
• Compensating controls implemented
• Timeline for remediation

4. Testing

Test Environment:

• Replicate production configuration
• Test patch installation process
• Verify system functionality
• Performance testing
• Integration testing with connected systems

Production-Like Scenarios:

• Simulate operational loads
• Test failure and recovery scenarios
• Verify redundancy and failover
• Test backup and restore procedures

5. Deployment

Planning:

• Schedule during planned outage
• Coordinate with operations
• Prepare rollback procedures
• Brief all stakeholders
• Have vendor support available

Execution:

• Follow documented procedure
• Monitor system during and after patching
• Verify functionality before returning to service
• Document any issues or deviations

Post-Deployment:

• Verify patch effectiveness
• Monitor for unexpected behavior
• Update documentation
• Close vulnerability tracking ticket

Special Scenarios

Zero-Day Vulnerabilities

When patches don't exist yet:

Immediate Actions:

• Assess exposure and risk
• Implement all possible compensating controls
• Increase monitoring
• Prepare incident response
• Coordinate with vendor and ICS-CERT

Short-Term:

• Isolate vulnerable systems if possible
• Apply workarounds if available
• Document risk acceptance
• Plan for rapid deployment when patch available

End-of-Life Systems

When vendor support has ended:

Options:

• Replace: Upgrade to supported system (preferred but expensive)
• Isolate: Air gap or severe network restrictions
• Virtualize: Move to supported OS in VM
• Accept Risk: Document risk acceptance with strong compensating controls

Compensating Controls:

• Maximum network isolation
• No internet or IT network connectivity
• Physical access controls
• Enhanced monitoring
• Regular vulnerability assessments
• Incident response specific to legacy systems

Emergency Patches

Critical vulnerabilities under active exploitation:

Accelerated Process:

• Abbreviated but not eliminated testing
• Higher approval authority
• Coordinate with operational leadership
• Accept higher implementation risk
• Enhanced monitoring post-deployment
• Be prepared to roll back

Patch Management Documentation

Vulnerability Tracking

• Unique ID for each vulnerability
• Discovery date and source
• Affected systems
• Risk assessment
• Patching plan or compensating controls
• Status and timeline
• Closure date

Patching Procedures

• System-specific patching procedures
• Pre-patch backups
• Testing requirements
• Rollback procedures
• Vendor contact information
• Expected downtime

Change Records

• What was patched and when
• Who performed the patching
• Testing results
• Issues encountered
• Verification of successful patch

Organizational Considerations

Roles and Responsibilities

• IT Security: Vulnerability monitoring, risk assessment
• OT Engineers: Technical implementation, testing
• Operations: Outage scheduling, operational approval
• Vendors: Patch provision, technical support
• Management: Risk acceptance, resource allocation

Metrics and Reporting

• Open vulnerabilities by criticality
• Time to remediation
• Compensating controls in place
• Systems with delayed patches
• Patch deployment success rate
• Regular reporting to management

Best Practices

1. Maintain asset inventory - Can't patch what you don't know about
2. Subscribe to advisories - Early awareness enables planning
3. Test everything - No shortcuts on testing for OT
4. Document decisions - Especially risk acceptances
5. Plan outages strategically - Bundle patches when possible
6. Maintain backups - Verified, tested backups before patching
7. Engage vendors early - Get their guidance and support
8. Use compensating controls - Don't leave systems unprotected while waiting

Next Module: Implementing these controls in real energy environments.