Implementing Safe Firewall Rollback and Auto-Recovery on AlmaLinux 8

Overview

Firewall misconfiguration is one of the most common causes of server lockouts in Linux environments.
A single incorrect firewalld or iptables rule can immediately block SSH access, requiring console or rescue intervention.

Enterprise environments mitigate this risk by implementing a firewall rollback and auto-recovery mechanism that automatically restores known-good firewall rules if connectivity is lost.

This document describes how to implement a safe firewall change workflow with automatic rollback on AlmaLinux 8 using systemd and firewalld.

Why Firewall Rollback Is Important

• Prevents permanent SSH lockouts

• Enables safe remote firewall changes

• Reduces dependency on console access

• Required in regulated and production environments

• Common in enterprise change-management processes

Despite its importance, firewall rollback automation is rarely documented in public guides.

Prerequisites

• AlmaLinux 8

• firewalld enabled

• Root or sudo access

• SSH access to the server

Step 1: Verify firewalld Status

systemctl status firewalld

Ensure the firewall service is running.

Step 2: Create a Known-Good Firewall Backup

Export the current firewall configuration:

firewall-cmd --runtime-to-permanent
firewall-cmd --list-all-zones > /root/firewalld_backup_$(date +%F).txt

This backup represents the last known working state.

Step 3: Create Firewall Restore Script

Create a restore script that resets the firewall to a safe state.

nano /usr/local/sbin/firewall-rollback.sh

Add the following content:

#!/bin/bash

# Restore default firewall configuration
firewall-cmd --complete-reload

# Ensure SSH is allowed
firewall-cmd --permanent --add-service=ssh
firewall-cmd --reload

logger "Firewall rollback executed"

Make it executable:

chmod +x /usr/local/sbin/firewall-rollback.sh

Step 4: Create systemd Rollback Service

Create a systemd service unit:

nano /etc/systemd/system/firewall-rollback.service

Add:

[Unit]
Description=Firewall Auto Rollback Service
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/firewall-rollback.sh

[Install]
WantedBy=multi-user.target

Step 5: Create Firewall Change Timer (Failsafe)

Create a timer that triggers rollback unless manually canceled.

nano /etc/systemd/system/firewall-rollback.timer

Add:

[Unit]
Description=Firewall Rollback Timer

[Timer]
OnActiveSec=5min
AccuracySec=1s

[Install]
WantedBy=timers.target

Step 6: Enable Rollback Mechanism

Enable the timer:

systemctl daemon-reexec
systemctl enable firewall-rollback.timer

Step 7: Safe Firewall Change Procedure

Before making firewall changes:

systemctl start firewall-rollback.timer

Apply firewall changes:

firewall-cmd --permanent --add-port=8080/tcp
firewall-cmd --reload

Verify SSH access in another session.

Step 8: Confirm Changes and Cancel Rollback

If access is confirmed:

systemctl stop firewall-rollback.timer

This prevents the rollback from executing.

Step 9: Verify Rollback Status

Check timer status:

systemctl list-timers | grep firewall

Check rollback logs:

journalctl -u firewall-rollback.service

Failure Scenario (Automatic Recovery)

If firewall changes block SSH:

• SSH session drops

• Timer expires

• Rollback service executes

• Firewall is restored

• SSH access resumes automatically

No console access required.

Security Considerations

• Rollback script must be root-owned

• Protect systemd unit files

• Limit script permissions

• Log every rollback execution

• Combine with change approval workflows

Enterprise Enhancements (Optional)

• Require confirmation via health-check script

• Integrate with monitoring systems

• Use IP-based SSH validation

• Add email or webhook alerts on rollback

Conclusion

Firewall auto-rollback is a critical safety mechanism that allows administrators to make firewall changes without risking permanent lockout.

This approach:

• Enables safer remote administration

• Aligns with enterprise change-control standards

• Reduces downtime and recovery effort

• Is simple, reliable, and auditable

Despite its value, this pattern is rarely documented outside enterprise runbooks.