During active incident response, there should be considerations toward working on recovery. The actual breach will dictate the course of recovery. This is when having backups on offline systems will prove invaluable. For recovery, the response team should be planning to bring back online any downed systems or applications, such as authentication servers, database servers, and any other production resources.
Having production backup hardware ready for use is highly recommended, such as extra hard drives, hot-spare servers, and the like. Ready-made systems should have all production software loaded and ready for immediate use. Perhaps only the most recent and pertinent data would need to be imported. This ready-made system should be kept disconnected from the rest of the potentially affected network. If a compromise occurs and the backup system is a part of the network, then it defeats the purpose of having a backup system.
System recovery can be a tedious process. In many instances there are one of two options. Administrators can perform a clean reinstallation of the operating system followed by restoration of all applications and data. Alternatively, administrators can patch the system of the offending vulnerability and bring the affected system(s) back into production.
Performing a clean reinstallation insures that the affected system will be clean from any trojans, backdoors, or malicious processes and that any data (if restored from a trusted back up) is cleared of any malicious modification. The drawback to total system recovery is the time involved in rebuilding systems from scratch. However, if there is a hot backup system available for use where the only action to take is to dump the most recent data, then system downtime is greatly reduced.
The alternate course to recovery is to patch the affected system(s). This method of recovery is more dangerous to perform and should be undertaken with great caution. The danger with patching a system instead of reinstalling is whether or not you have sufficiently cleansed the system of trojans, holes, and corrupted data. If using a modular kernel, then patching a breached system can be even more difficult. Most rootkits (programs or packages that a cracker leaves to gain root access to your system), trojan system commands, and shell environments are designed to hide malicious activities from auditors. If this approach is taken, only trusted binaries should be used (for example, from a mounted CD-ROM).