Software Vendor Patch Crashes SCADA System

Event Year: 2005 Reliability: Likely But Unconfirmed
Country: United Kingdom
Industry Type: Food & Beverage
Description:

A software patch was supplied from the vendor to correct an existing problem. This patch caused the SCADA server to fail, stopping the entire system. Operations of 3 plants were carried out on the one working PC, a standalone backup PC that had yet to be patched.

The system was restore to full operation by inserting a hard disk that was removed from the server prior to the patch being installed, this was the first time that the disk had been removed.

The software re-issued the patch with revised instructions detailing it was not to be used on the SCADA server.

Impact:

Production was severely effected during this period, and the on-call engineer was called to the site. An additional operator was also called in to help the existing personnel run the site using the one PC.

Action Description: Each patch installation is scrutinised and unless required urgently, the patch is not installed until it is at least a month old. All future patch installations require the second hard disk on the server to be removed and replaced with a blank one (mirrored disks are used). The period that the server is considered to be under test for was extended to a month. This means that the disk removed cannot be used elsewhere until the end of this period.