Software Vendor Patch Crashes SCADA System

A software patch was supplied from the vendor to correct an existing problem. This patch caused the SCADA server to fail, stopping the entire system. Operations of 3 plants were carried out on the one working PC, a standalone backup PC that had yet to be patched. The system was restore to full operation by inserting a hard disk that was removed from the server prior to the patch being installed, this was the first time that the disk had been removed. The software re-issued the patch with revised instructions detailing it was not to be used on the SCADA server.

Production was severely effected during this period, and the on-call engineer was called to the site. An additional operator was also called in to help the existing personnel run the site using the one PC.

Each patch installation is scrutinised and unless required urgently, the patch is not installed until it is at least a month old. All future patch installations require the second hard disk on the server to be removed and replaced with a blank one (mirrored disks are used). The period that the server is considered to be under test for was extended to a month. This means that the disk removed cannot be used elsewhere until the end of this period.