Keeping your experiment on the Internet

If you get a message from TIssue about vulnerabilities in your experiment’s data acquisition system, don’t ignore it. Work with your computing liaison to address the alert and fix the problem.

With Fermilab’s accelerator complex getting into operation again, experiments are running data acquisition systems. Sometimes these must be configured rapidly to take advantage of limited time available in test beams. It is important for experimenters to be aware of ways that computer security policies and operations can affect them, even to the extent of blocking their data acquisition systems from the Internet.

The lab cybersecurity team operates a set of scanners looking for systems that are vulnerable to common exploits. It is important that we quickly either fix these vulnerabilities or remove the vulnerable systems from the network to prevent being taken advantage of, potentially creating a serious computer security incident.

Our system to notify users about such detected vulnerabilities is called TIssue (issue tracker). Upon detection, it will issue a message to the manager of the system involved, warning the user that the system will be blocked from the Internet if the vulnerability is not fixed. At that point the system manager is supposed to mitigate the vulnerability and respond to the TIssue alert to avoid a network block from being issued. Unfortunately, however, there are several ways in which this process can fail.

First, the notification email may go to the wrong person if the experiment manager information is not up to date. Also, the manager might decide that this message can be ignored and does not really apply to the experiment. He or she may think the experiment deserves an exemption (and may even create a Service Desk ticket requesting such an exemption), not realizing that such exemptions are granted only when the request is accompanied by a list of compensatory security controls, such as firewalls or access controls, that will allow secure operation even with the vulnerability. Or the manager might fix the vulnerability but fail to inform TIssue. In any of these cases, the system will be blocked from the network, likely interfering with the experiment taking data. This applies not only to computers but also to any device connected to the lab network, such as oscilloscopes or other vendor devices whose configurations are difficult to manage.

How can an experiment avoid getting blocked? Make sure the listed system manager is correct. Do not ignore messages from TIssue. Involve your experiment computing liaison as quickly as possible when a TIssue alert is issued; the liaison can assist you in dealing with the situation. Be especially careful about third-party devices with unknown configurations: Get them onto the network well before data taking starts so issues can be dealt with while there is plenty of time to fix a vulnerability or explore options.

We know blocking devices from the network will get your attention. Paying close attention to earlier, more subtle alerts can avoid this drastic alternative.

Irwin Gaines