networks (3) resized
By Tim Craven, Technical Manager H3iSquared

Over the past few years there has been a large scale migration away from using serial communications in the utility environment, and towards using TCP/IP over Ethernet as the communications technology of choice. Most, if not all, utility providers worldwide are revamping substations to use Ethernet, as well as connecting all of these remote sites to a central control room, capable of monitoring and controlling the entire electric grid from a single location. However, as with all technological jumps, there are both pros and cons to changing to a distributed communications technology such as Ethernet. Knowing the pro’s and how to implement them, as well as the cons and how to mitigate them, is crucial to running an efficient, stable and reliable network for mission critical applications.

Ethernet is a distributed communications technology, meaning that a single network can be spread over a wide area and will allow connected devices to potentially communicate with any other connected device (not taking into account VLANs, firewalls etc.). Unlike with point to point technologies such as serial, which require a direct link between devices, and are quite limited in distance, Ethernet allows devices to utilize a single physical connection to the network in order to communicate with multiple other devices on the same network. This means that physically connecting a new device to the network is a simple matter of plugging a single cable from a switch to the device, rather than having to run a separate serial or direct connection cable between that device and any other device it needs to communicate with.

Previously the Ethernet standard could not properly cater for the quick response times required for mission critical applications, such as control and automation. However advances in the technology and the introduction of new standards (such as IEC61850) mean that now Ethernet is able to cater for the latency requirements, and in most testing cases have been found to be equal, if not even quicker, than electrical cabling options. However it must be ensured that proper planning, commissioning and testing of the network take place to ensure it properly caters for the highly critical traffic.

One of the biggest downfalls to serial communications is the limitations in distance, with a maximum of about 1.5km. Using single mode fibre optics Ethernet devices can easily reach 120km or more, while other types of physical lines such as E1/T1, ADSL or 3G allow much greater distances depending on the infrastructure available. Using the Internet, one can potentially reach anywhere in the world, however this does carry potential security risks which should be secured correctly.

Previously with serial communications, substations were generally stand alone in terms of monitoring and control (although they would be wired to adjacent substations for control). Using Ethernet, we can now achieve truly centralized control and monitoring of the grid, leading to the concept of a Smart Grid. Being able to monitor and collate data from the entire grid in a central location allows this data to be easily shared with the consumer, such as usage statistics. Maintenance of the grid is also greatly simplified, and large amounts of time and cost savings become quickly apparent. Rather than having to send a technician to site to collect information directly from a device in the event of a failure, operators will now have the data delivered directly to them, as well as corresponding data from anywhere else on the grid if required. This allows problems to quickly be identified and their physical location narrowed down so that repair teams may be dispatched directly to the fault location. The result of this is a decrease in the downtime and maintenance time required, and thus an increase in the overall efficiency of the grid.

Safety is also greatly increased as operators and technicians have available information about the entire grid, rather than just the section they are working on. This means that before opening/closing a section, it is a lot easier than before to check the condition of adjacent stations etc. Using special monitoring protocols built into the TCP/IP standard, such as SNMP (Simple Network Management Protocol), operators can be notified about the slightest change in an end device’s operation that may be cause for alarm, before these issues become sever enough to cause possible damage to hardware or personnel. Sections of the grid can be shut down remotely as soon as a problem is detected, rather than needing to wait for a technician to reach the site, which can often make the difference in critical systems.

Remote access does not necessarily have to be restricted to users within the control room or connected directly to the local network. If the local network has a connection point to the internet, remote access from anywhere in the world is possible. However as these are critical networks, this outside access needs to be highly secured, and often will be kept completely separate from the control side of the network as an added precaution.

The common way of securing remote access to any network is using a VPN (Virtual Private Connection) technology, such as IPSec or PPTP (Internet Protocol Security). These technologies create a highly secure ‘tunnel’ through an unsecured network (such as the internet) between two points. A client trying to connect in to the tunnel would need the correct credentials (for instance a simple username/password combination or a secure digital certificate provided by an administrator).

VPNs extend the benefits of using Ethernet even further, allowing technicians and engineers to connect to the network in emergency cases without even needing to leave their own home or travel to site. Troubleshooting of devices and systems can also be greatly simplified, especially in the event that technical support for these products is only available outside the country/province. Simply have a controlled access VPN connection that allows outside support technicians to connect in remotely and only access the devices relevant to them.

VPN tunnels can also be used to permanently connect a very remote site to the network where it is impractical to lay dedicated fibre optics. Using a 3G or satellite connection these remote sites can tunnel to the rest of the network, allowing them to be virtually connected directly onto the network.

However, the topic of remote access and VPN tunnels brings up probably the biggest con of Ethernet when compared to serial communications, and that is security. This is not to say that Ethernet networks are not secure, in fact they can be more secure than serial communications, however more forethought and planning has to go into the implementation of Ethernet security.

As serial is not a distributed technology, a malicious user would need to physically be able to tap onto a serial cable in order to view and change any data on that cable. As these cables are generally located within a secure substation yard, the biggest concern is direct physical access, which is catered for by standard physical security, such as fences, gates, alarm systems and access control systems.

Ethernet, as we have discussed, is a distributed communication technology, meaning that if a malicious user gains access to any part of the network, they can potentially gain access to any other connected part of the same network. This means that the first concern when planning for a secure Ethernet network is that all points of access are secured. For instance any point that connects to another, unsecure network (i.e. any network to which the control/production network that may be accessed by malicious users) must be secured by a properly configured firewall, which denies all traffic except for the required traffic as determined by the network administrators.

It is also just as important to protect against physical access to any networking hardware, as damaging or resetting these can result in network failure leading to damage to or failure of the grid. This must extend not only to personnel, but also to external storage devices such as USB sticks, which could contain a virus. Plugging this into a machine on the network could unleash said virus, which has now effectively bypassed any external security such as firewalls. For this reason it is recommended to (a) disable any USB ports that are not in use on networking hardware and (b) run a strong anti-virus on the network that is constantly kept up to date.

However if properly planned, commissioned and maintained the benefits provided by an Ethernet based communications network far outweigh the downsides, and for this reason the world is seeing a definite change to and standardization on Ethernet as the communications technology of choice for mission critical networks.