Reed the article in ABB Review

Despite introduction of advanced navigational technologies, oceangoing vessel navigation still relies heavily on human perception. People are excellent at handling uncertainty: They solve problems with creativity, applying their knowledge and experience in making judgements, however human senses and capabilities are suboptimal for many situations faced at sea [1-2].

As a global leader in advanced digital and auto­mation technology supporting the shipping industry to achieve safe and efficient operation, ABB posed the question: How can recent developments in sensor technology, data analytics and computing power be used to provide seafarers with better situational awareness and improved vessel control?

ABB’s answer is to introduce digital, autonomous and remote control capabilities to enable machines and humans to work together for superior performance. By using autonomous technologies crew can be freed for supervisory tasks, or to tend to alarms or navigation notifications, anything that might arise – all working together to ensure optimal performance, whether during a long ocean transit or docking at a harbor.

ABB Ability™ Marine Pilot family of smart products: ABB Ability™ Marine Pilot Control and ABB ­Ability™ Marine Pilot Vision, have been developed to support seafarers in achieving safer, more efficient, consistent and predictable navigation and operation →01.

The bridge is the stage
Despite the availability of mandatory navigational aids such as radar, global navigation satellite systems (GNSS), Automatic Identification System (AIS), Gyro compass and Electronic Chart Display System (ECDIS) on the bridge, navigation is still heavily reliant on human senses [1-2].

People known as a lookouts peer out the bridge windows, perhaps with binoculars – 400 year old technology – to make observations. This information is relayed to the officer of the watch (OOW), who combines it with information delivered independently by navigational tools and domain knowledge to form a “mental image” and assess the situational risk (based on relationships between different inputs and information reliability). Risk can then be mitigated by, for example, adjusting the ship’s speed to ensure safe and efficient operation. Today’s systems rely on human perception, understanding and interpretation of information etc, and such dependence is challenging [1-5] →02. For example, the navigational aids might not detect small objects, or those that do not reflect radar frequency. If the lookout does not see these objects, for all practical purposes they do not exist.

Another challenge is the independence of navigational devices onboard that supply independent data points. While this siloed information prevents single-point-of-failure, it results in unnecessary duplication and complexity for the crew, who must observe, process and utilize the information manually.

The human factor
Human senses are suboptimal for slow, continuous or wide-angle observations; this combined with subjective manual observations, human-to-human communication and intermittent flow of information, can increase the risk of missing an event or a conflicted understanding of challenging situations, common at sea.

For instance, restricted bridge visibility [2,5] requires additional crew during docking and tug operations; the crew relies on subjective data about size and distance of obstacles, manually, eg, via walkie talkie, to the bridge. Adverse weather, fog and darkness can impair vison and concentration while a ship’s motion can interfere with the crew’s ability to detect situational changes, eg, an approaching vessel.

Monotonous situations, for example, a calm, sunny day at sea with “nothing” on the radar, are also challenging. Boredom and dwindling focus can result in a situation where a slowly developing event is not recognized, leading to a near miss on a virtually empty sea even under favorable conditions. Such situations challenge the crew’s ability to observe, combine, and process information, and to act appropriately.

More autonomy
Autonomous solutions exist today that can support crew in ways that were previously impossible. Objective, accurate, repeatable, continuous, durable, and with improved system redundancy; with the right sensors, autonomous systems can perform observations and initiate consistent and predictable control operations to minimize risks in any situation.

Designed to sense and perceive the environment, ABB’s Marine Pilot products further understanding, and solutions for any situation →02 – 03, thereby enabling a safe trajectory and optimal control of the ship [1,3-4] →02. The operator is provided with a complete situational overview, novel awareness →04a and enhanced predictive control →04c for safer, more efficient operations – a real boon to seafarers.

 Imagine a ship crossing the open ocean; the OOW can easily spend their entire shift watching, viewing radar screens, without needing to touch any equipment. The monotony can lead to mental and physical fatigue; and reduced alertness, so that when critical tasks must be performed, eg, approaching heavily trafficked regions, reaction times may become too slow [3-5]. By automating observations, combining data, risk assessment and decision-making, the crew could rest, or review mission goals, increasing alertness for the critical work ahead. The OOW could use their expertise when required [4].

By supporting humans – complementing their strengths – ABB’s Marine Pilot products perform tasks beyond the comfort zone of crew onboard; humans can save energy and focus on their strengths, eg, judgment. The resulting collaboration – a human-machine team improves safety and efficiency, facilitating new ways to operate.

Perceptive Visions
Relying on navigational aids and visual perceptions, the crew determines the position and motion of the vessel. While a robust and fault-­tolerant assessment is possible even if one input, eg, GPS, provides inconsistent data, the process is prone to human error. Marine Pilot Vision enables situational assessment by providing data fusion and information processing capabilities automatically, without relying solely on human performance – crucial for demanding operations.

Docking Assistance
Docking Assistance is ideal for close-range operations eg, port maneuvering that typically requires several deck crews to estimate clearances, distance and alignment to the quay. Automatic close-range monitoring in real-time uses fused data from multiple sensors; eval­uating the actual vessel position, alignment to quay and vicinity, without relying on GPS, eg, in coastal environments and port areas where satellite-based positioning is prone to jamming.

Lookout Assistance
Emulating the human lookout, the Lookout Assistant performs visual monitoring automatically, continuously, relentlessly, objectively, and at a wide- or full-angle →04 [3].

Utilizing convolutional neural networks (CNN), trained specifically to detect and classify marine-relevant objects, the Lookout Assistant analyzes incoming video streams in real time; processes frames (correcting for disturbances, eg, lens effects), locates multiple objects and assigns detection confidence values. Because relative range and bearing to each detected object is based on camera¹ data, collision avoidance is enabled for objects typically missed by navigational radars, such as small boats, etc. →04.

Collision Avoidance
Oceangoing vessel transits typically use a preplanned route, charted in ECDIS, etc., that is executed via autopilot, to ensure safe operations. Despite this, confounding situations occur, eg, loss of attention or accidents. In such situations, collision avoidance will mitigate the actual situational risk.

What if multiple vessels are encountered in a space-limited environment →05? Currently, risk assessment, decision-making and avoidance- maneuver planning is manual, and therefore prone to human error. The Collision Avoidance module resolves these challenges by performing these processes automatically, continuously, and objectively →06.

 The system considers all vessels along the planned route, evaluating risk and calculating a safe plan. Fused data from sources, fairway space based on ENC, and COLREGs2 rules are used to plan the avoidance maneuver – a safe and efficient trajectory is calculated, course and/or speed are adjusted →06. Distances are configurable and adjustable based on multiple criteria; different behaviors are programmed based on various targets, navigational statuses, etc. for exceptional situations and local variations in the COLREGs rules.

The ultimate advisory system for the crew to assist in safe navigation for any type of vessel, this module with Pilot Control, enables autonomous collision avoidance actions.

Gaining control
Despite their prevalence, autopilot and dynamic positioning (DP) are two separate navigation and maneuvering control systems with contrasting utility. Developed for controlling course, heading and forward speed during open sea transits, autopilot assumes smooth operation and slowly changing conditions: it is insufficient for precise control and maneuvering operations in tight fairways or ports. In contrast, DP systems, designed with a zero-speed assumption, are ideal for low speed-maneuvering or maintaining a position automatically [6]. At low speeds, ship and thruster hydrodynamic models are simplified: speed-dependent phenomena, eg, nonlinear damping or rudder and drag effects of thrusters are neglected – Linear Quadratic optimal control solutions are common. Hence, DP systems are not suitable for dynamic maneuvering control situations performed at speed.

ABB’s Marine Pilot Control approach allows the use of the same control system throughout the journey. By relaxing the zero-speed assumption of a traditional DP system, a consideration of the speed-dependent hydrodynamic effects in the control actions is possible – important for ­Azipod® propulsion. Although complex, the resultant nonlinear model predictive control (MPC) algorithms enable automatic control of the vessel at zero speed, maneuvering at speed, or transit in open waters [6].

By providing the crew with a single control system for the entire voyage, Marine Pilot Control mimics the control behavior of experienced captains who utilize the vessel speed, rudder effect of the thrusters, and dynamic operational conditions to their advantage.

Moreover, events can be anticipated – critical for matching human performance. If a captain knows (s)he will stop or turn the vessel soon, (s)he will adjust the thrusters beforehand to a direction where the required force is anticipated. Marine Pilot Control’s nonlinear model predictive control algorithms enable this capability. The result is faster and more precise control in dynamic operations, eg, port maneuvering, docking; and accurate trajectory in restricted spaces.

Marine Pilot Control also features all-speed joystick control, and automated operations, eg, docking, transit, voyage, and crash-stop. The result is increased consistency and operational predictability – better schedule-keeping and decreased fuel consumption →07 [7].

Enabled as a class-approved DP system for offshore vessels with DP2 requirements, Pilot Control’s automatic operations can be upgraded to autonomous if deployed with Pilot Vision and Collision Avoidance, thereby facilitating the ability to react to changing surroundings and dynamic situations while the crew is being informed of anticipated situations and planned actions →07.

Autonomous and remote-controlled tug operation
To evaluate increased safety of Pilot Control’s fault-tolerant design and joystick control for maneuvering around a berth [4], ABB and Keppel Offshore & Marine successfully tested the autonomous and remote control of a tug in the congested harbor of Singapore in 2021 →08 [4,7]. Marine Pilot Vision created a virtual view of the tug’s location relative to obstacles by integrating navigational data, streamed to the onshore command center, where the operator received the augmented situational awareness →04a, 06 [4,7] while successfully controlling the vessel, autonomously; Collision Avoidance tests are currently underway. Because Pilot Control follows a single-point fault tolerant principle, the risk of failure is mitigated. Such real-world tests are crucial because any autonomous ships will need to operate safely around real ships, buoys, etc, not in artificially quiet zones.

Future waves
All mariners will benefit from innovations like ABB’s Marine Pilot solutions providing better situational awareness, easier, safer and more efficient operations and predictable consistent control [4,7]. Once deployed, software updates can enable autonomous and remote functions later as regulations evolve [1]. Although an unattended bridge on an ocean-going vessel might be elusive today [5], ABB sets the stage for this future reality by developing products that meet real-world conditions for autonomous navigation. 

Footnotes:
1Safety of Life at Sea (SOLAS) requirements can also be fulfilled as the forward-looking camera field of view can be extended by adding more cameras.
2COLREGS stands for the Convention on the International Regulations for Preventing Collisions at Sea; it was adopted in 1972 and entered force on July 15, 1977.

 Links for more information:

 Electric, Digital and Connected solutions for the marine industry

Subcribe to ABB Review

For more information contakt:
Jacob Lundholm on mail: jacob.lundholm@dk.abb.com or mobil phone: +45 2070 6195

For information on new buildings contact:
Michael D Christensen on mail: michael.d.christensen@dk.abb.com or 
mobil phone: +45 2012 0848

References
[1] ABB round table news, “The digital journey to autonomy: taking smarter steps”, Generations: ABB marine & ports, Singapore, April 26, 2021, Available: https://new.abb.com/news/detail/77389/the-digital-journey- to-autonomy-taking- smarter-steps [Accessed March 29, 2022].
[2] Press release, “ABB Puts Forward Guiding Hand For Autonomous Shipping; Awaits Required Regulations”, Marine Salvage News, June 11, 2020, Available: https://marine-salvage.net/en/maritime-news/abb-puts-forward- guiding-hand-for- autonomous-shipping- awaits-required- regulations/ [Accessed March 29, 2022].
[3] K. Tervo and E. Lehtovaara, “Electronic lookout for increased ship safety”, ABB Web story, Helsinki, Finland, February, 22, 2021, Available: https://new.abb.com/news/detail/74475/electronic-lookout-for-increased-ship-safety [Accessed March 29, 2022].
[4] K. Tervo, “Tug project: putting ideas into action” in International Tug & OSV, January/February, 2020, pp. 26 – 27.
[5] K. Tervo and E. Lehtovaara, “B0 – a conditionally and periodically unmanned bridge”, in ABB Generations: Marines & Ports, Available: https://new.abb.com/news/detail/24651/b0-a-conditionally-and-periodically-unmanned-bridge [Accessed March 29, 2022].
[6] A. BÄRLUND, et al., “Nonlinear MPC for combined motion control and thrust allocation of ships” in 21st IFAC World Congress Special issue, Vol. 53, Issue 2, Ed., R. Findeisen et al., Berlin, Germany, July 11-12, 2020, pp. 14698 – 14703. [Accessed March 29, 2022].
[7] Innovation highlight, “Autonomous and remote-controlled vessel operation with ABB Ability Marine Pilot”, ABB Review 1/2022, pp. 13.

ABB (ABBN: SIX Swiss Ex) is a leading global technology company that energizes the transformation of society and industry to achieve a more productive, sustainable future. By connecting software to its electrification, robotics, automation and motion portfolio, ABB pushes the boundaries of technology to drive performance to new levels. With a history of excellence stretching back more than 130 years, ABB’s success is driven by about 105,000 talented employees in over 100 countries. www.abb.com

DNV is the world’s leading classification society and a recognized advisor for the maritime industry. We enhance safety, quality, energy efficiency and environmental performance of the global shipping industry – across all vessel types and offshore structures. We invest heavily in research and development to find solutions, together with the industry, that address strategic, operational, or regulatory challenges. For more information visit: www.dnv.com/maritime

01 In 2018, ABB successfully trialed the ABB Ability™ Marine Pilot products on the Suomenlinna II, a passenger ferry (shown here), in Helsinki harbor.