The Drift: NavyCon Edition

Sponsored by:

Navigation Brief

WASHINGTON – Good Afternoon, Drifters

My dear friend Claude Berube over at the Naval Academy Museum put on a fantastic event last night, the second NavyCon. NavyCon is a group of people sitting around talking about Sci-Fi, how it has been influenced by the Navy and what lessons we can draw from it in the future.

I’ll post the link next week so you all can watch it.

Instead of the normal format this week, I thought I’d send you the detailed version of my presentation for the vast majority of you who didn’t watch last night.

My talk was on the 1979 film Alien, and it focuses on my time at sea, and how integrating greater levels of autonomy into our ships and giving those systems expanded responsibilities comes with obvious pitfalls explored in the film.

This is not an official Drift with a volume number attached, it’s a one-off special edition. We’ll get back to our regularly scheduled program next week.

I hope you enjoy! Let’s Drift.

-DBL

Alien and the Operators

In 2002, I was a goofy, awkward 18-year-old Operations Specialist Seaman Apprentice and a new check-in on board USS Normandy. As many of you are aware, that’s a Ticonderoga-class cruiser purpose built to operate the US Navy’s world-class AEGIS Combat System.

Now AEGIS is primarily an air-defense system designed to protect our aircraft carriers from incoming missiles, but it synthesizes and displays, for operators, inputs from all the sensors coming into the Combat Information Center.

And my job was to be the surface radar controller. Now, when you grow up reading Tom Clancy and Stephen Coonts novels, you imagine the military as being pretty high-tech and cutting edge. So, walking into CIC for the first time to be confronted with what, at the time, was still considered the world’s most sophisticated air defense system, was a bit of a shock.

It ran on computers the likes of which I hadn’t seen since the late 1980s, the displays looked like my brothers’ Atari and when I found out that I had to manually track surface contacts, I was appalled.

Now, the new installs of AEGIS, Baseline 9, which can be found on Normandy today, is much better and looks like something a kid raised on videogames would recognize as high-tech. But the more I learned about even our clunky old bit of 70s tech, the more I realized it was an elegant system. And on the air warfare side is where the magic happens.

For the uninitiated, that AEGIS has what’s known as “auto-special mode.” For the vast majority of circumstances that a cruiser will find itself in, you want full manned control of the weapons. Most scenarios are below the threshold of an all-out missile fight. But when you do get into that missile fight, that’s when the real beauty of AEGIS Comes into play.

This is an oversimplification but basically any aircraft or missile that the system has identified as a threat gets shot down until the ship runs out of missiles. This technology was a recognition that in the dawning era missiles, human brains would not always be able to react fast enough to protect the ship without the help of computers.

So the things we are talking about today, and the ethical decisions we are wrestling with about how much we want to trust computers with our life and death decisions, we’re not – I repeat – not talking about a revolution. This is an evolution from where we already are today.

Having AEGIS decide what to kill and having an already launched loitering munition decide for us: That technology exists, it’s already here, now we’re talking about how best to employ it.

But as crews get smaller and more things get automated, how might a sophisticated piece of software like the Aegis Combat System adapt to new and emerging technologies or even be expanded to other functions?

Think about SIRI or Alexa: computers we talk to and they respond to our orders. Instead of the tactical action officer telling the surface warfare commander to “Kill track 50675 with guns,” what if he said: “AEGIS, kill track 50675 with guns” and the ship did it automatically. Does that sound like a stretch?

Now let’s say the TAC wants to wake up the oncoming watch section? Instead of having the watch supervisor send some random OS down to the berthing to go wake folks up, does it seem like much of a technological heavy lift to have him say: “AEGIS, wake up the oncoming watch section,” and little alarms go off in all their racks?

I don’t think it does.

Now what if, instead of the OOD or TAC telling AEGIS to wake up the oncoming watch, AEGIS just knew to do it at 2315 to give them time to shower, hit midrats and get on watch. Now, what if, instead of AEGIS, we called that computer … “Mother”?

Ridley Scott’s 1979 movie Alien shows two potential pitfalls of artificial intelligence and autonomous systems:

1)These systems can breed dependence and overreliance on a system that could be either gamed or reprogrammed by a malign actor;

2) and perhaps even more dangerously, we risk the introduction of a system with an agenda not fully understood by the operators

For the purposes of this presentation, I will just briefly outline the premise of the movie.

The Nostromo is commercial towing vehicle hauling haul lots of deep-space mineral ore with a crew of seven and it’s returning to earth with its cargo. The Nostromo is a vessel that belongs to The Weyland-Yutani Corporation.

Because of the vast distances required for deep space travel, the human crew is put into “hypersleep,” which basically keeps their cellular system in stasis for an extended period of time. They sleep in pods in their underwear.

The crew is under a mandate – which they appear to not fully understand – to respond to any distress signals and when the ship comes into contact with such a signal, the ship’s autonomous computer brain, known as “Mother,” wakes the crew to respond.

Ripley is the star of the show, but for the purposes of this presentation, I want to focus on Dallas, whose decisions lead to the unfortunate business with the Xenomorph.

So, when the crew is woken up by Mother, they are unhappy to be sent to investigate, arguing that they’re just a freighter.

Here’s the piece of dialogue I want to pull out for you:

DALLAS

Some of you may have figured out that we're not home.

BRETT

What the hell?

DALLAS

Mother's interrupted the course of the voyage. Mother is programmed to in interrupt the course of out voyage if certain conditions arise. They have...

(pause)

We've received intermittent transmission from quadrant points QBR 157, 052. Somebody's gone down.

BRETT

So what?

KANE

We're obligated under Section B2...

PARKER

Christ. We're a commercial ship, not some rescue team. This kind of duty's not in our contract.

ASH

You better read your contract. Transmissions received in non- commercial lanes...

DALLAS

We're going in, that's it.

You see, Dallas doesn’t question Mother’s actions nor does he wait to gather full intel on the nature of the signal, which Ripley interprets as a warning (not a distress signal) only after Dallas sends a party to go investigate.

They come across a derelict spacecraft and inside, executive officer Kane finds some fun eggs. But then one of the eggs hatches and a face sucking thing attaches itself to his face.

When Dallas gets back with the investigation party, dragging along a smothered Kane, Dallas again shows he’s not the world’s greatest commanding officer and demands Ripley open the door, which would break entirely sensible quarantine protocols around letting any unknown lifeforms aboard the ship. Ripley refuses.

Ash – the officious science officer aboard the ship who we’ll learn more about in a bit – overrides Ripley by opening the door, breaking quarantine and letting in the facesucker.

When Ripley challenges Ash later, he pretends he was letting Kane in for humanitarian reasons.

There’s another key moment later, after the sucker detaches from Kane’s face and before the Alien pops out of poor Kane’s chest at the dinner table where we see the kind of CO Dallas is. Dallas lets let’s Ash keep the dead facesucker over Ripley’s strenuous objection, saying it was unsafe to have it onboard.

Dallas defers again to Ash, saying “You're the science officer. It's your decision.”

When Ripley challenges him again, we get our next great moment: See here

RIPLEY

How could you leave that kind of decision to him? 

DALLAS

I just run the ship. Anything that has to do with science division, Ash has the final word.

 RIPLEY

How does that happen? 

DALLAS

Same way everything else happens: Orders from the Company. 

RIPLEY

Since when is that standard procedure? 

DALLAS

Standard procedure is do what they tell you... Besides, I only about flying... I haul cargo for a living.

So, fast-forwarding to the end: Alien pops out of Kane’s chest and goes on a killing spree, ultimately killing poor, feckless Dallas.

Critically, we find out that Ash, who insisted on letting the lifeform on the ship in the first place, was an android all along and was collaborating with Mother to, at all costs, return any alien lifeforms to the good people at the Weyland-Yutani bio-weapons division.

The crew was, of course, expendable in that pursuit.

Ripley takes command after Dallas gets murdered by the Alien and ends up being the lone survivor, along with Jonesy the Cat, ultimately managing to get Mother to trigger the Nostromo’s self-destruct mechanism as she escapes in an escape pod.

Conclusions

Now there’s a couple things I want to draw out of this parable of autonomy and artificial intelligence:

As the Navy moves to ever-greater levels of autonomy, there is a significant risk that humans become too dependent on autonomous systems, deferring, like Dallas, to Mother rather than using his head for more than a hat rack.

Of course, Mother has our best interests at heart, right?

Not necessarily. And there are regular stories of people who follow their GPS into rivers or lakes because of programming errors.

There is a significant risk that in a life-or-death situation, either through cyber warfare, espionage and sabotage, or even an honest mistake on the part of the programmers, future autonomous systems could have an agenda that the crew won’t know about and the consequences could be a disaster.

Secondly, crews must be fully familiar with the systems on board their ships and have an exquisite understanding of how their systems will behave in any material circumstance.

These systems will, like Ash and Mother, be part of the crew and that’s exactly how the Navy needs to think about it. The tools of the past were incredible for their day but as we evolve into ever-more advanced systems, it becomes even more important for sailors to understand them. No surprises!

Ash was on a mission that nobody knew about. On a ship in combat, you need the crew in complete alignment as to what the mission is. That’s going to be key to being successful integrating greater levels of autonomy.

As we close, it’s important to note that the voyage of the Nostromo was a total mission failure for Weyland-Yutani, and it goes back to the misalignment of its crew – including Mother and Ash.

They didn’t deliver their payload of mineral ore to earth, and the secret mission the crew didn’t even know they signed up for was a failure as well: Ripley ultimately kills the Xenomorph in the escape pod.

It’s my worry that as we move down this path, without total alignment of the crew and its autonomous little helpers, the Navy could be setting itself up for accidentally allowing giant murderous Aliens on board its vessels that kill the whole crew.

Thank you.