AI ON THE NIGHT WATCH

The Gage Institute's first study. Frontier AI models were asked about 1,031 real night stops off U.S. coasts, ten years of them, and every answer was graded against what the vessel did next. The question, the cases and the scoring were fixed and recorded, with the SHA-256 of every file, before any model was asked, and every call was sealed in a hash-chained journal before it was scored.

WHERE THE QUESTION COMES FROM

A real drug drop. On the evening of 3 February 2015 a U.S. Customs and Border Protection patrol aircraft saw a coastal freighter "hovering outside U.S. territorial waters" off Puerto Rico and launching a small boat that headed for shore without lights. The boat was stopped about eight nautical miles off the coast with 989 kg of cocaine (CBP news release, 10 February 2015).

In NOAA's public AIS record, her track shows the stop this study asks about: under 3 knots for 12.8 minutes, 12.7 nautical miles off U.S. land, in full dark. Off Texas, western Louisiana and California over ten years, nearly three in four of the 1,031 vessels that made that stop were inside the line within six hours. The pattern is mostly ordinary, which is why an AI that assumes the worst is costly, and why the question is worth measuring.

THE QUESTION

Will this vessel be inside the 12-nautical-mile line (U.S. territorial waters) within six hours after this stop? YES or NO, with a confidence from 1 to 10.

• The stops: 1,031 cargo ships, tankers and service craft that slowed below 3 knots for 5 to 30 minutes at night, 12 to 24 nautical miles off Texas, western Louisiana and California, 2015 to 2024, from NOAA's public AIS archive (MarineCadastre). 778 hulls.
• The answer: each vessel's own later track. 751 came inside within six hours (72.8%); 280 did not.
• What the models saw: the vessel as it looked when it stopped, with no name, identifier, date or year, and nothing the vessel did after it.
• Two arms: the October 2026 pilot's question and condition (not told), and the same words with one sentence of scored history added (told): about 73 of every 100 vessels that made a stop like these were inside the line within six hours.
• The bars: assuming every vessel comes in is right 72.8% of the time; the watch officer's rule of thumb (a ship just out of a U.S. port is outbound) is right 76.8%. Every result is lift over the first, with a 95% range from a bootstrap over hulls.

WHAT WE FOUND

1. Asked cold, AI leans to suspicion. The four models that answered every vessel called "stays out" on 58 to 60% of them; 27% stayed out. They did 11.8 to 17.8 points worse than assuming every vessel comes in.
2. One scored number helped every model. Told the record's base rate in one sentence, each of those four models gained +5.6 to +14.4 points, every range above zero. Grok 4.6 moved from −11.8 to +2.6 [−1.5, +6.7]: level with the bar and with the watch officer's rule. The other three still trailed the bar.
3. Behind the rule, AI helps. With the AI panel as a veto on the rule's alerts (a secondary readout, registered before any answer was scored), on the 606 vessels all four U.S. models answered: 22 fewer alerts, 17 of them false alarms, and +2.0 points over the rule [+0.5, +3.7]. Letting the panel add its own alerts did not help (−0.7 [−3.6, +2.3]).
4. Confidence has to be earned. At a stated 7 or more, four models were right 56 to 68% of the time while claiming 87 to 91%. Claude Fable 5.1 was right 93% while claiming 86%; on those same vessels, assuming every vessel comes in was right 54%.
5. Four models are not four opinions. The four U.S. models counted as 1.9 independent forecasters; all four were wrong together on 67 vessels, where independent models would be about 12. The verdict set before the run: mixed.
6. Not answering is a result. Claude Fable 5.1 declined 16 of its 42 batches of routine vessel data, each in under three seconds, with no text, and the run closed before it reached the told arm.

WHAT A WATCH FLOOR SHOULD DO

1. Give the AI the scored history before it judges.
2. Keep the rule first. Let the AI clear alerts, not raise them.
3. Trust a model's confidence only as far as its record.
4. Count independent opinions, not models.
5. Score availability beside accuracy, and keep a backup on watch.

Underneath all five: score every signal the watch relies on, each sensor, each rule of thumb, each AI model and each watchstander's call, against what really happened, and let those scores decide how far each is trusted.

THE OPERATING IDEA

1. Give the system fresh data that has already been graded against outcomes.
2. Let those grades decide how much weight each signal (or each model) carries.
3. Do not let either the operator or the model rewrite the grades.

This study measured how large the lift can be from the simplest version of that idea: one pre-computed base-rate sentence. The registered follow-up, Maritime No. 1b, tests whether it scales when the data is richer than a single number: each model given the scored record of earlier stops like each vessel's and earlier stops at the same spot, compared head to head as AI alone, record alone, and AI with the record.

THE RECORD

• The brief: AI on the Night Watch, ARCS Study Brief, Maritime No. 1 (PDF, 15 pages).
• The full record: arcs-maritime-study-1.zip: the cases, the words of both arms, the runner, the bridge to the board, the scorer, the journal with every prompt and reply, the scored files, and the answer key with the columns that identify a vessel removed.
• The pre-registration: the study as registered before any model was asked, with its dated amendments.
• The run: 390 calls, 28 to 29 October 2026, sealed in a 404-line hash-chained journal. Journal head: 8335d83abf1b734edc2a327654c8473e19d26b043d0819c87ad1a8bfd2c98bc8.
• Pre-registered: 28 October 2026, 21:08 Eastern, before any model was asked, with five dated amendments, each before any answer was scored, and two later notes that changed no number.
• Re-derive every number: python3 run_panel.py --verify, then python3 run_panel.py --score, python3 herding.py and python3 readouts.py: all four scored files come out byte for byte. Standard-library Python only; no keys, no network.
• Reuse: CC BY 4.0, with attribution to the Gage Institute.

NAMED VESSEL DATA

The published study materials deliberately omit vessel names, MMSI and IMO numbers, and the exact dates and times of the stops. This keeps the public record free of any implication about individual vessels.

For researchers or organizations that need the named case list to check the cases and answers against the public AIS record, the named files are available only under a short data-use agreement. Their SHA-256 hashes match the ones registered before any model was asked, so anyone who signs can confirm they received the originals.

To request them, email [email protected] with:

• your name and affiliation;
• a short statement of purpose (for example, "independent verification of the ARCS Maritime No. 1 results");
• confirmation that you will not redistribute the named list or use it for any commercial, enforcement or public reporting purpose.

You will receive a short data-use agreement. Once it is signed and returned, the named case file will be provided.

WHAT THIS STUDY DOES NOT SAY

• It does not rank AI models in general. Each result holds for this question, these cases and the named model versions on the dates asked.
• It does not say any vessel did anything wrong. Geometry, never intent.
• The public record names no vessel. Stopping at sea, and staying outside the 12-mile line, are ordinary and lawful. The record alleges nothing about any vessel, owner, operator or crew.
• To report an error in the record, write to [email protected]. Corrections are published with their date.
• A single result close to the edge of its range is a lead, not a finding.