The earliest structure here, possibly dating to around 1280, was built by the Cantillon family in the form of a promontory fort, a type of enclosure that uses a natural headland or peninsula and defends only the landward approach with earthworks and walls. In this case, a semi-circular bank roughly 120 metres long and 10 metres wide still survives on the mainland side, the remnant of a gateway and curtain walls that once guarded access to the isthmus.

The site passed through several hands across the following centuries, each transition marked by conflict. In 1585 the Cantillon lands, including Ballingarry, were granted to George Isham, who collected rents without disturbing the existing occupants. That relative quiet did not last. By 1602 Garrett Roe Stack, brother-in-law to Lord Kerry, had garrisoned the castle, and Sir Charles Wilmot laid siege to it for more than a year. In 1603 Gerald Mac Morris surrendered and was executed, along with several of the garrison's leaders. The most eventful chapter came during the Cromwellian wars, when the castle came under the control of Colonel David Crosbie, a loyalist who strengthened the position considerably. He dug two trenches leading to a drawbridge, built houses on the peninsula for English settler families, and relied on supplies sent by his ally Lord Inchiquin of Clare to hold out against repeated sieges between 1641 and 1645. The end, when it came, was through betrayal rather than force: on 15 February 1645, a Sergeant James Kelly lowered the drawbridge to allow the besieging Irish forces entry. Crosbie, seriously ill at the time, was taken to Ballybeggan Castle near Tralee to be executed, but was rescued through the intervention of his niece Katherine MacGillicuddy and his nephews, Colonels MacElligott and MacGillicuddy. He recovered his estates in 1651 and died seven years later, and was buried at Ardfert.

On the landward side of the site, the remains of Crosbie's turret still stand, ruined, at roughly 1.8 metres by 1.2 metres high. Extending from it are two long trenches arranged in a V-shape, which the eighteenth-century antiquary Charles Smith recorded as subterranean passages used to shelter Crosbie and his followers during the sieges. They are modest remnants for a site that witnessed so much, but the earthworks and the surviving bank give the place a quiet, legible logic once you know what you are looking at.