The site, known historically as Ballingarry Castle, sits on a promontory, and its basic logic follows that of an Iron Age promontory fort, a type of enclosure that uses the natural sea cliffs on three sides and concentrates its defences at the narrow landward neck. The earliest visible element is a semi-circular bank on the mainland side, roughly 120 metres long and 10 metres wide, which dates the site's origins to around 1280.

That earliest phase is associated with the Cantillon family, who appear to have adapted the promontory fort form into something resembling a gatehoused curtain wall. The Cantillons held the land until 1585, when Ballingarry and their other territories were granted to a George Isham, who collected rents without disturbing the existing occupants. Things grew considerably more violent by 1602, when Garrett Roe Stack, brother-in-law to Lord Kerry, garrisoned the castle, prompting Sir Charles Wilmot to lay siege for over a year. In 1603, Gerald Mac Morris surrendered and was executed along with several of the garrison's leaders. The site's next major chapter belongs to Colonel David Crosbie, a loyalist who came into possession of Ballingarry during the Cromwellian wars and set about refortifying it. According to Charles Smith, writing in 1756, Crosbie built a small castle and extended the fortress with two long trenches radiating out in a V-shape from a turret on the landward side. A fosse, meaning a defensive ditch, runs along the interior of each trench; the north-west trench extends for around 120 metres and the north-east for around 175 metres, the latter originally terminating at a watchtower on the cliff edge that has since vanished entirely. Crosbie also settled English families on the peninsula and relied on his ally Lord Inchiquin of Clare to keep the garrison supplied with food and ammunition. Despite all of this, the fortification fell on 15 February 1645, when a Sergeant James Kelly lowered the drawbridge and let the besieging Irish forces in. Crosbie, seriously ill at the time, was taken to Ballybeggan Castle near Tralee to be executed, but was rescued through the intervention of his niece Katherine MacGillicuddy and two nephews who held the rank of colonel. He recovered his estates in 1651 and died seven years later, buried at Ardfert.

What remains on the ground today is a palimpsest of these successive phases. Crosbie's turret still stands in ruined form at around 1.8 metres high, and the V-shaped trenches are legible in the landscape, though a westward extension noted by the antiquary Westropp in 1909 has since been obscured by modern field boundaries. The earlier semi-circular bank is cut through by the later Cromwellian trenches in at least two places, making the stratigraphy of the site visible, if not immediately obvious, to anyone who walks the full extent of the earthworks.