A promontory fort uses the natural advantage of a headland, protecting its landward side with a bank, ditch, or wall while the sea does the work on the remaining flanks. The Cantillon family appear to have understood this well, and around 1280 they constructed a gateway with curtain walls that effectively updated this ancient form for medieval use.

The site accumulated centuries of contested occupation. In 1585 Ballingarry and the surrounding Cantillon lands were granted to George Isham, who was content to collect head rents and leave things otherwise undisturbed. That arrangement did not last. By 1602 Garrett Roe Stack, brother-in-law to Lord Kerry, had garrisoned the castle, prompting Sir Charles Wilmot to lay siege to it for more than a year. When Gerald Mac Morris finally surrendered in 1603, he was executed, along with several of the garrison's leaders. Later, during the Cromwellian wars, the castle passed to Colonel David Crosbie, a loyalist who substantially enlarged the complex. Crosbie cut two trenches leading to a drawbridge, extended occupation beyond the isthmus, and built a street of houses on the peninsula to settle English families there. His position was only held with outside help; Lord Inchiquin of Clare supplied the garrison with food and ammunition against ongoing Irish resistance.

What remains today is largely at foundation level, the houses along the eastern side of the interior reduced to their red sandstone footings. The ruins of the castle itself form a large rectangular structure measuring approximately 38 metres by 13 metres, with a possible entrance on the east side around 3 metres wide and two small rooms projecting from the southern wall. An Ordnance Survey map from 1841 to 1842 recorded a sheepfold in the north-east corner, suggesting the enclosure had found quieter uses by then. The southwest interior is still defined by a low bank, around a metre high and three metres thick, a modest but legible remnant of what was, for a time, one of the more fiercely defended positions on the Kerry coast.