The site's strategic importance was recognised early in the Norman conquest of Ireland when, according to historical records, King John ordered a castle built here in 1185. The land was granted to Philip of Worcester, who subsequently passed a portion to Gerald fitz Maurice, establishing a pattern of noble ownership that would continue for centuries.

The castle complex reveals its multi-period construction through its varied architecture. The oldest surviving structure is the 13th-century keep, positioned at the northeast corner of the defensive bawn wall. Two centuries later, a tower house was added to the southeast angle, reflecting the changing military needs and architectural fashions of medieval Ireland. The bawn itself, built from locally sourced limestone rubble with a distinctive external batter at its base, once formed a complete rectangular enclosure around the castle buildings. Today, only portions remain; the southern wall runs dramatically along the cliff edge, whilst the eastern wall stretches for just over 25 metres and contains what may be an original gateway, though its current form possibly dates to the 18th or 19th century.

The castle's military significance endured well into the modern period, facing siege during the Cromwellian wars of the mid-17th century. By 1640, according to the Civil Survey, the property belonged to the Bishop of Waterford and Lismore, with Richard Butler of Ardfinnan serving as lessee. The survey paints a picture of a partially ruined complex even then, describing "a castle & the walls of a castle, ye walls of a stone house & a bawne", suggesting that whilst still imposing, parts of the fortress had already begun their slow transformation into the atmospheric ruins visitors encounter today.