The site began as a timber fortress in 1213, built controversially on church property; the local bishop was so incensed he threatened to excommunicate the justiciar and his men, only relenting after promises of compensation. That wooden structure, described in 1245 as a 'moat and wooden tower', gave way to the impressive stone castle we see today, rebuilt between 1278 and the early 1280s under John de Lydyard's direction.

The castle's design represents what architectural historian Harold Leask termed a 'keepless castle', centring on a formidable three-storey gatehouse rather than a traditional keep. The complex spans roughly 60 metres square, enclosed by curtain walls up to 3.4 metres thick and punctuated by angle towers at the east and west corners. The gatehouse itself is a masterpiece of medieval military architecture, originally defended by a drawbridge, portcullis and machicolations. Its first floor houses a magnificent chamber with triple-bay groin vaulting and the remains of an ornate fireplace featuring engaged shafts with dogtooth capitals. Throughout the walls run spiral staircases, mural passages and garderobes, whilst cross-loop arrow slits pierce the curtain walls at strategic points, their large segmental-arched embrasures allowing defenders to rain fire on attackers below.

The castle remained in Butler hands from 1315, when Edmund Butler was created first Earl of Carrick, until 1703. It witnessed some of Ireland's most dramatic moments; Owen Roe O'Neill captured it in 1646, only for Cromwell's forces to seize it four years later. The Cromwellians found it 'in Repayre' with two turrets and a bawn, maintaining a garrison of 37 soldiers there as late as 1659. In the early 18th century, the elegant Damer House was built within the courtyard, transforming part of the medieval fortress into a Georgian residence. After serving as army barracks throughout much of the 19th century, the castle now stands as National Monument No. 211, its walls incorporating everything from reused Early Christian cross slabs to Jacobean chimney stacks, each addition marking another chapter in eight centuries of continuous use.