Built around 1306 by John Butler, son of Edmond Butler, the first Earl of Carrick, this Anglo-Norman fortress has witnessed centuries of adaptation and modification. The original castle's impressive curtain walls, measuring roughly 45.8 metres northeast to southwest and 42 metres northwest to southeast, still dominate the landscape, though the Slieveardagh hills provide an equally dramatic backdrop to the south and east.

The castle's most striking features are its two original three-storey angle towers; a circular tower at the southern angle and a D-shaped tower at the eastern angle. These towers underwent significant alterations during the 16th or early 17th century when new windows were inserted, including one with a decorative hood-moulding, and a quatrefoil gun-loop was added at the second-floor level of the eastern tower. The curtain walls, constructed from roughly coursed limestone rubble with a distinctive base batter, remain well preserved along the southeast and southwest sides, though fire damage has affected the latter in recent years. The entrance, located towards the southern end of the southeast wall, once featured a machicolation added at parapet level during the Tudor period, though only its supporting corbels survive today.

By 1640, the property was owned by Pierce, Lord Viscount Ikeryn, described in contemporary records as an 'Irish Papist'. The Civil Survey of 1654-6 paints a picture of the castle in its prime, noting 'a good castle, a slate house with a large bawne' at Clonomilcon. What the survey described as a bawn was actually the modified curtain wall of the original medieval castle, repurposed as part of a fortified house complex. This transformation from Anglo-Norman stronghold to early modern fortified residence reflects the changing nature of Irish castle architecture and the complex political landscape of medieval and early modern Tipperary.