The site's history stretches back centuries; by 1654, the Civil Survey recorded it rather dismissively as "the stump of an old castle neere the River Annor". At that time, the property belonged to Phillip Walle of Newtown-Anner, described in the records as a gentleman and Irish Papist, offering a glimpse into the complex religious and social dynamics of 17th-century Tipperary.

The castle as it appears today is largely the creation of Reverend N. H. Mandeville, who embarked on an ambitious rebuilding project in the 1860s. According to architectural historian Bence-Jones, Mandeville incorporated what remained of the old square castle that had previously been known as Ballinahy, belonging to the Mandeville family. The Victorian reconstruction resulted in an imposing structure built on a square plan, featuring two octagonal battlemented towers at the front and two rectangular battlemented towers at the rear, creating an impressive, if somewhat fanciful, medieval revival aesthetic.

Tragedy struck in 1926 when fire gutted the castle, leaving much of it in ruins. Only the front section was rebuilt, whilst the rear towers remain as ivy-covered shells, lending the site a rather romantic, if melancholy, atmosphere. Close examination of the southwest tower, which measures 15.6 metres across its eastern to western walls, reveals rubble limestone masonry that differs markedly from the irregular limestone blocks used elsewhere in the building; this architectural detective work suggests this tower may well incorporate elements of the original medieval castle, making it a rare survivor that bridges the centuries between the Walles' time and the Mandevilles' Victorian reinvention.