Originally a reputed Templar site and later an appanage of Keynsham Abbey, the castle changed hands numerous times throughout the centuries. In 1255, it was under the ward of Walter Wansell, the sheriff, before Thomas an Appagh granted it to Gibbon of Callan, whose descendants became known as the MacGibbons or White Knights. The castle continued to be a point of contention, with various legal disputes arising over its ownership, including a 1307 suit between the Prior of Bathkeale and Hugh Purcell regarding manor dues, and a 1309 claim by Sybil de la Chapelle for her widow's third.

The castle's fortunes took a dramatic turn during the rebellions of the 16th century. By 1583, what had been described as "an excellent and ample residence" was reduced to stone walls after being completely wasted during the rebellion. Sir Henry Oughtred received a grant for the castle in 1592 and attempted to rebuild, creating what was noted as a "fair house" by 1611, though this too was defaced during subsequent rebellions. The property saw military action again in 1598 when Oughtred fled to Limerick with his wife, leaving sixteen men to defend the house; they abandoned it after just two days, allowing Irish rebels to capture it. Sir Francis Berkeley reclaimed it in 1600 whilst pursuing Garrett Fitz Nicholas, seizing both the castle and the rebels' corn stores.

By the mid-17th century, the castle was recorded as being out of repair under Colonel Francis Courtnay's ownership, with both the castle and an associated mill in poor condition. The old fortifications were eventually repurposed as a barrack before 1827, whilst traces of what were thought to be ecclesiastical buildings remained visible in 1837, though their military characteristics were more apparent by then. A new Mayne House was built on or near the site, described in 1840 as a neat two-storey residence with offices, plantation and garden, home to Bryan Sheehy, though no surface remains of this house survive today.