The castle appears on the Down Survey map of Ballingaddy parish, a testament to its once prominent position in the local landscape. From at least 1290, Castleroberts was frequently mentioned alongside Adare as an important manor, with its detailed survey from 1331 preserved in the Red Book of the Earl of Kildare.

The castle's history reflects the turbulent changes in Irish land ownership across centuries. In 1559, the minister of Trinity Abbey held a castle here on what were termed "lawless lands". By 1595, the structure had fallen into ruin and was granted to Wallop. The site continued changing hands through the 17th century; in 1617, Sir J. Jephson held it as a possession of the Trinitarian or White Abbey, complete with valuable salmon and eel weirs on the river. The property passed to N. Lylles in 1638, though he died before 1657, and was later confirmed to Sir E. Ormsby in 1666 and George Evans of Ballyphilip, County Cork, in 1669.

The castle met an unceremonious end when both it and the local church were demolished, their stones repurposed as building material for a bridge. This practical recycling of medieval structures was common practice, yet it meant the complete erasure of what had been a significant fortification and religious site. Today, only historical records and old maps preserve the memory of Castleroberts, a manor that once stood alongside Adare in importance but now exists only as a name on the landscape.