Records of the O'Kelly family's connection to this land stretch back to 1585, when they held sway over the castle and approximately 500 acres of surrounding territory. By 1617, the estate was in the hands of Egnaham McDonell O'Kelly, who maintained ownership through to 1641, navigating the complex political landscape of early modern Ireland.

The O'Kelly family's fortunes were intertwined with both Gaelic and English power structures of the time. Around 1639, Egnaham's son Daniel married Elizabeth Osbalston, daughter of Edward Osbalston, who had served as Chief Justice of Connaught in 1602. This union represented the kind of strategic alliance between old Irish families and the new English administration that characterised the period. However, like many Catholic landowners, the O'Kellys were dispossessed during the Cromwellian conquest of the 1650s. Despite their attempts to reclaim their ancestral lands through legal channels into the 1660s, their petitions were apparently unsuccessful.

The castle itself has largely vanished from the landscape, reduced to what was described in 1972 as a stony mound at the eastern corner of a moated site. Today, even this remnant is no longer visible at ground level, leaving only documentary evidence and field names to mark where it once stood. In the early 18th century, the Lyster family acquired these lands and built their own house at Lysterfield, about 700 metres north-northwest of the original castle site, establishing a new chapter in the area's history whilst the O'Kelly stronghold faded into memory.