The original fortification here was a motte, an earthen mound typical of Norman defensive structures, but by the 16th century this had been replaced or supplemented by a proper stone castle. According to historical records from the Civil Survey of 1654 to 1656, the castle and its substantial estate of 1,000 acres belonged to William Plunkett of Dublin in 1640, marking it as a significant landholding during a turbulent period in Irish history.

By the late 17th century, however, the castle's fortunes had clearly waned. Robert Leigh, writing in 1684, noted that the structure was already falling into disrepair, suggesting that the political and social upheavals of the period had taken their toll on the building and perhaps its owners. The 1839 Ordnance Survey map places the castle just south of the original Norman motte, indicating that the stone structure was built as a separate edifice rather than directly atop the earlier earthwork.

Today, little remains of the stone castle itself, though two intriguing remnants have found their way to Dublin. A pair of gargoyles, presumably recovered from the castle ruins, now reside at the premises of the Royal Society of Antiquaries in Merrion Square. These carved stone waterspouts serve as tangible links to Old Ross Castle's grander days, when such architectural flourishes would have adorned what was clearly once an impressive fortified residence in the Wexford countryside.