This strategic location served as their main residence from the 13th to the 16th centuries, beginning when Conchobar Ó Briain constructed an earthen stronghold here around 1267. His predecessor, Donnachad Cairprech Ó Briain, had already built a similar fortification on the river's north bank some 57 years earlier, around 1210, establishing the area's importance to the ruling dynasty.

The site evolved considerably over the centuries, with a tower house likely added around 1470, followed by the construction of a large mansion attached to it by 1558. A sketch from 1681 by Thomas Dineley captures the impressive complex in its heyday, showing the tower house, a six-bay mansion, and defensive bawn walls complete with an arched entrance protected by a machicolation. The castle endured a siege in 1600, and records from 1658 mention the 'great ould house' being roofed with slate or shingle. Two bridges spanning the river and a stone causeway connecting the castle to the Franciscan Friary a kilometre west in the town centre were integral parts of this medieval powerbase.

Today, no visible remains of this once-mighty stronghold survive above ground, though the OS Letters of 1839 noted that remnants were still 'indistinctly traceable' at that time. The site has since been transformed; in the 19th century, a brewery and Clonroad House were built on portions of the original castle grounds, leaving only historical records and archaeological surveys to tell the story of this significant medieval royal residence.