According to historical records, the castle and manor house belonged to Sir Hardress Waller, who found himself trapped within its walls in May 1642 when General Purcell's forces surrounded the stronghold. After six weeks of entrenchment and siege, the defenders were forced to surrender; not through military defeat, but simply for want of water. The castle complex was substantial for its time, boasting a 110-foot long great hall, a stone stable with fine timber work, a barbican for defence, and even a well-stocked pigeon house. Today, nothing remains of either the castle or Castletown Manor, the country house that later occupied the site until its demolition in 1935.

The lands of Castletown have a complex history of ownership that reflects the turbulent nature of 16th and 17th century Ireland. Originally held by Thomas FitzGerald, Knight of the Valley, the estate was attainted and passed through various English hands as part of the Munster Plantation. In 1592, William Carter received an extensive grant of over 600 acres with strict conditions typical of plantation undertakers; he was required to build houses for twenty families and establish a proper settlement. The estate's strategic importance is evident from a 1573 record mentioning that a mason named James Dore was employed to "raze Castleton", suggesting earlier fortifications were demolished to prevent their use by rebels.

The Civil Survey of 1654-56 paints a picture of decline following the siege, recording only "a ruinous Castle, sixteene cottages a pigeon house, one orchard and one weare seate on the River of Shanen" on what was then sometimes called Islandmore. Sir Hardress Waller's fortunes fell dramatically after the Restoration; as one of the judges who signed Charles I's death warrant, his estates were forfeited, though his descendants somehow managed to retain possession. The 1657 Down Survey map shows a large house in the area, but by the time Thomas Westropp was writing in the early 20th century, he noted that the modern house stood on the castle site, a building that itself would disappear within a few decades.