Local tradition, recorded by Gleeson in 1915, suggests the castle was first built in the 14th century by O'Madden of Galway, though it later became a stronghold of the O'Kennedy clan. The tower house appears on the 1654-56 Down Survey map of Terryglass parish, and by 1640, Donnogh Kenedy of Lackeene held the two ploughlands of Drominagh, comprising 400 Irish acres of mixed arable, pasture, woodland and bog worth £20 per annum.

The castle's turbulent history is well documented in the Annals of the Four Masters, which record that in 1598, O'Neill's forces seized five Ormond castles, with Redmond Burke claiming Drominagh as his base for raiding Clanricard. The Civil Survey of 1654-56 described it as 'an old ruined castle and bawne, the walls onely standing', accompanied by a decayed orchard and twenty cottages. Following the Cromwellian plantations, the castle passed to a follower named Biggs, lending it the alternative name of Castle Biggs, before James Esmonde purchased the property in 1866.

Today's remains consist of a partially collapsed rectangular four-storey tower house measuring approximately 11.85m by 10.6m, with 1.1m thick walls built from limestone rubble with dressed quoins and a pronounced base batter. Originally five storeys high according to a 1943 survey, the structure featured projecting towers at its northeast and southeast corners, creating what Maurice Craig described as a 'tall, narrow arch-headed central recess' similar to Listowel Castle. The southeast corner containing the spiral stairs collapsed in 1979, burying the original entrances; one at first floor level between the projecting towers defended by a machicolation, and another at ground level in the south wall protected by a murder hole. The interior once boasted three stone vaulted ceilings, with the ground floor divided into two chambers, whilst the surviving first floor shows evidence of a twin-light ogee-headed window with trefoil spandrels, later converted to a single light, likely after 1700.