The castle forms part of a remarkable cluster of historic structures in the area, with a dovecote positioned 100 metres to the east-northeast, Brown's Stone lying 530 metres to the north, and rather confusingly, another castle also called Drombanny Castle situated 800 metres away in the same direction. The nearby Cahernarry church, 1.4 kilometres to the east, completes this intriguing historical landscape.

The castle's documented history reveals a succession of ownership that reflects the turbulent nature of 16th and 17th century Ireland. In 1584, Donnell Mac Canna held the lands, and by 1587 had entailed the castle to his sons. The property changed hands multiple times over the following century; by 1621, it had fallen into ruin and was granted to H. Holcroft as the former estate of Edmund McCany. Despite this transfer, livery on Drombanny was granted to Edmund McCanny in 1629 following his father Donough's death. By 1655, Piers Creagh, son of Andrew, held what was described as a "broken castle", and in 1669 the property was granted to the Duke of York.

The Down Survey maps from the 1650s provide valuable visual evidence of the castle's appearance and status. On the map of South Liberties Barony, the structure is depicted as a house with a prominent central chimneystack in the townland of 'Brownestowne' within Cahernarry parish. The accompanying terrier describes it as "a good stone house at Brownestowne", suggesting it had been converted to domestic use by this period. Meanwhile, its namesake to the north appears on the same maps as a more traditional tower house type castle in Donaghmore parish, highlighting the different architectural fates of these two fortifications.