The castle appears on historical maps dating back to the 1654-57 Down Survey, where it's depicted standing beside the river, and on the 1840 Ordnance Survey map, shown within the northern section of a woodland plantation.

The castle belonged to the Fox family, also known as Boskagh, from at least 1574 until 1655. Edmund Fox received a pardon in 1574, and the family maintained control of what was variously called Ballygorman or Gormanstown in Poblebuskagh, also known as Fox's country. By the time of the 1654-56 Civil Survey, Edmund Fox of Bulgaden held the townlands of Adamstown and Gormanstown, which contained two decayed castles, twelve cabins, and a mill seat. The survey's description of the castle as "decayed" suggests it was already in poor condition by the mid-17th century.

According to Thomas Westropp's 1906-7 description, the castle measured approximately 5.3 metres by 2.4 metres internally, with walls 1.14 metres thick. It stood 7.3 metres high and featured a vaulted lower room. The building had two doorways, one facing east and another west; the western door was particularly notable for its carved decoration of crosses and circles. By 1942-3, archaeologist M.J. O'Kelly reported that only a fragment of the west wall remained standing, and even this has since vanished from the landscape.