The castle's origins are tied to the Archbishops of Tuam, who apparently sought to suppress the bishopric of Annaghdown through its construction. The last bishop appointed to Annaghdown in 1421 is likely responsible for erecting the current structure, though by 1574, the castle had passed into the hands of Nicholas Lynch.

This almost square tower, measuring 11 metres long by 10.3 metres wide, rises five storeys high and remains in fairly good condition. Visitors entering through the pointed arch doorway in the southeast wall would have found themselves in a lobby overlooked by a murder hole; a rather unfriendly welcome for uninvited guests. The guardroom lies to the south, whilst a northern passage leads to intramural stairs accessing the first floor, where a spiral staircase continues upward. The castle features a stone vault between the second and third floors, and architectural details include a partially destroyed fireplace on the first floor's southwest gable and an inaccessible passage within the southeast wall.

The defensive features of Annaghdown Castle reveal its turbulent past: simple rectangular window slits and robbed out two light windows punctuate the walls, whilst corbels that once supported angled bartizans remain visible at the north, east, and south corners. Slop stones, those practical medieval drainage features, survive on all walls except the northeast gable, and both gables are crowned with chimney stacks. The garderobe chute near the base of the northwest wall offers a glimpse into the more mundane aspects of castle life. Traces of a bawn wall, standing less than 2 metres high and 0.85 metres thick, extend southward from both the east and west corners, with a slit window and gun loop still visible in the eastern section.