The castle appears on the 1654-7 Down Survey map as Caislean Barrcha or Carrick Castle, when it belonged to Barnaby O'Bryan, a Protestant landowner. Local tradition, recorded by the Ordnance Survey in 1837, claims that both Garrycastle and nearby Coosan Castle were built by the O'Bryan family, though the surviving ruins tell a more complicated story of destruction and reconstruction spanning several centuries.

What remains today are the footings and partial walls of a long rectangular building measuring approximately 15 metres by 9 metres, with the best preserved section being the southwest gable wall, which still stands 4 to 5 metres high and features a substantial fireplace with a ruined brick chimney flue. These ruins likely date from the mid to late 17th century, when the Galborne family constructed a fortified house here, possibly on or very near the site of an earlier medieval Gaelic castle. Historical records mention an attack on a castle at this location in 1442, which was apparently levelled, though some late 19th century sources claimed parts of those earlier ruins were still visible in their time.

Archaeological testing in 2001 revealed more layers to the site's history, uncovering the foundations of a late 18th or early 19th century farmhouse about 70 metres southwest of the castle ruins; this building, constructed with stone base courses and brick upper walls, was demolished in the first half of the 20th century and later used for silage storage. The same building appears on the 1837 Ordnance Survey map as 'Garrycastle Old Ho.', though no surface remains are visible today. The castle itself has deteriorated since the 19th century maps were drawn; by 1910, the northwest wall had already collapsed, and today the ruins are largely obscured by mature tree cover, preserving this multilayered historical site as a hidden remnant of Athlone's past.