Built around 1583, possibly by Jason Fant who held the property at that time, this imposing structure measures approximately 10.36 metres by 5 metres, with walls an impressive 1.75 metres thick. The castle's defensive features include bartizans (small turrets) on the northwest and southeast corners containing rooms, gunloops for musket fire, and corbels that once supported a machicolation over the entrance; a projecting gallery from which defenders could drop objects on attackers below.

The tower house's five-storey layout reveals the comfortable lifestyle of its inhabitants, with multiple fireplaces warming the upper chambers and a sophisticated internal arrangement that included a latrine cunningly built into the southeast wall. An intriguing architectural detail is the east doorway, which presents a pointed arch on the interior and a rounded arch on the exterior, demonstrating the transitional building styles of the period. The castle changed hands several times throughout its history; from the Fant family it passed to the Burgat family, who by 1640 had developed the site to include not just the castle but also an orchard and a gristmill powered by the River Lubagh, along with six cabins for workers or tenants.

Perhaps the castle's most unusual feature is the sheela-na-gig carved into a quoin stone at first-floor level on the northeast angle. These enigmatic female figures, found on various Irish medieval buildings, continue to puzzle historians about their exact purpose and meaning. The 1654-56 Down Survey map depicts Fantstown as a classic tower house complete with a gabled chimneystack, and by 1666 the property had been confirmed to Captain Ponsonby and Lord Colloony. Though the parapet has been lost to time, reducing the castle from its estimated original height of 18.3 metres, the structure remains remarkably intact, with its spiral staircase, vaulted ceilings, and even some original plastering still visible on the interior walls.