What makes Barrystown quietly unusual is the layering: a tower house, the kind of fortified medieval residence common across Ireland, once stood here and was gradually absorbed into later construction work, until that later work was itself removed. The result is a site where the castle has, in a sense, been erased twice over, leaving the more modest Georgian-era house as the primary survivor of a sequence of occupation stretching back centuries.

The land at Barrystown had originally been held by the Barry family, from whom the townland takes its name. In 1655, following the upheaval of the Cromwellian settlement, Nicholas King received a grant of the forfeited Barry lands. His son Jonas had established himself at Barrystown by 1709, and it was likely the King family who oversaw the transition from castle to country house, possibly occupying the earlier structure while the new one took shape around or beside it. Archaeological testing in the area of the walled garden, a feature typical of gentry households of this period, produced no material directly related to the earlier phases of occupation. One structure that may carry older fabric is a coach-house situated roughly twenty metres east of where the tower house stood; it is thought to retain early features, though the full extent of what survives there remains uncertain.