The Down Survey, the ambitious land-mapping project carried out between 1655 and 1656 under the direction of William Petty, placed that symbol on its barony map of the Barony of Crannagh. Yet the same townland goes unrecorded on the accompanying parish map, and the reason is a telling one: the land was the property of the Countess of Ormond in 1640 and had not been forfeited during the Cromwellian confiscations. Because it remained in safe hands, the surveyors had no administrative need to document its buildings, and so they were quietly omitted. The castle indicated on the barony map appears to correspond with what is now the townland of Damma Upper, where Damma House stands today.

The Smyth family seem to have been the resident gentry here across several generations, and their story gives some shape to what the building may contain. The historian William Carrigan, writing in 1905, traced the history of a small church known as St Michael's, or 'Mickle's Church', located about a kilometre to the north-north-west of the house. He recorded that it was built by William Smyth of Damma, who died in 1655, then rebuilt by his grandson Valentine Smyth of Damma, who died in 1700, and rebuilt again by Valentine's daughter-in-law, Jane Smyth, otherwise Read, of Damma, who died in 1747. Three generations of the same family, all identified by their connection to this place, across nearly a century. That continuity makes it plausible that Damma House either dates from the seventeenth century outright or, if the structure was later remodelled, that it retains within its fabric some remnant of the earlier castle that the Down Survey's cartographers thought worth marking on their map.