The castle stood midway between the medieval church and graveyard, which still survive 60 metres to the east-northeast, and what locals called 'the Moat'; a circular earthwork or barrow that appeared on early Ordnance Survey maps but was removed in 1880. From this elevated position, the site commands excellent views along the valley from north to south, though the rising ground to the west limits the western prospect.

The castle's history spans several centuries and multiple occupants. According to the historian Carrigan, writing in 1905, the fortress was occupied successively by the Pays family, the Byrnes, and finally Charles Kavanagh in the 18th century. Local tradition preserved a dramatic account of its demise: the castle collapsed on St. Martin's night around 1780, tragically killing a herd and his family who were living within its walls at the time. This catastrophic end to the building's life meant that by 1839, when the Ordnance Survey Letters were compiled, only ruins remained of what locals then called 'the Barrack', with observers noting merely 'the site of an old Castle'.

Today, nothing visible remains of either the castle or the mysterious moat that once stood nearby. The first edition Ordnance Survey map of 1839 marked the castle as 'site of', indicating it was already gone by then, whilst the circular feature shown on the same map, likely the moat or barrow mentioned in historical accounts, disappeared forty years later. What survives is the medieval church and graveyard of Kilmacar, silent witnesses to the castle that once shared this commanding position above the Gloshia valley.