Today, the site is nothing more than a field locally known as 'Castle Hill', with no visible traces of the fortification that once dominated this landscape. Historical maps tell the story of what has been lost; the 1837 Ordnance Survey marks it as 'Site of Castle', whilst earlier parish maps refer to it as the 'Site of Old Castle'. The most vivid record comes from the 1657 Down Survey map of Fartullagh Barony, which depicts a tower house at 'Miltowne', with the accompanying terrier explicitly stating that 'upon Miltowne Stands a Castle'.

The castle and its surrounding lands belonged to William Bermingham, an Irish Catholic landowner who also held the neighbouring townland of Corcloon in 1640. The Down Survey maps, created in the aftermath of the Cromwellian conquest, recorded Bermingham's properties before they were seized following the Irish Confederate Wars of 1641 to 1653. Like many Catholic landowners of the period, Bermingham lost his estates as punishment for supporting the wrong side in the conflict.

The forfeited lands, including the castle site at Milltown, were subsequently granted to Nathaniel Micklethwaite, who appears in the records as the Protestant owner by 1670. This transfer of property from Catholic to Protestant hands was typical of the massive land redistribution that followed the Confederate Wars, fundamentally altering the social and political landscape of Ireland. Though the physical castle has long since disappeared, its location continues to bear witness to this turbulent period of Irish history, when ancient Gaelic and Old English families lost their ancestral holdings to new Protestant settlers.