According to the 1657 Down Survey map of Rathconrath parish, a tower house once stood here, owned by Oliver Dalton, who was recorded as an 'Irish papist' in 1641. This designation tells us much about the religious and political tensions of the period, when Catholic landowners faced increasing pressure from Protestant authorities. The castle's appearance on the Down Survey, commissioned by Oliver Cromwell to redistribute confiscated lands, marks a pivotal moment when many Irish Catholic families lost their ancestral homes.

Today, visitors to the site will find it directly north of a farmhouse, at the northeast corner of a farmyard. Though the castle itself no longer stands, its footprint remains visible as a rectangular area measuring approximately 34 metres northwest to southeast and 22 metres northeast to southwest. Low, grass-covered wall foundations, possibly medieval in origin, can still be traced on the ground, enclosed by walls on three sides with the southwest side open. The Ordnance Survey maps tell the story of the castle's gradual disappearance; the 1837 edition marked it as 'Milltown Castle in ruins', whilst by the 1913 revision, it had been reduced to merely 'Milltown Castle (Site of)'.

Adding to the site's historical interest, a dovecote lies about 120 metres to the southeast, suggesting this was once a substantial estate. Dovecotes were symbols of wealth and privilege in medieval and early modern Ireland, as only landowners with significant holdings had the right to keep pigeons. The presence of both structures paints a picture of a once-prosperous estate that fell victim to the upheavals of the Cromwellian conquest and subsequent centuries of neglect.