The site's strategic position on a platform above the Boro River made it an ideal location for fortification, though the exact whereabouts of the original medieval castle of Keyer remains something of a mystery. What we do know is that in 1247, William de Denne controlled this barony, holding it through the service of three knights. By 1376, the castle had passed to Stephen Furlong, and sometime during the 16th century, the powerful Butler family took possession of the estate.

The Butlers held onto Keyer until the tumultuous 17th century, when Pierce Butler owned an impressive 2,800 acres here in 1641, encompassing lands in Keyer (Wilton), Clonmore, Edermine, and possibly Ballyelland. Following the forfeiture of Butler lands, Captain Robert Thornhill received the estate as a grant, only to sell it to William Alcock in 1695. Alcock constructed a house on the site, which stood until 1840 when it was replaced by a Gothic mansion designed by Daniel Robertson of Kilkenny; a fashionable architect of the period known for his work on several Irish country houses.

The Gothic revival mansion met its end during the Irish Civil War when it was burnt in 1923, leaving behind the atmospheric ruins that stand today. While these ruins offer no visible evidence of the medieval castle that once commanded this river crossing, they serve as a poignant reminder of the cycles of construction, destruction, and abandonment that characterise so many Irish historic sites. The platform overlooking the river bend, where the Boro turns northward, continues to mark this historically significant spot in the Wexford landscape.