The castle's history stretches back to at least 1247, when Hugh, son of Martin, held the lands from the Vallence estate in exchange for a quarter of a knight's service. By 1324, it had passed to Philip Franceys, and around 1425, the heir of Nicholas Frenshe controlled what was then called Ballytormyn. The Down Survey barony map from 1656-8 depicts the structure as a substantial house, suggesting it was still a significant dwelling in the mid-17th century.

The French family connection to Ballymachtorny is particularly intriguing. Nicholas French, who owned 174 acres here in 1640, was no ordinary landowner; he served as the Catholic Bishop of Ferns from 1645 to 1678, navigating the treacherous waters of Irish religious politics during the Cromwellian period. His father was John French of Rathmacknee, and his mother Christina Rossiter. After Nicholas, the estate passed to Constantine Neale, but by 1750 the Codd family of Castletown had acquired it. Around 1800, John Codd dramatically altered the medieval structure, reducing the tower's height and incorporating it into a new five-bay, two-storey Georgian house.

Today, what survives of the original castle is the base of a rectangular tower measuring approximately 8.4 metres northwest to southeast and 6.9 metres northeast to southwest, complete with a defensive base batter. The southeast wall still features two original slit windows, whilst a pointed doorway on the northwest wall, measuring 0.86 metres wide and 2.2 metres high, now provides access to the 19th-century addition. Most remarkably, the vault over the first floor remains intact, a tangible link to the medieval fortress that once commanded this quiet Wexford valley.