Originally known as Ballymagir, this fortified enclosure was built by the powerful Devereux family who controlled it from the 13th century onwards. The site came into their hands through Alina de Heding, daughter of Sir Alexander de Heddon, who married Sir Hugh Devereux in the mid-1200s. Together with the nearby castle at Adamstown, Ballymagir formed the heart of the Devereux family's extensive Wexford estates, held by the feudal service of two knights' fees.

The rectangular earthwork measures roughly 90 metres east to west and 70 metres north to south, defined by substantial earthen banks that still rise over four metres high on the southern and western sides. A water-filled moat, or fosse, surrounds what remains of the defensive perimeter; widening to 11 metres on the eastern approach. Within the entrance on the northern side lies a curious feature: a separate 73-metre-long inner moat that doesn't connect to the outer defences, likely serving as a medieval fish pond to supply the castle kitchen. At the heart of the enclosure stands the tower house of Ballymagir, a testament to the site's former importance as a fortified residence.

The landscape around Richfield has changed dramatically since medieval times. When the Devereux family held sway here, the tidal waters of Ballyteige Lough, then called Mablen Haven, reached to within 500 metres of the castle walls. Victorian drainage schemes pushed the sea back beyond Ballyteige Burrow, leaving today's castle ruins over two kilometres from the coast. The family's fortunes shifted too; Nicholas Devereux was marked for transportation to Connaught with 31 dependents during Cromwell's conquests in 1653, though he managed to reclaim his lands through the Court of Claims in the 1660s. By the 19th century, the estate had passed to the Loftus family, who renamed Ballymagir as Richfield, the name it bears today.