Built by the Keating family who held these lands for centuries, the castle represents the enduring power of Anglo-Norman families in medieval Ireland. The Keatings first established themselves here through the service of half a knight's fee from the Bigod estate, with the nearby motte at Kilcowan serving as their initial stronghold before this more sophisticated castle was constructed. By 1641, James Keating owned 180 acres and the castle at Baldwinstown, though the family's fortunes would change dramatically when John Keating was marked for transplantation to Connaught in 1653 with 46 dependents.

The tower house, measuring approximately 9.5 metres by 8 metres externally, showcases the defensive architecture typical of late medieval Ireland. Its round-headed entrance doorway on the southwest wall was once protected by machicolation from the parapet above and a portcullis, though no murder-hole was included in the design. Inside, the castle reveals its domestic arrangements across multiple floors: the ground floor served defensive purposes with its double-splayed loops, whilst the upper floors housed the family's living quarters. The second floor features a fireplace and garderobe chamber, whilst the third floor boasts rectangular windows with built-in seats and another fireplace of dressed stone. Each floor's ceiling was supported by corbels set into the walls, and a sophisticated system of mural and newel stairs provided access throughout the tower.

Despite serious structural cracks in the northeast and southwest walls, the castle retains many original features including its stepped crenellations and a gable wall rising above the northwest wall-walk. A bawn wall extends southeast from the tower's eastern angle, terminating in a circular tower approximately 3.5 metres in diameter and 7 metres high. This defensive wall once enclosed a courtyard to the west, between the tower house and the stream, though little else of this outer defence survives. Archaeological testing conducted in 1996 found no related materials in the immediate vicinity, but the castle itself stands as an impressive testament to the military architecture and social ambitions of the Anglo-Norman gentry in medieval Wexford.