The Rossiters arrived in 1169 with the Norman invasion, landing at Bannow with Maurice de Prendergast, and by the early 14th century had established Rathmacknee as their principal seat. From here they expanded their influence, building castles at Slevoy, Tomhaggard, Bridge Bargy and Newcastle. The family reached the pinnacle of their power in the 15th century when John Rossiter and later his grandson Thomas served as seneschals, or governors, of County Wexford. It was likely one of these men who built the impressive castle that survives today.

The castle itself is a formidable five-storey tower house, built in the late 15th century and measuring roughly 10 by 9 metres. Its defensive features include murder holes, machicolations for dropping projectiles on attackers, and a pointed doorway protected from above. Inside, the tower rises through five floors connected by a mural staircase built into the thick walls. The second floor served as the main living quarters, complete with a fireplace topped by an oversailing canopy, elegant ogee-headed windows with built-in seats, and a garderobe tower for sanitation. Hidden within the barrel vault is an oubliette, a secret chamber that could serve as either a prison or hiding place. The tower is enclosed within a rectangular bawn, or fortified courtyard, with walls standing over 7 metres high and a defended gateway that would have controlled access to this stronghold.

The Rossiters' fortunes changed dramatically during the turbulent 17th century. Thomas Rossiter, who inherited as a four-year-old in 1592, grew up to play a leading role in the 1641 rebellion and hosted Cardinal Rinuccini, the Papal Nuncio, at Rathmacknee. His wife Ann was aunt to the famous Patrick Sarsfield, and her brother was son-in-law to Rory O'Moore, the rebellion's chief instigator. Despite owning almost 700 acres in 1640, Thomas lost nearly everything after the rebellion's failure, receiving just 66 acres in distant Connaught under Cromwell's plantation scheme. Though his grandson Walter briefly reclaimed the castle during the Williamite War in 1689-90, the Rossiters never recovered their ancestral home. Today the castle stands as a National Monument, remarkably complete with its crenellated battlements, look-out platforms at each corner, and the full circuit of its bawn walls, preserving this piece of medieval Ireland for future generations.