Two medieval castles once stood here, both recorded on the 1657 Down Survey map of the barony of Fore. These fortifications belonged to Thomas FitzSymons, an Irish Catholic landowner who held the estate until the Cromwellian confiscations stripped him of his property in 1641. The lands subsequently passed to Henry Pakenham in the later 17th century, marking a significant shift in the area's ownership from old Irish Catholic families to the new Protestant ascendancy.

Pakenham wasted little time establishing his mark on the landscape, constructing a square fortified house that may have incorporated elements or foundations from one of the earlier medieval structures. This new castle would eventually evolve into the grand Tullynally Castle that stands today. The second medieval castle likely stood about 150 metres west of what became Pakenham Hall; local tradition, passed down through generations and confirmed by the current owner Thomas Pakenham, maintains this spot as the location of Tullynally's original castle.

Today, visitors searching for traces of these medieval fortifications will find only subtle hints in the landscape. The suspected site of the western castle presents as a broad, low natural rise with a steep scarp along its southern edge. While the eastern and western boundaries follow gentle natural slopes modified by centuries of tree planting, the northern side shows signs of old quarrying work. The flat platform atop the rise offers little obvious archaeological evidence to the casual observer, though it continues to mark a spot where Irish history shifted hands during one of the country's most turbulent periods.