The castle, a later construction associated with the powerful Butler dynasty, is said to occupy the ground where a convent of Poor Clares once stood. The Poor Clares were an order of Franciscan nuns founded in the thirteenth century, living under a rule of strict enclosure and communal poverty, and their houses were relatively rare in medieval Ireland.

Just how rare is suggested by a passing reference in late fourteenth-century ecclesiastical correspondence. A Franciscan friar named Peter de Trau, writing or reporting sometime between 1385 and 1399, mentioned only three convents of Franciscan nuns in the whole of Ireland. The Tipperary house may have been one of them, though the identification is not certain. If it was, then what is now the footprint of a castle was once a place of enclosed religious life at the edge of a medieval town, overlooking the same river that runs there still. The substitution of a castle for a convent is not unusual in the broader pattern of Irish medieval history, where one powerful institution frequently displaced or absorbed another, but the near-total disappearance of the earlier foundation, leaving only a cautious "is said to have been", gives this particular site an unusually elusive quality.