The settlement on the eastern bank of the River Nore at a shallow fording point was once known as Tullabarry, or Tulachbarri, a thoroughly Irish place-name that quietly persisted in official documents even as the town accumulated a new identity. By the early thirteenth century, an Anglo-Norman named Richard le Ragged had established himself here, and the association proved permanent enough that the place eventually shed its older name entirely and took his instead.

The earliest firm evidence for le Ragged comes from a dispute in 1220 between the Canons of St Thomas's Abbey in Dublin and a local priest over the tithes of his lands, which belonged to the church of Tulachbarri. The Abbey itself had been granted the churches and tithes of the district around 1200, with that grant confirmed in 1202. Over the following centuries, ownership of the town passed through a busy succession of hands, and the 1542 Ormond Deeds alone record four separate conveyances within the space of a few months, involving figures including Stephen Pembroke, James Purcell, William Seysse, Leonard Blaunchvile, and Margaret FitzGerald, the widowed Countess of Ormond, who entered a bond of one thousand pounds in March of that year concerning its possession. By 1580, Richard Butler, the youngest son of Piers Rua, the ninth Earl of Ormond, had inherited the town and was raised to the peerage as Viscount Mountgarret, with Ballyragget becoming the principal seat of that junior Butler line. In 1600 the town was garrisoned by thirty foot soldiers under the authority of Sir George Carew, Lord-President of Munster, after the sons of Lord Mountgarret joined a rebellion and conspired with O'More to arrest the Earl of Ormond. King James I formally constituted it a manor in 1619, granting Richard, the third Viscount Mountgarret, the right to hold two fairs. The mid-seventeenth-century Down Survey barony map of Fassadining shows a modest cluster of buildings at the site, with a planted area to the east, and it is suggested that the town was reorganised around a triangular green at roughly the same period. At the southern end stands the fifteenth-century tower house known as Ballyragget Castle, set within a substantial bawn, the walled enclosure that typically protected such structures and the households dependent on them.