A motte, the earthen mound raised by a Norman lord as the foundation of a castle, sits about 130 metres northeast of the bridge that carries traffic over the Dinin river, now tucked within the grounds of Castlecomer House. The medieval church that once served the settlement no longer survives above ground, its location only tentatively identified near the site of St Mary's church. The town as it appears today is largely the product of a plantation scheme, a 1798 burning, and a subsequent rebuilding, and the widening of the main street towards its western end still reflects the deliberate geometry of a seventeenth-century planner.

The story of how Castlecomer became a plantation town is tangled with some of the more turbulent decades of early modern Ireland. By the mid-fourteenth century the O'Brenans held the territory known as Idough, of which Castlecomer formed part. The area's coal deposits, iron, and timber made it worth coveting, and in 1635 Thomas Wentworth, the Lord Deputy of Ireland, confiscated Idough for the Crown, displacing Catholic families including Richard, Viscount Mountgarret, and Richard Butler of Castlecomer. Two years later Wentworth sold the territory to Christopher Wandesforde, and in 1639 Charles I formally granted it to him as the manor of Castlecomer, along with the right to hold weekly markets and three annual fairs. Wandesforde set about building what a 1778 biography described as an elegant town modelled on an Italian original, with freestone houses, a market place, a church, a minister's house, an inn, and a small castle on Colliery Hill to guard the coal workings. When the 1641 Rising reached Castlecomer, some five hundred English Protestant settlers took refuge in those same buildings; the church and a house belonging to a man named Parkinson fell quickly to Confederate forces under Captain John Bryan and Captain Phillip Purcell, while the castle held out for more than three months before surrendering in March 1642. A bastioned fort, a defensive enclosure with projecting corner works, stood at the northeastern end of town between the Dinin and its tributary the Cloghogue, and it is thought Richard Butler was directed to take control of both the castle and this fort in the aftermath of the siege. The town was burned again during the 1798 rebellion, when United Irishmen and Crown Forces clashed in what became known locally as the Battle of Comer. The rebuilding that followed was driven by Anne, Countess of Ormond and Ossory, and by 1802 a survey recorded 211 houses, many of them good and slated, set along one very broad and well-built street.