Thomas Dineley's 1681 drawing captures what was once a modest but formidable structure; a multi-storey tower standing southwest of the local church and graveyard. The castle featured distinctive defensive elements typical of its era, including a round-headed entrance on its eastern wall protected by a projecting box-machicolation, essentially an overhanging stone gallery from which defenders could drop objects on unwelcome visitors below. A possible angle-machicolation at the second floor's southeast corner suggests additional defensive capabilities, allowing the castle's inhabitants to protect the building's vulnerable corners.

Historical records indicate that Caherconlish once boasted four castles, though these fortified town houses have long since vanished, with traces apparently surviving only until the early 19th century. These structures were likely similar to surviving examples in Fethard, County Tipperary, and in Ardee and Carlingford in County Louth; urban strongholds that served as both residences and defensive positions for wealthy merchants or minor nobility. The Ordnance Survey maps mark the site of one castle southeast of the churchyard, whilst another is noted to the north of the village, though no physical remains survive at either location.

The third castle may have stood where Caherconlish House was later built, southwest of the village, whilst references to a "college" add another layer of mystery to the town's medieval past. Today, visitors to Caherconlish must rely on imagination and historical documentation to picture these lost fortifications, which once formed an integral part of this Limerick town's defensive and social landscape. The vanished castles serve as a reminder of Ireland's turbulent medieval period, when even modest market towns required substantial fortifications to protect their inhabitants and trade.