According to the Ordnance Survey Letters, Littletown Castle once belonged to the Dillon family, though its exact location has puzzled historians for generations. When antiquarian John O'Donovan surveyed the area in the 1830s, he noted somewhat sceptically that there was "no castle in Littletown", though he did observe ruins just south of Littletown House that might have been the elusive fortress.

The 1837 Ordnance Survey maps offer tantalising clues about the castle's possible location. A small rectangular structure appears on the six-inch map, positioned about 80 metres south of the ruins of Littletown House. This northeast to southwest aligned building, depicted as a ruin on the Fair Plan map from the same year, may well have been the remains of the medieval stronghold. The rectangular footprint suggests a typical tower house design, common amongst the Anglo-Norman and Gaelic Irish nobility who dominated the Westmeath landscape.

Today, visitors searching for Littletown Castle will find no dramatic ruins or crumbling battlements. Where the 1837 maps once marked a structure, only a small hollow depression in the ground remains, a subtle reminder of the building that once stood here. This absence of physical remains is not uncommon in Irish archaeology; many castles and tower houses were dismantled over the centuries, their stones repurposed for field walls, houses, and farm buildings. The story of Littletown Castle survives primarily through these historical documents, compiled by researchers Frank Coyne and Caimin O'Brien, offering a glimpse into Westmeath's medieval past even when the stones themselves have vanished.