What remains is the chancel of a late twelfth-century Romanesque nave and chancel church, roughly five and a half metres across, covered by a barrel-vaulted roof, and now wedged between two later buildings as though the town simply grew around it and forgot to clear it away. For centuries it has served as a porch to the adjoining Synod Hall rather than as a place of worship in its own right, which goes some way to explaining why something so architecturally arresting can be so easily overlooked.

The church was built in the late twelfth century, and for several hundred years the full structure, nave and chancel together, would have formed a significant presence in the town. In 1767, fire destroyed the nave entirely, leaving only this chancel standing. The surviving section is a National Monument, and the reason for that designation is immediately apparent in the detail of the stonework. The chancel arch is Romanesque in style, meaning it is round-headed and typically richly carved, and here it runs to six decorative orders supported by columns, each order being a successive receding band of carved stonework around the arch opening. The east gable retains three decorated round-headed windows, and a fourth sits in the south wall. What surrounds this fragment now is almost equally strange: on one side, the fourteenth-century Synod Hall; on the other, the nineteenth-century Protestant cathedral of Saint Mary, which shares its dedication and its name with the much older structure it effectively replaced.