Lt. Col. Thomas James of the Royal Artillery visited the dwellings around a decade before the devastation of the Great Siege and described them thus:5
‘The storekeeper and ordnance clerks dwellings, commonly called the bomb-house, was once a fine Moorish building: I take it to have been the residence of their governors, because I have seen the same kind of structures in Spain, and never but one in that style in each town: and that which is peculiar, is the top of the house, which is a flat oblong terrace; round it is a wall of three feet high, and on the wall are stone pillars that support a roof: there houses are much higher than any other building in the town, and command the whole: this upper apartment is at prevent a dwelling room, the spaces between the pillars being filled, and now has windows, and a door place. The cellars remain in their old state, one of which I take to have been the family mosque; the inside is an oblong square, and round the centre are pillars that support a handsome cupola.
Round the architrave is an inscription, but so defaced that I could not make anything of it. This house, when we took the place, was quite entire, and very large; and the complete remains of the Moors, as a dwelling, in the town; but the changes it has since undergone, have almost diverted it of its ancient beauties; ancient I call it, because it might have been built soon after the coming of those people in seven hundred and eleven ...’
Further modifications and additions were made to the house from the 18th century onwards, particularly due to damage suffered during the Great Siege, leading to its present layout. According to Drinkwater part of the building was hit by cannon shot and burnt down and group of officers dining in the house at the time were injured - one of them, Captain Burke, the Town Mayor was killed as a result.
After the siege, the reconstructed grounds were divided into two distinct military quarters one of which became the official residence of the Assistant Director of Ordnance Stores from which Ordnance House derives its name. Throughout the 19th Century these properties remained in use as military quarters for the Officers of the Garrison.
However, by the early 19th Century property No. 7 situated at the north-west corner housing the area of the Moorish Baths appears to have been converted into a small museum with an entrance opposite the synagogue as can be seen in the 1908 survey map.