The original Dockyard Clock Tower stood on a rocky outcrop which formed the foundations of a medieval tower known as la Torre del Tuerto which formed part of the defences of the New Mole. This tower protected the approaches to the New Mole along the Colorada (Red Sands). During the Anglo-Dutch assault on the Rock in 1704 the tower, which the Spaniards had mined before retreating, was completely destroyed causing huge deaths and casualties. John Drinkwater in his History of the Siege of Gibraltar described the action:
By daybreak on the 23rd [Julian Calendar], the ships appointed to cannonade the town, under admirals Byng and Vanderdussen, with those that were destined to batter the new mole, commanded by captains Hicks and Jumper, were at their several stations. The admiral made the signal to begin the cannonade, which was performed with great vivacity and effect, so that the enemy, in five or six hours, were driven from their guns, especially from the new mole head.
The admiral, considering that by gaining that fortification the town might sooner be reduced, ordered captain Whitaker, with the armed boats, to possess himself of it; but captains Hicks and Jumper, who lay next the mole, pushed ashore with their pinnaces, before the rest came up; whereupon the Spaniards sprung a mine, which blew up the fortifications, killed 2 lieutenants and 40 men, and wounded 60. The assailants nevertheless kept possession of the work, and being joined by captain Whitaker, advanced and took a small redoubt half-way between the mole and the town.