The use of water safety plans (WSPs) has been recommended by the World Health Organization (WHO) since 2004 as a highly effective means of improving water safety management. Experience with the implementation of WSPs is increasing worldwide, but there is no unified or standardized methodology for how the verification of a WSP should be conducted. In this article, we present a thorough evaluation of a specific WSP five years after its initial implementation. We reviewed the risk assessment methodology used by a water utility in Salta, Argentina, and assessed the implementation of control measures. To objectively evaluate the effectiveness of the WSP, we evaluated water quality parameters and customer complaints using a time-series analysis. We show that although some control measures were implemented, and a reduction in risk values was observed, it was not possible to improve long-standing problems in the water supply of the city of Salta, such as the number of consumer complaints or high turbidity levels in the water during the rainy season. We discuss the role of rigorous scientific assessments and the importance of legislation and regulatory bodies in implementing the WSP.