While the city is working on procedures to more efficiently and securely manage its data centers, the changes that Denver Auditor Timothy M. O’Brien’s office urged last year remain largely incomplete, a follow-up report released last week discovered.
In January 2019, auditors reviewed Denver’s Technology Services agency’s use of two data centers, one primary and one secondary, and their findings raised several red flags about protocols.
Among other concerns, environmental controls at one of the centers failed to provide consistent temperatures and humidity, and Technology Services did not have an accurate inventory of hardware, databases and applications. Employees made occasional unauthorized changes to the data centers, and there was no formal way of ensuring that software applications still provided value to the city over time.
“By fully implementing three of our recommendations, the Technology Services agency is now ensuring the city data centers are collaborating with other agencies to share tools and knowledge and are implementing standards to align with TIA-942, the Telecommunications Industry Association’s standard for data center quality,” O’Brien’s office wrote in the September progress report.
Among the steps taken, Technology Services has trained certain employees to avoid making unauthorized system changes, and is transitioning both data centers to one “hosted center” from a dedicated service provider, to be completed by December.
For the hosted center, however, “the city could not migrate certain services — such as mainframe servers — to the hosted data center at this time, and Technology Services had no formal solution for migrating the unsecure servers,” auditors wrote.
The 2019 report documented water damage to the ceiling of the secondary data center as a concern about the status quo. A lack of geographical distance between the primary and secondary sites fails to safeguard against environmental hazards, and the secondary facility “was initially designed for another purpose and was not appropriately renovated to house a data center.”
The ability to provide an “easy-to-run report” of the city’s technology inventory is also dependent upon migration to the hosted center, so O’Brien’s office considered that recommendation “partially implemented.” The follow-up report did not consider the tool Technology Services purchased for inventory services to be able to “periodically assess the completeness and accuracy of the data center inventory,” though.
The tool, NetZoom, is capable of tracking energy costs at the hosted data center. But device-level energy consumption is not currently available for analysis, and consequently “Technology Services will miss opportunities for improving data center energy cost.”
Finally, Technology Services did create a checklist for maintaining the data sites on a weekly, monthly and annual basis, but auditors found the details lacking.
“For example, one of the weekly checks is titled ‘generator status check,’ but it does not provide the specific procedures that should be performed,” wrote O’Brien’s office.
The original report also addressed Denver International Airport's data centers, but because those recommendations extended into 2021, auditors did not review their progress in the follow-up report.