The City’s algorithms task force released a report of recommendations in November on how the local government should oversee its algorithm use, calling for organizational restructuring and better public engagement, though some former members wish they could have accomplished more.
Established in 2018 by Local Law 49, the automated decisions systems (ADS) task force convened for 18 months on a volunteer basis in order to review how hidden algorithms affect New Yorkers’ lives.
Several city agencies use datasets and algorithms to assist with decision-making, from the Administration for Children’s Services’ child risk and safety assessments, to the Department of Corrections’ classification system of inmate housing arrangements and confinement conditions.
The AI Now Institute at New York University published a list of reported city agency uses of ADS in April, including “risk assessment tools,” which use criminal justice data to “score” accused offenders on how likely they may re-offend, and “facial recognition” used by the Police Department.
Among the task force’s chief goals was to give recommendations on how to determine whether a city agency ADS disproportionately impacts persons based upon age, race, color, religion, national origin, gender, disability, citizenship status, and other characteristics.
“We know that certain decisions, whether made by a human or a computer, carry with them a risk of certain implicit or explicit biases,” the report states. “Furthermore, new technologies and systems—while providing new efficiencies—might introduce issues that may take some time for organizations to understand and correct.”
The November report arrives six months after claims of “fracturing” and “stalling” among the task force, due in part to confusion over what defines “ADS” and the city’s reluctance to share real-life examples of its data systems, according to reports by the Verge and Curbed.
In the report, the task force gives around 27 recommendations on how the city could approach its ADS use and manage its processes. 12 of the recommendations underline a need to educate the public and policymakers on how the systems work and to provide channels for public inquiry.
The report, however, doesn’t reference any actual city agency ADS’s currently in use, nor does it recommend any specific policy changes.
While the report states the city did ultimately provide ADS examples from the Department of Transportation, the Police Department, the Department of Education, and the Fire Department, it doesn’t share how the task force reviewed these examples, what they were and what they learned from them.
“Our goal was not to change policy immediately,” said Julia Stoyanovich, former task member and Professor in the Department of Computer Science and Engineering at the NYU Tandon School of Engineering.
“Rather, it was to be in an advisory sort of role, to provide support, and then the city would decide how to respond,” she said.
While the city could have done more from the beginning to give task force members access to the algorithm systems in question, the report represents “significant progress,” said Stoyanovich.
“We really need to start investing time, resources, and thought into our public education efforts,” she said. “This is something that is a prominent part of the task force report, and I personally feel is a very, very important set of statements.”
Not every former member of the task force agrees on the report’s outcome.
Meredith Whitaker, co-founder of NYU’s AI Now Institute, told the Verge the report “reflects the city’s view and disappointingly fails to leave out a lot of the dissenting views of task force members,” and adds that efforts to change specific policies were “scrapped.”
The Mayor’s office maintains that the task force’s meetings and its subsequent report are a success.
“Of the 20 members and chairs of the Task Force, 19 reached consensus on our recommendations. The fact is, this report made an impact,” said the Mayor’s office in a statement to Kings County Politics. “After review of the recommendations, the Mayor created a new position for an Algorithms Management and Policy Officer, showing swift action in response to the Task Force’s primary recommendations.”
The Algorithms Management and Policy Officer will have the role of ensuring city agency use of algorithms will be “fair, equitable, and transparent,” the statement said.
