In something that reads like science fiction but serves notice on the future of how the city governs through algorithms, a “shadow report” on New York’s automated decision systems (ADS) reveals infighting within the ADS task force and offers a bevy of recommendations, contrasting the city’s report in a number of ways.
New York University’s AI Now Institute published the “shadow report,” published two weeks after the city released the ADS Task Force Report, to document what happened over the two-year period during which the task force formed and convened.
The ADS Task Force was established in 2018 through Local Law 49 to review how hidden algorithms affect New Yorker’s lives. It was the first of its kind in the country, and included government officials, such as Director of the Mayor’s Office of Operations Jeff Thamkittikasem, and non-governmental experts, such as Meredith Whittaker of the AI Now Institute, a research center dedicated to understanding the social implications of artificial intelligence.
City agencies, such as the Administration for Children’s Services (ACS) and the Department of Corrections (DOC), rely on data and use algorithms to aid in decision-making. Part of the task force mandate was to determine how to hold city agency algorithms accountable for bias. The issue became a major concern in 2016, after a ProPublica investigation revealed New York’s predictive policing software was biased against blacks.
AI Now Institute’s report, “Confronting Black Boxes: A Shadow Report of the New York City Automated Decision System,” diverges from the city’s official report and reveals a less satisfactory picture of how the task force engaged the public. It suggests more urgent policy changes from city agencies and calls for greater transparency.
“They didn’t really seem to interrogate any of the issues or provide any insight that required convening the amount of experts that they had and the resources that they were given,” said Rashida Richardson, Director of Policy Research and editor of the shadow report.
Richarson calls the ADS Task Force Report “a disappointment.” She drafted her report alongside several experts as a “last resort,” following months of failed attempts to collaborate with the task force throughout its 18-month review period, she said.
“It just felt like we’d reached a wall, whereas on the advocacy end we tried everything we could, and there still didn’t seem to be any type of good faith effort,” said Richardson.
AI Now Institute’s report is broken into three main parts. The first part overviews the history of when the ADS Task Force formed to the end of its 18-month lifespan. The second part offers recommendations for the city’s algorithm usage, and the third part recommends how task forces reviewing ADS should act in the future.
The shadow report describes a “lack of consensus” among the task force and a “lack of public communication” in how the task force engaged the public throughout 2019.
“On the executive side, I don’t think they were very transparent,” said Richardson. “I don’t think the mayor’s office appreciated the gravity of this bill or this issue.”
The shadow report’s timeline shows Mayor Bill de Blasio did not sign or veto the original bill. It became law automatically after a 30-day waiting period.
The city report states the task force held community meetings in Queens, Staten Island and the Bronx, in order to include the public in the review process. The shadow report goes a step further and claims information about these meetings was not provided to the public on any government websites.
What is also omitted from the city’s report, but is detailed in the shadow report, are the ways in which civil rights advocates, including some non-governmental task force members, unsuccessfully tried to bring in more outside experts into the fold. These attempts include two public letters to the mayor’s office, signed by several task force members and non-members, calling for more public engagement.
One paragraph describes a public forum in late April which showed a lack of consensus among the task force. Governmental task force members “abruptly ended” the meeting when a series of “direct questions about the lack of public engagement” were “answered candidly” by a non-governmental task force member.
“Every single comment received by the Task Force in person or in writing was documented and included in deliberative materials that served as the basis for the Task Force’s summer conversations and decision-making process,” said Mayoral spokesperson Laura Freyer.
The city’s newly established Algorithms Management and Policy Officer will include public input as he or she carries out their work, Freyer said.
The shadow report’s second section outlines recommendations for the city and contrasts the official ADS report in its boldness and specificity.
The official report recommends the city establish an “organizational structure” to manage algorithm usage, but it makes no mention of how agencies should act, and it doesn’t suggest specific protocol.
In more exact terms, the shadow report details ways city agencies should move forward, including suggestions for ACS, DOC, the Department of Education (DOE), the Police Department (NYPD) and several others.
“There are a lot of recommendations targeted at agencies because there are very specific things they can do within their mission and authority, right now, that doesn’t require legislation being passed, that can help address some of the concerns that already exist,” said Richardson.
The shadow report also calls for city agencies to publish all of their algorithm source code, print the source code alongside “plain-language” explanations of its usage into physical copies, and then distribute the copies to branches of the New York Public Library.
The ADS Task Force Report doesn’t suggest city agencies publish source code. Instead, it asks the city to publish “agency reported information,” when “legally permissible,” after getting approval from a legal and security team.
The shadow report’s third part gives recommendations “based on the New York City experience,” suggesting ways to smoothen the process for future task forces and other types of temporary governmental bodies.
“It’s making it a more global conversation and also making sure the research and insights learned from this experience can have utility both in New York City and elsewhere,” said Richardson.
Despite her disappointment with the city’s task force report, Richardson said she feels positive about the future.
“The fact that throughout the process while we were advocating, there were also people in government advocating, who want to do something and understand the importance of these issues, makes me hopeful that something’s going to happen,” she said.
When asked to respond to the claims in AI Now Institute’s shadow report, the mayor’s office stood by the task force’s official report, saying it represents a consensus from 19 of the 20 members and chairs, according to Freyer.
