The District has been unable to determine how at-risk children are performing in public schools because critical data “on attendance, dropout rates, in-school suspension rates, graduation rates, post-high school college and career success data disaggregated by race, gender, and socio-economic status over time are compromised or nonexistent.” That’s a conclusive finding of the newly released “Measuring What Matters: More and Better Data Needed to Improve DC Public Schools,” published by the Office of the DC Auditor and prepared with Data Ethics LLC.
DC Auditor Kathy Patterson said that the absence of such data has meant the city doesn’t know something as simple as “whether interventions are improvements or merely interventions.”
“Our ability to bring about racial equity through education policy and practice is thereby crippled,” she added in a March 10 letter to Mayor Muriel Bowser and DC Council Chairman Phil Mendelson, who has scheduled a public hearing for March 18.
A chief obstacle to serving the city’s at-risk students appears to be the failure to construct a Statewide Longitudinal Data System (SLDS).
OK, watch your eyes. Don’t let them glaze over.
An SLDS doesn’t involve the useless collection or accumulation of irrelevant information. The feds define it as a system that “collects and maintains detailed, high quality, student- and staff-level data that are linked across entities and over time, providing a complete academic and performance history for each student.”
To understand its importance, consider one explosive fact from the report: In 2018, more than 18 % of ninth graders were retained at their current academic level. A functional SLDS would include early warning signs that could identify students who are on a downward spiral. It could also guide the creation of a customized plan for each student, preventing such disastrous results.
DC’s Office of the State Superintendent of Education (OSSE) is responsible for the implementation and enforcement of local and national education laws, rules, regulations and policies. It is also the agency charged with collecting the data that would comprise the SLDS. It has never effectively fulfilled that duty — despite receiving a $10 million grant from the federal government and $25 million in local funds between 2007 to 2012 to construct such a system.
What happened to that money? Should there be an investigation?
The audit released Wednesday included an examination of current and missing data; a discussion of how the absence of data affects the quality of education in the city; a presentation of best practices in other states with longitudinal systems; and recommendations for what should be done to build an SLDS in DC.
The report was mandated by the Research Practice Partnership Establishment and Audit Act of 2018. That legislation followed the scandal surrounding the graduation of students at Ballou and other DC high schools who either had not attended the required number of days or had not completed course requirements to receive their diplomas.
Ward 3 Council member Mary Cheh co-introduced that bill and ushered it through the legislature. Speaking at the release of the report, she called the findings “deeply concerning.” She promised to introduce new legislation that would require OSSE to build a “data warehouse,” among other things.
Prior to the recent press conference, she worried about how the absence of reliable and broad data would affect the city’s ability to address documented learning loss that has occurred during the coronavirus pandemic. “We cannot provide appropriate services and supports for our students until we address this data problem — and the need is more urgent than ever,” she said in a prepared statement.
At-large Council member Robert White echoed those concerns while focusing on Black and brown students, many of them underperforming in DC’s public schools. He said the report contained very “concerning data points” while noting that “data has a heartbeat.” He asserted that the report could — and should — change the focus from one on schools to one on students.
In 2015, the National Academy of Sciences conducted an evaluation of the Public Education Reform Act and cited the absence of critical data. Evaluators said DC should have a single online data warehouse accessible to educators, researchers and the public that provides data about learning conditions and academic outcomes in both DCPS and charter schools. “Such a warehouse would allow users to examine trends over time, aggregate data about students and student groups, and coordinate data collection and analysis across agencies (education, justice, and human services).”
Apparently, education leaders’ oft-repeated mantra that their actions are data-driven has been mostly a false narrative that has jeopardized the future of thousands of students.
White students or middle-class African Americans with strong academic backgrounds may not have been significantly affected by that false narrative. However, other Blacks from mostly low-income families have struggled, left without the kind of targeted, effective assistance that could have been crafted using a warehouse of data collected over time that would provide a deeper and more complete profile of each of them and their particular needs.
For example, each year the public has received standardized test scores indicating continued poor performance at the same schools, many in predominantly Black communities. Concerned parents and advocates often were assured that the students were being tracked and customized support would be provided. The following year the results were nearly the same — or worse.
“The District currently relies heavily on aggregate and point-in-time data and does not use individual student test score data linked longitudinally over time for accountability, equity, or improvement-focused analyses,” auditors wrote in their report.
Lay the primary blame at OSSE’s door. In fact, as I read the report, I kept thinking parents should demand the superintendent whose tenure was covered by the report — Hanseul Kang — should return her salary, all of it.
The auditor said that while the education agency may have nominally required certain data, it permitted some charter schools to negotiate what eventually was submitted, which frequently was incomplete. In some cases, information never arrived. Further, Patterson and her team found that the District’s education leaders, particularly OSSE, “perpetuated flawed systems of collecting and reporting data on attendance and chronic absenteeism; failed to track the trajectory of large numbers of high school students assigned to alternative programs; and failed to adequately track transfer trends of high school students who leave charter and the District’s selective high schools for [traditional] public schools.”
Interestingly, auditors found that “high school students struggling in their first year were transferred out of [public charter schools] at disproportionately high rates.” Equally disturbing, “the graduation rates of students transferred out of PCS high schools is substantially lower than the graduation rate of students remaining in PCS schools.”
The OSSE failed to track course-taking, credit completion and other indicators that would allow for the development of an early warning system to support struggling students and keep them on track to graduate; failed to ensure data integrity, thus raising questions about the validity of data published in school report cards; and failed to ensure other essential data collection was complete.
And, if that weren’t enough, in an analysis of data collected prior to 2016, the auditor found that students missing altogether from enrollment records were more likely to be considered at-risk, to have disabilities or to be homeless. That suggested racial or economic bias in the city’s data collection.
Oddly, some council members and advocates have renewed a push to make OSSE independent from mayoral control. White said the reasons the SLDS had not been developed are the governance structure and politics. “No one appointed by a politician is going to jump out with information that is going to be perceived as critical.”
In my view, now is not the time to give OSSE any independence. An agency that has failed to complete a critical element of its mission shouldn’t be passed along until it has gotten itself in order and corrected the deficiencies. I am not persuaded that the State Board of Education is prepared to take on oversight or supervision of such a massive agency.
If anything, Bowser ought to become more intensely and directly involved in oversight of the education cluster.
“This audit is a perfect opportunity for the Mayor to use her authority over public education in the District — to direct her new state superintendent to finish the job of building a statewide data system,” Patterson told me. “Families need — families deserve — to know that their elected officials are generating the kind of information that can effectively guide our schools. Other states have early warning systems; other states don’t lose their kids like we do for failure to effectively track their progress.
“We can do better. We will do better,” added Patterson.
I am not so optimistic, especially given written responses from education leaders to the audit. Rick Cruz, chair of the DC Public Charter School Board (PCSB), said the report “misrepresents current data practices and key relationships in ways that imply bad faith on the part of DC PCSB, or even the entire public charter sector; gives analysis that [the board] cannot replicate or verify and thus cannot meaningfully respond to; and omits crucial context necessary to provide a nuanced understanding of the agencies, practices, and systems being examined.”
Ironically, Deputy Mayor for Education Paul Kihn admonished Patterson for failing to follow the intent of the law. “We regret this audit fell short of its charge to develop foundational data assets and insights to strengthen and ground the Research Practice Partnership going forward.”
Wasn’t it the job of the OSSE, an agency under his oversight, to create the foundation for the longitudinal system? Now, he’s upset that the auditor didn’t do the work for his team.
Shana Young, interim state superintendent of education, defended the work her agency has done and disputed several aspects of the audit. She variously told auditors they weren’t qualified to evaluate the agency’s data collection decisions and practices; didn’t know what they were talking about; and to please stay out of the way and let us do our work.
As Patterson was preparing to release her audit, Bowser announced in January 2021 the selection of the Urban Institute as the city’s education research partner. Was that move designed to shield the administration from any blowback caused by the report?
Given the absence of significant data, don’t be surprised if the new research and policy group finds itself facing the same challenges as those encountered in 2015. “Good research starts with good data,” Cheh said during the press conference. “You can’t fix what you don’t measure accurately and honestly.”
And the audit provides sufficient evidence to doubt the integrity of what has been collected and distributed by OSSE and other education agencies.