Find a back issue

Making Dallas Even Better

The Dallas Miracle: How Data Show Mike Miles Deserves a New Contract

Charts showing Dallas's dramatic improvement in 2013 (downward movement=better).
Charts showing Dallas’s dramatic improvement in 2013 (downward movement=better).

The most important question parents, teachers, staff, and trustees in DISD can ask themselves this summer is this: Did Dallas ISD perform a miracle in 2013? Because according to one of the most comprehensive performance indexes you’ll find, the answer is yes. And if that’s the case — and it is, you can choose to acknowledge it or not — it should affect everything about how we hold the school board accountable next year in supporting more of Mike Miles’ reform efforts. As well, it means there is just more proof that the school board should extend Miles’ contract for at least two to three years.

The data come from a group out of Houston called ERG Analytics (ERG stands for Education Resource Group). The company is well known to school boards, administrators, and legislators across Texas, to whom they pitch their data analysis tools. (In fact they spoke to the DISD board not long ago, but only to report on the district’s financial performance; the company was brought in by DISD’s CFO, Jim Terry, one of the smartest people in the school district.)

I talked to ERG’s Paul Haeberlen last week so he could explain what had been described to me as “the Dallas Miracle”: that a district that had been mired in sub-mediocrity (by ERG’s numbers) for a decade took a dramatic turn for the better in 2013.

Here’s what happened: ERG has a performance index that measures a school district’s performance based on seven measures of student outcomes, and THEN controls for poverty (which you must do to have any meaningful discussion about how well a school district is doing the job of educating its kids). And THEN ERG uses that index— the final score the district gets — to rank districts against each other.

You see why all this is important? First, you have to take several measurements to get a realistic picture of student achievement across the vast ecosystem of a large school district. They take three difficulty levels of the STAAR assessment, two graduation measures (rate and college-readiness), and two college readiness test scores (mean scores of SAT and ACT). They are weighted appropriately (not equally), and this produces a number. That’s why, for example, graduation rates only count 10 percent toward the final measurement number — because it’s only one metric, albeit a high-profile one. Judging a district based solely on any one metric would be like judging the effect of a storm system by counting lightning strikes.

Then, by adjusting for poverty levels, ERG provides context — whether the storm takes place over an ocean or a dessert. As we’ve talked about, a school district could be doing an average job of teaching rich kids and its test scores will still be outstanding. You could also be doing a great job teaching poor kids, and your district’s test scores are much lower. ERG adjusts for this so everyone is graded on how well they’re teaching the kids in front of them.

Then, ERG understands that the performance score is not the end of the story. Districts aren’t trying to improve in a vacuum. They’re competing against other districts across the state that are also trying to improve. Think about it like ships in an ocean. If your ship is trailing all the others, but knew information helps you figure out a way to increase its speed by 5 knots, you’re still losing ground if everyone else uses that same knowledge to increase their speed by 6 knots. (In this case, using new methods of instruction, or implementing the latest research, what have you.)

Now that we understand how in-depth ERG’s numbers are, how does DISD fare in the company’s performance index? By ERG’s performance measures, DISD had been in the third quartile [of the 200 largest ISDs statewide] for the past decade, Haeberlen said.

The charts and table at the top of this post show just that. But look at what happened in 2013: Dallas ISD, the second-largest ISD in the state, one which has never moved out of the third quartile, shot to mid-second quartile, placing 69th among the 200 largest districts in Texas. For comparison, Houston ISD, a current Broad Award winner, is ranked 62nd.

“What happened in 2013?” Haeberlen asks. “The district improved across the board.” In other words, in six of the seven metrics measured, DISD showed improvement. For a district this size, that’s astonishing.

All DISD really did, Haeberlen points out, is catch up with the many outstanding districts around it. Richardson ranked 2nd by ERG measures (and has a student population almost as poor and challenging as DISD’s, but without the board dissension). HEB, Garland, Carrollton-Farmers Branch, Mesquite, Highland Park — they all show well in ERG’s data.

Why should we all care? Because the quality of leadership in a school district has a profound impact on student learning. And it’s really hard to measure the outcomes of our leaders’ decisions when so much achievement data is just noise, thrown out piecemeal, usually cited to back up a conclusion that an interest group arrived at years ago. That’s why we need to look at the smartest data analysts out there, the ones not involved in the political muck surrounding school districts, to seek clarity — so we can hold our school leaders accountable.

In this case, Mike Miles has earned more time to see his reforms through. We need to track ERG data over the next three years to see if his reforms continue to have a positive outcome in ERG’s performance index. For now, it’s clear to me that Mike Miles is doing what he was hired to do: make DISD a much better district.

  • Michael MacNaughton

    The districts overall ranking in the 2-1 quartile continues a steady improvement as measured by the ERG for the last 5 years. But understand that the ranking is based on the districts combined academic and financial performance. In other words, the district provides a high relative academic performance at the lowest relative cost per student. We have caught up to other districts in our “chase group” by reducing the cost to educate each child. How? There are 1,000 fewer teachers as of January 2104 than when Miles came aboard…and 3,000 fewer teachers than 6 years ago even though the number of students has increased. Voilà!

  • Hopping mad

    1,179 emails to DISD trustees telling them not to renew Miles contract says you are wrong. There are many seasoned and young teachers in DISD and we know what we are doing. They do not teach us useless junk when we go to college to earn our degree or advanced degree. We are not afraid to do our job to the best of our ability and we are VERY concerned for our students! We are not lazy and we are good teachers. Miles does not know what he is doing and our students are suffering for it. That is the truth.

  • Todd Williams

    in 2012-13, Dallas ISD also closed the gap with the state by 1 to 3 points across reading, math, writing, science and social students for all students, 2 points across all subjects for African American students and improved in math, science and social studies for Hispanic students while remaining flat in Writing. Terrific achievement. We won’t understand 2013-14 progress until we get state scores in September, but this is something to be cheered and acknowledged. This is not just about cost reductions.

  • EastDallasDad

    We should withhold judgement until all the data is in. You also failed to mention that Miles fell far short of what he promised the school board last year with regard to the Destination 2020 schools. I’d also like to know why we (taxpayers) should pay any superintendent a salary close to that of the President of the United States.

  • Eric Celeste

    Actually, it’s not at all about cost reductions, because Michael MacNaughton is 100 percent wrong. MM, you’re confusing ERG’s District Productivity Index (which takes both academic and financial performance into account) with the District Performance Index, which only considers the academic performance measurements I describe in the post. (ERG also calculates a District Financial Index, separate from academic performance.) Here’s the webpage that shows the top 10 districts in each performance category to help you keep them separate in your head:

    http://www.educationresourcegroup.com/page_bestdistricts_2014.html

    I know you’re not just trying to find damaging information about Mr. Miles, MM, and care only about what the data show. Honest mistake on your part.

  • Eric Celeste

    I should note that if anyone were just looking for out-of-context damaging data out of this, you should know that ERG’s District Achievement Index (which does not control for poverty/demographics) shows Dallas at 180 out of 200 schools. Take that and run with it!

  • Eric Celeste

    200 districts, I mean.

  • Eric Celeste

    I’m sure you’d type that same first sentence if the ERG data showed a huge drop in 2013 performance. As to your last question, there are several online courses you can take that explain marketplace economics.

  • Eric Celeste

    Oh no. Emails. That number! So specific! So much data to counteract what I’m saying! Heaven forfend, what shall I do?

  • Eric Celeste

    Oh no. Emails. That number! So specific! So much data to counteract what I’m saying! Heaven forfend, what shall I do?

  • Senate Loser

    Mr. Celeste I attended Frazier, Anderson and graduated from Lincoln High School. My son attended Polk, Griener and graduated from Townview. We both attended Dallas ISD from grades 1-12. Do you have children enrolled in a Dallas ISD school? If so identify the school. Are there any subs or uncertified teachers teaching at that school?

    (If you would like to have my email, I will gladly email you)

  • Eric Celeste

    According to the bio at the top of the page, it says “Celeste’s daughter was a K-12 DISD student who graduated high school in 2012.” Not sure why it matters, but she went to Stonewall, Irma, Long, Booker T. Did you have a point? If so, identify it now. I command it thus be done!

  • Eric Celeste

    (That was in response to Senate Loser comment below.)

  • Eric Celeste

    (Not sure why I can’t reply to the below, so here’s my reply, above Senate Loser’s comment):

    According to the bio at the top of the page, it says “Celeste’s daughter was a K-12 DISD student who graduated high school in 2012.” Not sure why it matters, but she went to Stonewall, Irma, Long, Booker T. Did you have a point? If so, identify it now. I command it thus be done!

  • Diane Birdwell

    SO, where is your column demanding that I ALSO get a raise and a contract extension?

    With all deference to my ultimate boss here, he runs a district, but he does not teach in it, and gains or losses do not happen overnight.

    2,000 teachers, most of the EXCELLENT ones, left DISD, and many more did this year, too.

    Where is our pay raise? Where is our longer than a one year contract? I will keep asking.

    And just for the record, do you get your talking points form the DCC, the Regional Chamber, SFC or the two PACs… or just the same four guys behind all of them?

    p.s. Eric—not to bring in your kid, but when your kid goes only to special DISD schools, where they screen the applicants, BTW or have a LAKEWOOD address, Long and Stonewall… kinda hard for you to have much street cred on what we endure on a regular basis in the comp schools.

  • Diane Birdwell

    SO, where is your column demanding that I ALSO get a raise and a contract extension?

    With all deference to my ultimate boss here, he runs a district, but he does not teach in it, and gains or losses do not happen overnight.

    2,000 teachers, most of the EXCELLENT ones, left DISD, and many more did this year, too.

    Where is our pay raise? Where is our longer than a one year contract? I will keep asking.

    And just for the record, do you get your talking points from the DCC, the Regional Chamber, SFC or the two PACs… or just the same four guys behind all of them?

    p.s. Eric—not to bring in your kid, but when your kid goes only to special DISD schools, where they screen the applicants, BTW or have a LAKEWOOD address, Long and Stonewall… kinda hard for you to have much street cred on what we endure on a regular basis in the comp schools.

  • Eric Celeste

    So, I deleted Diane Birdwell’s comment partially because she’s just ranting about the same crap she always does, and also bizarrely tries to bring my kid into this, and say, I don’t know, that my experience isn’t DISD enough or something because of the schools my kid went to. It’s just asinine and troll-y and we don’t do that here. I’m fine with criticism and disagreement, but if you’re the same nine people who fill the comment sections of the other education/news blogs in town with your conspiracy theories, more power to you, but this ain’t your home.

    You are welcome to try again, Diane. Some advice: Stay on topic. Show your work. Leave my kid out of it.

  • marian

    I clicked to see Ms. Birdwell’s comment from 10 hours ago and all I see are your snarky exchanges with Mr. McNaughton, another DISD teacher and a DISD parent.. Your daughter went to Booker T – not exactly your typical DISD school. I believe their enrollment includes almost as many students from the suburbs as the Dallas district.

  • EastDallasTeacher

    Is there anywhere that gives more detail on what these numbers mean or how they were crunched? I see graphs, but have no idea how it was reached. I know you mentioned seven criteria but how did ERG achieve their rankings? And if they are going to say that it’s proprietary and they can’t share raw data, then can we really use this in any kind of discussion about our students?

  • Eric Celeste

    Well, you can do what I did: Call them, ask them for their time, take a webinar with the folks there and another education expert so you understand the numbers, see the raw data, and ask questions. Or you can engage in some fantasy that this is all part of the conspiracy.

  • EdFuller

    Unless you know the methodology behind the analyses, you cannot make any conclusions from such a study. How do they adjust for poverty? Through regression analysis or simple comparison? Do they examine student attrition or only student scores (the fastest way to improve scores is to push out the lower performing kids)? From what I know of their methodology, they do not properly control for factors outside the control of teh district. Further, SAT scores and ACT scores should never be used to compare districts (according to the College Board and ACT). Why? Because participation rate drives the scores. Was participation rate controlled for? You cannot tell from this story. So, essentially, the writer is asking us to believe her/him and s/he is being asked to believe ERG with no independent analysis of the methodology. That is just bad journalism. As for my knowledge of this? I have studied Texas education for 25 years, am executive director of an evaluation center, and teach research design, evaluation, and quantitative analysis at Penn State.

  • EdFuller

    You absolutely CANNOT compare districts or assess growth on changes in percentage of students passing. It gives you erroneous results every single time. You have to compare scale scores and even then you need to adjust for student demographics and other factors. A district looking better on a flawed metric doesn’t mean the district is actually getting better.

  • EdFuller

    For some reason my posts immediately disappeared.

  • Eric Celeste

    Who said they are? And there is an entire section about how they scale for demographics.

  • Eric Celeste

    Yes, Ed, I know who you are. Someone who has a history of cherry-picking data (or cherry-picking your analysis of data) to reach a very predictable conclusion.

    Examples:
    http://www.kipp.org/news/statement-examining-high-profile-charter-middle-schools-in-texas-by-dr-ed-fuller

    http://www.austinchronicle.com/daily/news/2011-12-08/fuller-versus-idea/

    But to amuse myself, I’ve asked the folks at ERG — whom no one on the school board, even those who disagree with everything I write, doubt in terms of their data analysis — to respond. They’ve already given me a lot of their valuable time and I’ll understand if they say, sorry, we don’t need to please every Diane Ravitch wannabe out there, but we’ll see.

  • Greg Pulte

    It was a legitimate question. Why are you so defensive?

  • Eric Celeste

    Great comment. I understand that their analysis is not the be-all, end-all. However, it is very rigorous, and respected by administrators across the state. (A lot of districts who do well on TEA achievement don’t like them, because they do more poorly when you factor in demographics.) And you’re right, college completion is the single metric most people say is the one best indicator of success. (And that is not factored into their formula.) Great comment, thank you.

  • Eric Celeste

    Because I like to fight? And because if the district, the board, other districts, and legislators say their data is clean and their methodology good, why would I have reason to suggest otherwise? And why has no one, ever, in any forum, in any way questioned ERG’s data for a decade — until I point out that it showed a dramatic jump for DISD last year? Because they don’t like that answer.

  • EdFuller

    well, you know, I got that study published through a peer-review process. I also had several experts from around the country review. And they all approved it. You, however, don;t back up anything. No statistician has backed up ERG. I worked with them on a project and I know their methodology won;t pass peer review. And neither would your “reporting” which is why you write this rag.

  • Eric Celeste

    Reply from the ERG folks:

    Fuller is “dead wrong.”

    “We have spoken with him.”

    His analysis “does not makes sense.”

  • Paul Haeberlen

    Mr. Fuller appears to have issues with the article.

    Let’s examine his arguments.

    From what I know of their methodology, they do not properly control for factors outside the control of teh district.

    Just what does Mr. Fuller know about the methodology used to evaluate the performance of Dallas ISD? What is the “proper control” for factors outside the control of the district? Who is it that determines “proper control?” Is Mr. Fuller in charge of this? Can any other person or organization have an approach, or is there a rule book somewhere that defines performance management rules? The ERG methodology has been in use for over a decade and validated by school districts representing millions of students. ERG founders have extensive (many more than 25 years) of experience in performance management methodologies.

    Mr. Fuller seems to be making an accusation that Dallas ISD is “pushing out low performing students” more aggressively than other school districts. Perhaps he should share his evidence? Has he brought up his concerns to the administration at Dallas ISD?

    Further, SAT scores and ACT scores should never be used to compare districts (according to the College Board and ACT). Why? Because participation rate drives the scores. Was participation rate controlled for?

    We have a contradiction here. Mr. Fuller states “from what I know about the methodology” then makes the argument about use of SAT and ACT scores without knowing that ERG does control for participation rates. So why does he make the statement at all, since its source is the College Board and ACT and he clearly does not understand the methodology used? If “you cannot tell from the story,” then why make the accusation? If Mr. Fuller did not independently verify his charge, then who is really out of line here? Is commenting on methodology that one does not understand ok, when publishing methodology that one does understand is not ok? ERG understood the impact of participation rate by evaluating the SAT and ACT data. In doing so, we learning how the data can be used to improve the clarity of performance. Following broad guidelines without digging into the data to understand what can and cannot be done would reduce the integrity of the analysis.

    The author is well aware of the qualifications of ERG. What additional “independent analysis” is Mr. Fuller suggesting? Must all journalists get independent analysis prior to publishing? Who qualifies as an independent analyst?

    What is Mr. Fuller’s justification for determining what is “bad journalism?” Certainly not the weak suggestive arguments that he presents. Does 25 years of experience studying Texas education qualify him as a journalism critic? Does Mr. Fuller have an independent evaluation of the performance of Dallas ISD that he would like to submit?

    More importantly, the article was intended to give credit to the board, administration, principals, teachers, and all of the support organizations that did the hard work that resulted in improved performance. If you disagree, bring your information to the table.

  • Paul Haeberlen

    Keep in mind that all districts and campuses are evaluated on the percentage of students passing in both the past and current accountability systems. Perhaps someone should notify the TEA about the attendant weaknesses in this approach.

    At the risk of overstating the obvious, the metrics used to evaluate performance should be in line with the stated expectations of the leadership. Districts are judged by the TEA on the percentage of students passing, therefore ERG includes that metric in the performance index.

  • Bill Betzen

    Eric, given the disasterous TEA ratings for DISD made public 8-7-14, do you feel a bit guilty for publicising this year old data just in time for the debate about the extension of Mike Miles contract, and just before the TEA data was made public? DISD knew the TEA data would be negative for DISD. Is that why they rushed these contract decisions? How do you reconcile this “Dallas Miracle” idea with the rest of the data as collected at http://www.dallasisd.us ? Sounds like the “Texas Miracle” of 2000 when they were claiming dropout rates near 0% in Houston. Eric, we can laugh about this until we realize that such lies are hurting our students ultimately. You need more integrity in your reporting.

  • mikedub99

    Bill,
    I’m a teacher in DISD. I took a look at your website you recommended, and it is awful. Your website clearly has an agenda, and it presents a number of poor, misleading statistics. For instance, teacher turnover rate in 2012-13 was 17.8 percent. “Man, that’s a lot!” But, then you look at Region 10 (Dallas area) and the average was 16.4 percent. “Oh, that’s not quite as much as it sounded like.” And don’t forget, during that time, Dallas was paying teachers to retire and working to improve it’s teaching staff. Not to debate that 1.4% isn’t important, but a statistic without some comparison is worthless…

    Whatever your thoughts on ERG, the data has been consistently mediocre (over a decade) when it comes to Dallas ISD results. When you measure anything with the same metric year over year and consistently get the same results (DISD in the 3rd quartile) and have a huge jump (mid 2nd quartile) that is newsworthy. DISD needs to repeat that again this year to show it was not a fluke, but that’s impressive! By any measure, graduation rates are up, drop out rates are down, and the minority achievement gap is slowly improving. If you don’t care for Mr. Miles, you can point out that much of this started under Hinojosa, but don’t belittle the improvements made. It makes you look ignorant and appear to have an agenda.

    Michael Walker

  • mikedub99

    Bill,
    I’m a teacher in DISD. I took a look at your website you recommended, and it is awful. Your website clearly has an agenda, and it presents a number of poor, misleading statistics. For instance, teacher turnover rate in 2012-13 was 17.8 percent. “Man, that’s a lot!” But, then you look at Region 10 (Dallas area) and the average was 16.4 percent. “Oh, that’s not quite as much as it sounded like.” And don’t forget, during that time, Dallas was paying teachers to retire and working to improve it’s teaching staff. Not to debate that 1.4% isn’t important, but a statistic without some comparison is worthless…

    Whatever your thoughts on ERG, the data has been consistently mediocre (over a decade) when it comes to Dallas ISD results. When you measure anything with the same metric year over year and consistently get the same results (DISD in the 3rd quartile) and have a huge jump (mid 2nd quartile) that is newsworthy. DISD needs to repeat that again this year to show it was not a fluke, but that’s impressive! By any measure, graduation rates are up, drop out rates are down, and the minority achievement gap is slowly improving. If you don’t care for Mr. Miles, you can point out that much of this started under Hinojosa, but don’t belittle the improvements made. It makes you look ignorant and appear to have an agenda.

    Michael Walker