Under Kids First and Superintendent Toback- Traditional Hoboken Public School SAT Composite Scores Lower than Expected Based on National Family Income Estimates

CLICK TO ENLARGE

Note: I would like to acknowledge and thank a number of anonymous bloggers and colleagues who have provided useful data, readings, perspectives, url's, and analysis on this topic. Edits have been made to the original post on 3/23 and 3/24 and now include the use of Cohen's d, effect size, links to free lunch (FL) and free and reduced lunch (FRPL) criteria, deleting of county comparisons (the FL% when including any of the charter schools within the district's borders), explanation of some assumptions, an alternative one-tailed t test analysis, and modifications to some of the original text. 

It has become popular to have the public expect less from schools and districts that educate students from families with lower than average socio-economic status (SES). Indeed, there is research and data that support the fact the there is a high correlation between SES and national tests like the SAT. Recently, a report was released that outlined very specifically the correlation of SES and the SAT. And, as one might expect, as family income increases so too does SAT scores (reading, math, writing, total or composite-see diagram). Using these numbers and charts, we can find an "expected" SAT score for each of 10 socio-economic categories. If a district's scores are higher than expected, we can reasonably  assume something good is going on. Students are performing at levels higher than expected by socio-economic status alone. Conversely, if a district's SAT scores are lower than expected by socioeconomic status, we can reasonably assume there are factors working against student success within the school/district etc.

As an example, let us look at the Hoboken School District in Hoboken, New Jersey. According to latest figures, the free lunch percentage (as opposed to "free and reduced percentage") for the Hoboken School District shows Hoboken High has 592 students; 411 qualify for Free Lunch, which is about 69% (about 81% for FRPL). The district FL% is about 64%.  Some researchers like free lunch and have said that "free lunch" is a better indictor of poverty on academic performance than "free and reduced" while others prefer the "free and reduced" percentages. 
For income eligibility guidelines please click here. 
For New Jersey: Click Here. 
For federal income eligibility guidelines for July 1, 2012 - June 30, 2013 please CLICK HERE. 

Click to Enlarge

If we look at the LOWEST income category ($0- $20,000) we notice that we would expect a school district with students from this category to receive a Total SAT score of 1326. If we look at the second lowest income category ($20,000-$40,000) we would expect students from this demographic to receive a Total SAT score of 1402. Hoboken scores a SAT Total score of 1192. The majority of students in the Hoboken School District who take the SAT certainly fall somewhere in the lowest and second lowest categories. Some assumptions/estimations have been made including a national population of about 300,000 (based on the Washington Post article) and a local population of about 50. From published data, the standard deviation for the SAT composite is 319 and was used for all analyses. It is possible that the standard deviation locally differs from 319 but not clear whether that difference would be above or below the national standard deviation for the SAT composite score. 

Finding 1 (red line): Hoboken scores on the composite SAT are lower than would be expected by students from families making between $0-$20,000 a year. Moreover, this difference is VERY STATISTICALLY SIGNIFICANT (not occurring by chance). Effect size = .208 (small/medium)

Finding 1 (blue line): Hoboken scores on the composite SAT are lower than would be expected by students from families making between $20,000-$40,000 a year. Moreover, this difference is EXTREMELY STATISTICALLY SIGNIFICANT (not occurring by chance). Effect Size = .315 (medium)

Note: 

Effect size is a simple way of quantifying the difference between two groups that has many advantages over the use of tests of statistical significance alone (like a t-test). Effect size emphasises the size of the difference rather than confounding this with sample size. 

This analysis tries to bring some objectivity to the discussion of quality of education, especially in the Hoboken Public Schools. While its easy to have articulate, well intentioned parents to speak of the wonderful experiences their children are having in the schools and to hear the good news that often takes place at Board meetings, this analysis attempts to examine objective data in some context.

What does the SAT composite measure? According to the College Board: 

According to the College Board, the SAT measures literacy and writing skills that are needed for academic success in college. They state that the SAT assesses how well the test takers analyze and solve problems—skills they learned in school that they will need in college. The SAT is not an aptitude test. 

What does this mean? In common language it means that students taking the SAT are performing at levels lower than would be expected on the SAT Composite score based on the socio-economic status of their national cohort. 

Kids First campaign literature 2009

One must consider that there is a non trivial number of students taking the SAT's in Hoboken from families making considerably more than $40,000 a year. 

Recently, Hoboken Superintendent Toback has discussed the so called "segregating" impact of charter schools on the school district. Board President Leon Gold has been quoted discussing the "white flight" occurring in the Hoboken School District because of charter schools. Perhaps these two gentlemen should concentrate on providing a better educational experience for the students already attending their schools and be less concerned with statements of segregation, "white flight",  and diminished expectations. -Dr. Petrosino

"The fact is that it is not an attack...It's just a suggestion that things should change." - Hoboken Superintendent of Schools -Dr. Mark Toback 

Please leave a comment if you would like.

-------------------------------------------------------

Unpaired t test results Finding I 

P value and statistical significance:   The two-tailed P value equals 0.0026   By conventional criteria, this difference is considered to be very statistically significant. Confidence interval:   The mean of $0-$20.000 minus HHS equals 136.00   95% confidence interval of this difference: From 47.36 to 224.64 Intermediate values used in calculations:   t = 3.0144   df = 300048   standard error of difference = 45.117 Review of data:

Group	$0-$20.000	HHS
Mean	1326.00	1190.00
SD	319.00	319.00
SEM	0.58	45.11
N	300000	50

Cohen's d = .426
Effect Size (r) = .208

Unpaired t test results Finding II

P value and statistical significance:   The two-tailed P value is less than 0.0001   By conventional criteria, this difference is considered to be extremely statistically significant. Confidence interval:   The mean of $20K-$40K minus HHS equals 212.00   95% confidence interval of this difference: From 123.36 to 300.64 Intermediate values used in calculations:   t = 4.6989   df = 300048   standard error of difference = 45.117 Review your data:

Group	$20K-$40K	HHS
Mean	1402.00	1190.00
SD	319.00	319.00
SEM	0.58	45.11
N	300000	50

Cohen's d = .6645
Effect Size (r) = .315

ALTERNATIVE ANALYSIS 

Another way to analyze the data would be to use a one-sample t-test, where you treat the district as a sample from the larger population, in this case the national SAT Composite average at the $0-$20,000 and $20,000-$40,000 family income categories. Using this approach, one would not want to compare the means as two independent samples via the more typical t-test comparison. In this scenario, it would be fine to talk about the difference in terms of standard deviations, since the SAT is normalized. 

For $0-$20,000 Analysis: 

For the $20,000- $40,000 Analysis: 

Summaries from this alternative analysis do not differ significantly from the original analysis along t-statistic, critical values, one-tailed probability, Cohen's d, or effect size.