Wednesday, March 19, 2014

Under Kids First and Superintendent Toback- Traditional Hoboken Public School SAT Composite Scores Lower than Expected Based on National Family Income Estimates

CLICK TO ENLARGE
Note: I would like to acknowledge and thank a number of anonymous bloggers and colleagues who have provided useful data, readings, perspectives, url's, and analysis on this topic. Edits have been made to the original post on 3/23 and 3/24 and now include the use of Cohen's d, effect size, links to free lunch (FL) and free and reduced lunch (FRPL) criteria, deleting of county comparisons (the FL% when including any of the charter schools within the district's borders), explanation of some assumptions, an alternative one-tailed t test analysis, and modifications to some of the original text. 

It has become popular to have the public expect less from schools and districts that educate students from families with lower than average socio-economic status (SES). Indeed, there is research and data that support the fact the there is a high correlation between SES and national tests like the SAT. Recently, a report was released that outlined very specifically the correlation of SES and the SAT. And, as one might expect, as family income increases so too does SAT scores (reading, math, writing, total or composite-see diagram). Using these numbers and charts, we can find an "expected" SAT score for each of 10 socio-economic categories. If a district's scores are higher than expected, we can reasonably  assume something good is going on. Students are performing at levels higher than expected by socio-economic status alone. Conversely, if a district's SAT scores are lower than expected by socioeconomic status, we can reasonably assume there are factors working against student success within the school/district etc.

As an example, let us look at the Hoboken School District in Hoboken, New Jersey. According to latest figures, the free lunch percentage (as opposed to "free and reduced percentage") for the Hoboken School District shows Hoboken High has 592 students; 411 qualify for Free Lunch, which is about 69% (about 81% for FRPL). The district FL% is about 64%.  Some researchers like free lunch and have said that "free lunch" is a better indictor of poverty on academic performance than "free and reduced" while others prefer the "free and reduced" percentages. 
For income eligibility guidelines please click here
For New Jersey: Click Here
For federal income eligibility guidelines for July 1, 2012 - June 30, 2013 please CLICK HERE
Click to Enlarge
If we look at the LOWEST income category ($0- $20,000) we notice that we would expect a school district with students from this category to receive a Total SAT score of 1326. If we look at the second lowest income category ($20,000-$40,000) we would expect students from this demographic to receive a Total SAT score of 1402. Hoboken scores a SAT Total score of 1192. The majority of students in the Hoboken School District who take the SAT certainly fall somewhere in the lowest and second lowest categories. Some assumptions/estimations have been made including a national population of about 300,000 (based on the Washington Post article) and a local population of about 50. From published data, the standard deviation for the SAT composite is 319 and was used for all analyses. It is possible that the standard deviation locally differs from 319 but not clear whether that difference would be above or below the national standard deviation for the SAT composite score. 

Finding 1 (red line): Hoboken scores on the composite SAT are lower than would be expected by students from families making between $0-$20,000 a year. Moreover, this difference is VERY STATISTICALLY SIGNIFICANT (not occurring by chance). Effect size = .208 (small/medium)

Finding 1 (blue line): Hoboken scores on the composite SAT are lower than would be expected by students from families making between $20,000-$40,000 a year. Moreover, this difference is EXTREMELY STATISTICALLY SIGNIFICANT (not occurring by chance). Effect Size = .315 (medium)

Note: 
Effect size is a simple way of quantifying the difference between two groups that has many advantages over the use of tests of statistical significance alone (like a t-test). Effect size emphasises the size of the difference rather than confounding this with sample size. 


This analysis tries to bring some objectivity to the discussion of quality of education, especially in the Hoboken Public Schools. While its easy to have articulate, well intentioned parents to speak of the wonderful experiences their children are having in the schools and to hear the good news that often takes place at Board meetings, this analysis attempts to examine objective data in some context.

What does the SAT composite measure? According to the College Board: 
According to the College Board, the SAT measures literacy and writing skills that are needed for academic success in college. They state that the SAT assesses how well the test takers analyze and solve problems—skills they learned in school that they will need in college. The SAT is not an aptitude test. 
What does this mean? In common language it means that students taking the SAT are performing at levels lower than would be expected on the SAT Composite score based on the socio-economic status of their national cohort. 
Kids First campaign literature 2009

One must consider that there is a non trivial number of students taking the SAT's in Hoboken from families making considerably more than $40,000 a year. 

Recently, Hoboken Superintendent Toback has discussed the so called "segregating" impact of charter schools on the school district. Board President Leon Gold has been quoted discussing the "white flight" occurring in the Hoboken School District because of charter schools. Perhaps these two gentlemen should concentrate on providing a better educational experience for the students already attending their schools and be less concerned with statements of segregation, "white flight",  and diminished expectations. -Dr. Petrosino


"The fact is that it is not an attack...It's just a suggestion that things should change." - Hoboken Superintendent of Schools -Dr. Mark Toback 

Please leave a comment if you would like.

-------------------------------------------------------

Unpaired t test results Finding I 

P value and statistical significance:
  The two-tailed P value equals 0.0026
  By conventional criteria, this difference is considered to be very statistically significant.

Confidence interval:
  The mean of $0-$20.000 minus HHS equals 136.00
  95% confidence interval of this difference: From 47.36 to 224.64

Intermediate values used in calculations:
  t = 3.0144
  df = 300048
  standard error of difference = 45.117

Review of data:
  Group   $0-$20.000     HHS  
Mean 1326.00 1190.00
SD 319.00 319.00
SEM 0.58 45.11
N 300000     50    

Cohen's d = .426
Effect Size (r) = .208

Unpaired t test results Finding II


P value and statistical significance:
  The two-tailed P value is less than 0.0001
  By conventional criteria, this difference is considered to be extremely statistically significant.

Confidence interval:
  The mean of $20K-$40K minus HHS equals 212.00
  95% confidence interval of this difference: From 123.36 to 300.64

Intermediate values used in calculations:
  t = 4.6989
  df = 300048
  standard error of difference = 45.117

Review your data:
  Group  $20K-$40K    HHS  
Mean1402.001190.00
SD319.00319.00
SEM0.5845.11
N300000    50   


Cohen's d = .6645
Effect Size (r) = .315

ALTERNATIVE ANALYSIS 

Another way to analyze the data would be to use a one-sample t-test, where you treat the district as a sample from the larger population, in this case the national SAT Composite average at the $0-$20,000 and $20,000-$40,000 family income categories. Using this approach, one would not want to compare the means as two independent samples via the more typical t-test comparison. In this scenario, it would be fine to talk about the difference in terms of standard deviations, since the SAT is normalized

For $0-$20,000 Analysis: 

For the $20,000- $40,000 Analysis: 

Summaries from this alternative analysis do not differ significantly from the original analysis along t-statistic, critical values, one-tailed probability, Cohen's d, or effect size. 


5 comments:

Dr. Anthony Petrosino said...

Please leave a comment-

bob jones said...



-48% is the total free and reduced lunch population and includes both Charter and Hoboken Board of Education student populations.

-SAT test data was based only on Hoboken High School SAT scores, not the total population you presented to the reader. HHS has an 80% free and reduced lunch rate.

-The subset used in your t-test was not representative of the total population (nor reliable for such usage).

-Your data used ONLY the FREE LUNCH rate when providing your t-test data outcome.Not the 48% nor the 80% free and reduced lunch rates.

-None of your t test outcomes are relevant to correlate free and reduced lunch rates and academic achievement for Hoboken Board of Education students.

-SAT tests are aptitude (ability) tests not academic achievement(learning) tests. Aptitude tests test natural ability they do not test learned academics.


Your decision to utilize SAT test data outcome only indicates the average innate ability of students who were enrolled in the Hoboken Board of Education and chose to take the SAT test that particular year and nothing more.



PhillipMarlowe said...

Listen, I enjoy a good (?) t-test as the next guy, but let's be clear: you can't compare the average outcomes for a single high school against the average outcomes for an entire national income group. It's false and it's wrong and it's irrelevant and it's wrong and just no and wrong and don't do it, please. We just don't know how Hoboken High's -- or any school's -- income is distributed through its population of kids who take the SAT (which is a voluntary test). If you want to see how a particular high school does against other high schools -- when controlling for student population socio-economic status -- then that's what you need to look at. Compare a high school against others by some metric that can control for all of their students' poverty status -- that makes sense. But saying a high school doesn't do as well as a particular family income group of students nationwide is like saying your lawnmower sucks because it doesn't go as fast as your bicycle. Why would you ever think to compare the two? - See more at: http://jerseyjazzman.blogspot.com/#sthash.Q5BxZ00w.dpuf

Dr. Anthony Petrosino said...

bob jones- regardless of whether you use FREE AND REDUCED LUNCH or just FREE LUNCH, neither percentage is calculated into the t-test. I simply used the percentages to characterize the general population in terms of family income.

To be clear, I do not use the t test outcomes to correlate free and reduced lunch rates and academic achievement for Hoboken Board of Education students. I use them for the SAT composite scores. Any assumption of the SAT composite score to academic achievement is an interesting conversation but not what the t test was trying to accomplish.

Your claim that the SAT is an aptitude test needs support. According to the College Board, the SAT measures literacy and writing skills that are needed for academic success in college. They state that the SAT assesses how well the test takers analyze and solve problems—skills they learned in school that they will need in college. That is NOT the definition of an aptitude test.

Dr. Anthony Petrosino said...

PhillipMarlowe- I have included Effect size and Cohen d for some additional information. If a school has a FREE LUNCH % of 47% and a FREE AND REDUCED LUNCH % of > 75%, we may not know the exact specifics of the population distribution but we can make some reasonable assumptions.

If more and specific data was available, I have no doubt a more detailed analysis could be conducted.

I believe comparing a high school and a national income group along the same measure (SAT composite) using a t-test, while not perfect, is reasonable. I understand you may disagree.

The chart comparing Hoboken to the sample of NJ high schools is appreciated and welcomed.