What's the Deal with BLBC?

• BLBC is a hotly contested policy and has been the source of much political conflict for several decades
• Researchers have been unable to evaluate BLBC programs to a rigorous degree
• Several studies evaluating Proposition 227 in California which show mixed results
• Parrish et al. (2006) find no effect but student selection is rampant
• Gordon and Hoxby (2002) slight positive effect in early grades for BLBC on academic achievement
• Jepsen (2010) looks at California English Development Language Test (CELDT) finds BLBC reduced English language proficiency by 0.3 standard deviations in early grades
• Controls for prior achievement and student selection; does not look at academic performance in other subjects
• Mixed results in other subjects looking at large national survey data (Federman 2000; Cheng 1996; Lopez 2003)
• Greene 1998 meta analysis finds small positive effect in reading

Why Look at BLBC in Wisconsin?

• BLBC is mandated in Wisconsin with a cutoff threshold that makes the policy ideal to evaluate
• Some students receive BLBC and some do not merely by being on one side of the cutoff or the other, potentially within the same district
• Districts with a cluster of students at the right cutoff must provide BLBC services

• BLBC programs are highly contentious and have been the source of several academic evaluations.
• States like California have mandated strong impediments to offering BLBC programs, while states like Wisconsin mandate them.
• Most BLBC evaluations focus on English acquisition, not at other academic outcomes.

Research Question

• What we aren't answering:
• Do BLBC programs work?
• Should school districts offer BLBC programs?
• Which instructional method provides the best outcomes for ELL students?
• What are the best pathways to educating ELL students?
• What we are answering:
• Does a state policy mandating BLBC programs for clusters of students in the same language group and grade range improve student outcomes on standardized tests--specifically ACCESS, WKCE Math, and WKCE reading?
• This is called "intent to treat" or an ITT estimate, which is different than the "treatment-on-the-treatment" estimate, TOT in the rest of the literature
• Does mandating a BLBC program increase student outcomes for students exposed to such a program? Should policymakers explore a policy like that in Wisconsin to improve educational outcomes of ELL students?

Data Used

• Using Wisconsin DPI data we could leverage the statewide nature of this policy to evaluate the overall impact of offering BLBC on student outcomes

• To analyze this data we used state assessment results on the WKCE and the ACCESS exam for ELL
• We used administrative records from the BLBC programs themselves required under state statute
• Student level data, statewide, is the key.

Variables Included

• Economic disadvantage
• Age
• Prior year test scores
• Race
• Gender
• Years in ELL
• SPED
• BLBC opt out
• Some program level controls

BLBC in Wisconsin

Wisconsin Districts

• Only about 13% of districts offer one or more BLBC programs
• 2/3 of students are eligible for BLBC program in their district
• 2/3 of those students speak Spanish, 30% speak Hmong, smaller numbers speak Mandarin, Korean, Lao, Khmer, Tibetan, Somali, Russian, Arabic, or Albanian

Example

• Consider two students A and B
• Identical on all observable characteristics including race, gender, age, starting score on the test, etc.

• Student A is in an ESL program and Student B is in a BLBC program

• Student A appears to be performing better in subsequent grades

What is wrong with this?

Caveats and Concerns

• They may be in different districts with different instructional practices, different teacher quality, and different levels of resources
• Because BLBC programs are contentious, families may choose schools and districts based on availability / lack of programs
• Due to student mobility districts' status for requiring BLBC instruction is not perfect
• Families have possible control over program participation, districts have control around cutoff of student count
• Need a technique to control for simultaneous selection bias

plot of chunk growthplot2

• Despite their similarity in the data Student A and Student B are not the same

Two Possible Techniques

• Value-added. Try to control for observable time-variant factors like prior achievement to model student growth.
• Fixed effects. Control for all unobservable time-invariant student attributes (like ability) and observable time-variant factors (like FRL status)
• These two models are shown to bracket the true causal effect (Angrist and Pischke 2009)
• They have different tradeoffs to obtain their estimates
• All models use district fixed effects to avoid comparing students across school districts

Value Added

• Value added uses the score of student X in year t-1 to predict student X's score in year t

• Value added here is combined with district fixed effects so students are compared to students within their district

• Comparing how a student performed this year compared to other students in the district with similar performance last year in BLBC and non-BLBC settings

Fixed effects

• What if we take selection out and look only at changes within the same student?

• With a large state dataset we might be able to find enough students to do this!

• Compares students who were eligible to receive BLBC in one year but not another
• The 70,000 unique students in the dataset is reduced to 4,000 which raises questions of external validity

• Aren't students who switch between BLBC and not-BLBC eligible different from students who are stable on these measures?

What we do?

• We fit statistical models to all the students in Wisconsin modeling their future test score as best as possible through each strategy described above
• We also include an indicator of whether the student was eligible for BLBC instruction or not
• We compare to see if the average student receiving BLBC did better than the average student receiving other ESL services, all else equal
• We report the range of uncertainty around this difference between student groups and compare them to see if any meaningful differences emerge
• Due to our large sample size we expect our results to be biased in favor of finding statistically significant results, so we also examine the magnitude of findings to see if they are substantively significant; i.e. is the difference big enough to matter in the lives of students?

Results

• Results are presented in effect sizes, or standard deviation units of change in test scores.

• 0.1 is small, 0.2 to 0.4 is reasonable and is about a year of education in most cases. Bigger than 0.4 is huge.

• The bars represent the 95% confidence interval around the estimate. The VAM model is consistently statistically significant, not overlapping 0, and negative.
• The length of the bars represent the uncertainty about the estimate.
• But the mean effect size is quite small, less than 0.1 standard deviations in most cases.

• This represents a year-to-year change in a student's score between BLBC and non-BLBC instruction.

Results (II)

• Here we directly compare the two statistical models that are known to ``bracket'' the true causal effect

• On all three outcomes--English proficiency, math, and reading--the true causal effect overlaps 0

• We can rule out a positive effect of any meaningful size, and a negative effect of any meaningful size looks quite unlikely.

Results (III)

• Here we look at the variation in the effect across grades. In the previous literature effects were shown to be larger in earlier grades

• Again, the SFE and VAM models are bracketing the true causal effect. There is little meaningful variation across grades though the uncertainty around the true effect on English proficiency shrinks

Results (IV)

• Language is different. Wisconsin has a large sample of both Hmong and Spanish speakers and they have different results when analyzed separately

• BLBC has no effect for Spanish speakers on math, but a large negative effect for Hmong speakers
• BLBC may be slightly negative for reading among SPanish speakers, but not for Hmong speakers
• BLBC may be slightly positive for English proficiency for Hmong speakers and not for Spanish speakers

• More precision is needed

Conclusions and Next Steps

Conclusions

• In Wisconsin BLBC does not have the negative effects found in recent research on California, though a slight negative effect may exist in some cases
• Unlike other states where BLBC has been studied, Wisconsin has substantially different results between language groups on mathematics and possibly on English proficiency achievement
• There is still a lot of imprecision in the estimates used here and more precision would be helpful, but effects are not substantively large in terms of relative student performance, even in the upper and lower bounds

Next Steps

• Get more data over more years and use a more precise estimation technique to reduce uncertainty about effects
• Explore the variation across BLBC programs in addition to the mean effect
• Learn more about the non-cognitive non-academic outcomes for BLBC in order to understand the costs and benefits of BLBC programs more fully
• Estimate a "treatement-on-the-treated"" parameter to more directly compare to prior research
• Survey teachers and merge teacher/program practice data with student outcomes to begin exploring the effective components of BLBC and ESL programs

Questions

Contact Jared Knowles