Chapter 7: The Synthetic Control Method
Story: Rebuilding the Unseen Baseline in Santa Barbara
In the spring of 2020, as COVID-19 swept across the United States, California's Judicial Council issued an unprecedented order: to reduce jail crowding and viral transmission, the state mandated a zero-cash bail policy for most non-violent offenses. Across California, arrestees who once would have needed to pay bail were now released automatically, free of charge. Santa Barbara County, like all counties, adopted the emergency rule. When the statewide mandate ended in June 2020, Santa Barbara elected to continue zero bail independently.
Residents were worried. Would releasing defendants without bail spur a crime wave? Local officials pointed to anecdotal upticks in thefts and assaults. Yet causal inference requires more than anecdotes. Without a counterfactual — what would crime have looked like without zero bail? — we cannot know if the policy truly changed outcomes.
The pandemic made this question even harder: COVID-19 disrupted normal life, work, policing, and crime rates everywhere. Simply comparing Santa Barbara before and after the reform would confound policy effects with pandemic effects. Nor could we easily compare Santa Barbara to other counties: every county had initially implemented zero bail, and only a handful made different policy choices afterward.
Faced with no clear untreated control group, but many possible partial comparators, we need a method that can construct a custom, data-driven counterfactual. This is where the Synthetic Control Method (SCM) comes to the rescue.
Concept: Building and Interpreting a Synthetic Control
The Synthetic Control Method (SCM) offers a powerful, principled solution to one of the most difficult problems in causal inference: estimating the treatment effect for a single treated unit when no clear untreated twin exists. First developed by Abadie and Gardeazabal (2003) and later formalized by Abadie, Diamond, and Hainmueller (2010), SCM builds a data-driven counterfactual by constructing a weighted combination of untreated units that mirrors the treated unit’s pre-treatment behavior.
Rather than comparing Santa Barbara to a single other city or to an unweighted average of many cities, SCM assigns optimal non-negative weights to control units to create a Synthetic Santa Barbara. The goal is simple but profound: make the synthetic unit’s pre-intervention outcomes match the treated unit as closely as possible. If this matching is successful, we can plausibly attribute post-intervention differences between the treated and synthetic unit to the effect of the treatment itself—in this case, Santa Barbara’s decision to reinstate zero bail after the statewide order ended.
Assumptions of SCM
To properly interpret SCM estimates, several assumptions are critical:
No Interference (SUTVA): The treatment for Santa Barbara does not spill over to the control cities. That is, zero-bail policies in Santa Barbara should not affect crime in other cities.
Convex Hull Assumption: The treated unit lies within the convex hull of the controls. In other words, Santa Barbara’s pre-treatment characteristics can be reproduced by a weighted combination of other cities, without needing extrapolation.
Stable Relationship Over Time: The relationship between predictors and outcomes remains stable. If, absent treatment, Santa Barbara would have continued to evolve like its synthetic control, then SCM yields a valid counterfactual.
Good Pre-Treatment Fit: A close match in the pre-intervention period is critical for credible inference. SCM’s validity relies on achieving a low root mean squared prediction error (RMSPE) before treatment.
Each Plot as a Conceptual Step
The SCM framework is visualized through a sequence of six plots, each corresponding to a critical step in causal inference reasoning:
(a) Outcome Trends: Santa Barbara vs. California Average
The first plot compares Santa Barbara’s actual crime trend to the average across California. It shows the problem: Santa Barbara’s trend is unique. It does not move in lockstep with the state average, underscoring why a simple Difference-in-Differences (DiD) approach would fail. No single city or simple average can serve as an adequate counterfactual. This motivates the need for a customized synthetic control.
Interpretation: California's broad trends are not a suitable control. A better counterfactual must be built.
(b) Synthetic Outcome Trends: Santa Barbara vs. Synthetic Santa Barbara
This plot overlays the real Santa Barbara against its synthetic twin. In the pre-treatment period (before June 2020), the two lines are nearly indistinguishable, demonstrating that the synthetic control closely replicates Santa Barbara’s history. After treatment, however, a visible divergence appears.
Interpretation: A strong pre-treatment fit validates the credibility of the synthetic control. Post-treatment gaps suggest potential causal effects.
(c) Gap Plot: Treated Minus Synthetic
The third plot zooms in on the difference—also called the gap—between Santa Barbara’s real outcomes and the synthetic control's outcomes over time. Ideally, the gap is near zero before treatment (showing good pre-fit) and then diverges post-treatment if the intervention had an effect.
Interpretation: This plot isolates the estimated treatment effect over time, emphasizing changes attributable to the policy.
(d) Placebo Gaps: Treating Control Cities as if They Were Treated
The fourth plot repeats the SCM procedure for every control city, pretending each one had received the treatment. This produces a distribution of placebo gaps. Santa Barbara’s gap is plotted in bold, while placebos are grey.
Interpretation: If Santa Barbara’s gap post-treatment is much larger than the placebo gaps, it suggests that the effect is not due to random chance.
(e) Filtered Placebo Gaps: Good Pre-Fit Cities Only
Not all control units are equally credible placebos. Some may have poor pre-treatment fits. Therefore, the fifth plot filters out placebo units whose pre-treatment RMSPE was worse than twice that of Santa Barbara's synthetic fit. Only "good" matches remain.
Interpretation: Filtering increases validity. Among well-matched units, Santa Barbara’s post-treatment divergence appears even more distinctive.
(f) MSPE Ratio Plot: Statistical Inference
Finally, we compute the Mean Squared Prediction Error (MSPE) ratio—comparing post-treatment fit to pre-treatment fit—for Santa Barbara and all placebo units. A histogram displays these ratios.
Interpretation: If Santa Barbara’s MSPE ratio is unusually large compared to the placebo distribution, it provides evidence that the treatment effect is statistically significant, not due to random noise.
Summary of the Concept Section
Together, these plots build a complete causal argument:
Why SCM is necessary (plot a),
How the counterfactual is constructed (plot b),
Where and how gaps emerge (plot c),
Whether observed effects are unusual (plots d, e, f).
Each stage of visualization corresponds to a deeper conceptual safeguard: ensuring credible matching, verifying significant treatment effects, and transparently displaying assumptions.
Moreover, the Santa Barbara application used a full SCM script (included with the book’s companion materials), making the analysis fully reproducible: from setting up the donor pool (only cities that did not reinstate zero bail) to generating each figure and computing formal significance tests.
Predictors Balance Table
We ensure that treated and synthetic units match closely on key predictors
Property per 100k (Jan 2017)
5.665
5.621
5.255
Property per 100k (Dec 2017)
5.412
5.507
5.175
Property per 100k (Dec 2018)
5.399
5.445
5.136
Property per 100k (Dec 2019)
5.337
5.458
5.043
Median Home Value
13.820
13.614
12.609
City Weights Table
Only a few cities contribute to the synthetic control, reinforcing targeted matching:
Camarillo
0.256
Ojai
0.003
San Francisco
0.531
Thousand Oaks
0.208
Final Results Table
0.140544
0.113208
0.223891
53
Insight: Seeing Synthetic Control as Rebuilding the Past
Imagine trying to reconstruct a shattered mirror. No single shard captures the whole reflection, but with care, patience, and the right adhesives, you can assemble a new, cohesive image. Some shards fit perfectly; others are discarded. The Synthetic Control Method operates similarly: piecing together fragments of other units to rebuild the untreated trajectory of the treated unit.
In Santa Barbara’s case, no single city could have served as an adequate control. San Francisco resembled its urban profile; Thousand Oaks its suburban crime dynamics; Ojai its size and demographics. SCM assigns weights selectively, taking a fraction of each "shard" to recreate a synthetic version of Santa Barbara before the zero-bail policy. Because the pre-treatment mirror is nearly perfect, any crack that appears after June 2020—the crime rate gap—can be plausibly attributed to the new policy.
This insight also reveals why SCM is superior to traditional methods like DiD in certain contexts. DiD would assume that Santa Barbara and an average of other counties evolved similarly before and after treatment. SCM tests this assumption directly, constructing a synthetic that actually matches pre-trends, rather than hoping for parallelism. It makes visible what DiD leaves implicit.
Moreover, SCM clarifies how the control group is formed—openly showing that most cities contribute zero weight, while a few selected cities form the synthetic. It respects the real-world complexity of heterogeneity, not forcing all comparisons into a single mold.
Finally, the SCM framework reminds us of a deeper philosophical point in causal inference: we can never observe counterfactuals directly. But we can reconstruct them rigorously, transparently, and credibly by blending observed realities. Synthetic Control is, in this sense, a scientific method of imagination—rooted not in speculation but in optimization.
Practice: Reconstructing the Counterfactual for Santa Barbara
In this hands-on section, you will apply the Synthetic Control Method (SCM) to study the effects of Santa Barbara's adoption of the zero-bail policy during the COVID-19 pandemic. Using real crime data, you will build a synthetic Santa Barbara, analyze the effect of zero bail, and learn the step-by-step logic of SCM in practice.
Step 0: Setup
Exercise 1: Why do we create "plots" and "tables" folders before starting?
Step 1: Parameters to Set
Exercise 2: If you change outcome_var to "ForRape_per_100k", what might you expect differently in the results?
Step 2: Load and Prepare Data
Exercise 3: Why do we create numeric city_id and time_id?
Step 3: Define Treated and Control Units
Exercise 4: Why must Santa Barbara be excluded from the donor pool?
Step 4: Create Lagged Predictors
Exercise 5: Add another lagged predictor (e.g., Property_Jan2018) to improve the pre-fit.
Step 5: Prepare for SCM
Exercise 6: Why is matching on lagged outcomes so important in SCM?
Step 6: Run SCM
Exercise 7: What if the optimizer cannot find a good match?
Step 7: Visualizing Treated vs. Synthetic Control
Step 8: Gap Plot (Treated - Synthetic)
Step 9: Placebo Gaps for Control Cities
Step 10: Filtered Placebo Gaps (Good Fit Cities Only)
Step 11: ATT, MSPE Ratio, MSPE Histogram
Reflection Questions
How does SCM construct a better counterfactual than simple averages?
What do the ATT and pseudo-p value tell you about Santa Barbara’s zero-bail effect?
How do placebo tests strengthen the causal claim?
Conclusion
Congratulations! You have completed the SCM workbook for Santa Barbara. You have:
Built a synthetic control
Visualized outcomes and gaps
Conducted placebo tests
Interpreted causal effects
You are now ready to extend these skills to other contexts, like the upcoming San Jose Gun Insurance study!
✏️ TASK: Applying Synthetic Control – Gun Insurance in San Jose
In this hands-on exercise, you will apply the Synthetic Control Method (SCM) to a real-world policy intervention: the implementation of San Jose’s Ordinance No. 30716, which required firearm owners to carry liability insurance and pay an annual harm-reduction fee. Your goal is to estimate the effect of this ordinance on firearm assaults per 100,000 population in San Jose, relative to a synthetic control made from similar cities that did not adopt the law.
Dataset Provided:
chapter1log.csv
This dataset contains monthly crime, demographic, and police staffing data for multiple cities in California over the period 2017–2024.
📖 Instructions
You will build your own SCM analysis by completing the following steps.
1. Define the Study Units:
Treated unit: San Jose
Control units: All other cities in the dataset except Watsonville. (Watsonville should be excluded due to missing data issues and prior interventions.)
2. Choose the Outcome Variable:
Outcome to model: FASSact_per_100k (Firearm Assault Serious Injury or Death per 100,000 population)
3. Specify Predictors:
Use heavy lagged outcomes: that is, include past values of FASSact_per_100k at several important months as predictors.
Example: January 2017, December 2017, December 2018, December 2019, December 2020, December 2021, and December 2022.
Add time-varying covariates:
Number of Police Officers (per capita or per 100k)
Median Household Income
Thus, your predictor set will combine lagged firearm assault rates and these two key socioeconomic indicators.
4. Define the Timing:
Treatment implementation date: January 2023.
Pre-treatment period: January 2017 – December 2022 (first 72 months).
Post-treatment period: January 2023 onward.
5. Prepare the Data:
Create a time_id variable indexing each month sequentially.
Create city_id variables.
Clean any missing data carefully (e.g., ensure predictors and outcomes are complete during the pre-period).
6. Set Up the dataprep Object:
Use the treated unit (San Jose) and donor pool (control cities).
Use the selected predictors.
Match on the pre-treatment period only.
7. Run the SCM:
Estimate the synthetic San Jose using the donor pool.
Extract the path.plot (outcome trends over time).
Extract the gaps plot (difference between San Jose and synthetic over time).
8. Perform Placebo Tests:
Conduct a placebo study: run SCM on each control city pretending it was treated.
Compare the size of the treatment effect for San Jose to the distribution of placebo effects.
9. Check Model Fit:
Report the pre-treatment Root Mean Squared Prediction Error (RMSPE).
Highlight whether the synthetic San Jose closely tracked the real San Jose prior to 2023.
10. Interpret the Findings:
Based on the gap plot and placebo distribution, conclude whether the ordinance likely had a causal impact on firearm assaults.
Was the gap large and sustained after treatment?
Was San Jose’s gap larger than most placebo gaps?
✏️ Hints and Guidance
A good SCM depends critically on the pre-treatment fit — if synthetic San Jose tracks real San Jose closely before 2023, you can be more confident in causal claims.
Police staffing and median income are important controls because they influence crime independently of gun policies.
You might want to standardize predictors (e.g., median income) so that large-scale variables don't dominate smaller-scale ones.
Filtering placebos: You may want to restrict attention to "good" placebo units whose pre-treatment RMSPEs are similar to San Jose's.
Visual storytelling: Use graphs to compare actual vs synthetic trends, plot gaps, and plot placebo distributions.
● Use the example SCM code from the Santa Barbara case to guide your code!
📂 Deliverables
At the end of this task, you should produce:
A plot showing firearm assault rates for San Jose vs. synthetic control.
A gap plot showing the difference over time.
A placebo plot showing San Jose’s effect compared to placebo effects.
A summary table showing ATT, RMSPE, pseudo-p value, and sample size.
A short written interpretation (2–3 paragraphs) summarizing the causal effect you find.
📌 Final Tip
Remember: Causal inference is detective work. Your SCM will only be credible if the synthetic control truly mimics the treated unit's past. Check your assumptions carefully, diagnose your fits visually, and think critically about any observed gaps.
Good luck, detective! 🔍
7b – Staggered Adoption and Partially-Pooled Synthetic Control
You may be wondering: what if I have more than one treated unit, and they are treated at different times? That situation is called staggered adoption.
The Synthetic Control Method (SCM) script and approach we discussed earlier works beautifully when only one unit is treated. However, it doesn’t automatically extend to cases where multiple units receive the treatment at different points in time. Relax—there are advanced methods for that.
In 2021, Ben-Michael, Feller, and Rothstein introduced the Partially-Pooled Synthetic Control Method specifically for this setting (link to paper here).
If you want the TL;DR version, here it is: Think of SCM as a spectrum.
On the extreme left, we have unit-specific SCM. This focuses on minimizing the imbalance between each treated unit and its corresponding synthetic control. The upside: each treated unit has an excellent pre-treatment fit. The downside: the overall, or pooled, fit across all treated units could be poor.
On the extreme right, we have pooled SCM. This focuses on minimizing the imbalance between the average treated unit and the average synthetic control. The upside: you achieve good global (pooled) fit. The downside: some individual treated units might be poorly matched.
The problem with staggered adoption is that treatment occurs at different times for different units, meaning:
Different pre- and post-treatment periods for each unit
Different donor pools available at each point
These differences make applying traditional SCM unit-by-unit quite messy and inconsistent. Ben-Michael and co-authors solved this by proposing Partially-Pooled SCM, which blends the unit-specific and pooled fits into a single optimization problem.
The key idea is introducing a hyperparameter, called ν\nuν ("nu"), that regulates the trade-off between:
Unit-specific fit (when ν=0\nu = 0ν=0)
Pooled (global) fit (when ν=1\nu = 1ν=1)
Thus, by adjusting ν\nuν somewhere between 0 and 1, you can balance individual and pooled goodness of fit. In practice, you usually pick ν\nuν through cross-validation or based on minimizing pre-treatment mean squared error.
At the end of the day, this hybrid approach gives you a better overall estimate than either extreme (purely unit-specific or purely pooled SCM) could by itself.
Moreover, Partially-Pooled SCM formalizes additional assumptions:
The counterfactual outcomes are assumed to be generated by an autoregressive and linear factor model, which provides a structure for bounding estimation errors.
Stability conditions ensure the estimator behaves well as the number of treated and control units grows.
If you're interested, I also have R code and a presentation.
Last updated