Impact Evaluation in Practice

Session Report
Vaishali Singh

The five-day online training course, “Impact Evaluation in Practice,” organized by IMPRI (Impact and Policy Research Institute), kicked off with an enlightening session led by Mr. Rakesh Pandey. As a doctoral candidate at the Pardee RAND Graduate School and an assistant policy researcher at the RAND Corporation, USA, Mr. Pandey guided participants through Basic Statistics & Econometrics in Practice and Introduction to Impact Evaluation Methods. The course, conducted from December 18 to December 22, 2023, aimed to empower researchers, policymakers, and professionals in refining their skills for effective impact evaluation.

Day 1: Basic Statistics & Econometrics in Practice and Introduction to Impact Evaluation Methods

The day began with a methodological journey, immersing participants in theoretical intricacies and practical exercises that form the bedrock of impact evaluation. Ordinary Least Squares (OLS) took center stage, with participants gaining profound insights into its structure and theoretical underpinnings. The session underscored the importance of OLS as a powerful method for estimating relationships between variables, particularly in establishing causal linkages.

Mr. Pandey navigated through OLS basics, unraveling its components, assumptions, and nuances. The relationship between standard error and sample size was explored, emphasizing the importance of the exogeneity of treatment, particularly in the context of Randomized Control Trials (RCT).

The session seamlessly transitioned from theory to practical application, with participants actively engaging in hands-on examples using STATA. This practical immersion served as a capstone, solidifying the theoretical understanding gained throughout the day. In essence, day one laid a robust foundation for the course, providing participants with not only a comprehensive understanding of statistical techniques but also practical skills vital for impactful and data-driven decision-making in the realm of development outcomes.

Day 2: Randomized Control Trials (RCTs)

The second day continued the enriching learning experience with a focus on Randomized Control Trials (RCTs). Ms. Vasanthi Subramonia Pillai and Mr. Ankit Agarwal conducted insightful sessions, delving into the fundamental nature of RCTs as prospective, comparative, and quantitative studies conducted under controlled conditions.

Ms. Pillai provided an in-depth exploration of RCTs, emphasizing their prospective orientation, quantitative methodology, and meticulous random allocation of interventions. The components of RCTs, including treatment and control groups, were dissected, along with various randomization approaches such as stratified, block, cluster randomized trials, factorial randomization, and adaptive trials.

Blinding in RCTs emerged as a critical aspect, and practical applications in RCT design were emphasized. Addressing challenges such as attrition, the analytical framework, including intention-to-treat analysis, and data management were highlighted. In conclusion, the second day significantly contributed to participants’ understanding of RCTs, equipping them with valuable insights and skills for impactful research in impact evaluation.

Day 3: Difference-in-Difference (DiD) and Panel Data Method

Mr. Rakesh Pandey led the third day’s session, providing a comprehensive overview of the Difference-in-Difference (DiD) and Panel Data Method. The significance of Panel Data in retrospectively evaluating policies and addressing challenges in evaluating non-randomly assigned treatments was emphasized.

The Fixed Effects Model and Panel Data Regressions were discussed, with the Rwandan Genocide case serving as an illustrative example. Mr. Pandey explained the distinction between time series and panel data, highlighting the utility of panel data in analyzing trends and controlling for unobserved variables.

Omitted Variable Bias and the use of panel data in controlling for non-time-varying variables were elucidated. Fixed Effects were presented as a tool to control for time-invariant unobservable factors, and methods to estimate a regression with individual intercepts were covered.

The session concluded with insights into graph analysis and real-life examples, providing participants with a deeper understanding of Difference-in-Difference and Panel Data Methodology.

Day 4: Introduction to Impact Evaluation Methods

On the fourth day, Mr. Rakesh Pandey conducted a pivotal session on “Introduction to Impact Evaluation Methods.” The session unfolded as part of a two-month immersive online hands-on certificate training course, aiming to equip participants with the knowledge and tools required for rigorous impact evaluations.

The session commenced with a deep dive into the Difference-In-Difference (DiD) method, a widely used approach in impact evaluation studies. Mr. Pandey highlighted the method’s significance and potential pitfalls, emphasizing its prevalence and capacity to yield insightful results or, conversely, misleading outcomes.

Static and dynamic treatment designs were meticulously explored within the realm of impact evaluation, using seat belt laws as a practical illustration. The static approach, rooted in datasets of state seat belt laws, was contrasted with the dynamic treatment design that considered changes over time. Mr. Pandey illustrated potential biases in past studies analyzing dynamic treatments before 2018.

The session further distinguished between the Difference-In-Difference method and Randomized Control Trials (RCT). Using the education department of Bihar as an example, Mr. Pandey elucidated the limitations of RCT when treatments are not randomly assigned, underscoring the importance of nuanced methodologies.

The mathematical core of Difference-In-Difference estimation was unravelled, with Mr. Pandey explaining the key elements and delving into the intricacies of comparing treatment and control groups over time. The formula was elucidated, emphasizing the interpretation of results with consideration of baseline and endline data points.

The foundations and assumptions of the Difference-In-Difference method were explored, emphasizing the need for parallel trends, pre-intervention trend data, and at least three observations over time. The session underscored the importance of observing trends before interventions and the necessity for multiple observations per period.

Mr. Pandey engaged with the audience, highlighting five key aspects that make the Difference-In-Difference method appealing to researchers. The method’s focus on trends rather than levels, its similarity to Fixed Effects (FE) as a within estimator, and its ability to estimate differences between treatment and control groups while mitigating bias from time-invariant differences were emphasized.

The session reached its zenith as Mr. Rakesh Pandey applied the Difference-In-Difference method to real-world scenarios, dissecting graphs and analyzing past data. The comprehensive discussion concluded with a robust Q&A session, providing participants with a solid foundation for their exploration into impact evaluation methodologies and practical applications. The session was hailed as an enriching learning experience, enhancing participants’ understanding and clarity on the nuances of impact evaluation methods.

Day 5: Regression Discontinuity Design (RDD)

On the fifth day of the Generation Alpha Data Centre’s immersive online training course, Mr. Rakesh Pandey delivered a comprehensive session on the intricacies of Regression Discontinuity Design (RDD). The day commenced with an engaging exploration of research papers, with a notable example being the study of the Persistent Effects of Mita Mining in Latin America. Melissa Del’s economic paper, delving into Mita’s economic system from 1500-1600 AD, served as a compelling illustration of employing RDD for studying historical phenomena.

Mr. Pandey skillfully navigated through the application of RDD, unraveling its core components, including the running variable, cutoff, and bandwidth. The session provided profound insights into the significance of accounting for how the running variable influences outcomes and the necessity of focusing on observations around the cutoff point. Practical examples elucidating cut-offs in geographic, political, and age-related contexts were discussed, elucidating the versatility of RDD across diverse scenarios.

The session offered a nuanced understanding of setting up regression discontinuity, emphasizing the importance of selecting appropriate methods for predicting outcomes and choosing the bandwidth. Mr. Pandey further addressed the concept of fuzzy regression continuity, shedding light on scenarios where the probability of treatment undergoes gradual changes instead of a sharp jump.

The key takeaways emphasized the crucial assumption of a smooth outcome at the cutoff and the nuanced estimation of the effect attributed to this critical juncture. Mr. Pandey presented an empirical model, providing additional clarity on the methodology and reinforcing the concept of nearly random assignment on either side of the cutoff in RDD.

The day concluded with a vibrant discussion, where students had their queries addressed, adding an interactive dimension to the learning experience. Overall, Mr. Rakesh Pandey’s session on Regression Discontinuity Design on Day 5 served as a pivotal juncture in the training program, offering participants a profound understanding of the methodology’s application through historical and real-world examples.

Read more event reports of IMPRI here:

Why Young Women are the Key to Public Policy’s Future



    IMPRI, a startup research think tank, is a platform for pro-active, independent, non-partisan and policy-based research. It contributes to debates and deliberations for action-based solutions to a host of strategic issues. IMPRI is committed to democracy, mobilization and community building.

    View all posts
  • vaishali singh