Assessing and Improving the External Validity of Experiments in Software Engineering

11 Jun 2026

Most scientific manuscripts describing empirical studies in software engineering (SE) - particularly those reporting controlled experiments - almost ritualistically acknowledge threats to external validity in their discussion section. Like a preemptive defense mechanism against an overly critical reviewer 2, authors backpedal on their claims with excuses like “Because our sample of subjects is limited, we cannot generalize the results.” Acknowleding this limitation is important, but not even striving for some level of generalizability runs the risk of producing isolated results with negligible usability and relevance to practice. So when is the external validity of a result actually threatened? And is acknowledging all that we can do about it?

Understanding Threats to External Validity

Many studies in SE aim to quantify the effect of an independent variable x on a dependent variable y (called the average treatment effect), e.g., the effect of using LLM-support (x) on the productivity of developers (y). Since it is impossible to involve all software engineers or artifacts in any empirical study, we need to draw a sample to involve in a study. External validity is the ability to transport the results from this experimental sample to the much larger target population, i.e., the group of people, artifacts, etc. that we want to reason about.

External validity is “our ability to generalize the results of our experiment to industrial practice” (Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2012). Experimentation in software engineering (Vol. 236, p. P3). Berlin: Springer.)

So what actually threatens external validity? There is a surprisingly straight-forward answer to this question. The external validity of the results from an empirical study are threatened when two conditions hold:

1. the average treatment effect of x on y is moderated by a third covariate z, and
2. this covariate’s distribution differs in the experimental sample from the target population.

Perfectly clear, right? Certainly, this may not be straightforward. So let’s break these conditions down.

Presence of a Moderator

A moderator, or treatment effect modifier, is a third variable z that changes the effect that x has on y. If no moderator is present, then the effect of x on y is consistent across all levels of z. But if z acts as a moderator, the effect that x has on y is dependent on the value of z, i.e., it changes over the range of values of z.

Consider the following example: When studying the effect that GenAI-usage (x) has on code-review performance (y), the experience of the code reviewer (z) may moderate the effect. Inexperienced code reviewers (i.e., low values of z) might benefit strongly from the use of GenAI (i.e., experience a positive effect) because GenAI-support is better than nothing. However, experienced code reviewers (i.e., high values of z) may actually be hindered by GenAI-support (i.e., experience a negative effect) because GenAI-suggestions may be slightly off, confusing, or no match for their domain experience. In such a case, z moderates the effect of x on y.

Different Distributions

The second condition implies a covariate shift, i.e., that the distribution of a variable is different in a sample from a target population. This happens very easily in SE studies. For example, students are easier to recruit than practitioners, but then the covariates experience and skill will differ in the experimental sample from the target population.

Co-Occurrence of both Conditions

When these two conditions co-occur, i.e., the moderator is also subject to a covariate shift, then we have a threat to external validity. If our experimental sample focuses only on a particular subset of the target population (e.g., very junior code reviewers) for whom the effect is particularly strong due to the moderator (see the yellow box in the figure below), then the average treatment effect calculated from the experiment will overestimate the effect as it would manifest in the total target population, where the covariate is distributed differently.

Mitigating the Threat to External Validity

But is acknowledging them the only thing we can do? Turns out, there are statistical techniques to improve the external validity of a controlled experiment as long as

1. we collect data about the moderator in question, and
2. we have data about the distribution of this moderator from the target population.

The latter could be an observational data set just containing data about z. Picking up the previous example, it could be a sample study recording the experience in number of years working as a code reviewer of a larger target population (e.g., a company-wide profile).

When this data is available, we can apply transportability methods. In essence, these work by re-weighting the data points collected in the controlled experiment based on the sampling probability: when a data point has a covariate value z that is rare in the experimental sample but common in the target population, then it will be weighted much stronger in the estimation of the average treatment effect than a data point where z is common in the experimental sample and rare in the target population. For example, a few more senior code reviewers in a controlled experiment will influence the results much more than the many students, because these more senior subjects represent the target population much better than the juniors.

Application

How can you use this? Arguably, the application of these transportability methods is non-trivial, but there are some direct applications available to you today:

1. Discussing threats to validity: The presented concepts provide a frame to discuss threats to external validity. Whenever acknowledging threats to external validity in your empirical studies, check: is it plausible to assume that the two conditions (moderator & covariate shift) are present? If not - and if reviewer 2 can also not provide evidence for it - then there might just be no threat to external validity, and critique of your study unjustified.
2. Studying threats to validity: If you design an empirical study and have reasons to assume that a (potentially covariate-shifted) moderator is at play, be sure to collect data about it in your study. By visualizing the effects of x on y across z, you can visually confirm this hypothesis and actually bring evidence to the claim.
3. Improving validity: Once you have collected your data and you also obtained data about the distribution of the covariate in the target population (e.g., via a survey or MSR-type study), you can apply transportability methods to improve the external validity of your results, effectively producing more relevant conclusions from your research.

A tutorial and library to apply transportability methods is on our agenda and will be published soon. A note on the final application: we anticipate that collecting (observational) data about the distribution of a moderating covariate will become a very worthwhile research activity given the framework of transportability. Once we become more aware of what moderators typically affect phenomena in SE research (e.g., developers experience and skill, their AI affinity, etc.), census-style data characterizing our target population could become reusable across several empirical studies for the purpose of transportability. We envision a productive, collaborative future in empirical SE research when adhering to this frame.

Conclusion

Not every factor threatens external validity, only when it acts as a moderator and is distributed differently than in the intended target population. If we want to aim for useful and relevant results in SE, we should

• systematically investigate the presence and effect of moderators on SE phenomena, and
• recruit samples accordingly, i.e., try to maximize the coverage of the moderators,

in order to minimize or mitigate threats to external validity. We hope

We presented this work at the International Conference on Evaluation and Assessment in Software Engineering (EASE) 2026¹ in beautiful Glasgow, Scotland, where our paper received the “Best Paper Award” on the Short Papers and Emerging Results track. You can find a publicly accessible pre-print on arxiv at https://doi.org/10.48550/arXiv.2604.08200.

with Transportability Methods. In Proceedings of the 31st International Conference on Evaluation and Assessment in Software Engineering. https://doi.org/10.48550/arXiv.2604.08200.