Using Administrative Data to Explore the Effect of Survey Nonresponse in the UK Employment Retention and Advancement Demonstration

Dorsett, R., Hendra, R. and Robins, P.K. 2018. Using Administrative Data to Explore the Effect of Survey Nonresponse in the UK Employment Retention and Advancement Demonstration. Evaluation Review. 42 (5-6), pp. 491-514. doi:10.1177/0193841X16674395

TitleUsing Administrative Data to Explore the Effect of Survey Nonresponse in the UK Employment Retention and Advancement Demonstration
TypeJournal article
AuthorsDorsett, R., Hendra, R. and Robins, P.K.
Abstract

Background: Even a well-designed randomized control trial (RCT) study can produce ambiguous results. This paper highlights a case in which full-sample results from a large-scale RCT in the United Kingdom (UK) differ from results for a sub-sample of survey respondents.

Objectives: Our objective is to ascertain the source of the discrepancy in inferences across data sources and, in doing so, to highlight important threats to the reliability of the causal conclusions derived from even the strongest research designs.

Research design: The study analyzes administrative data to shed light on the source of the differences between the estimates. We explore the extent to which heterogeneous treatment impacts and survey non-response might explain these differences. We suggest checks which assess the external validity of survey measured impacts, which in turn provides an opportunity to test the effectiveness of different weighting schemes to remove bias. The Subjects included 6,787 individuals who participated in a large-scale social policy experiment.

Results: Our results were not definitive but suggest non-response bias is the main source of the inconsistent findings.

Conclusions. The results caution against overconfidence in drawing conclusions from RCTs and highlight the need for great care to be taken in data collection and analysis. Particularly, given the modest size of impacts expected in most RCTs, small discrepancies in data sources can alter the results. Survey data remain important as a source of information on outcomes not recorded in administrative data. However, linking survey and administrative data is strongly recommended whenever possible.

JournalEvaluation Review
Journal citation42 (5-6), pp. 491-514
ISSN0193-841X
Year2018
PublisherSage
Accepted author manuscriptDorsett, Hendra and Robins 2016 - Using Administrative Data to Explore the Effect of Survey Nonresponse in the UK Employment Retention and Advancement Demonstration - accepted manuscript.pdf
Digital Object Identifier (DOI)doi:10.1177/0193841X16674395
Web address (URL)http://journals.sagepub.com/doi/abs/10.1177/0193841X16674395
Publication dates
Published online25 Oct 2016
Published in print2018

Related outputs

Intergenerational and interethnic mental health: an analysis for the United Kingdom
Dorsett, R., Rienzo, C. and Weale, M. 2019. Intergenerational and interethnic mental health: an analysis for the United Kingdom. Population, Space and Place. 25 (2), p. e2195. doi:10.1002/psp.2195

Young people's labour market transitions: the role of early experiences
Dorsett, R. and Lucchino, P. 2018. Young people's labour market transitions: the role of early experiences. Labour Economics. 54, pp. 29-46. doi:10.1016/j.labeco.2018.06.002

What young English people do once they reach school-leaving age: A cross-cohort comparison for the last 30 years
Dorsett, R. and Anders, J. 2017. What young English people do once they reach school-leaving age: A cross-cohort comparison for the last 30 years. Longitudinal and Life Course Studies. 8 (1), pp. 75-103. doi:10.14301/llcs.v8i1.399

HMP Peterborough Social Impact Bond - learning exercise
Dorsett, R. 2017. HMP Peterborough Social Impact Bond - learning exercise.

A review of the matching process for the impact analysis of the HMP Peterborough Social Impact Bond
Anders, J. and Dorsett, R. 2017. A review of the matching process for the impact analysis of the HMP Peterborough Social Impact Bond.

Peterborough social impact bond: final report on cohort 2 analysis
Anders, J. and Dorsett, R. 2017. Peterborough social impact bond: final report on cohort 2 analysis. Ministry of Justice.

The effect of lifelong learning on men's wages
Dorsett, R., Lui, S. and Weale, M. 2016. The effect of lifelong learning on men's wages. Empirical Economics. 15 (2), pp. 737-762. doi:10.1007/s00181-015-1024-x

The Work Programme: factors associated with differences in the relative effectiveness of prime providers
Dorsett, R. and Lucchino, P. 2016. The Work Programme: factors associated with differences in the relative effectiveness of prime providers. Department for Work and Pensions.

Breaking the low pay, no pay cycle: the effects of the UK Employment Retention and Advancement programme
Hendra, R., Riccio, J.A., Dorsett, R. and Robins, P.K. 2015. Breaking the low pay, no pay cycle: the effects of the UK Employment Retention and Advancement programme. IZA Journal of Labor Policy. 4 (14). doi:10.1186/s40173-015-0042-9

Empirical research on youth transitions to and within the labour market
Speckesser, S., Anders, J., De Coulon, A., Dorsett, R., Espinoza Bustos, H., Kirchner Sala, L. and Nafilyan, V. 2015. Empirical research on youth transitions to and within the labour market. Department for Business, Innovation and Skills.

Education and its Effects on Income and Mortality of Men aged Sixty-five and over in Great Britain
Dorsett, R., Lui, S. and Weale, M. 2014. Education and its Effects on Income and Mortality of Men aged Sixty-five and over in Great Britain. Labour Economics. 27, pp. 71-82. doi:10.1016/j.labeco.2014.02.002

A Multi-Level Analysis of the Impacts of Services Provided By the UK Employment Retention and Advancement Demonstration
Dorsett, R. and Robins, P.K. 2014. A Multi-Level Analysis of the Impacts of Services Provided By the UK Employment Retention and Advancement Demonstration. Evaluation Review. 37 (2), pp. 63-108. doi:10.1177/0193841X13517383

Explaining patterns in the school-to-work transition: An analysis using optimal matching
Dorsett, R. and Lucchino, P. 2014. Explaining patterns in the school-to-work transition: An analysis using optimal matching. Advances in Life Course Research. 22, pp. 1-14. doi:10.1016/j.alcr.2014.07.002

The effect of temporary in-work support on employment retention: evidence from a field experiment
Dorsett, R. 2014. The effect of temporary in-work support on employment retention: evidence from a field experiment. Labour Economics. 31, pp. 61-71. doi:10.1016/j.labeco.2014.10.002

Mind the Gap
Dorsett, R., Rienzo, C., Rolfe, H., Burns, H., Robertson, B-A., Thorpe, B. and Wall, K. 2014. Mind the Gap. Education Endowment Foundation.

Breaking the low-pay, no-pay cycle: Final evidence from the UK Employment Retention and Advancement (ERA) demonstration, DWP Research Report No 765
Hendra, R., Riccio, J.A., Dorsett, R., Greenberg, D., Knight, G., Phillips, J., Robins, P.K., Vegeris, S., Walter, J., Hill, A., Ray, K. and Smith, J. 2011. Breaking the low-pay, no-pay cycle: Final evidence from the UK Employment Retention and Advancement (ERA) demonstration, DWP Research Report No 765. London Department for Work and Pensions.

Breaking the low-pay, no-pay cycle: Final evidence from the UK Employment Retention and Advancement (ERA) demonstration
Hendra, R., Riccio, J.A., Dorsett, R., Greenberg, D., Knight, G., Phillips, J., Robins, P.K., Vegeris, S., Walter, J., Hill, A., Ray, K. and Smith, J. 2011. Breaking the low-pay, no-pay cycle: Final evidence from the UK Employment Retention and Advancement (ERA) demonstration. Sheffield Department for Work and Pensions.

Employment Retention and Advancement (ERA) demonstration: delivery, take-up, and outcomes of in-work training support for lone parents
Hendra, R., Ray, K., Vegeris, S., Hevenstone, D. and Hudson, M. 2011. Employment Retention and Advancement (ERA) demonstration: delivery, take-up, and outcomes of in-work training support for lone parents. London Department for Work and Pensions.

Pathways to work for new and repeat incapacity benefits claimants: evaluation synthesis report
Dorsett, R. 2008. Pathways to work for new and repeat incapacity benefits claimants: evaluation synthesis report. Norwich Department for Work and Pensions.

Earnings Top-up evaluation: effects on unemployed people
Smith, A., Dorsett, R. and McKnight, A. 2001. Earnings Top-up evaluation: effects on unemployed people. Leeds Corporate Document Services.

Earnings Top-up evaluation: effects on low-paid workers
Marsh, A., Stephenson, A., Dorsett, R. and Elias, P. 2001. Earnings Top-up evaluation: effects on low-paid workers. Leeds Corporate Document Services.

An econometric analysis of the prevalence of smoking among lone mothers
Dorsett, R. 1999. An econometric analysis of the prevalence of smoking among lone mothers. Journal of Health Economics. 18, pp. 429-441. doi:10.1016/S0167-6296(98)00045-9

Permalink - https://westminsterresearch.westminster.ac.uk/item/q2xw6/using-administrative-data-to-explore-the-effect-of-survey-nonresponse-in-the-uk-employment-retention-and-advancement-demonstration


Share this
Tweet
Email