Post on 07-Oct-2018
Session 5A: Field work and quality assurance
Beatriz Godoy, Mario Navarrete
Sistemas Integrales Delhi, march 20th 2013
What are the most frequent problems you have encountered in fieldwork?
A. Inadequate selection of staff
B. Poor training C. Weak supervision D. Bad or inconsistent data E. High non response rate F. Fieldwork takes longer
than expected G. Delayed payments H. Excessive paperwork
2 A. B. C. D. E. F.
21%23%
5%
13%
26%
13%
A horror story Survey Month
Households reporting illnesses, accidents (Q407)
Households reporting chronic
deseases (Q401)
Households reporting
agricultural activities (Q901)
Number of crops
reported (Q911)
Households reporting livestock
activities (Q918)
Households reporting fishing activities (Q924)
Total Nb of durables
(Section 12)
Nb of HH having credit (Section 13)
HH size mean
Number of lines with food consumption
1 1,645 705 572 1,352 563 17 9,538 1,017 7.8 95,687
2 1,352 624 503 1,299 530 13 8,853 937 7.5 99,491
3 996 577 507 1,421 502 12 9,818 910 7.6 96,028
4 898 642 486 1,469 504 25 9,301 880 7.9 96,139
5 816 545 436 1,243 464 3 9,180 811 7.1 97,094
6 691 513 477 1,442 465 23 9,667 841 7.5 97,315
7 625 529 465 1,338 494 26 9,621 738 7.5 108,432
8 658 498 439 1,264 433 17 9,437 707 7.3 100,888
9 769 552 433 1,244 428 11 9,584 740 7.2 98,893
10 858 534 468 1,227 436 25 9,224 719 7.4 98,415
11 743 517 399 1,165 411 10 9,294 737 7.2 97,138
12 693 464 356 1,133 440 25 9,722 705 7.1 96,052
Survey month
No. of households reporting illnesses
1 1,645
2 1,352
3 996
4 898
5 816
6 691
7 625
8 658
9 769
10 868
11 743
12 693
The factors of fieldwork quality
Field staff selection and
training
The fieldwork organization
Data management
Supervision and control
4
Field staff selection and training
5
Field staff selection and
training
The fieldwork organization
Data management
Supervision and control
Call for applicants
Pre-selection of candidates
based on profiles
Training
Staff selection based on
performance and integrity
Selection approval by hiring entity
Final formalization of hiring with the selected
personnel
6
Field staff selection and training
The fieldwork organization
7
Field staff selection and
training
The fieldwork organization
Data management
Supervision and control
The fieldwork organization
Fieldwork Organization
Organization of field workers
The fieldwork
plan
Survey Plan
8
Organization of field workers
• Based on the notion of independent teams
• Specialized personnel within each team, or even specialized teams
• Computer-based quality controls integrated into the field work
9
Côte d’Ivoire 1984
Peru 1985
Ghana 1987
Jamaica 1988
Bolivia 1989
Paraguay 1997
Argentina 1997
Bangladesh 1995, 2001
Timor-Leste 2001, 2007
Niger 2008
Namibia 2006
Mongolia 2008
Iraq 2006-2012
India (UP) 1998-2000
Pakistan 1991 - 2002
Vietnam 1996
Honduras 1998
Papua New Guinea 2008
Polynesia 2000
Guatemala 2000
Nicaragua 1989, 2000, 2005
Panama 1998, 2003
Nepal 1994 – 2002 - 2010
Ethiopia 2011
Uganda 2010
Integrating computer quality controls to fieldwork in LSMS Survey’s
(around the world in three decades)
Data entry in fixed locations Data entry in the field CAPI
Organization of field workers
• An interview team 1 team supervisor 2-4 interviewers 1-2 specialized people in specific measurements
(anthropometry, cognitive development testing, hemoglobin, etc.)
1 data entry operator 1 driver
11
Team supervisor Interviewers Specialist Data entry operator
The fieldwork plan at a sampling point
Things to consider: – Number of interviews per sampling point – Geographical scattering of households within each sampling point – Conformation of teams – Household listing operation, if applicable – Quality controls – Biometric and cognitive test, measurements, etc., if applicable – Interview duration, number of visits and correction visits – Other questionnaires, for example, at the community level – Rest time – Approximate travel time to the next sampling point
12
Optimistic scenario Pessimistic scenario $
$
What is the team supposed to do in each sampling point, and how long does it take
Example of a work plan at a sampling point
Día Operador de datos
H1 H2 H3 H4 H5 H6 H7 H8 H9 H1 H2 H3 H4 H5 H6 H7 H8 H92 V1 V1 V1 V1 V1 V1 V1 V1 V1 Ingreso Q. H1 a H93 VC1 VC1 VC1 V2 V2 V2 V2 V2 V2 V2 V2 V2 Ingreso Q. H1 a H94 VC1 VC1 VC1 V3 V3 V3 V3 V3 V3 V3 V3 V3 Ingreso Q. H1 a H95 VC1 VC1 VC1 V4 V4 V4 V4 V4 V4 V4 V4 V4 Ingreso Q. H1 a H96 VC2 VC2 VC2 V5 V5 V5 V5 V5 V5 V5 V5 V5 Ingreso Q. H1 a H97 VC2 VC2 VC2 V6 V6 V6 V6 V6 V6 V6 V6 V6 Ingreso Q. H1 a H98 VC2 VC2 VC2 V7 V7 V7 V7 V7 V7 V7 V7 V7 Ingreso Q. H1 a H99 VC3 VC3 VC3 VC3 V8 V8 V8 V8 V8 V8 V8 V8 V8 Ingreso Q. H1 a H910 VC3 VC3 VC3 VC3 VC3 V9 V9 V9 V9 V9 V9 V9 V9 V9 Fin ingreso H1 a H91112
Envío de datos a Niamey
DescansoDesplazamiento
1Consolidación y
envio de datos ola
Jefe de equipo Encuestador 1 Encuestador 2 Encuestador 3Listado de hogares
Selección de hogares a encuestar
Survey plan (Who visits each sample point and when)
A bad survey plan
Summer
Spring Autumn
Winter
A better survey plan
Team 5
Team 1
Team 4
Team 3
Team 2
Even better (if you can)
Teams 1, 2 and 3
Teams 4 and 5
Inter-penetrating samples (Mahalanobis,1953)
Survey Plan • Which team will visit each sampling point and when • The survey plan should consider:
– Time spent by a team at a sampling point (work plan) – Number of sample points – Total time available for fieldwork – Spatial and temporal distribution of the sample – Number of field teams – External constraints
• Weather • Security • The calendar of the evaluated project
18
Example of a survey plan
19
Piso ecológico Departamento Municipio Provincia UPM Viviendas
UPM Población
UPM Equipo Fecha de inicio Fecha de fin
1 LA PAZ Achacachi Omasuyos 20065 308 1002 1 1-Nov-11 10-Nov-11
1 LA PAZ Achacachi Omasuyos 20073 341 1304 1 11-Nov-11 21-Nov-11
1 LA PAZ Achacachi Omasuyos 20088 167 440 1 22-Nov-11 2-Dec-11
1 LA PAZ Achacachi Omasuyos 20097 360 1211 1 3-Dec-11 13-Dec-11
1 LA PAZ Caquiaviri Pacajes 20179 213 416 2 14-Dec-11 24-Dec-11
1 LA PAZ Caquiaviri Pacajes 20188 256 760 2 25-Dec-11 4-Jan-12
1 LA PAZ Puerto Acosta Camacho 20245 204 511 2 5-Jan-12 15-Jan-12
1 LA PAZ Puerto Acosta Camacho 20253 297 521 2 16-Jan-12 26-Jan-12
1 LA PAZ Puerto Carabuco Camacho 20232 266 709 3 27-Jan-12 6-Feb-12
1 LA PAZ Chuma Muñecas 20328 310 1260 3 7-Feb-12 17-Feb-12
1 LA PAZ Chuma Muñecas 20336 332 1100 3 18-Feb-12 28-Feb-12
1 LA PAZ Viacha Ingavi 20983 249 551 3 29-Feb-12 10-Mar-12
1 LA PAZ Jesus de Machaca Ingavi 21002 298 473 4 11-Mar-12 21-Mar-12
1 LA PAZ Pucarani Los Andes 20638 204 498 4 22-Mar-12 1-Apr-12
1 LA PAZ Pucarani Los Andes 20646 276 721 4 2-Apr-12 12-Apr-12
1 LA PAZ Pucarani Los Andes 20658 223 814 4 13-Apr-12 23-Apr-12
Good organization is key to:
• Minimize data collection errors – Correction of errors directly in the household – Increase in response rate – Decrease problems due to protocol adherence
• Timeliness
20
Data management
21
Field staff selection and
training
The fieldwork organization
Data management
Supervision and control
Interview
Data entry
Quality report
Supervisor review
Re-visit
Data Entry in the Field
INTERNET
Consolidated data base
Data transfer
22
FieldworkMonitoring
Production of a data base: • Must be timely • Must be well documented • Exported to statistical software Partially, to monitor field work Finally, for analytical use
23
Data Management
Characteristics of the data entry program
• Development platform available: CsPro; LSD; Blaise; etc.
• Variables dictionary • Validations
– Range checks – Checks against reference tables (anthropometrics,
food composition, etc) – Skip controls – Consistency checks
• Data consolidation and export • All programing must be ready before fieldwork
24
What happens without integration?
• The long and frustrating “data cleaning” process becomes unavoidable
The data loose their policy relevance
• Data quality is not guaranteed
“Data cleaning” converges, at best, to a database that is
only internally consistent, but does not necessarily represent reality
• “Data cleaning” entails a myriad of decisions, generally
undocumented
Users mistrust the data
25
Benefits of integration
• Reliable and timely databases • Permanent monitoring of fieldworker performance, allowing
early warning of inadequate behaviors • Guarantee that all fieldworkers apply uniform criteria
throughout the full period of data collection • Inconsistencies solved by direct observation of reality, rather
than by office guesswork • Foster the total quality culture
26
SUPERVISION AND CONTROL
27
Field staff selection and
training
The fieldwork organization
Data management
Supervision and control
Quality Control
Human Supervision
• HH selection • Check up-visit • Observation of
interviews
Computer -Assisted
• Scrutiny of questionnaires
Computer Assisted
• Monitoring of data quality indicators
28
In the field
In the central Office
Quality assurance during the field phase
Fiel
d Te
am
Interviewer controls questionnaire Supervisor verifies quality of interviewers Supervisor controls questionnaires with the aid of PED messages Return to household to correct errors, verification of inconsistencies and complete any missing information
Cent
ral S
uper
visio
n
Supervising Missions Verification of standard compliance and work schedule Verification of errors or remaining inconsistencies Statistical observations by interviewers and teams Telephone check ups
Inve
stig
atin
g Te
am
Supervising Missions Verification of errors or remaining inconsistencies remanentes Statistical observations by interviewers and teams Verification of BD conformation
29
Quality assurance during the field phase
30
INTERVIEWERS
SUPEVISION OF IMPLEMENTING FIRM
SUPERVISION OF INVESTIGATORS
SUPERVSION OF FIELD TEAM
Enquêteur Valeurs 0 et
5 Autres valeurs Total
112 70.0% 30.0% 100.0% 113 23.8% 76.2% 100.0% 114 38.9% 61.1% 100.0% 122 45.5% 54.5% 100.0% 123 38.1% 61.9% 100.0% 124 33.3% 66.7% 100.0% 132 12.5% 87.5% 100.0% 133 13.0% 87.0% 100.0% 134 22.2% 77.8% 100.0% 142 13.3% 86.7% 100.0% 143 12.9% 87.1% 100.0% 144 42.9% 57.1% 100.0% 152 47.1% 52.9% 100.0% 153 30.0% 70.0% 100.0% 154 20.0% 80.0% 100.0% 162 84.6% 15.4% 100.0% 163 35.0% 65.0% 100.0% 164 41.7% 58.3% 100.0% 171 66.7% 33.3% 100.0% 172 44.4% 55.6% 100.0% 173 33.3% 66.7% 100.0% 174 77.3% 22.7% 100.0%
Follow-up of quality indicators
Last digit distribution of de anthropometric measurements
If the measurements have
been made correctly, it would be expected that the last digit
will be equally distributed between 0 to 9, that is, 10%
each.
31
Unknown Total
0 1 0 Team 0 100.0% 100.0%
1 84.8% 15.2% 100.0% 2 51.5% 48.5% 100.0% 3 100.0% 100.0% 4 53.8% 46.2% 100.0% 5 53.3% 46.7% 100.0% 6 97.8% 2.2% 100.0% 7 41.4% 58.6% 100.0% 9 86.8% 13.2% 100.0% 11 78.9% 21.1% 100.0% 12 100.0% 100.0% 13 83.7% 16.3% 100.0%
Total 77.4% 22.6% 100.0%
Follow-up of quality indicators
32
Percentage of « unknown » answers by
team
33
Example of development of quality indicators
Example of a follow-up EQUIPO REGION ola 1 ola2 ola 3 ola4 ola5 ola 6 ola 7 ola 8 ola 9 ola 10 ola 11 ola 12 ola 13 ola 14 ola 15 ola 16
11 Agadez 2,9 2,8 1,5 1,0 1,3 0,8 1,2 0,5 0,6 0,5 0,2 1,4 1,5 0,2 0,3 0,212 Diffa 1,0 0,9 2,0 0,1 2,2 3,5 0,7 1,1 0,8 1,1 1,0 1,2 1,4 0,8 0,0 0,013 Maradi 16,1 11,5 10,8 1,2 1,5 0,9 1,2 0,5 1,7 0,7 0,5 0,2 0,2 0,1 0,2 0,414 Maradi 28,8 11,8 9,3 4,1 6,0 2,6 3,0 2,4 2,4 0,9 0,3 0,1 0,4 0,5 0,8 7,215 Tahoua 12,6 3,4 5,4 1,6 1,5 2,7 1,0 0,6 0,5 0,1 0,2 0,6 0,4 0,4 0,5 0,116 Tahoua 5,1 3,7 5,0 1,5 3,0 0,8 0,7 0,9 0,7 0,8 0,8 0,4 0,2 0,2 0,2 0,117 Tillaberi 10,6 12,3 6,0 1,0 0,7 0,7 0,2 0,1 0,2 0,1 0,1 0,1 0,4 0,2 0,2 0,518 Tillaberi 3,5 2,3 1,6 1,4 1,2 1,9 1,8 1,2 0,5 0,6 1,7 0,8 2,7 1,3 0,8 2,119 Zinder 5,7 9,3 19,6 1,8 2,0 1,9 1,8 0,5 0,1 0,3 0,2 0,4 0,3 0,3 0,2 0,620 Zinder 12,6 5,5 6,2 4,1 4,6 3,2 2,7 0,8 0,5 0,8 0,8 0,2 0,4 0,1 0,2 0,921 Zinder 71,9 9,9 8,2 3,2 2,5 1,3 4,2 1,7 1,0 2,9 2,9 2,5 2,5 1,6 1,0 0,922 Dosso 12,3 4,9 2,3 2,3 1,9 1,4 4,5 2,9 1,5 1,7 2,4 2,6 1,0 1,0 1,1 1,623 Dosso 62,1 4,1 1,9 1,0 0,3 0,2 0,1 0,0 0,0 0,2 0,1 0,3 0,1 0,1 0,3 0,124 Niamey 4,0 9,5 1,2 2,3 1,4 0,5 0,9 0,9 1,4 1,4 1,3 0,8 1,5 1,5 1,6 1,425 Niamey 6,5 4,7 1,7 1,9 1,9 0,7 0,7 0,3 0,6 0,3 0,3 1,6 1,5 0,8 0,5 0,9
Después de re entrenamientoAntes de re entrenamiento Re Entr.
34
• Requirements: – Age, gender, language, education, etc.. – Credentials – depending on the regulations of each
country- to carry out specific duties, such as: • Biometric testing • anthropometry • Cognitive testing
• Time availability and disposition to work in the field
• Selection of more candidates than needed
35
Preselection of candidates
• Strategy – Centralized/ Decentralized – Duration in weeks (2 weeks or longer) – Formation of groups(groups of no more than 30-40
people) – Specific training for each area (questionnaires, biometric
and cognitive tests, anthropometric measurements, data entry etc.)
• Stages of Training – Preparation of training material and agenda – Training of master trainers – Training of field staff
36
Training
• Detailed agenda of training – Lectures – In-classroom practice – Field practice (households, children, measurements…)
• Materials and logistics – Adequate infrastructure – Standardized slide presentations – Vignettes / case studies – Quizzes / final evaluation – Sufficient questionnaires, manuals and equipment – Refreshments – very important for morale – Transport to and from training site and per diem allowance
37
Training