Malawi population and housing census technological trajectory: Unpacking 2018 experience

1.Census undertaking in Malawi

Census undertaking dates back to as early as 3800 BC by the Babylonians that focused on counting all persons and livestock within a pre-defined area (Population Reference Bureau, 2019). Globally, with increase in the need for evidence for relevant and effective responses to situations, the holistic stocktaking and documentation of subjects of interest has been heightened and embraced in this century as discussed by Davoudi [4] and Cronin and Sadan [3].

In the same vein Malawi successfully conducted the first comprehensive Population and Housing Census (PHC) in 1966 and it was followed by other rounds in 1977, 1987, 1998 and 2008. Unlike the 2008 PHC, the previous censuses utilized paper-based tools, processes and products. With this approach, enumeration and data processing proved to be time consuming and costly as it required manual conversion of the questionnaire responses into compatible data files for analysis. Further, data back-up was a challenge resulting in more reliance on the used paper questionnaires.

In 2008, paper questionnaires were used for data collection and heavy-duty questionnaire scanners were employed for data capturing and initial data processing which equally was labor intensive, time-consuming and had a substantial financial implication emanating from engagement of data-entry clerks, editors and reviewers for not less than 3 months on-site. Additionally, just like manual processing, this approach was prone to human errors which affected data quality. In particular, it did not have automated data consistency and accuracy checks in the field and during data processing resulting in traceable incidences of missing responses, wrong entries and omitted cases. Further, in terms of geo-referencing and mapping, all these censuses utilized a clear backdrop. This negatively affected map interpretation and enumeration as the field personnel struggled to relate to features on the ground and to identify pre-defined boundaries of enumeration areas, more especially imaginary ones.

2.Approach for the 2018 PHC

Census trajectories in many countries including Malawi have not been easy to sail through due to massiveness and complexity of the entire process. To navigate through such a challenging and multifaceted course requires grounded methodology and an informed and effective implementation plan.

In the face of the previous experiences, lessons drawn and financial implications, NSO started preparations for the 2018 PHC in the year 2015. Firstly, a census strategic plan aimed to implement a cost-effectiveness methodology, safeguard data quality and ensure timely release and dissemination of printed and electronic census products was developed. It opted for adoption of innovative strategies and incorporation of technology in some of the census activities to significantly reduce processing time and improve reliability and timeliness of census results [2].

For the latest census, NSO decided to embrace use of Satellite imagery for mapping, Computer Assisted Personal Interviewing (CAPI) technology for data capturing and wireless and on-line data transmission. To support the adoption of these technologies, NSO conducted a situation analysis on the technical capacity of the office, resources and budget implication, equipment required and geography and research boundaries. Results of the analysis highlighted the need for additional staffing, procurement of equipment, increase of office space, provision of storage rooms and security, installation of electrical power back-up system and need to update maps.

In respect of the foregoing, Mapping Strategy was developed and approved by the Secretary to the Treasury (ST) for Malawi Government. The document detailed the structural framework of the proposed mapping methodology, highlighted activities and expected timelines, stakeholders and a total budget of US$3,263,779 for implementation. During the 2015/ 2016 fiscal year, the Ministry of Finance (committed approximately US$550,000) and other stake holders mobilized resources for the initial phases of the census. The resources covered procurement of computers, servers, ArcGIS mapping software, field work and initial payment for 6000 satellite imagery from Regional Centre for Mapping of Resources for Development (RCMRD). As per objective of the mapping exercise, NSO set a condition that satellite images were to be latest. For urban areas, they were to be captured within 1 year and a maximum of 3 years in rural areas whilst being of a prescribed 0.5 meters resolution.

Additional resources were earmarked for the collaborative implementation of the mapping component. Malawi Government through NSO roped in Department of Surveys under Ministry of Lands, Ministry of Local Government and Rural Department, US Census Bureau and RCMRD. RCMRD were to provide on-site and remote technical support for mapping processes. RCMRD is an inter-governmental organization established to develop capacity and provide technical support in the use of space technology and has 20 Contracting Member States in the Eastern and Southern Africa Regions, including Malawi; and US Census Bureau, as the name suggests, is an official statistics agency of the United States of America.

3.Census mapping

Relative to 2008 PHC mapping, boundary shifts had taken place hence mapping updates were very necessary. Precisely, some Group Village Headmen had been elevated to the status of Sub Traditional Authority or Traditional Authority which in turn affected their jurisdiction areas. City boundaries had been shifted, land area for wards in cities had increased and rural areas had been dented due to urbanization and other drivers. A comprehensive mapping exercise had to be undertaken to reflect current boundaries and ultimately, create a geographic frame with distinct boundaries that could allow a complete count of persons, households and structures; avoid duplication of data and facilitate disaggregation of data at any geographic and administrative level. Additionally, the mapping component would feed into facilitation of timeliness, universality and simultaneity of census processes and activities at the smallest feasible level.

Effective updating of administrative boundary layers, enumeration area’ boundary layers and map attributes and corresponding shape files necessitated capacity building of the mapping team. In this regard, NSO identified the Regional Centre for Mapping of Resources for Development (RCMRD), ESRI-South Africa, other experts and avenues to build capacity of the team in all aspects of mapping. For knowledge and skills in interacting with maps (in terms of symbology, visualization, and finding/ identifying/selecting features), working with tabular data and performing spatial analysis, the team went to ESRI South Africa for short term trainings. As part of the RCMD contract for census technical support, the NSO team underwent a number of short-term trainings on-site (NSO Headquarters) as well as at RCMD base in Nairobi, Kenya. The trainings focused on creating, displaying and evaluating data for analysis, designing map layouts and automating the mapping process. To complement this aspect, US Census Bureau offered and deployed GIS experts to organize and facilitate working sessions on mapping. Further, NSO capitalized on international meetings and interactions with experts from other Statistical Offices to equip the GIS team with other required skills.

In line with the highlighted trainings and also recognizing the technological innovations and advances, detailed assessment on the technology and equipment that would contribute to efficiency of the mapping processes was proposed and undertaken. Census study tours were prioritized as they provide professional insight through knowledge sharing [5]. The team travelled to Kingdom of Eswatini, formerly Swaziland for a hands-on access and live demonstration of the processes and systems used. From the visit, details of specific electronic devices and other equipment that could ably handle the Malawi census mapping and other functionalities were documented. Missions to Egypt and Ethiopia were also planned as the former had just conducted a full CAPI census in the year of 2017 whilst the latter was at an advanced stage of census preparation. Unfortunately, the rest of the study tours did not materialize mainly due to logistical challenges and other technicalities. Additionally, consultations with experts from US Census Bureau, other technical experts on census and specialists within Malawi were arranged and took place.

Based on these consultations and situation analysis, NSO procured forty high performance HP desktop computers, 20 Laptops, 1 HP Proliant DL60 Gen 9 server, 200 handheld global positioning devices and 2 plotters for map printing. The server was for central storage and controlled access to satellite images by the computers for processing satellite imagery, updating 2008 PHC Enumeration Area (EA) maps, and developing dwelling structure frame using ArcGIS software. The computers were also for converting map products into formats compatible with tablets for field work phase (EA boundary identification and data collection). Additionally, Garmin GPSMAP78 global positioning gadgets were procured for geo-referencing of features of interest and field validation.

By design, field and office processes of mapping were run concurrently to allow field verification and updating of map features and revision of the maps. The in-office aspect focused on the 2008 mapping processes and products. It concentrated on EAs that were flagged as being so vast, had poor terrain, reportedly the numbers of households were above the threshold (150 in rural areas and 250 in urban areas), boundaries were problematic (had imaginary boundaries, erroneously missed features and other references, etc.) and other highlighted issues. Further, as part of the office component, boundary layers of 2008 base maps were overlaid on the satellite imagery to aid identification and marking of structures, potential households, physical features and where necessary update attribute tables. This phase also led into development of frame of dwelling structures using dotting approach (BBBB, 2008). Finally, using ArcGIS software and respective satellite imagery the team adopted mapping propositions and produced latest maps.

For the other part of the mapping phase, the teams were deployed for field verification and validation of the updated administrative and EA maps printed using heavy-duty plotters. To ensure local recognition/ownership of the process and more significantly the administrative boundary changes, local authorities or representatives were involved. The mappers reported to the head of District Police Station, District Commissioner and other key stakeholders for debriefing on the work and the list of gazetted TA’s from the Ministry of Local Government.

The field teams physically checked features on the printed administrative and EA map. They randomly checked sizes of segments highlighted on the maps and where necessary they proposed new boundaries and recommended structures and other features to be marked on the maps. This phase allowed the teams to calibrate the handheld GPS receivers with ground reference points on the satellite imagery to minimize errors. For areas not yet updated, the teams capitalized on their presence in the district or neighboring district to collect information for updating the 2008 base map layers (roads, hospital, etc.).

4.Use of CAPI in the census

Prior to 2018 Population and Housing Census, NSO decided to migrate and adopt Computer Assisted Personal Interview technology (CAPI) over Pencil and Paper Interviewing (PAPI) system. The decision was driven by the need to improve efficiency in terms of timeliness and data quality; ease data capturing; minimize human errors [8]; ease data processing and transferring; and enhance availability of geo-referenced data. By its nature CAPI facilitates real time logic checks, skip patterns, and validations which makes data collection more efficient and achieves high data quality (World Bank Group, 2019). In Malawi this technology has been used in a number of surveys including Malawi Demographic Health Survey, Feed the Future Survey, Malawi Communications Regulatory Authority (MACRA) Survey and Integrated and Household surveys (IHS).

A consultative approach was used to identify a software for the structural framework of the proposed CAPI system for the census. In-house assessment was conducted and CSPro was preferred as no major issues have been experienced; on-site and remote technical support was assured; and it has no financial implication as it can be downloaded free of charge and requires no user licenses. Initially, CSPro version 7.0 was used in the programming of the CAPI system, however with the complex set-up for 2018 PHC application new versions (7.1, 7.2 Beta and 7.2) were adopted along the way.

5.Data management

Adoption of CAPI technology for data collection has often led to the thinking that we can do away with data processing. Indeed, in an ideal scenario, data processing should be by-passable and arguably data analysis can follow completion of the enumeration phase and/or data concatenation of unique data files. This inclination is obviously attributable to misconception that the use of CAPI technology sufficiently enhances technical implementation of the census in terms of distribution and management of work. Further, with the built-in checks and skips/routing capabilities, CAPI application completely effect data flow consistency checks and logical harmonization of responses of an individual case/respondent in real time.

On the contrary and in the face of CAPI and built-in data quality control features, data processing is irreplaceable and key. As hinted by Kordos [6], censuses are never perfect and coverage and/or content errors and other technical irregularities end up on the data files. For Malawi, the main issues included assignment of wrong ID parameters to assigned enumeration area’ case, partial or incomplete data files attributable to erroneous launch of data entry application (e.g. calling household questionnaire application instead of census structures capturing application, etc.), enumeration of households beyond the EA boundaries, inappropriate tablet sharing/transferring among enumerators, missing data files from unsuccessful data synchronization, data structuring, etc.

6.Census field operation

The initial field plan was that five enumerators would be assigned 1 field supervisor and 40 field supervisors (FS) would work under 1 Control Centre Supervisor (CCS) who would report to the census district supervisor/manager who in turn reported to the Census Zone manager. The technical census implementation, in terms of data transfer, was also devised to follow the same structure only that from the Control Centre Supervisor the data would go straight to NSO Headquarters. With the logistical challenges that led to failure to procure additional 5000 tablets in time, NSO changed the data transfer plan. Though field supervisors were maintained, their direct role in data transfer was scrapped off such that CCS synchronized data directly with 40 interviewers.

Despite the technical dent on data transfer, data collection and supervision structure of the 2018 PHC remained intact. Every Control center supervisors (CCS) recruited 8 field supervisors (FS) to assist in manning the 40 enumerators (5 enumerators each). They acted as link for ensuring smooth collection of data, located boundaries of each enumeration area, verified and certified completion of the assigned workload. The FSs also safe-guarded well-being and safety of enumerators. To ensure all front-line staff were updated regularly with relevant information, the FS regularly held briefing meeting with CCS and formed part of their respective census district WhatsApp groups.

Since the field supervisors were already trained on CAPI and supervisory roles, they were allowed to log-in on the interviewer’s tablet. This accorded them an opportunity to review the structures and households enumerated, re-interview random households in the EA, troubleshoot problems and ensure that data backing-up was taking place.

7.Census results dissemination

The census results were as per protocol presented to the public through various reports. This phase involved hiring of private printing companies who designed and printed several copies of the successive reports, which were then stored at the NSO Headquarters in readiness for launch and distribution. The preliminary report was first released and launched on 10th January 2019, which was then followed by the main report released and launched on 30th May 2019. Additional reports on specific areas in the form of thematic reports, district monographs, and constituency reports are underway and will in the same spirit be released to the public with special interest to policy makers when completed. The released reports have been shared with government sectors, academia institutions as well as other interested parties and stakeholders at large.

NSO with help from UNFPA, has plans to further disseminate data for the 2018 PHC. It has planned to conduct district dissemination of reports and infographics, and disseminate corporate products (media briefs, data wheels, fliers and posters), and also develop data localization platform. Conceptually, the platform will be a mobile phone application that will allow users to query the database for information at different levels including enumeration area level.

8.Challenges experienced

The use of advanced technology and incorporation of innovative systems is increasingly transforming the way censuses and surveys are conducted (Castano, 2018). Despite the strengths and merits underscored by the endless success stories, the technological migrations bring along some hiccups on the census trajectory as described below.

Firstly, Malawi has 28 districts which were demarcated into 40 census districts, 485 control centers and 18,772 Enumeration Areas (EA) for purposes of censuses and surveys. By design, this required approximately 20,000 tablets as one control center supervisor manned and synchronized data with 40 front line personnel each tasked with enumeration of an EA. This implementation plan obviously had a financial implication in addition to creating a logistical challenge in terms of sourcing, distributing and provisioning as well as ensuring security for such a huge number of tablets. Similar cost challenges were faced in procuring the satellite imagery and mapping equipment; heavy-duty plotters, scanners, GIS software(s) and other IT infrastructure. Therefore, a thorough cost-benefit analysis and risk enquiry for embarking on the technology is required in the planning of a census project.

Secondly, the pilot census informed NSO that from the onset of census implementation, back-up supply of electric energy will be required by field personnel as quite a substantial part of the country has not been connected to the electric grid. Further, intermittent electric power interruption was persistent in many parts of the country. To counter power shortages in remote areas with little or no access to electricity, the budget of 2018 MPHC had to be revised upwards to allow procurement and distribution of 2000 milliamps (mA) electric power banks and solar chargers.

Further, CAPI census also faced data transfer challenges. On one hand, data transfer from an enumerator to a control Center Supervisor was over Bluetooth connection deliberately opted to counter poor internet connectivity in very remote areas. This required physical presence of a CCS for every data synchronization which proved to be tedious and costly since one CCS was tasked with synchronizing data from 40 enumerators. On the other hand, on-line data transfer posed a high risk of data confidentiality breach resulting from unauthorized data interception as well as malicious access to the data bank. In view of this, network systems were strengthened and upgraded as demonstrated by opting for virtual private network for data synchronization with data center at the Head office which came at a substantial cost. Additionally, 3 servers safeguarded with passwords and an encryption mechanism were added to the data network.

9.Lessons learnt and recommendations

Census undertaking of 2018 has accorded NSO an opportunity to draw a number of lessons. Firstly, knowledge sharing within the African region in the spirit of South to South Cooperation is worthwhile. NSO greatly benefited from experiences and skills from countries that have pioneered incorporation of CAPI technology and use of satellite imagery during Population and Housing Censuses. In reciprocating the deed, NSO-Malawi has explicitly opened up to study tours and provision of remote and on-site support on CAPI development, Mapping and other areas of interest to other countries.

Incorporation of technological innovations in censuses comes with notable advantages on one hand and significant costs on the other hand. For instance, migration from paper questionnaire to CAPI technology contributed to over 30 percent of the overall budget whilst mapping component on the framework of satellite imagery utilized over 50 percent of the 1.5-billion-kwacha (US$2.3 million) budget. Fortunately, the cost of the Malawi census was greatly subsidized through bi-lateral agreements in which DFID procured 15,000 tablets and UNFPA-Malawi coordinated loaning of the devices and other gadgets under the umbrella of south-to-south cooperation. Following enumeration, data processing and cleaning, the tablets were forwarded in July 2019 to Zambia for their 2020 Population and Housing Census.

With regard to funding and late disbursement of resources, being optimistic, positive minded and pro-active is key. Major activities of the Malawi census were characterized by funding issues up to as late as 2 months before enumeration when the donor community through UNFPA-Malawi filled the 20 percent gap. However, irrespective of that situation, NSO implemented all the activities whose resources were already secured and conducted preparatory work for the activities that had resource gaps. This approach arguably contributed to the success story of implementing a full CAPI census.

10.Conclusion

The 2018 Malawi Population and Housing Census has been a success story. The 2018 PHC has been appreciated for smooth running and completion of implementation. Particularly, the overall timeliness of the census results has been far more effective. The preliminary report was released within 4 months after the data collection and the main report was released within 4 months after the preliminary report. With the speed and quality of the census results, the cost implication of using the CAPI system was worthwhile. Although the census incorporated extra costs in usage of tablets, there was an overall cost minimization compared to the 2008 PHC. According to the 2018 MPHC results, the total population enumerated was 17.5 million people against a total budget of US$19.8 million representing a high cost efficiency of US$1.1 per capita against US$1.4 in 2008 and US$1.74 average per capita for most censuses in Southern Africa.