Wednesday, July 6, 2016


It is easy for many to join the band-wagon of big data. Indeed, left, right, center, up, down, wherever you look, there is seemingly the "craze" about the benefits of big data. In this era, there are different analytical and visualization techniques around big data and these are changing the way we understand human behavior, society, cities, the flow of goods, services, money, and understand the complex patterns of interaction and communication. Things which were hitherto difficult to measure are being measured with sophisticated and sometimes simple technologies; generating data magnitudes beyond our imagination. 

Even now, there are ongoing debates about the dynamics, uniqueness, and relevance of what some call small data and big data, which are changing the debates around data. Don't get me wrong, I am neither saying it is a bad interest or emphasis, neither am I saying that such drive and efforts are irrelevant. My concern is not really about the relevance of either of the two since they both contribute to how we understand human behavior and society or if you are business person, how we understand and use data for business decisions. 

My concern relates to how developing countries struggle to even generate the basic but needed data to make critical decisions. In "Missing Millions and Measuring Progress towards the Millennium Development Goals with a focus on Central Asian States" and "Missing Millions and Measuring Development Progress", Roy Carr-Hill points our attention to the difficulty in measuring development progress in developing countries. In the former article, Car-Hill concludes that 
"It is impossible to assess progress towards or away from the MDGs in both the Central Asian Republics and worldwide. It is urgent to find solutions to the problem of the ‘missing' poor population sub-groups." 
In "The Millennium Development Goals Report" in 2013, the United Nations conceded that 
"Measuring poverty continues to be a barrier to effective policy making. In many countries, the availability, frequency and quality of poverty monitoring data remain low, especially in small states and in countries and territories in fragile situations" (See page 7).
A review of global database of development indicators also show that the challenge still remains. As I argued in one of my researches on the MDGs

"An in-depth overview of the data sources that inform the MDG database illustrates that much of the data is based on national reports, surveys and research by these countries. By implication, a challenge with these national data illustrates a missing link in data in the global database. For this reason, it is important to appreciate the magnitude of tracking global development and poverty as these basic indicators are not comprehensive enough to present a good trajectory of development interventions and their impacts in West African countries" (See page 180)
Evidently, because these global development databases rely on national reports and data communicated to them, any flaw or missing data illustrates challenges from source. This present some overview of the weaknesses in data generation and management capacities for many countries, especially developing countries.

This is even worse when one tries to access or find data at the micro-levels in developing countries; say a region, a city, a town, or a community. Even up-to-date profiles for such places are even difficult to access; and worse if one attempts to explore for specific development indicators.

It so seems that many developing countries, for a country I am more familiar with, Ghana, do not have such capacities. Micro data are seemingly unavailable. And when they exist, they are mostly not accessible, out of date or unreliable. This thus limits the kinds of analysis that can be adopted to understand the dynamics and complexities of human behavior and society in these countries as well as to track development progress. 

Data such as the number of jobs created by the economy remains a challenge, data about who can be verified as a Ghanaian, those who are within legal ages of making certain decisions, those who pay or do not pay taxes, poverty levels, among many others are either obsolete, unavailable or unreliable.

It is for this reason that the call by the UN Data Revolution group is welcomed and shows that there is recognition of the need to adequately measure development progress for the post 2015 Sustainable Development Goal agenda. Indeed, or at least, at the global level such appreciation has been identified but the success of achieving effective and reliable and valid data to measure development progress and demand more than a global appreciation and effort. Countries need to give a great attention and priority to generating and making such data available and "relatively" up-to-date at different levels within their countries.

So until the fundamentals of measuring development progress are set aright, it will be difficult for many countries to start to even engage in small data or big-data debates. For the private or business sector this may not entirely be true but for small data and big data relating to development, i.e. understanding human and societal development, this will remain true until the fundamentals are corrected.