By Andrew C Lemer, PhD, Senior Program Officer, the National Academies of Sciences, Engineering and Medicine; Chair, APWA Engineering and Technology Committee

The idea of 'big data' pops up just about everywhere these days, and most of us have experience with it, although we may not always be of its impact.

If I search online for information about some newly popular film or TV show, for days afterwards I will see ads and be offered deals on whatever I searched for and anything commercially available that is in some remote way similar to what I searched for. All because the technical obstacles and costs of collecting data about my online searches and those of other people somewhat like me—'in my demographic' as the marketers say—have come down so much that someone thinks it is worthwhile to analyse our browsing and shopping records to predict what we might be enticed to purchase.

The accumulation of large collections of data about consumer behaviour, medical records, bird-watchers’ reports, and more has given impetus to the techniques of 'data mining.' The term refers to searching through the mounds of data to find potential valuable correlations and trends that might tell us about how to sell more tickets, stop the spread of the flu, or spot a real change in wildlife migration patterns. Some researchers have used data mining to learn how to do a better job predicting settlement of soils in highway embankments and improve water treatment plant efficiency.

Researchers working in the United States and the Persian Gulf area (as ancient geographers named it, although some people today would prefer to call it the Arabian Gulf) have been exploring how data mining techniques, new communication technologies, and relatively low-cost sensors can be combined to track viruses and other interesting components of municipal sewage flow. The concept is that a community might be able to spot and track closely the appearance of pathogens and other artefacts in the sewage in specific neighbourhood areas and, when necessary, mobilise public-health resources to prevent the spread of infectious diseases. The researchers envision that the system might also be used to characterise the overall health of a city.

The underlying thought of using data to improve measurement and management of public services is not really so new, but our ability to do it quickly and over large areas is growing rapidly. For example, streamflow gauging to predict downstream flooding and crowdsourcing of road traffic speed to warn of congestion hotspots have been around for a while but remain fairly crude. On the other hand, while the 'Super Bowl flush'—a sharp nationwide drop in municipal water pressures at the start of the halftime break—may be more urban legend than fact, at least one system manager found he could indeed infer that the timing of televised events influenced main pressures. Once someone discovers such a relationship, others may ultimately find ways to use it. Time and continuing advances in monitoring, communication, and computation technologies are likely to make big data a growing part of public works management.

Andrew Lemer, Ph.D., is currently a Senior Program Officer with the National Academy of Sciences of the United States of America. In addition to technical papers and occasional articles for the Reporter, he writes on civil infrastructure and human settlement at www.andrewlemer.com.

This article was first published in the March edition of the American Public Works Association (APWA) Reporter, as part of the Imagination to Innovation series. 

Dennis Gabor, awarded the 1971 Nobel Prize in Physics for his discoveries underpinning the development of holography, once wrote, “The future cannot be predicted, but futures can be invented.” Imagination to Innovation is a periodic look at new technology and scientific discovery that we could be using to invent the future of public works.