Month: March 2014

Maps of Maps

[Cross-posted from OpenCrisis.org]

I amused myself last night by answering one of my burning questions, namely “can I make a better list of crisis maps out of all the partial lists I have lying around”.

Here’s the original map of Ushahidis:

Here’s a copy of the draft results (i you want edit access to the real thing, just ask… and blame the spammers for this – they’re even targetting maps now) – my other unanswered questions include whether there’s been a drop-off, rise or steady number of new maps, and how the categories lists have changed over the past few years (I’ll put the scraper for that into github).

And here, for balance, are some Esri crisis maps. Because I just downloaded their WordPress plugin, and it’s, kinda, playtime.

Processing all teh Files in Directory

[Cross-posted from ICanHazDataScience]

Okay. So we’ve talked a bit about getting set up in Python, and about how to read in different types of file (cool visualisation tools, streams, pdfs, apis and webpages next, promise!).  But what if you’ve got a whole directory of files and no handy way to read them all into your program.

Well, actually, you do.  It’s this:

import globimport os

 

datadir = “dir1/dir2”

csvfiles = glob.glob(os.path.join(datadir, ‘*.csv’))

 

for infile_fullname in csvfiles:

filename = infile_fullname[len(datadir)+1:]

print(filename)

That’s it.”os.path.join” sticks your directory name (“dir1/dir2”) to the filetype you’re looking for (“*.csv” here to pull in all the CSV files in the directory, but you could ask for anything, like “a*.*” for files starting with the letter “a”, “*.xls” for excel files, etc etc).  “glob.glob(filepath)” uses the glob library to get the names of all the files in the directory.  And “infile_fullname[len(datadir)+1:] ” gives you just the names of the files, without the directory name attached.

And at this point, you can use those filenames to open the files, and do whatever it was that you wanted to do to them all.  Have fun!

Sensor Shopping

[Cross-posted from Sensornews.com]

shoppinglist

Here’s the list of items that should arrive at home soon: a basic sensor set to supplement the Geiger counter, spectroscopes, cameras, microcontrollers (Arduino and RaspberryPi), accelerometers, temperature, IR and range sensors in my toolkit.

I’m most excited about the dust sensor because it’s it was a component in Matt Schroyer’s DustDuino sensor (as seen on PublicLab.org) that’s being trialled in the developing world by Internews’ Earth Journalism Network… will be interesting to see what I can get out of it.

Sensors in Weather Reporting

“We all know that the weather with which the barometer sympathises, is considered to consist of three independent variables – the velocity of the wind, its temperature, and its dampness. It is a question how far the direction of the wind need be reckoned as a fourth distinct influence” – Francis Galton (first weather reporter)  [Galton1870]

A Little History

Weather predictions date back millennia, to at least 4th Century BC Babylonians, and recorded weather measurement, on which forecasts are made, date back hundreds of years, to the Central England Temperature series, which was collected by amateurs and has continued to be recorded since 1659 [Saner 2007].

first weather report

Figure 1 First Weather Report, 1875

early weather symbolsFigure 2 1861 Weather Report with Symbols

Weather reporting in the media dates back to 1875 with Francis Galton’s weather observation maps in The Times (above, with Galton’s 1861 map using symbols); radio broadcasts of weather information started in 1916 at the University of Wisconsin-Madison’s 9XM studio, with UK radio broadcasts of weather information  in 1922 (British Broadcasting Company), and commercial US radio reporting of weather forecasts in 1923 (Edward Rideout’s reports on WEEI Boston;  [Kutler 2003]). Television weather reporting started in 1941 at WNBT-TV, New York [Monmonier 2000].

The sensors and platforms used in weather reporting vary from manual reading and reporting of simple sensors in Stevenson screens, to digital instruments, weather balloons and satellite-mounted Doppler radars.  Outputs from these sensors are usually gathered by national, local or global meteorological organisations (e.g. NOAA, or the World Meteorological Organisation); these outputs and/or detailed analysis of them, including weather forecasts, are passed to media outlets for use in weather reports.

Early US television weather reports (e.g. Weather Man in the 1950s) were simple textual descriptions (e.g. “Sunny, chance of showers”) without maps.  The first use of satellites in weather reporting was in 1960, when the TIROS-1 was launched to send back cloud cover images of Earth from two television cameras (one high-resolution, one low-resolution); later (1960-1965) TIROS satellites included radiometers (measuring infrared radiation) with missions including detecting cloud cover during hurricane seasons and detecting snow cover; news stories arising from their use included the early detection of Hurricane Esther in 1961, and the first complete view of the world’s cloud cover in 1965. The camera resolution of the last TIROS satellite launched was 2 miles at the camera centre (the area covered by a camera pixel is typically larger at the camera image’s edges than at its centre), with each image covering 640,000 square miles [NASA 2013].

What’s Available Online Today

Weather reporting using satellite radar pictures and outputs from weather stations are now commonplace.  The radar spectrum has several ‘notches’ where radar waves are absorbed by water molecules, making storm clouds easier to see; weather stations give temperature, rainfall, windspeed, etc.

The availability of personal weather stations (including wifi-enabled personal weather stations) has made it much easier for weather-based communities to form.  Examples of grassroots communities dedicated to sharing local weather reports include:

As personal weather stations become increasingly automated and include more sensor types, this trend for micro-weather reporting and its potential to fill in data gaps in macro reporting in most likely to continue.

References

What is a Sensor?

“Sensor. Noun. A device that detects or measures a physical property and records, indicates, or otherwise responds to it.” – Google

A sensor detects physical variations in the world, e.g. light, temperature, radiowaves, sound, magnetic fields, vibration, particles (e.g. pollution, radiation) or objects (e.g. water droplets).

Humans contain Sensors

Humans and other creatures contain sensors: eyes, ears, nose, tongue, skin.  Humans are very good general-purpose sensors:

These sensors are sometimes used in sensor journalism, but so too are manmade sensors like cameras, motion detectors and thermometers.

In many situations, manmade sensors are more appropriate:  humans tend to fail the “dull, dirty, dangerous” test: their attention wanders on boring tasks, and it’s not fair to put them into dirty or dangerous situations where a manmade sensor would be more appropriate; they also can’t detect much of the physical world -e.g. radiowaves – without help, and their outputs aren’t always accurate enough for the task in hand.

Manmade Sensors are Old News

Manmade sensors convert common physical quantities (light, temperature etc) into measurements, actions or stimuli (sounds, light etc).  Manmade sensors have been designed and used for centuries, including:

Low-Tech Still Works

One area of journalism where sensors are commonly used is weather reporting.  At the low-tech end of modern weather recording is a Stevenson screen with manually-read instruments.  A Stevenson screen is a wooden box designed in 1864 to shelter a thermometer from direct weather (rain, snow, wind) and other objects (leaves, animals) that might damage the instruments inside it or bias readings from them. Stevenson screens are used for weather reporting worldwide, and contain instruments like:

Stevenson Screens are a good example of the care needed to obtain minimally-biased sensor readings.  The box and its positioning is standardized by the World Meteorological Organisation, to minimize instrument bias, e.g. all boxes are mounted 1.25m high to minimize ground temperature effects, louvred to minimize the effects of still air (e.g. overheating), and with doors opening North, to minimize reading errors from direct sunlight.

But Electronics can be Convenient

Each of the instruments above has an electronic equivalent, e.g. a sensor that can provide data remotely without a person needing to visit the Stevenson screen.  Digital sensors (sensors that convert physical quantities into electrical signals) are more recent, including digital cameras (Sasson 1975) and other sensors whose outputs can be sent to electronic storage, over wifi links or directly to computer processors.  We’ll talk more about these later.