Crisis Data Management

Maps of Maps

[Cross-posted from] I amused myself last night by answering one of my burning questions, namely “can I make a better list of crisis maps out of all the partial lists I have lying around”. Here’s the original map of Ushahidis: Here’s a copy of the draft results (i you want edit access to the real thing, just ask… and blame the spammers for this – they’re even targetting maps now) – my other unanswered questions include whether there’s been a drop-off, rise or steady number of new maps, and how the categories lists have changed over the past few years (I’ll put the scraper for that into github). And here, for balance, are some Esri crisis maps. Because I just downloaded their WordPress plugin, and it’s, kinda, playtime.

Data Science

Processing all teh Files in Directory

[Cross-posted from ICanHazDataScience] Okay. So we’ve talked a bit about getting set up in Python, and about how to read in different types of file (cool visualisation tools, streams, pdfs, apis and webpages next, promise!).  But what if you’ve got a whole directory of files and no handy way to read them all into your program. Well, actually, you do.  It’s this: import globimport os   datadir = “dir1/dir2” csvfiles = glob.glob(os.path.join(datadir, ‘*.csv’))   for infile_fullname in csvfiles: filename = infile_fullname[len(datadir)+1:] print(filename) That’s it.”os.path.join” sticks your directory name (“dir1/dir2”) to the filetype you’re looking for (“*.csv” here to pull in all the CSV files in the directory, but you could ask for anything, like “a*.*” for files starting with the letter “a”, “*.xls” for excel files, etc etc).  “glob.glob(filepath)” uses the glob library to get the names of all the files in the directory.  And “infile_fullname[len(datadir)+1:] ” gives you just the names…


Sensor Shopping

[Cross-posted from] Here’s the list of items that should arrive at home soon: a basic sensor set to supplement the Geiger counter, spectroscopes, cameras, microcontrollers (Arduino and RaspberryPi), accelerometers, temperature, IR and range sensors in my toolkit. I’m most excited about the dust sensor because it’s it was a component in Matt Schroyer’s DustDuino sensor (as seen on that’s being trialled in the developing world by Internews’ Earth Journalism Network… will be interesting to see what I can get out of it.


Sensors in Weather Reporting

“We all know that the weather with which the barometer sympathises, is considered to consist of three independent variables – the velocity of the wind, its temperature, and its dampness. It is a question how far the direction of the wind need be reckoned as a fourth distinct influence” – Francis Galton (first weather reporter)  [Galton1870] A Little History Weather predictions date back millennia, to at least 4th Century BC Babylonians, and recorded weather measurement, on which forecasts are made, date back hundreds of years, to the Central England Temperature series, which was collected by amateurs and has continued to be recorded since 1659 [Saner 2007]. Figure 1 First Weather Report, 1875 Figure 2 1861 Weather Report with Symbols Weather reporting in the media dates back to 1875 with Francis Galton’s weather observation maps in The Times (above, with Galton’s 1861 map using symbols); radio broadcasts of weather information started in 1916 at the…


What is a Sensor?

“Sensor. Noun. A device that detects or measures a physical property and records, indicates, or otherwise responds to it.” – Google A sensor detects physical variations in the world, e.g. light, temperature, radiowaves, sound, magnetic fields, vibration, particles (e.g. pollution, radiation) or objects (e.g. water droplets). Humans contain Sensors Humans and other creatures contain sensors: eyes, ears, nose, tongue, skin.  Humans are very good general-purpose sensors: They detect noise over a wide set of frequencies, both with their ears and with the rest of their bodies (e.g. the deaf percussionist Evelyn Glennie accurately tunes her drums by ‘hearing’ with different parts of her body) Their visual systems see the world in high resolution at roughly 60 images per second (which is why films, games and light fittings are updated at the rates that they are), in stereo that gives decent depth and motion detection (although optical illusions mess with these beautifully) And similarly for smells, tastes and skin. These sensors are sometimes used in sensor…