Ruby day 9: Local power!

OSM Changes for Typhoon Ruby
OSM Edits for Typhoon Ruby

This. Just this: local mappers made more changes to the map of the Philippines during Typhoon Ruby than anyone else in the world (by a very very big margin). Anyone who doesn’t believe in the strength of local people to build their own resilience should look very, very hard at these numbers.

Ruby’s all over now for the mappers – DHN is de-activated, everyone’s gone back to work.  There’s still a lot of work to do on the cleanup: MarkC mentioned 35000+ houses destroyed and 200000 people without shelter, and there will still be OSM mapping to do for that.  This weekend Celina’s running a “train the trainers” OSM event in Manila: if you’re one of the people who created the figures above, please please go and help spread your skills further!

Ruby day 8: Next

Ruby has gone now – “goodbye Ruby, Tuesday” is apparently becoming a popular song here.  But the cleanup work is only just starting.  Celina spends a lot of the day trying to UAV stuff sorted out; we get word that the team is getting imagery in the worst affected areas, and she works on getting that data stored and back to the mapping groups that need it.

Response teams are moving into the field – many by boat because they can’t get flights. Requests still come in – one for an assessment of damage to communications and media stations (I suggest that Internews might have a list for this, then find that Agos is tracking communications outages. I stop with the lunchtime work and get back to the day job.

ACAPS puts out an overview map but still need a rainfall map for it. Maning from the Philippines observatory just happens to have one (he’s been working on this for a while).

6pm: A government sit rep has data in it that needs scraping: I get out the pdf tools. Cometdocs dies on it, so I move over to PDFTables. We will slowly teach governments to release tables as datasets not pdfs. We will.  The ACAPS HIRA work starts. First, I have some pdfs to scrape. Except the sitreps from http://www.ndrrmc.gov.ph are PDFs with images of excel spreadsheets cut-and-pasted in. Start reaching out, trying to find out who has the original datafiles, but rein in when gently reminded that these things are political.

Connecting Ushahidi data to the HDX repository

I’ve been talking to the HDX team for some time now (well, since before HDX was a thing, but then so have many of us).  HDX is a data repository for humanitarian data: basically, it’s a place to put machine- and human-readable datasets so that other people can use them too.

Ushahidi tools (Ushahidi platform instances, Crowdmap instances) often have datasets in them that could be useful to other people, so part of the conversation has been about how to share data from Ushahidi sites, both on the HDX site and in the HXL humanitarian standard.

Ushahidi CSV to HDX CSV

First let’s look at how to share the CSV file that Ushahidi creates when you click on the “download reports” button. Before you do this, please, please, please read my post about mitigating potential risks to people from sharing your Ushahidi data.

Converting that CSV into HXL format is pretty easy: it’s as simple as adding a row to your CSV file.  Under the heading row (“#”, “INCIDENT TITLE”, ”INCIDENT DATE”, “LOCATION”, “DESCRIPTION”, “CATEGORY”, “APPROVED”, “VERIFIED”, “LATITUDE”, ”LONGITUDE”), add another row with the HXL tags (#x_uniqueid, #x_reporttitle, #report_date, #loc, #x_description, #x_category, #x_approved, #x_verified, #lat_deg, #lon_deg).  Erm… that’s it.

Loading the CSV file into HDX is also pretty easy: you go to HDX, create an account (and an organisation, if you need to), then click on the ‘upload data’ button and follow the instructions.

Ushahidi API to HDX CSV

I did this recently for some Ushahidi location data (I used the API to pull out all latitude-longitude-placename rows from the Ushahidi instances used for the last Philippines typhoons, then wrote them to a CSV which I cleaned up a bit before posting manually to HDX).

If you can write (or run) Python, you can access a Ushahidi Platform API and get all the reports (or locations or categories etc) from an Ushahidi Platform instance – IF that instance has turned on its API (most have). This allows you to do more complex things, like get data from multiple Ushahidi instances, process it a bit (e.g. removing all those repeat locations) or merge it with other data (e.g. Pcodes).

Geeky bit (non-coders, look away now): This API to HDX transfer could be done as a 1-click button on the reports page, if we add an “HDX” plugin to the Ushahidi Platform.  The design for the plugin is basically: a) allow users to add their HDX settings using the plugin’s “settings” page, and b) add a  button to the reports page, and use the existing Ushahidi CSV creation code to create a CSV file, then push that file through the HDX API using the settings that the user’s already added. This would be a pretty sweet Ushahidi plugin for a community coder to write: it’s well-defined, and most of the code needed is already there.

Ushahidi API to HDX API call

HDX not only allows you to upload datasets, it also allows you to add a link to an API, so HDX sees the current information from a site (not a snapshot that you’ve turned into a CSV file).  You can do this by adding one of the Ushahidi API calls to the HDX “link” field, e.g. if your site is called www.yoursite.com, 

Again, please please please read my notes about mitigating potential risks to people from sharing your Ushahidi data before doing any of these.

HDX to Ushahidi

The HDX datastores include files in CSV, Json, XML and Shapefile formats.  You can import Ushahidi reports data from HDX into an Ushahidi instance (but be sure you have permission to do this before you do); you can also upload shapefiles as layers (converted to WMS or KML format: the ogr2ogr tool can do this for you) into an Ushahidi site.

Helping out with HDX and Ushahidi

HDX is an open-source project (https://github.com/OCHA-DAP/): if you’re a Python coder, please consider helping out with its  codebase.

The HDX team are also working on unlocking useful data that’s still in PDFs. If you want a small task to help them with that, try this: unlocking humanitarian planning data.

The Ushahidi Platform is also open-source (http://github.com/ushahidi): if you’re a Php coder, please consider helping out there too.  The plugin specification above could be a nice place to start…

Design: Knowing what you want to build and why

Cross-posted from http://icanhazdatascience.blogspot.com/.

I recently taught a coding course for international development students (yes guys, I’m still marking your assignments). But teaching coding isn’t useful in itself – it’s like teaching someone a new language without understanding what they’d want to communicate in it – so woven throughout the course was a design exercise, so students would end the course with a design for a system they wanted to build, and a tangible use for their new skills.

We focussed on user experience (UX): the process of designing a system from the user’s point of view, rather than the coder’s assumptions about what the user might like. And specifically on a set of UX tools that together define a system:

  • personas
  • user journeys
  • sitemaps
  • wireframes
  • graphic designs
  • user stories

I pointed students at two great resources on this: the beginner’s guide to UX and UX Apprentice, but here’s a quick run-down on the basics.

Design tools

If you’re doing visual designs (wireframes, site maps etc), pen and paper is both free and easy to edit, but not so good at storing and sharing edits between people electronically. Better tools for that are Balsamiq (free for students) and Pencil (less powerful, but free for everyone).  And Trello is a good tool for user stories.

Top-level design

The top-level UX elements are personas, user journeys and sitemaps.

Personas are examples of each type of user you expect to interact with your system. They’re used to get to know the people who will use your system, understand the problem that they’re trying to solve with it, and how people already solve that problem.

User journeys (aka scenarios: they’re almost the same thing) describe how the user will solve their problem.  A user journey contains elements including

  • Context – where?
  • Progression – what are the steps in this journey?
  • Devices – mobile? laptop? smartphone?
  • Functionality – what do they expect?
  • Emotion – angry? tired? scared? time-pressured

A sitemap lists all the pages on your site, and how they’re related to each other. Sitemaps can be generated by a “card sort”, where you write all your system functions (or user stories) on post-its, then sort them into groups.

Page design

The UX elements for individual pages on your site are wireframes, mockups and graphics.

A wireframe is a deliberately rough sketch of what’s on the page. It’s deliberately rough to force your stakeholders (users and customers) to concentrate on *what’s* on the page, not what colour it is, which images you’re using etc.  You might see the text “lorum ipsum” on mockups: this is standard ‘nonsense’ text used by designers, and can be found on sites like http://www.lipsum.com/.

A mockup is more detailed, and used as a proxy for the completed page in exercises like talking potential users through the functions of a site. Clickable mockups are good for this (Balsamiq wireframes are clickable, as in you can click a button on one wireframe and the code will display the page that the button goes to), as are printouts of mockups if you don’t have a machine available.

Graphics are the look-and-feel of your site, its branding and things that make it seem ‘familiar’ to your users.  This includes the color palette you use, fonts, shapes and images (icons, header images, backgrounds etc). Not everyone who codes can design graphics, so you main choice is either to find someone who can design and build all the graphic elements, or use packages of ready-made graphical elements like Bootstrap or Foundation with tools like DivShot.

When you design your pages, it helps to look at similar websites, and ask yourself what you do and don’t like about them, then consider that in your designs.

Back-end design

The above has been all about designing the front-end of a system, e.g. the things that a user can see and the interactions the user has with them.  Back-end design (e.g. algorithm design) is a different beastie, and one to be covered later.

Project management design

Design elements used for communicating with a coding team (including back-end coders) include user stories, kanban and minimum viable products.  These are the foundation of “agile development”, a process designed around building parts of each system quickly in “sprints” (short cycles of coding and feedback) so you can get user feedback on them (and adapt your design to that user feedback) as you go.

User stories are used for planning code sprints, and look like this: as a <role> I want to <goal> in order to <benefit>

Kanban charts are those cards (or post-its) on the wall that you see in every photo of a modern software company.  They usually have cards organized into 5 columns: backlog, ready, coding, testing and done, with a user story or task on each card. An agile project manager (“scrum master”) will pick the most important user stories on their list, and add them to the kanban chart at the start of each code sprint. As programmers write code for each user story, the story will move from left to right across the chart until it ends up in the “done” column.

A minimum viable product (MVP) is the smallest system that you can build and still get useful feedback on. You build this, get feedback, adapt your design and repeat.

That’s it: a basic run-through of design tools for coding.  Now go out there and design something!

Ruby Day 7: Back to Work

7am. Today is a work day, and my deal with work was that I’d be working-from-Philippines, not bunking-off-to-do-disasters, so I’ll only be popping in and out of the chats and here from now.

Jean Cailton from VISOV (French VOST) has popped up overnight: many VOSTies are online working under the flags of other groups, which is kinda normal for mappers. Someone asks for rainfall data: the image a couple of days ago was based on TRMM data, so I wonder if NASA has daily updates to share (I already have scraper code for this from another project, and I remember that Lea Shanley is working on communities there now): Maning points out that the data is updated 3-hourly; is good. SBTF data is going to be posted on the Rappler map. And ACAPS puts out their first briefing note: volunteers are working with them, gathering secondary data (news, data etc) for the notes.  Teams have all settled down into a rhythm, so I don’t feel too bad about having to drop out of helping for the day.

11am. Eat breakfast, with triple espressos all round (twice). Stock up on boko juice (coconut water) and head back to work.  Typhoon is winding down:  it’s now rated as a severe tropical storm (yawn worthy in this land of 20 typhoons a year), Borongan airport is open to receive disaster response teams and relief; Guiuan airport is being cleared, and “national roads affected by #RubyPH are now passable” (DWPH). Rappler people are reporting from some of the typhoon-hit areas (Dagami Leyte, Calarbazon, Camarines Sur) and connected to SBTF team.  Investigate flights to Dumaguete tomorrow, to catch up with some old friends (and work from local coffeeshop).

Do dayjob work today. Spend lunchtime being massaged: hunch over laptop 20/7 for a week has left me with two shoulder knots so big that I’ve given them names.  Try to book flight to Dumaguete: tomorrow’s flights get switched from airline to an online reseller whilst I’m trying to book, and the reseller’s site is down. *sigh*. Book flight for Wednesday – I haven’t come all this way *not* to check in on friends here.  It rains – drizzles – all day; at 5pm the sky is dark enough to need lighting on indoors, but there’s nothing more than a little light rain and wind going on outside. Skypechats are now silent, save for the occasional person coming in to start volunteering. Turning up 3 days later isn’t cool dudes: the volunteer help is usually needed as a disaster hits, not afterwards.

9pm.  Ruby isn’t quite done with me yet: find myself in another Skypechat, this time the one for helping ACAPS gather the background data needed for future event reports. Will be doing a bit each day on that for a couple of weeks, hopefully alongside lots of filipino volunteers too. It’s still raining a bit here in Manila, but if we didn’t know a typhoon <del><del>tropical storm was passing overhead, we really wouldn’t notice it.

Sharing community data

I’ve been thinking lately about how open data and community data fit together.  Actually, I’ve been thinking about it for a long time – since we launched OpenCrisis to try to get more data tools and ideas into the hands of crisismappers, and started the long work of trying to archive and share as much mapping data and information as we could. Here’s some first thoughts on restarting this work.

Some of the big questions that came up then, and continue to come up, are about ownership, risk and responsibility.  For instance:

  • Ownership. If a community of people add reports to a site, and that site also sucks in data from social media, who owns that data?  That question’s already been asked and answered in many open data and social media sites, often involving much work and lost data as licenses are transferred (see OpenStreetMap’s license moves, for example).  Having your mappers sign a contributor agreement, and having a data license statement on any dataset you put out is a good start here.
  • Risk. Is something we’ve also always dealt with. The types of information that mappers handle mean you need to do data risk analysis that covers population, mappers, organizations and leads.
  • Responsibility. If you make a dataset public, you have a responsibility to, to the best of your knowledge, skills and advice you can obtain, do no harm to the people connected to that dataset. That’s a big responsibility, and one that has to do a balancing act between making data available to people who can do good with it and protecting the data subjects, sources and managers.

This week, I used some Python code to share all the geolocations (latitude, longitude and location name) from 2013 Ushahidi instances for Philippines typhoons.  That was a pretty clear-cut example: it wasn’t a conflict situation, none of the locations were sensitive (e.g. personal homes or addresses of places like rape crisis centers that need to be kept secret), and the data was gleaned from public sites created by large NGOs and open data teams.  Life isn’t always that clear-cut.

Datasets: where and what

Some of the places that mappers hold data are:

  • Google spreadsheets (tables of information),
  • Skypechats (informal information transfers) and googledocs/emails (e.g. annotated highlights from a day’s Skype discussions),
  • OpenStreetMap
  • Micromappers data (often visible as google spreadsheets) and
  • Ushahidi instances (unfortunately afaik, there weren’t any Ushahidi instances updated for Typhoon Ruby, so I couldn’t compare the two sets of data).

Some of the data collected by those mappers includes:

  • Geolocation: Latitude and longitude for each location name in a list.  These are usually either a) found automatically by the platform using a gazetteer like Nominatim, b) input by site administrators, or c) input by the person submitting a direct report.
  • Direct reports: Messages sent by reporters or general public via SMS or web form.  These generally have the standard set of Ushahidi report entries (title, description, category list etc), but might also include custom form fields to capture items of specific interest to the map owner.
  • Indirect reports: Messages scraped from Twitter, Facebook etc.
  • Category lists: The categories that reports are tagged with; these lists are usually created by the site administrator.
  • API data: data input into a platform by another system, using the platform’s API. This includes sensor data, and datasets scraped and aggregated from another platform instance.
  • Media: Images, video, audio files added to a report by the reporter or an administrator.

Not all of this data is owned by the site owner.  For example, third party data (e.g. Twitter messages) has restrictions on storage and ownership that even with the original sender’s permission could make it illegal for you to distribute or keep on your site.

Who and why?

Open data is, by its nature, open, and it’s difficult to anticipate all the uses that people have for a dataset you release.  Some examples from experience are:

  • Academics – analysis of social media use, group dynamics etc.
  • People in your organization – for lessons learned reports, for illustration, for visualizations, for analysis of things like report tempo (how frequently did information arrive for processing, translation etc)
  • Data subjects – to check veracity of data about them, and to ask for data removal (sometimes under laws designed for this purpose, e.g. EU privacy laws). I haven’t seen this happen yet, but I really really want it to.

If you release a dataset to anyone, you have to assume a risk that that dataset will make its way into the public domain. We’ve seen too many instances of datasets that should have been kept private making it into the public domain (and, to be fair, also instances of datasets that should have become public, and datasets that have been carefully pruned being criticized for release too).  Always have the possibility of accidental release in mind when you assess the risks of opening up data.

How?

Sharing data shouldn’t just be about clicking on a “share” button. There are processes to think about, and risk analysis to do:

  • Ethical process: Assessing the potential risks in sharing a dataset; selecting which data you should and should not share. Alway think what the potential harms from sharing information from the deployment is, versus the potential good.  If you’re not sure, don’t share, but if you’ve checked, cleaned, double-checked and the risk is minimal (and ethical: you’re working with other people’s information here), seriously consider it. If it’s come from a personal source (SMS, email etc), check it. At least twice. I generally do a manual investigation by someone who already has access to the deployment dataset first, with them weeding out all the obvious PII and worrisome data points, then ask someone local to the deployment area to do a manual investigation for problems that aren’t obvious to people outside the area (see under: Homs bakery locations).
  • Legal process: choosing who to share with, writing nondisclosure agreements, academic ethics agreements etc.  You might want to share data that’s been in the media because it’s already out there, but you could find yourself in interesting legal territory if you do (see under: GDELT). In some countries, slander and libel laws could also be a consideration.
  • Physical process: where to put cleaned data, how to make it available.  There are many “data warehouses” which specialise in hosting datasets.   Data warehouses include the Humanitarian Data Exchange (HDX), which specialises in disaster-related data.  You can also share data by making it public on an Ushahidi instance (e.g. crowdmap), or by making data available to people on request. See crowdmap.com’s api and csv public download button.

Some of the things I look for on a first pass include:

  • Identification of reports and subjects: Phone numbers, email addresses, names, personal addresses
  • Military information: actions, activities, equipment
  • Uncorroborated crime reports: violence, corruption etc that aren’t also supported by local media reports
  • Inflammatory statements (these might re-ignite local tensions)
  • Veracity:  Are these reports true – or at least, are they supported by external information.

Things that will mess up your day doing this include untranslated sections of text (you’ll need a native speaker or good auto translate software), codes (e.g. what does “41” mean as a message?) and the amount of time it takes to check through every report by hand.  But if you don’t do these things, you’re not doing due diligence on your data, and that needs to happen.

Other questions that you might want to ask (and that could make your checking easier) include:

  • How geographically accurate does your data release have to be?  E.g. would it be okay/ better to release data at a lower geographical precision (e.g. to town level rather than street)?
  • Do you need to release every report?  Most deployments have a lot of junk messages (usually tagged unverified) – remember, the smaller amount of data you release, the less you have to manage (and worry about).
  • Would aggregated data match the need of the person asking for data?  e.g. if they just need dates, locations and categories, do you need to release text too?
  • Time. You might want to leave time after your deployment for the dataset to be less potentially damaging. When you release a dataset, you should also have a “data retirement” plan in place, detailing whether there’s a last date that you want that data to be available, and a process to archive it and any associated comments on it.

Further reading

There’s a worldwide discussion going on right now about how private individuals, groups and companies can release open data, and on the issues and considerations needed to do that. More places to look include:

Ruby day 6: clicking starts

Screen Shot 2014-12-07 at 2.43.02 PM
http://www.gov.ph/crisis-response/typhoon-ruby/#section-1

7am. Wake at 6:30 am – check self for hangover, then check overnight Skype / email traffic.  Andrej from UNOCHA has posted a whole pile of links to existing UNOCHA datastores (outside HDX), and a link to 2013 population estimates (up to now, we’ve only found 2010 figures).  Realise that there isn’t a long-list of datastores for the digital humanitarians (I’ve been sending links out, but they’re behind a group’s firewall).  And the HDX team now has a Ruby page at https://data.hdx.rwlabs.org/ruby.

The micromappers deployments have started (if you’re reading this and want something to do – that!).  I see a message that http://ourairports.com/countries/PH/ needs updating – Andrej has a link for that too.  Am asked to lead one of the remote mapping teams… point out that I might be a little short on internet soon.  One of the local mappers that the remote team has been worrying about has made it to a cellphone signal – he’s not only okay, he’s also volunteering in the regional disaster response office… yay, go local mappers etc!

This morning I’m planning to do a little micromapping (categorising incoming tweets) and scrape all the lat/long geolocation data from typhoon Pablo/Yolanda Ushahidi sites (I have code for this, will leave copy in github).  And drink lots of coconut water.

I do a little micromapping, classifying tweets.  This is still done by hand, even though the auto-classification algorithms are getting better with every disaster. The categories we’re sorting tweets into are:

  • Requests for Help / Needs – Something (e.g. food, water, shelter) or someone (e.g. volunteers, doctors) is needed
  • Infrastructure Damage – Houses, buildings, roads damaged or utilities such as water, electricity, interrupted.
  • Humanitarian Aid Provided – Affected populations receiving food, water, shelter, medication, etc. from humanitarian/emergency response organizations.
  • Other Relevant Information – Informative for emergency/humanitarian response, but in none of the above categories, including weather/evacuations/etc.
  • Not Informative – Not related to the Typhoon, or not relevant for emergency/humanitarian response.
  • N/A: does not apply, cannot judge – Not in English, not readable, or in none of the above categories.

I’m tempted to ask the micromapping team for a filter that removes any tweet with the word “pray” or “god” in it. Every time, this traffic happens: filipinos and americans start to pray; brits start with the dark humour.  At least I haven’t seen any spam on the hashtags yet – selling handbags and Ugg boots after a disaster seems to be a very popular online activity.

Start seeing requests from people who want to see what’s going on. I move list of websites about Ruby off a group site and into a HDX page. Do much annoying link-checking because urls don’t copy over.  Rose, who always has useful stuff to add, posts a list of typhoon landfall times (6 in total) in  PHT, EAT, GMT and EST times.  Timezones are both the digital humanitarians’ strength (always someone awake) and weakness (all those time conversions).

I hunker down with some Ushahidi API code.  I find a bug in my Ushahidi python code that’s been hanging around for a while: thank you Ruby, for supporting my dayjob. The code works (and is at https://github.com/bodacea/icanhaz/blob/master/rubytyphoon.py if you want to use it too) – the results are in HDX at https://data.hdx.rwlabs.org/dataset/geolocations-from-digital-humanitarian-deployments.

See lots of notes about missing aerial imagery; ask if anyone knows where SkyEye is operating today. Told to ask Celina (who’s asleep upstairs at the moment, exhausted).

10am. I learn stuff every time with these guys. Today it’s how easy reverse image searches are (photos from Haiyan are cropping up in the social media streams): use Google image search or tineye.com, or right-click in Chrome and use Search Google for this image.  My dayjob colleague Michelle checks in – she’s the only person there who’s asked how I’m doing, and none of my Ruby-related emails seem to have got through.  I guess they think I can look after myself.  Am doing what I can today because I’m back to the dayjob tomorrow – if you need anything from here, today is a good time to ask!

12pm. Brunch at Legaspi market. Hear people talking about the typhoon – calm, sounds more like gossip than panic.  Bump into a bunch of INGO folks who’ve just come out of their coordination meeting.  Eat great hot pad thai with homemade ginger beer; buy some christmas presents. Bump into agency person who needs some of the datasets that I’ve been helping with, and others from Andrej’s list this morning.

2pm. See yet more datasets hiding behind firewalls, some of them the data needed above.  Will keep trying to persuade people to share through HDX-linked googledocs, but watching replication between groups is annoying.  See that the HOT OSM instructions need a bit of gardening: try to create an account (I always forget my name and password there) but get Account creation error. Your IP address is listed as an open proxy in the DNSBL used by OpenStreetMap Wiki. You cannot create an account”.   I try to remember my account name by hitting “forgot password”, but get “Your IP address is blocked from editing, and so is not allowed to use the password recovery function to prevent abuse”.  Wonder how often that happens to people in developing countries. The HOT OSM team are wonderful about this, with suggestions of how to best deal with it.

New link comes in: Philippines government’s Ruby page. Google crisis map on it includes all the shelters across the islands (from DSWD data) – is beautiful, but can’t get KML download to work to cross-check list against ones acquired earlier (and take down old datasets as needed).  Someone asks about pdf to spreadsheet conversion software (please stop with the pdfs guys – please!); we recommend tabula for the easy and cometdocs for the difficult ones. I run the doc through cometdocs and upload to HDX just in case anyone else needs it (poverty figures for the Philippines).

4pm. Link field team with GPS locations up with OpenStreetMappers – sit back and enjoy watching them coordinate together. Check work chats – find some sweet comments from teammates on my note about Ruby, which is much heartening: perhaps I’m not forgotten after all (and to be fair, had forgotten about an offer of help there a day or two ago).  Download evacuation centre list from Google crisis map, then convert it from kml to csv (the command to use is “ogr2ogr -f CSV evacuation_centres.csv Typhoon\ RubyHagupit\ Evacuation\ Center\ Map.kml -lco GEOMETRY=AS_XY”; there’s a great ogr2ogr cheat sheet at https://github.com/dwtkns/gdal-cheat-sheet). Try to go for a nap.

Darn. Message from Redcross: Google map is missing evacuation centres on a couple of islands. Look over the 2 lists: there are other centres on NGO lists that aren’t on map either. Am now very sleepy: post notes in chat, add new shelter names to googledoc, drop note in chat for Google team and ask Jus to help fix it. Celina wakes up and starts a bridge between the areas that OSM is missing satellite data for and the local UAV team.  Finally go for nap.

8pm. Wake up to see the audit trail behind the DSDW shelters map in one of the chats: everyone’s contributed a part to it, and mysteries are solved. See a request for a list of towns/villages that the storm has passed through please; wonder if it’s possible to estimate that from a track+width (I know the storm walls are destructive, but not sure how far the damage area would extend; also see a request for road blockage data.  Maning points to photos on local news site (ABS-CBN); looks like home did after Sandy.

Darn again: Philippines government has UAV regulations amd not all local companies have got the licenses from CAAP to fly post-tornado (Andrej points at the UAV links on UNOCHA’s response page). Don’t tell me we have to break out the balloons and kites again? We check with the UAV teams: they’re cleared.

10pm. Pierre from OSM has the typhoon track on a map… off to turn that into a list of towns in a 50mile swath. Am exhausted and eyes are closing reading the Overpass API docs (although the test window for this is cool): post task in SBTF’s geolocation window for another mapper to pick up. Celina is talking to the UAV guys and mapping teams, getting lists of place that imagery is needed (damaged areas yes, but also areas that were covered by cloud in the satellite images).  Go off to bed after midnight, leaving Celina still hunched over her laptop.

Ruby Day 5: data day

http://reliefweb.int/sites/reliefweb.int/files/resources/TC_Hagupit_Impacts_05DEC14_1500UTC..pdf
http://reliefweb.int/sites/reliefweb.int/files/resources/TC_Hagupit_Impacts_05DEC14_1500UTC..pdf

6am Everyone was exhausted last night – crashed out, went to bed early.  Woke at 5am today to a note about sushi and coconut juice in the fridge. Ate supperbreakfast/breakupper/supfast/whatever and checked in on the Skypechats.  Lots of messages overnight: OSM team is organised and churning through mapping tasks; DHN is oriented, the SBTF is gearing up ready, some lovely new maps on Reliefweb and a few more links to add to Joyce’s online list but they’re slowing down in frequency now.

I map some residential roads and rivers on http://tasks.hotosm.org/. It’s time to start thinking about datasets. Maning is already working on shelter geolocation, someone’s repurposed the Yolanda geonode site, and I need to start checking through HDX for anything useful/ needed for Ruby.

Dayjob meets weekend: am asked about the OSM humanitarian style on Ushahidi V2. Start trying to remember what the workaround for the Ebola maps was. It was RobBaker writing a plugin: https://github.com/rrbaker/osm-humanitarian-tiles.  DSDW (the philippines agency that coordinates shelters etc) is working on Crowdmap; I check the plugins list there to see if Rob’s one has made it in. It hasn’t yet.

Josh from google person finder is looking for people in-country to test their SMS tool… yay! Am in country!  Does anyone else need something testing out here?

8am. Looks like a nice day out there – warm, slightly cloudy. Not a clue in the sky that there’s anything approaching.  Matchmake some Sahana-based design work.  Disasters aren’t the best time to try something new, but they can focus people’s minds beautifully on what’s needed for the next one.

11am. Starting to get little gusts of wind here. A local mapping group (Eric) makes contact via UNOCHA, but the remote mappers they need to talk to are all asleep. We have to get better at hyperlocal mapping – getting people the tools and techniques they need then getting the heck out of the way unless they need extra bodies or skills.  We all start micromapping the damage to trees from the last storm here (“come look at coconut trees int the Philippines – thank you, but I already saw a load from the car yesterday…) using imagery gathered by SkyEye, a great local UAV company. It’s frustrating, knowing that I could drive out and get corroborating imagery with my phone.  Results are being used as a training/test set for image classification algorithms at Simon Fraser Uni – is so good to see the bullshit “disaster application because it looks good in the publicity” work that used to happen in academia as cover for algorithm exploration or military work being replaced with genuine connections to people who actually need the help.

Dan checks in… he’s starting to worry about whether I’m safe; I say that right now it’s a lovely day and we’re all going to be fine.

4pm Normal life. Lunch at a hipster cafe that seems to have dropped intact out of midtown Manhattan.  Shopping at the market, visiting wine expert (margaux, yum), napping, spa.  Check in on Hagupit facebook page. (“We’re building a community of #RubyPH #Hagupit weather watchers and on-the-ground citizen journalists. Join up to share the latest info and meet people who want to make sure we’re all prepared!”).  Storm and power outage reports starting to come in (Tacloban, Dolores, Sulat, Calbayog).  Look through the SBTF micromappers deployment site.  More locals start connecting into the grid of international folks watching Ruby, including EngageSpark, who’ve been pushing out SMS messages.  I make a list of local crowd coordinators, but I’m not totally sure who to give it to.  There’s a new coordination skypegroup setting up – perhaps they all belong in there?

8pm. At dinner… everyone keeps popping away from the table to answer phones/emails and coordinate work.  See two datasets that should be out there for the digital humanitarians: evacuation centre list (messy, various forms, pdf) and expected storm surge heights in towns/ cities.  Convert both into PDF – team posts them up on HDX.  One of the dinner guests has a tech problem – field people using ODK on tablets who can’t get the data from them uploaded because they only have SMS available.  Ping the tech volunteer groups to see if anyone has an answer for them… Kate from OSM responds. Discover person with shelters list for 2 more areas – get promise of pdfs from them.