Category: Code for Good

FIXIT: Ushahidi location lists

Today, a conversation amongst crisismappers went something like: “when are we going to migrate to IRC?” (this in response to yet another Skype idiosyncrasy getting in the way of the team), then “not until we have a decent interface for it on all platforms” and “we asked the weekend hackathons for this, but it’s not sexy: you can’t tell people you’re saving the world, or helping starving children with it”.  That’s a whole pile of cynicism and frustration, but behind it are three things: 1) mappers still don’t have all the tools they need, and are relying on people-as-processors to get round that, 2) mappers don’t know how to ask for tools in ways that gets them what they need, and 3) hackathons may not be the best place to get those non-sexy tools built.

So. Options. Best is to find a good piece of existing open-source kit that almost meets mappers’ needs, and extend it with the things needed.  Less-good is to build all-new kit – mappers aren’t hackers, and new kit inevitably breaks and needs maintaining and training.  In-between is adding things onto proprietary kit using APIs (if they don’t get removed – yes, I’m looking at you, Skype team), and adapting existing open-source kit that doesn’t meet needs, but could be adapted (with the programming equivalent of a carefully-swung sledgehammer).  But that’s just options from the mapping teams’ perspective: another option is to team up with a coding company that wants to build tools for a similar (or adjacent) market.

I’m just as guilty for not documenting the things that I think need improving.  I’m used to writing “FIXIT” in my code when I know there’s a potential problem in it – so here’s the start of a set of posts about things that could be upgraded.  Some of them, I’ll start trying to fix – and document that in updates to each FIXIT blogpost.

 

ushahidi_find_location

There are lots of little things that bug me about the technologies I use in mapping.  One of them is repeating geolocation – specifically, having to find the same addresses multiple times in Ushahidi because I can’t upload a new list of addresses to the “find location” box (see above).  Now, I’m not going to go very far with this thought because it’s quite possible that someone in the Ushahidi community has already fixed this, but here’s the how-and-where (for Ushahidi version 2.7, which will be superseded sometime next year by 3.0):

  • When you press the “find location” button, Ushahidi calls (eventually) a function called geocode($address) in file application/helpers/map.php
  • This calls the google map api: for instance, for the address “Mount Arlington”,it calls http://maps.google.com/maps/api/geocode/json?sensor=false&address=Mount%20Arlington (don’t worry about the %20: it’s the internet’s way of saying “space”).  The API call produces json data (go ahead and look at the page – it’s all there), which Ushahidi then pulls the country, country id, location name, latitude and longitude from.
  • Erm. That’s it.

A geolocation team on a deployment is responsible for finding lat/longs for location names.  They usually keep a list of these lat/longs and location names, and that’s usually in a Google spreadsheet.   So what we need here is:

      • An Ushahidi variable holding the address of the geolocators’ spreadsheet.
  • That spreadsheet to be in a recognisable form – e.g. it has a column each for latitude, longitude and placename
  • A piece of code inserted into function geocode($address), that when the Google map API comes up blank, checks the Google spreadsheet for that location instead (or maybe checks the spreadsheet first: that depends on how the geolocation teams usually work). That piece of code will need to use the googledocs API, which is possibly the hardest part of this plan.
  • Maybe even (shock, horror), check the *other* map APIs (openstreetmap etc) too.

None of this is horrendous, but it does take time to do.  Perhaps someone somewhere will think that it’s worth it.

International Space Apps – view from the virtual room

The International Space Apps Challenge is happening this weekend, at cities, space stations and virtually, all around the world. The livestream for it is here: http://scrbliv.me/44762

Behind every global development hackathon is a team of people who act like glue, watching projects around the world and connecting up the ones that are related to each other, or can help each other or people who can just plain support each other when it’s dark and cold and they’ve discovered that there are limits to how long coffee and doughnuts can keep you awake for.

Up to now it’s down to how many connections and projects we can keep in our heads at once, and what we’ve got scribbled on paper and files all over our pcs, and how many streaming windows we can physically fit onto all our screens (because one laptop is never enough if you’re keeping tabs on a global event).

\"\"

But this time we’re spoilt. We have a list of projects that’s been moderated and sorted down to a mere 61. We have spreadsheets that show which teams (There are 81 of these at the moment, and about 1000-2000 people) in the 26 camps around the world are working on those projects (and we really mean around the world when we\’re talking about the international space station). And we have a piratepad for every project, so teams around the world (e.g. Jakarta, Oxford, Stuttgart and San Francisco on the Open Data API) can leave each other notes and hints and pointers to data and code.

 

\"\"

It’s working well. I have some favourite silly moments. Like seeing how the StartupBus moving-camps-on-a-map idea got adapted to the ISS moving across the map of camps for this hackathon. And the duck that tells when the ISS is overhead. And the tweet from a zucchini plant.

\"\"

And I have to give a big nod to the New York StartupBus Crew who came out in force when Mike Caprio (this year\’s New York conductor) asked them to.

\"\"

But mostly it’s the feeling that the multi-city hackathon has finally come of age. And pride that a lot of people around the world gave up their weekend to go make a difference to the world. And space. And beyond…

What is a hackathon?

I accidentally ended up organising a hackathon recently. RHOK NYC could have been a tragedy. I was too overloaded to help organize it, the local Crisiscommons lead was too busy, and the young man who stepped in to lead was inexperienced with hackathons and unsupported but managed to pull things together well until the fortnight before, when RHOK NYC lost its venue. Which is a big deal in New York – – they’re not easy to come by for a weekend event with a sleep-over (or rather a crash on the floor for a couple of hours in-between coding -over). Oh, and the young man was unexpectedly out of the country at another event.

So we cancelled, got talked back into trying again, and put out the call to the local volunteer technical community. The community answered us, in spades. Within a week, Phil from Open Plans found us a venue via Naomi and Beth at New York Law School; Josh from the UN Mission to UNICEF offered us space in their building, and Phil’s friends Danielle and Beth offered to take the whole RHOK NYC contingent wholesale into their Open Data Day hackathon on the upcoming US Farm Bill and its effects on agriculture. NY Hackers offered support, and JonMark from the StartupBus offered sandwich money and advice from his new position as Twilio’s developer evangelist.

We then had a week to go and no main organizer, so we took the hackathon-sharing option (where hardcore hackers met gourmet macrobiotic pizza, but that’s a story for another day). There’s a whole post of thankyous to be had (it’s on the RHOK website), bun in short it worked out really really well. And the most unexpected benefit came from mixing the Farm Bill people who’d never run or been to a hackathons before with our little band of hard-core techs and hackathons veterans (‘0’ in the previous hackathons box met ‘over 10’).

And I met Sarah from Oxfam who said “I had no idea what a hackathon was”.

Wow. After years of evangelizing, it’s easy to forget that for many people outside the tech community, this is utterly completely new (and perhaps a little terrifying too – props to Sarah for coming along anyway).

The short answer is that a hackathon strengthens your volunteer technical community and teaches you more about your specialist subject – whether that be how to build better iphone apps or distribute food aid around the world. And each others’ subjects too. My favourite moment was a Farmbill lady who’d come expecting hand-designed graphics walking by a RHOK hacker, saying “could you just do a scraper for (some random farming subject) for me please?”, to be met with “sure, it’ll take about 10 minutes”. Transfer of skills and information – it’s beautiful to watch.

The long answer is that eventually hackathons run like this. A lot goes on before a hackathon: finding sponsors, finding a venue, finding subject matter experts, advertising the event to potential attendees, organizing catering, sorting security, planning. But on the day (or weekend),  typically:

  • Everyone turns up , drinks coffee and chats with other attendees (don’t worry – hackers don’t bite, and we generally have a good sense of humour. Even at 1am when our computer crashes and loses our code). Then the hacking starts.
  • Subject matter experts (people who understand the problems you’re trying to address with the hackathon) come up one by one to describe a problem that they want to work on, or a system they want to build, or even their area in general (not all great hacks are pre-determined).
  • People gravitate to the problems that interest them to form teams.
  • Then spend the day/weekend working on the problem: building designs, code, visualisations using whatever skills, data, code and knowledge they can glean (problem providers: please have someone on standby for the hackathon, even if they’re at home – it’s frustrating to make design decisions without a user around, and it’s rude not to help the hackers once you’ve asked the community for help yourself).
  • And at the end of the day/weekend, each team stands up and presents their work. There might be prizes, but usually the biggest prize is to go home knowing that you’ve touched the world in some way. Oh, and know (and have been through a hard day/night with) a lot more interesting people.

Your first hackathon may be different. In your first hackathon, you generally turn up and learn what a hackathon is, by working with the other participants. That learning takes time, so don’t beat yourself up if you’re not instantly brilliant at the first one you go to.

I was privileged to have been there when a whole subject area – food – went through their first hackathon together. They got it, they did brilliantly, they had some great ideas I hadn’t seen before here (like micro-lectures in a room off to the side) and Sarah not only learnt about hackathons and how to create problem statements for them, she also (with her team) won a prize. I’m glad I was there.

Global groups

And so onto the global groups that I’m tracking or part of.

This is only the start of a list that I’ll add to when I remember all the job-related groups that I’m involved in.

New York tech groups

I’ve been talking to quite a few people as part of the dayjob.  And I knew I was talking to quite a few communities, but even I was surprised when our communications officer asked just how many, and we counted over 100.  Which since I’ve only been doing this since January is rather a lot.  So first, if you’re part of a community that I talk to, please bear with me if I seem a little distracted. And second, part of the job is to create stronger links between these communities and the UN, so I thought I’d write about who I was talking to and why, to give other UN peeps some possibly-needed leads.

This is going to take a while, so I’ve split it into three posts: international communities, local New York communities and UN projects and communities.  Later on, there are academics, companies and conferences, but for now the focus is on groups.

So. Starting with the New York communities (and this is a post that would be appropriate for each PulseLab).

I found these communities through personal contacts (it helped to have shared an apartment with a major community member – thank you John, and thanks to the crisismappers for introducing us), and by searching meetup.com.  Garys Guide was also useful, as was searching eventbrite and the New Work City and General Assembly events and classes pages. But after a while, most useful leads come from people I meet.

Community events happen, usually in the evenings, at tech company buildings all over the city, but there are two co-working spaces that are very active at hosting community events and well worth a mention: New Work City and General Assembly.  I’ve been a member of both of these – for a once-weekly desk space at New Work City, and for community notices and some excellent tech training classes at GA.

For algorithms, I go to:

  • Hacks/Hackers New York – hackers and journalists together: a very active group, always with interesting ideas
  • NYC Predictive Analytics – applied machine learning and big data
  • NYC data nerds. Not so much a group, as an informal data drinking club, brought together by Drew Conway and Hilary Mason. We’ve also heard about a Sunday-morning data nerds’ brunch club by the New York Times building, but we haven’t managed to track them down yet.
  • Data Without Borders – not so much a meetup as a movement that runs hackathons and projects, but a good one worth watching.

For programming, I go to:

  • NY Hacker – the local coders group. Holds monthly townhalls and weekly drop-ins
  • NY Tech – the big meetup for NYC technologists, hosts huge meetings that I haven’t managed to get to yet.
  • NY Ruby, nyc.rb and NYC on Rails – the Ruby programming language. I also like Ruby Nuby, a group that meets to train people in Ruby.
  • NY Python and Django-NYC – the Python programming language
  • NY R – the R statistical programming language (useful for big data)
  • HPC & GPU Supercomputing Group NY – these guys can run up some awesome computing resources very very quickly
  • NY Hadoop user group – one of the more widely-used big data frameworks
  • NY Scala – a newish framework that I’m tracking as potentially useful to us in the future

For development (as in human development) technology and Open Data:

  • NY Tech4Good
  • OpenNY – our local Open Data group, and a strong part of the New York open data initiatives.
  • Volunteer Coders – doing some great work on social projects.
  • CrisisCampNYC – runs a monthly social that gets the local crisismappers away from their keyboards for a while.

For hardware:

And for sanity, I go to the NY CTO Club.

I will have missed some groups off that list, some of them because they’re not quite up and running yet. It’s not that I don’t love you, it’s just that I’m tracking a lot of groups here – so please please tell me about anyone active that I’ve forgotten.