This is the detailed post to go with yesterday's quick discussion about proximity search. All the code is available on GitHub.

This assumes a bit of NodeJS knowledge, a working copy of homebrew or something similar.

Install

• MongoDB - brew install mongodb
• NodeJS
• NPM (included in NodeJS installer these days)

These are included in the package.json but it can't hurt to mention them here:

• npm install twitter (node twitter streaming API library)
• npm install mongodb (native mongodb driver for node)
• npm install express (for convenience with API later)

Start mongod in the background. We don't quite need it yet but it needs done at some point, may as well do it now.

Create a Twitter App

Fill out the form Then press the button to get the single-user access token and key. I love that Twitter does this now, rather than having to create a full authentication flow for single-user applications.

ingest.js

(open the ingest.js file and read along with this bit)

Using the basic native MongoDB driver, everything must be done in the database.open callback. This might lead to a bit of Nested Callback Fury but if it bothers you or becomes a bit too furious for your particular implementation, there are a couple of alternative Node-MongoDB modules that abstract this out a bit.

// Open the proximity database
db.open(function() {
    // Open the post collection
    db.collection('posts', function(err, collection) {
        // Start listening to the global stream
        twit.stream('statuses/sample', function(stream) {
            // For each post
            stream.on('data', function(data) {
                if ( !! data.geo) {
                    collection.insert(data);
                }
            });
        });
    });
});

Index the data

The hard work has all been done for us: Geospatial Indexing in MongoDB. That's a good thing.

Ensure the system has a Geospatial index on the tweets.

db.posts.ensureIndex({"geo.coordinates" : "2d"})

Standard Geospatial search query:

db.posts.find({"geo.coordinates": {$near: [50, 13]}}).pretty()
(find the closest points to (50,13) and return them sorted by distance)

By this point, we've got a database full of geo-searchable posts and a way to do a proximity search on them. To be fair, it's more down to mongodb than anything we've done.

Next, we extend the search on those posts to allow filtering by query

---

db.posts.find({"geo.coordinates": {$near: [50, 13]}, text: /.*searchterm.*/}).pretty()

API

Super simple API, we only have two main query types:

• /proximity?latitude=55&longitude=13
• /proximity?latitude=55&longitude=13&q=searchterm

Each of these can take an optional 'callback' parameter to enable jsonp. We're using express so the callback parameter and content type for returning JSON are both handled automatically.

api.js

(open the api.js file and read along with this bit)

This next chunk of code contains everything so don't panic.

db.open(function() {
  db.collection('posts', function(err, collection) {
    app.get('/proximity', function(req, res) {
      var latitude, longitude, q;
      latitude = parseFloat(req.query["latitude"]);
      longitude = parseFloat(req.query["longitude"]);
      q = req.query["q"];

      if (/^(-?d+(.d+)?)$/.test(latitude) && /^(-?d+(.d+)?)$/.test(longitude)) {
        if (typeof q === 'undefined') {
          collection.find({
            "geo.coordinates": {
              $near: [latitude, longitude]
            }
          }, function(err, cursor) {
            cursor.toArray(function(err, items) {
              writeResponse(items, res);
            });
          });
        } else {
          var regexQuery = new RegExp(".*" + q + ".*");
          collection.find({
            "geo.coordinates": {
              $near: [latitude, longitude]
            },
            'text': regexQuery
          }, function(err, cursor) {
            cursor.toArray(function(err, items) {
              writeResponse(items, res);
            });
          });
        }
      } else {
        res.send('malformed lat/lng');
      }

    });
  });
});

If you've already implemented the ingest.js bit, the majority of this api.js will be fairly obvious. The biggest change is that instead of loading the data stream then acting upon each individual post that comes in, we're acting on URL requests.

app.get('/proximity', function(req, res) {

For every request on this path, we try and parse the query string to pull out a latitude, longitude and optional query parameter.

if (/^(-?d+(.d+)?)$/.test(latitude) && /^(-?d+(.d+)?)$/.test(longitude)) {

If we do have valid coordinates, pass through to Mongo to do that actual search:

collection.find({
  "geo.coordinates": {
    $near: [latitude, longitude]
  }
}, function(err, cursor) {
  cursor.toArray(function(err, items) {
    writeResponse(items, res);
  });
});

To add a text search into this, we just need to add one more parameter to the collection.find call:

var regexQuery = new RegExp(".*" + q + ".*");
collection.find({
  "geo.coordinates": {
    $near: [latitude, longitude]
  },
  'text': regexQuery
}

This makes it so simple, making it it kind of feels like cheating. Somebody else did all the hard work first.

App.net Proximity

This works quite well on the App.net Global Timeline but it'll really become useful once the streaming API is switched on.

Of course, the code is all there. If you want to have a go yourself, feel free.

Now that geolocated posts are beginning to show up around app.net, I found myself wondering about proximity search. Twitter provides one themselves for geotagged tweets. What a proximity search does, essentially, is provide results from a data set ordered by increasing distance from a given location. This can be further enhanced by combining it with a text search either before or after the distance sorting. This would give you a way to search for a certain query within a certain area.

Here, for example, is a proximity search centred on Berlin including any tweet in 1000 miles containing the word 'toast'.

When I first started thinking about the tech required for a proximity search, I remembered Lukas Nowacki back in our old Whitespace days implementing the Haversine formula in MySQL (Alexander Rubin has a good overview of how to do this). As much as I love my trigonometry and logarithms, I must admit, I was looking around for a simpler solution. Actually, I was looking around for a copy-paste solution, to be honest. I may even have spent some time going down that route if Max hadn't pointed me in the direction of MongoDB.

I'd been putting off digging into NoSQL databases for a while because, well, I had no real reason to. Recently, I've either been focused on front-end dev or hacking away at Java and never really had any good reason to investigate any of these new-fangled technologies get off my lawn you kids.

MongoDB

After 10 minutes of messing around with Mongo, I pretty much just found myself saying “No... way. There's no way that's actually working” I'm sure those of you experience with document-oriented databases are rolling your eyes right now but for those few of us left with an entirely relational concept of databases, let me just explain it like this: you know those things you want to do with a database that are just a hassle of multiple joins and confusing references? Document databases do some of those things really really well.

The biggest selling point for me, however was the native geospatial indexing. That pretty much made the majority of my proximity search complete. All I needed to do was wrap it in a nice interface and call it a day...

I'll follow up tomorrow with a more detailed 'How-to' guide.

After Twitter started announcing changes to its API usage and people started to get worried about the future of the developer eco-system, App.net appeared. It provides a streaming data-platform in much the same way Amazon provides a web hosting platform. The reason for the Twitter comparison is that one of the things you can make with it is a Twitter-like public short message system. This was, in fact, the first thing the App.net developers made to showcase the platform: Alpha although that example seems to have convinced many people that the whole point was to build a Twitter-clone instead of a service whose purpose is to stream small, discrete blocks of meta-tagged data. The community-developed API spec is an important aspect of App.net as well although feature discussions can devolve into musings on Computer Science theory a bit too easily.

For the first couple of weeks, it was fun to just hack around with the APIs, post a bit, build a little test app (disabled now that more App.net clients have push notifications). It all became much more interesting, however, when two new features were added to the platform – Machine-only posts and Well-known Annotations.

Machine-only posts

Pretty much exactly what they sound like. These are posts that are not intended to be viewed in any human-read conversation stream. They still get pushed around the network exactly the same as all other posts but to see them, you must explicitly say 'include machine-only posts' in your API calls. For developers who have been building silly toys on Twitter for a couple of years, this is fantastic. You don't need to create a separate account purely for data transfer. This can be part of your main account's data stream. I have quite a few Twitter accounts that I just use for outputs from various applications. I have one, in fact, that does nothing but list the tracks I listen to that I created for this side-project.

By classifying these as 'machine-only', they can stay associated with the user and subject to whatever privacy controls they have set in general. This makes it far easier for the user to keep control of their data and easier for the service to track accurate usage. For devs, it also means you can hack away at stuff, even if it is to be public eventually, without worrying too much about polluting the global stream with nonsense.

Well-known Annotations

Annotations were part of the spec from the beginning but only implemented at the end of August. Annotation is just another word for per-post meta-data – each post has a small object attached which provides extra information. That's it.

The annotations can be up to 8192 bytes of valid JSON which is quite a lot of meta-data. Even with this, however, the usefulness is limited to per application use-cases until some standards start to appear. There's nothing to stop a popular application being the one to set the standard but for the more general cases, it is most fitting to continue with the community-led development model. This is where Well-known Annotations come in.

Well-known annotations are those attributes which are defined within the API spec. This means that the community can define a standard for 'geolocation', for example, and everyone who wants to use geolocation can use the standard to make their application's posts compatible with everybody else's.

Obviously, I'm quite into my geolocated data. I love a bit of map-based visualisation, I do. Here's a sample of jQuery that will create a post with a standard geolocation annotation:

$.ajax({
  contentType: 'application/json',
  data: JSON.stringify({
    "annotations": [{
      "type": "net.app.core.geolocation",
      "value": {
        "latitude": 52.5,
        "longitude": 13.3,
        "altitude": 0,
        "horizontal_accuracy": 100,
        "vertical_accuracy": 100
      }
    }],
    machine_only: true
  }),
  dataType: 'json',
  success: function(data) {
    console.log("Non-text message posted");
  },
  error: function() {
    console.log("Non-text message failed");
  },
  processData: false,
  type: 'POST',
  url: 'https://alpha-api.app.net/stream/0/posts?access_token=USERS_OAUTH_TOKEN'
});

In the same way as the machine-only posts, these annotations aren't provided on a default API request, you have to specifically ask for them to be included in the returned data. This is to make sure that the majority of use cases (public streaming, human-readable conversation) don't have to download up to 8KB of unnecessary data.

Retrieving both

This is an API call to retrieve posts marked machine-only and with annotations

Potential use cases

You might have noticed, the API call above had the machine-only attribute as well as the well-known geo annotation. If I wanted to create an app that would run on my phone and track my routes throughout the day, all I would need to do would be to run that $.ajax call periodically with my current geolocation. The data would be saved, distributed, streamed and could be rendered onto a map or into a KML at the end of the day. I could record hiking trails or museum tours, or share my location with someone I'm supposed to be meeting so they can find out where I'm coming from and why I'm late. That's just a couple of the single-user cases. Having a shared standard means that the potential for geo-tagged posts opens up to at least equal that of Twitter's. Heatmap-density diagrams showing areas of the most activity; global and local trends; location-based gaming. Add a 'news' annotation to geotagged posts and suddenly you've got a real-time local-news feed. Add 'traffic' and you've got community-created traffic reports.

There are so many clever things you can do with location-tagged data. I hope others are just as enthused about the possibilities as I am.

I'm almost managing to keep to my intended schedule of one map-based web experiment per week. Unfortunately, I've mostly been working on internal Nokia Maps projects over the weekends recently so I've not had much to post here.

I can share my latest toy, though: CoverMap.me

Using just the public APIs over a couple of hours last Sunday afternoon, I made this to allow you to set a Nokia Map as your Facebook Timeline cover. The whole process is really straightforward so I thought I'd go over the main parts.

The exact aim of CoverMap.me is to allow the user to position a map exactly, choose from any of the available map styles and set the image as their cover image.

Make a Facebook App

Go to developers.facebook.com/apps/ and click 'Create New App', fill in the basic details – name of the app, URL it will be hosted on, etc – and you're done.

Facebook login

I've used the Facebook JS SDK extensively over the summer for various projects but I wanted to try out the PHP one for this. Super, super simple. Really. Include the library (available here), set your appId and secret and request the $login_url.

$facebook->getLoginUrl(array('redirect_uri' => "http://example.com/index.php"));

That will give you a link which will take care of logging the user in and giving you basic access permissions and details about them.

Nokia Maps JS API

When I'm hacking together something quick and simple with the Nokia Maps API, I usually use the properly awsm jQuery plugin jOVI written by the equally awsm Max. This makes 90% of the things you would want to do with a map extremely easy and if you're doing stuff advanced enough to want the extra 10%, you're probably not the type who'd want to use a jQuery plugin, anyway. Either way, you need to register on the Nokia developer site to get your Nokia app_id and secret.

To create a map using jOVI, simply include the plugin in your page then run .jOVI on the object you want to contain the map along with starting parameters:

$(window).on('load', function() {
  $('#mapContainer').jOVI({
    center: [38.895111, -77.036667], // Washington D.C.
    zoom: 12,           // zoom level
    behavior: true,     // map interaction
    zoomBar: true,      // zoom bar
    scaleBar: false,    // scale bar at the bottom
    overview: false,    // minimap (bottom-right)
    typeSelector: false,// normal, satellite, terrain
    positioning: true   // geolocation
  }, "APP_ID", "APP_SECRET");
});

This gives us a complete embedded map.

As I mentioned above, part of the idea for CoverMap.me was to allow the to choose from any of the available map styles. This was an interesting oddity because the public JS API gives you the choice of 'Normal', 'Satellite', 'Satellite Plain' (a.k.a. no text), 'Smart' (a.k.a. grey), 'Smart Public Transport', 'Terrain' and 'Traffic' while the RESTful Maps API (the API that provides static, non-interactive map images) supports all of these plus options to choose each of them with big or small text plus a 'Night Time' mode. Because of this, I decided to go for a two-step approach where users were shown the JS-powered map to let them choose their location then they went through to the RESTful static map to allow them to choose from the larger array of static tiles.

RESTful Maps

The RESTful Maps API is relatively new but does provide a nice, quick map solution when you don't need any interactivity. Just set an img src with the query parameters you need and get back an image.

(this should be all on one line)
http://m.nok.it/
    ?app_id=APP_ID
    &token=APP_TOKEN
    &nord       // Don't redirect to maps.nokia.com
    &w=640      // Width
    &h=480      // Height
    &nodot      // Don't put a green dot in the centre
    &c=38.895111, -77.036667 // Where to centre
    &z=12       // Zoom level
    &t=0        // Tile Style

That URL produces this image:

Upload to Facebook

Given the above, we've now got an image showing a map positioned exactly where the user wants it in the tile style the user likes. We just need to make the Facebook API call to set it as Timeline Cover Image and we're done.

You'd think.

Facebook doesn't provide an API endpoint to update a user's profile image or timeline cover. It's probably a privacy thing or a security thing or something. Either way, it doesn't exist. Never fear! There's a solution!

With the default permissions given by a Facebook login/OAUTH token exchange/etc... (that thing we did earlier), we are allowed to upload a photo to an album.

The easiest way to do this is to download the map tile using cURL then repost it to Facebook. The clever way to do it would be to pipe the incoming input stream directly back out to Facebook without writing to the local file system but it would be slightly more hassle to set that up and wouldn't really make much of a difference to how it works.

// Download from RESTful Maps
$tileUrl = "http://m.nok.it/?app_id=APP_ID&token=APP_TOKEN&nord&w=640&h=480&nodot&c=38.895111,%20-77.036667&z=12&t=0";
$ch = curl_init( $tileUrl );
$fp = fopen( $filename, 'wb' );
curl_setopt( $ch, CURLOPT_FILE, $fp );
curl_setopt( $ch, CURLOPT_HEADER, 0 );
curl_exec( $ch ); 
curl_close( $ch );
fclose( $fp );

//Upload to Facebook
$full_image_path = realpath($filename);
$args = array('message' => 'Uploaded by CoverMap.me');
$args['image'] = '@' . $full_image_path;
$data = $facebook->api("/{$album_id}/photos", 'post', $args);

The closest thing we can do then is to construct a Facebook link which suggests the user should set the uploaded image as their Timeline Cover:

// $data['id'] is the image's Facebook ID 
$fb_image_link = "http://www.facebook.com/" . $username . "?preview_cover=" . $data['id'];

Done

There we go. Minimal development required to create a web app with very little demand on the user that gives them a Nokia Map on their Facebook profile. Not too bad for a Sunday afternoon.

Go try it out and let me know what you think.