GhostStory

During all the research I did for my CSS testing talk, I couldn't help but spot another gap where a testing tool could be useful.

Cucumber

Cucumber is a technology used widely in automated testing setups, mostly for acceptance testing - ensuring that the thing everybody agreed on at the beginning was the thing delivered at the end.

This is accomplished by having a set of plain text files containing descriptions of different scenarios or aspects of the application, usually with a description of the actions performed by an imaginary user. You describe the situation (known as a 'Given' step), describe the user's action ('When') and describe the expected outcome ('Then').

The language (properly known as gherkin) used in these files is deliberately simple and jargon-free so that all the key stakeholders in the project - designers, developers, product owners - can understand but the files are also written in a predictable and structured style so that they can, behind the scenes, be turned into testable code.

What occurred to me when looking into this area was that there wasn't an agreed terminology for specifying the layout/colour/look and feel of a project in plain text. Surely this would be the perfect place to drop in some cucumber salad.

What we've got now is a project based on SpookyJS - a way of controlling CasperJS (and, therefore PhantomJS) from NodeJS - which contains the GhostStory testing steps and their corresponding 'behind the scenes' test code. There are only two steps at the moment but they are the most fundamental which can be used to build up future steps.

By the way, the name 'GhostStory' only just won out over the original 'CucumberNodeSpookyCasperPhantomJS'. It kinda makes sense because it's built using a lot of ghost-named technologies (Spooky, Casper, Phantom) and cucumber test files are sometimes called "Story" files because they describe a user story.

Implemented Steps

Here, "Element descriptor" is a non-dev-readable description of the element you want to test - "Main title", "Left-hand navigation", "Hero area call-to-action". In the project, you keep a mapping file, selectors.json, which translates between these descriptions and the CSS selector used to identify the element in tests.

Then the "Element descriptor" should have "property" of "value"

This is using the computed styles on an element and checking to see if they are what you expect them to be. I talked about something similar to this before in an earlier post. This is related to the 'Frozen DOM' approach that my first attempt at a CSS testing tool, cssert, uses but this way does not actually involve a DOM snapshot.

Then the "Element descriptor" should look the same as before

This uses the 'Image Diff' approach. You specify an element and render the browser output of that element to an image. The next time you run the test, you do the same and check to see if the two images differ. As mentioned many times before, this technique is 'content-fragile' but can be useful for a specific subset of tests or when you have mocked content. It can also be particularly useful if you have a 'living styleguide' as described by Nico Hagenburger. I've got some ideas about CSS testing on living styleguides that I'll need to write up in a later post.

Future Steps

Off the top of my head, there are a couple of other generic steps that I think would be useful in this project.

Then the "Element descriptor" should have a "property" of "value1", "value2", ..., or "valueN"

This variation on the computed style measurement allows an arbitrary-length list of values. As long as the element being tested matches at least one of the rules, the step counts as a pass. This could be used to ensure that all text on a site is one of a certain number of font-sizes or that all links are from the predefined colour palette.

Then the "Element descriptor" should look the same across different browsers.

This would build on the existing image diff step but include multiple browser runners. Just now, the image diffs are performed using PhantomCSS which is built on top of PhantomJS which is Webkit-based. This would ideally integrate a Gecko renderer or a Trident renderer process so that the images generated from one could be checked against another. I still feel that image diff testing is extremely fragile and doesn't cover the majority of what CSS testing needs to do but it can be a useful additional check.

The aim

I'm hoping this can sit alongside the other testing tools gathering on csste.st where it can help people get a head-start on their CSS testing practices. What I'm particularly keen on with the GhostStory project is that it can pull in other tools and abstract them into testing steps. That way, we can take advantage of the best tools out there and stuff it into easily digested Cucumber sandwiches.

Try it

The GhostStory project is, naturally, available on GitHub. More usefully, however, I've been working on a fork of SpookyJS that integrates GhostStory into an immediately usable tool.

Please check out this project and let me know what you think. I might rename it to distinguish it from the original SpookyJS if I can figure out exactly how to do that and maintain upstream relationships on GitHub.

Some App.net recipes

This is a collection of code snippets for various common tasks you might need to accomplish with the App.net API. Most of these are focused on creating or reading geo-tagged posts. They require a developer account on app.net and at least one of an App ID, App Code, App Access Token or User Access Token. The calls here are implemented using jQuery but that's just to make it easier to copy-paste into the console to test them out (so long as you fill in the blanks).

An important thing to bear in mind is the possibility for confusion between a 'stream' and 'streams'. By default, a 'stream' is a discrete chunk of the 20 latest posts served at a number of endpoints. This is the open, public, global stream:

https://alpha-api.app.net/stream/0/posts/stream/global

On the other hand, 'streams' are long-poll connections that serve up any matching posts as soon as they are created. The connection stays open while there is something there to receive the response. Streams are available under:

https://alpha-api.app.net/stream/0/streams

Totally not confusing. Not at all.

Creating a user access token

Required for any user-specific data retrieval. The only tricky thing you'll need to think about here is the scope you require.

scope=stream email write_post follow messages export

should cover most requirements.

Requires

• client_id

Visit this URL:

https://alpha.app.net/oauth/authenticate
    ?client_id=[your client ID]
    &response_type=token
    &redirect_uri=http://localhost/
    &scope=stream email write_post follow messages export

Using a user access token to create a post (with annotations)

Requires

• User Access Token
• text to post

The text is essential if you don't mark a post as 'machine_only'. The annotations here are optional. Annotations don't appear in the global stream unless the requesting client asks for them.

$.ajax({
  contentType: 'application/json',
  data: JSON.stringify({
    "annotations": [{
      "type": "net.app.core.geolocation",
      "value": {
        "latitude": 52.5,
        "longitude": 13.3,
        "altitude": 0,
        "horizontal_accuracy": 100,
        "vertical_accuracy": 100
      }
    }],
    "text": "Don't mind me, just checking something out."
  }),
  dataType: 'json',
  success: function(data) {
    console.log("Text+annotation message posted");
  },
  error: function() {
    console.log("Text+annotation message failed");
  },
  processData: false,
  type: 'POST',
  url: 'https://alpha-api.app.net/stream/0/posts?access_token={USER_ACCESS_TOKEN}'
});

Using a user access token to post a machine_only post (with annotations)

Requires

• User Access Token

In this example, we're creating a post that won't show up in user's timelines and adding the 'well-known annotation' for geolocation.

$.ajax({
  contentType: 'application/json',
  data: JSON.stringify({
    "annotations": [{
      "type": "net.app.core.geolocation",
      "value": {
        "latitude": 52.5,
        "longitude": 13.3,
        "altitude": 0,
        "horizontal_accuracy": 100,
        "vertical_accuracy": 100
      }
    }],
    machine_only: true
  }),
  dataType: 'json',
  success: function(data) {
    console.log("Non-text message posted");
  },
  error: function() {
    console.log("Non-text message failed");
  },
  processData: false,
  type: 'POST',
  url: 'https://alpha-api.app.net/stream/0/posts?access_token={USER_ACCESS_TOKEN}'
});

Retrieve the global stream, including geo-annotated posts if there are any

Requires

• User Access Token

This is a very basic call to retrieve the global stream but it also instructs the endpoint to return us all annotations and include machine-only posts.

var data = {
  "include_machine": 1,
  "include_annotations": 1,
  "access_token": "{USER_ACCESS_TOKEN}"
};

$.ajax({
    contentType: 'application/json',
    dataType: 'json',
    success: function(data) {
      console.log(data);
    },
    error: function(error, data) {
      console.log(error, data);
    },
    type: 'GET',
    url: 'https://alpha-api.app.net/stream/0/posts/stream/global',
    data: data
});

Creating an App Access Token

This is necessary for many of the streams operations. It is not used for individual user actions, only for application-wide actions.

• App.net API wiki on App Access Tokens

Requires

• client_id
• client_secret

client_credentials is one of the four types of grant_type specified in the OAuth 2.0 specification. I had difficulty getting this to work when using a data object:

var data = {
    "client_id": "{CLIENT_ID}",
    "client_secret":"{CLIENT_SECRET}",
    "grant_type": "client_credentials"
};

The client_credentials kept throwing an error. Instead, sending this as a string worked fine:

$.ajax({
  contentType: 'application/json',
  data: 'client_id={CLIENT_ID}&client_secret={CLIENT_SECRET}&grant_type=client_credentials',
  dataType: 'json',
  success: function(data) {
    console.log(data);
  },
  error: function(error, data) {
    console.log(error, data);
  },
  processData: false,
  type: 'POST',
  url: 'https://alpha.app.net/oauth/access_token'
});

One other thing to note is that this bit should be done server-side. This will throw a bunch of "…not allowed by Access-Control-Allow-Origin…" errors if you do it via jQuery.

Returns

{
    "access_token": "{APP_ACCESS_TOKEN}"
    }

Creating a streams format

Now you have your app access token, you can use it to tell the service what kind of data you want back. The streams offered in the API have two quite powerful aspects. Firstly, filters allow you to run many kinds of queries on the data before it is streamed to you so you don't need to recieve and process it all. Secondly, the decoupling of filters from streams means you can specify the data structure and requirements you want once then just access that custom endpoint to get the data you want back any time.

Requires

• App access token

This first example just creates an unfiltered stream endpoint

$.ajax({
  contentType: 'application/json',
  data: JSON.stringify({"object_types": ["post"], "type": "long_poll", "id": "1"}),
  dataType: 'json',
  success: function(data) {
    console.log(data);
  },
  error: function(error, responseText, response) {
    console.log(error, responseText, response);
  },
  processData: false,
  type: 'POST',
  url: 'https://alpha-api.app.net/stream/0/streams?access_token={APP_ACCESS_TOKEN}'
});

Returns

{
    "data": {
        "endpoint": "https://stream-channel.app.net/channel/1/{LONG_RANDOM_ENDPOINT_URL}",
        "id": "77",
        "object_types": [
            "post"
        ],
        "type": "long_poll"
    },
    "meta": {
        "code": 200
    }
}

Using Filters to create a stream of geotagged posts

We'll specify some requirements for our filter now so that it only returns back a subset of posts. The rules we're specfying here are:

At least one item in the "/data/annotations/*/type" field
must "match"
the value "net.app.core.geolocation"

Requires

• User access token

The field is specified in 'JSON Pointer' format. Within the response, there is a 'data' object and a 'meta' object. The data contains an 'annotations' object which contains an array of annotations, each of which has a type. This is represented as /data/annotations/*/type.

$.ajax({
  contentType: 'application/json',
  data: JSON.stringify({"match_policy": "include_any", "clauses": [{"object_type": "post", "operator": "matches", "value": "net.app.core.geolocation", "field": "/data/annotations/*/type"}], "name": "Geotagged posts"}),
  dataType: 'json',
  success: function(data) {
    console.log(data);
  },
  error: function(error, responseText, response) {
    console.log(error, responseText, response);
  },
  processData: false,
  type: 'POST',
  url: 'https://alpha-api.app.net/stream/0/filters?access_token={USER_ACCESS_TOKEN}'
});

Returns

The filter rules you just specified, the id of the filter (remember that for later) and the details of the application used to make the request.

{
"clauses": [
    {
        "field": "/data/annotations/*/type",
        "object_type": "post",
        "operator": "matches",
        "value": "net.app.core.geolocation"
    }
],
"id": "527",
"match_policy": "include_any",
"name": "Geotagged posts",
"owner": {
    "avatar_image": {
        "height": 200,
        "url": "https://d2rfichhc2fb9n.cloudfront.net/image/4/Pr63PjEwJ1fr5Q4KeL3392BMgSnIAYlHxv8OkWwzx75V8quNfpaFp4VPpKnDRxdXtYYPtIutrDVdU9NbJn7hKApQL84T5sfB1D9bWTgtizMWInignv0WyPPfM2DpqSThQgvkB68vbPzjZ8VeKM02M2GySZ4",
        "width": 200
    },
    "canonical_url": "https://alpha.app.net/thingsinjars",
    "counts": {
        "followers": 30,
        "following": 65,
        "posts": 96,
        "stars": 0
    },
    "cover_image": {
        "height": 230,
        "url": "https://d2rfichhc2fb9n.cloudfront.net/image/4/UWZ6k9xD8_8LzEVUi_Uz6C-Vn-I8uPGEBtKb9jSVoFNijTwyEm1mJYpWq6JvnA6Jd4gzW76vFnbSWvM3jadhc1QxUl9qS4NTKiv3gJmr1zY_UpFWvX3qhOIyKrBPZckf2MrinqWay3H0h9rfqY0Gp9-liEg",
        "width": 960
    },
    "created_at": "2012-08-12T17:23:44Z",
    "description": {
        "entities": {
            "hashtags": [],
            "links": [],
            "mentions": []
        },
        "html": "<span itemscope="https://app.net/schemas/Post">Nokia Maps Technologies Evangelist; CreativeJS team member; the tech side of museum140; builder of The Elementals; misuser of semi-colons;rn</span>",
        "text": "Nokia Maps Technologies Evangelist; CreativeJS team member; the tech side of museum140; builder of The Elementals; misuser of semi-colons;rn"
    },
    "id": "3191",
    "locale": "en_GB",
    "name": "Simon Madine",
    "timezone": "Europe/Berlin",
    "type": "human",
    "username": "thingsinjars"
}
}

Listening to the geotagged post stream

This wil return a link to a long-lasting connection to the app.net stream that will only return posts with the geolocation annotation.

Requires

• filter_id from the previous call

Note: the filter_id was returned as id in the previous response.

$.ajax({
  contentType: 'application/json',
  data: JSON.stringify({"object_types": ["post"], "type": "long_poll", "filter_id": "527"}),
  dataType: 'json',
  success: function(data) {
    console.log(data);
  },
  error: function(error, responseText, response) {
    console.log(error, responseText, response);
  },
  processData: false,
  type: 'POST',
  url: 'https://alpha-api.app.net/stream/0/streams?access_token={APP_ACCESS_TOKEN}'
});

Returns

The same kind of response as the 'Creating a streams format' example except the data coming down on the stream is filtered.

https://stream-channel.app.net/channel/1/{LONG_RANDOM_ENDPOINT_URL}

Open that URL up in your browser (seeing as we're testing) and, in a different tab, create a geo-tagged machine-only post (see above). Your post will appear almost instantly after you've submitted it.

Location-based time

Inspired by the simplicity of implementing a proximity search using MongoDB, I found myself keen to try out another technology.

It just so happened that I was presented with a fun little problem the other day. Given a latitude and longitude, how do I quickly determine what the time is? Continuing the recent trend, I wanted to solve this problem with Node.JS.

Unsurprisingly, there's a lot of information out there about timezones. Whenever I've worked with timezones in the past, I've always gotten a little bit lost so this time, I decided to actually read a bit and find out what was supposed to happen. In essence, if you're doing this sort of task. you do not want to have to figure out the actual time yourself. Nope. It's quite similar to one of my top web dev rules:

Never host your own video.

(Really, never deal with video yourself. Pay someone else to host it, transcode it and serve it up. It'll will always work out cheaper.)

What you want to do when working with timezones is tie into someone else's database. There are just too many rules around international boundaries, summer time, leap years, leap seconds, countries that have jumped over the international date line (more than once!), islands whose timezone is 30 minutes off the adjacent ones...

To solve this problem, it needs to be split into two: the first part is to determine which timezone the coordinate is in, the second is the harder problem of figuring out what time it is in that timezone. Fortunately, there are other people who are already doing this. Buried near the back of the bottom drawer in very operating system is some version of the tz database. You can spend hours reading up about it, its controversies and history on Wikipedia if you like. More relevantly, however, is what it can do for us in this case. Given an IANA timezone name – "America/New_York", "Asia/Tokyo" – you can retrieve the current time from the system's tz database. I don't know how it works. I don't need to know. It works.

Node

Even better for reaching a solution to this problem, there's a node module that will abstract the problem of loading and querying the database. If you use the zoneinfo module, you can create a new timezone-aware Date object, pass the timezone name to it and it will do the hard work. awsm. The module wasn't perfect, however. It loaded the system database synchronously using fs.readFileSync which is I/O blocking and therefore a Bad Thing. Boo.

10 minutes later and Max had wrangled it into using the asynchronous, non-blocking fs.ReadFile. Hooray!

Now all I needed to do was figure out how to do the first half of the problem: map a coordinate to a timezone name.

Nearest-Neighbour vs Point-in-Polygon

There are probably more ways to solve this problem but these were the two routes that jumped to mind. The tricky thing is that the latitude and longitude provided could be arbitrarily accurate. A simple lookup table just wouldn't work. Of course, the core of the problem was that we needed to figure out the answer fast.

Nearest Neighbour

1. Create a data file containing a spread of points across the globe, determine (using any slow solution) the timezone at that point.
2. Load the data into an easily searchable in-memory data-structure (such as a k-d tree)
3. Given a coordinate, find the nearest existing data point and return its value.

Point in Polygon

1. Create a data file specifying the geometry of all timezones.
2. Given a coordinate, loop over each polygon and determine whether this coordinate is positioned inside or outside the polygon.
3. Return the first containing polygon

This second algorithm could be improved by using a coarse binary search to quickly reduce the number of possible polygons that contain this point before step 2.

Despite some kind of qualification in mathematic-y computer-y stuff, algorithm analysis isn't my strong point. To be fair, I spent the first three years of my degree trying to get a record deal and the fourth trying to be a stand-up comedian so we may have covered complexity analysis at some point and I just didn't notice. What I do know, however, is that k-d trees are fast for searching. Super fast. They can be a bit slower to create initially but the point to bear in mind is that you only load it once while you search for data lots. On the other hand, while it's a quick task to load the geometry of a small number of polygons into memory, determining which polygon a given point is in can be slow, particularly if the polygons are complex.

Given this vague intuition, I settled on the first option.

If I wanted to create a spread of coordinates and their known timezones from scratch, it might have been an annoyingly slow process but, the Internet being what it is, someone already did the hard work. This gist contains the latitude and longitude for every city in the world and what IANA timezone it is in. Score! A quick regex later and it looks like this:

module.exports = [
  {"latitude": 42.50729, "longitude": 1.53414, "timezone": "Europe/Andorra"},
  {"latitude": 42.50779, "longitude": 1.52109, "timezone": "Europe/Andorra"},
  {"latitude": 25.56473, "longitude": 55.55517, "timezone": "Asia/Dubai"},
  {"latitude": 25.78953, "longitude": 55.9432, "timezone": "Asia/Dubai"},
  {"latitude": 25.33132, "longitude": 56.34199, "timezone": "Asia/Dubai"},
  etc…

(See the original on GitHub)

All that's left is to load that into a k-d tree and we've got a fully-searchable, fast nearest neighbour lookup.

Source

The source for this node module is, of course, available on GitHub and the module itself is available for install via npm using:

npm install coordinate-tz

When combined with the zoneinfo module (or, even better, this async fork of the module), you can get a fast, accurate current time lookup for any latitude and longitude.

Not a bad little challenge for a Monday evening.

Building a Proximity Search

This is the detailed post to go with yesterday's quick discussion about proximity search. All the code is available on GitHub.

This assumes a bit of NodeJS knowledge, a working copy of homebrew or something similar.

Install

• MongoDB - brew install mongodb
• NodeJS
• NPM (included in NodeJS installer these days)

These are included in the package.json but it can't hurt to mention them here:

• npm install twitter (node twitter streaming API library)
• npm install mongodb (native mongodb driver for node)
• npm install express (for convenience with API later)

Start mongod in the background. We don't quite need it yet but it needs done at some point, may as well do it now.

Create a Twitter App

Fill out the form Then press the button to get the single-user access token and key. I love that Twitter does this now, rather than having to create a full authentication flow for single-user applications.

ingest.js

(open the ingest.js file and read along with this bit)

Using the basic native MongoDB driver, everything must be done in the database.open callback. This might lead to a bit of Nested Callback Fury but if it bothers you or becomes a bit too furious for your particular implementation, there are a couple of alternative Node-MongoDB modules that abstract this out a bit.

// Open the proximity database
db.open(function() {
    // Open the post collection
    db.collection('posts', function(err, collection) {
        // Start listening to the global stream
        twit.stream('statuses/sample', function(stream) {
            // For each post
            stream.on('data', function(data) {
                if ( !! data.geo) {
                    collection.insert(data);
                }
            });
        });
    });
});

Index the data

The hard work has all been done for us: Geospatial Indexing in MongoDB. That's a good thing.

Ensure the system has a Geospatial index on the tweets.

db.posts.ensureIndex({"geo.coordinates" : "2d"})

Standard Geospatial search query:

db.posts.find({"geo.coordinates": {$near: [50, 13]}}).pretty()
(find the closest points to (50,13) and return them sorted by distance)

By this point, we've got a database full of geo-searchable posts and a way to do a proximity search on them. To be fair, it's more down to mongodb than anything we've done.

Next, we extend the search on those posts to allow filtering by query

db.posts.find({"geo.coordinates": {$near: [50, 13]}, text: /.*searchterm.*/}).pretty()

API

Super simple API, we only have two main query types:

• /proximity?latitude=55&longitude=13
• /proximity?latitude=55&longitude=13&q=searchterm

Each of these can take an optional 'callback' parameter to enable jsonp. We're using express so the callback parameter and content type for returning JSON are both handled automatically.

api.js

(open the api.js file and read along with this bit)

This next chunk of code contains everything so don't panic.

db.open(function() {
  db.collection('posts', function(err, collection) {
    app.get('/proximity', function(req, res) {
      var latitude, longitude, q;
      latitude = parseFloat(req.query["latitude"]);
      longitude = parseFloat(req.query["longitude"]);
      q = req.query["q"];

      if (/^(-?d+(.d+)?)$/.test(latitude) && /^(-?d+(.d+)?)$/.test(longitude)) {
        if (typeof q === 'undefined') {
          collection.find({
            "geo.coordinates": {
              $near: [latitude, longitude]
            }
          }, function(err, cursor) {
            cursor.toArray(function(err, items) {
              writeResponse(items, res);
            });
          });
        } else {
          var regexQuery = new RegExp(".*" + q + ".*");
          collection.find({
            "geo.coordinates": {
              $near: [latitude, longitude]
            },
            'text': regexQuery
          }, function(err, cursor) {
            cursor.toArray(function(err, items) {
              writeResponse(items, res);
            });
          });
        }
      } else {
        res.send('malformed lat/lng');
      }

    });
  });
});

If you've already implemented the ingest.js bit, the majority of this api.js will be fairly obvious. The biggest change is that instead of loading the data stream then acting upon each individual post that comes in, we're acting on URL requests.

app.get('/proximity', function(req, res) {

For every request on this path, we try and parse the query string to pull out a latitude, longitude and optional query parameter.

if (/^(-?d+(.d+)?)$/.test(latitude) && /^(-?d+(.d+)?)$/.test(longitude)) {

If we do have valid coordinates, pass through to Mongo to do that actual search:

collection.find({
  "geo.coordinates": {
    $near: [latitude, longitude]
  }
}, function(err, cursor) {
  cursor.toArray(function(err, items) {
    writeResponse(items, res);
  });
});

To add a text search into this, we just need to add one more parameter to the collection.find call:

var regexQuery = new RegExp(".*" + q + ".*");
collection.find({
  "geo.coordinates": {
    $near: [latitude, longitude]
  },
  'text': regexQuery
}

This makes it so simple, making it it kind of feels like cheating. Somebody else did all the hard work first.

App.net Proximity

This works quite well on the App.net Global Timeline but it'll really become useful once the streaming API is switched on.

Of course, the code is all there. If you want to have a go yourself, feel free.

Proximity Search

Now that geolocated posts are beginning to show up around app.net, I found myself wondering about proximity search. Twitter provides one themselves for geotagged tweets. What a proximity search does, essentially, is provide results from a data set ordered by increasing distance from a given location. This can be further enhanced by combining it with a text search either before or after the distance sorting. This would give you a way to search for a certain query within a certain area.

Here, for example, is a proximity search centred on Berlin including any tweet in 1000 miles containing the word 'toast'.

When I first started thinking about the tech required for a proximity search, I remembered Lukas Nowacki back in our old Whitespace days implementing the Haversine formula in MySQL (Alexander Rubin has a good overview of how to do this). As much as I love my trigonometry and logarithms, I must admit, I was looking around for a simpler solution. Actually, I was looking around for a copy-paste solution, to be honest. I may even have spent some time going down that route if Max hadn't pointed me in the direction of MongoDB.

I'd been putting off digging into NoSQL databases for a while because, well, I had no real reason to. Recently, I've either been focused on front-end dev or hacking away at Java and never really had any good reason to investigate any of these new-fangled technologies get off my lawn you kids.

MongoDB

After 10 minutes of messing around with Mongo, I pretty much just found myself saying “No... way. There's no way that's actually working” I'm sure those of you experience with document-oriented databases are rolling your eyes right now but for those few of us left with an entirely relational concept of databases, let me just explain it like this: you know those things you want to do with a database that are just a hassle of multiple joins and confusing references? Document databases do some of those things really really well.

The biggest selling point for me, however was the native geospatial indexing. That pretty much made the majority of my proximity search complete. All I needed to do was wrap it in a nice interface and call it a day...

I'll follow up tomorrow with a more detailed 'How-to' guide.

CoverMap - Nokia Maps on Facebook

I'm almost managing to keep to my intended schedule of one map-based web experiment per week. Unfortunately, I've mostly been working on internal Nokia Maps projects over the weekends recently so I've not had much to post here.

I can share my latest toy, though: CoverMap.me

Using just the public APIs over a couple of hours last Sunday afternoon, I made this to allow you to set a Nokia Map as your Facebook Timeline cover. The whole process is really straightforward so I thought I'd go over the main parts.

The exact aim of CoverMap.me is to allow the user to position a map exactly, choose from any of the available map styles and set the image as their cover image.

Make a Facebook App

Go to developers.facebook.com/apps/ and click 'Create New App', fill in the basic details – name of the app, URL it will be hosted on, etc – and you're done.

Facebook login

I've used the Facebook JS SDK extensively over the summer for various projects but I wanted to try out the PHP one for this. Super, super simple. Really. Include the library (available here), set your appId and secret and request the $login_url.

$facebook->getLoginUrl(array('redirect_uri' => "http://example.com/index.php"));

That will give you a link which will take care of logging the user in and giving you basic access permissions and details about them.

Nokia Maps JS API

When I'm hacking together something quick and simple with the Nokia Maps API, I usually use the properly awsm jQuery plugin jOVI written by the equally awsm Max. This makes 90% of the things you would want to do with a map extremely easy and if you're doing stuff advanced enough to want the extra 10%, you're probably not the type who'd want to use a jQuery plugin, anyway. Either way, you need to register on the Nokia developer site to get your Nokia app_id and secret.

To create a map using jOVI, simply include the plugin in your page then run .jOVI on the object you want to contain the map along with starting parameters:

$(window).on('load', function() {
  $('#mapContainer').jOVI({
    center: [38.895111, -77.036667], // Washington D.C.
    zoom: 12,           // zoom level
    behavior: true,     // map interaction
    zoomBar: true,      // zoom bar
    scaleBar: false,    // scale bar at the bottom
    overview: false,    // minimap (bottom-right)
    typeSelector: false,// normal, satellite, terrain
    positioning: true   // geolocation
  }, "APP_ID", "APP_SECRET");
});

This gives us a complete embedded map.

As I mentioned above, part of the idea for CoverMap.me was to allow the to choose from any of the available map styles. This was an interesting oddity because the public JS API gives you the choice of 'Normal', 'Satellite', 'Satellite Plain' (a.k.a. no text), 'Smart' (a.k.a. grey), 'Smart Public Transport', 'Terrain' and 'Traffic' while the RESTful Maps API (the API that provides static, non-interactive map images) supports all of these plus options to choose each of them with big or small text plus a 'Night Time' mode. Because of this, I decided to go for a two-step approach where users were shown the JS-powered map to let them choose their location then they went through to the RESTful static map to allow them to choose from the larger array of static tiles.

RESTful Maps

The RESTful Maps API is relatively new but does provide a nice, quick map solution when you don't need any interactivity. Just set an img src with the query parameters you need and get back an image.

(this should be all on one line)
http://m.nok.it/
    ?app_id=APP_ID
    &token=APP_TOKEN
    &nord       // Don't redirect to maps.nokia.com
    &w=640      // Width
    &h=480      // Height
    &nodot      // Don't put a green dot in the centre
    &c=38.895111, -77.036667 // Where to centre
    &z=12       // Zoom level
    &t=0        // Tile Style

That URL produces this image:

Upload to Facebook

Given the above, we've now got an image showing a map positioned exactly where the user wants it in the tile style the user likes. We just need to make the Facebook API call to set it as Timeline Cover Image and we're done.

You'd think.

Facebook doesn't provide an API endpoint to update a user's profile image or timeline cover. It's probably a privacy thing or a security thing or something. Either way, it doesn't exist. Never fear! There's a solution!

With the default permissions given by a Facebook login/OAUTH token exchange/etc... (that thing we did earlier), we are allowed to upload a photo to an album.

The easiest way to do this is to download the map tile using cURL then repost it to Facebook. The clever way to do it would be to pipe the incoming input stream directly back out to Facebook without writing to the local file system but it would be slightly more hassle to set that up and wouldn't really make much of a difference to how it works.

// Download from RESTful Maps
$tileUrl = "http://m.nok.it/?app_id=APP_ID&token=APP_TOKEN&nord&w=640&h=480&nodot&c=38.895111,%20-77.036667&z=12&t=0";
$ch = curl_init( $tileUrl );
$fp = fopen( $filename, 'wb' );
curl_setopt( $ch, CURLOPT_FILE, $fp );
curl_setopt( $ch, CURLOPT_HEADER, 0 );
curl_exec( $ch ); 
curl_close( $ch );
fclose( $fp );

//Upload to Facebook
$full_image_path = realpath($filename);
$args = array('message' => 'Uploaded by CoverMap.me');
$args['image'] = '@' . $full_image_path;
$data = $facebook->api("/{$album_id}/photos", 'post', $args);

The closest thing we can do then is to construct a Facebook link which suggests the user should set the uploaded image as their Timeline Cover:

// $data['id'] is the image's Facebook ID 
$fb_image_link = "http://www.facebook.com/" . $username . "?preview_cover=" . $data['id'];

Done

There we go. Minimal development required to create a web app with very little demand on the user that gives them a Nokia Map on their Facebook profile. Not too bad for a Sunday afternoon.

Go try it out and let me know what you think.

The code is now available on GitHub

Using CSS :target to control state

There can be a fair amount of discussion provoked by the phrase "CSS only" or "Built entirely in HTML and CSS" when it comes to tech demos. The discussion generally revolves around the fact that what the phrase actually means 99% of the time is "Using CSS for the graphical bits but tracking the mouse using JS" or "Using CSS transforms for most of it but using JS to calculate the values" or any number of variations on that theme.

Now, I'm not saying that's a bad thing at all, I quite happily called my interpretation of Super Mario Kart CSS-only even though without JS, Yoshi wouldn't make it to the first bend. What these demos are doing, essentially, is using CSS as the rendering layer where previously, JS would have been used. By doing so, you even get some nice hardware acceleration thrown in. Not too bad.

Why fall back?

The reason we fall back to JS for a lot of things is because CSS is a declarative language (see my post on CSS testing to learn more). However you say things should be is how they are. In CSS, you can't set a variable to one value, do something with it then set it to a different value and do something else. For a start, you don't have variables. Even if you did, the style sheet is read top-to-bottom before anything is done with it. The variable would always have the final value assigned to it, even the first time you used it. I'm simplifying a bit but not much. If you want to do anything clever, you generally need to resort to JS, an interpreted language.

Really CSS only

If you want to make something properly interactive but entirely CSS, you have to find a way to modify the state of the DOM so that different sets of styles can be applied to the same sets of elements. We actually do this all the time when we change the colour of a link on :hover. The user interacts with the page, the state of the page is changed slightly, a different style applies. There's a lot you can do now that you have a way to modify the page based on user interaction. Hooray for pseudo-classes!

An extremely important addition to the set of pseudo-classes available in CSS3 is the :target class. This becomes set on an element when the element is the target of a clicked link. Think of in-page jump links. When you click on one and the URL changes to 'blah.html#the-thing-you-clicked', the element with id="the-thing-you-clicked" gets the :target pseudo-class. Now you can affect it and its children with new styles. Now it becomes interesting, you can click on something on one bit of the page and it cause something to happen on another bit of the page.

Multiple-nested states

By nesting several elements around the thing you're intending to modify, you can now create a set of states entirely controlled by CSS. For example, with this HTML:

<div id="red">
  <div id="blue">
    <div id="green">
      <div id="yellow">
          <p>Hello.</p>
      </div>
    </div>
  </div>
</div>
<ul>
  <li><a href="#red">Red</a></li>
  <li><a href="#blue">Blue</a></li>
  <li><a href="#green">Green</a></li>
  <li><a href="#yellow">Yellow</a></li>
  <li><a href="#default">Default</a></li>
</ul>

p {
    color:black;
}
#red:target p {
    color:red;
}
#blue:target p {
    color:blue;
}
#green:target p {
    color:green;
}
#yellow:target p {
    color:yellow;
}

You can change the colour of the text by clicking on the link. Without any JS.

See this on JSFiddle.

Shiny demos

This still isn't perfect. There are still going to be many things that JS is best for, calculations being one, keyboard input being another. To try and find the best way to show this off, I tried to update CSS Mario Kart to be entirely CSS and I almost got there but wasn't 100% successful.

• CSS-only Mario Kart

I then started to build a zoetrope in CSS, found someone else had already done it and decided to modify it.

• CSS-only Zoetrope

Maybe soon, the phrase "CSS only" will really mean "CSS only. And a bit of HTML."

Web Audio API Part 1

ImpressJS Tools

While putting together last week's promo video for Museum140 (a vote'd be awsm, btw), I knocked up a few tools to make my life easier. As I always say, if you find something you like, make a plugin for it. Seriously, I always say that. That might even be how I proposed to my wife, I'll have to check.

Anyway.

Play

This is a simple timing helper. It just provides a little array you can push slide durations into and at the end, you call 'play'. I can't see many uses for this other than in creating videos.

ImpressJS.play(3000);  //Set the first slide duration for 3 seconds
ImpressJS.play(1000);  //Set the second slide duration for 1 second
ImpressJS.play(); //Play from the start

Edit

This is much more useful.

If this script is included in the page (after the impress.js script), you can drag your slides around, rotate and scale them into exactly the position you want. Once you're happy, you can get the current HTML output onto the console for pasting back into your original slideshow file. I could have snuck in a drag-n-drop file handler but that would make it Chrome only.

Disclaimer

These tools rely on ImpressJS having a public API which it currently doesn't have. It's obviously high on the author's priority list (given the amount of discussion it's raised) but, as too many pull requests spoil the broth, I've made a little fork of my own, added the public functions the tools require and I'll update them once the main repository's settled down a bit.

Download

These are available in the tools folder of this fork of impress.js. Each one contains an example. Hopefully, these will be updated to use the public API as soon as it's available.

Create a draggable file on-the-fly with JS

Here's a useful little code snippet if you're building a web application. It's a simple way of making the boundary between web-browser and platform a bit smaller. It allows you to create a file (text, html, whatever) in in your page which the user can drag onto their desktop (if their browser supports the dragstart event and dataTransfer methods).

document.getElementById('downloadElement').addEventListener("dragstart", function (e) {
  e.dataTransfer.setData("DownloadURL", "text/html:filename.html:data:image/png;base64," + btoa(fileContent));
});

A description of the code:

• attach an event listener to the draggable element you specify (downloadElement)
• when you start to drag it (dragstart),
• it creates a dataTransfer object (with the type DownloadURL)
• and sets the content of that to be whatever you pass it (fileContent)
• It uses btoa() to encode the string data as a base64-encoded string.
• When you combine this with the MIME-type (text/html),
• you can create a file with the specified name (filename.html) when the user releases the drag in their local file system.
• The fake MIME-type (image/png) is there as part of the object data to convince the browser this is a binary file (which it is, even though it's not an image).

Note: The element this is attached to (downloadElement above) should either be draggable by default (image, link or text selection) or have the draggable attribute set otherwise this'll do nothing.

Credit goes to Paul Kinlan for using this in Appmator which is where I first saw this done this way. He actually uses it alongside some extremely clever JS zip-file creation stuff, too, it's definitely worth digging through the source there.

You can find out more about the drag events on the MDN.

cssert – Like ‘assert’ but with CSS at the front

This continues on from my post about CSS verification testing

cssert – pronounced however you feel like it – is my attempt at bringing some kind of style verification into an automated build process. If you've read the previous article, you'll know that this falls into the second group of CSS test frameworks, style measurement and comparison. The system works exactly as I described above – your test files have a basic HTML structure created by traversing the DOM from the element being tested upwards. You could also include your entire HTML in the test file if you liked, it would just be unnecessary in most cases.

I've created a couple of (for me, at least) helpful utilities which allow these test cases to be generated from a page via a bookmarklet and then run in the browser or on the command-line. Running the tests in the browser is useful for quick human verification while the command-line interface can be integrated into an automated build system if you like that kind of thing. The test file structure is quite simple (all samples here taken from the Twitter Bootstrap project:

First, we have the test file opening structure:

<!doctype html><html><head><title>cssert test page</title><link rel="stylesheet" href="../lib/cssert.css"></head><body><h1>cssert Test cases</h1><p>click to expand test</p><script type="text/html">/*==

Then we have the name of the test:

Intro Paragraph

Then we have the skeleton DOM:

<!doctype html><html><head><meta charset="utf-8"><base href="http://twitter.github.com/bootstrap/examples/hero.html"><link href="../1.3.0/bootstrap.css" rel="stylesheet"><style type="text/css">
      body {
        padding-top: 60px;
      }
    </style><style>#cssert-style-modal {display:none;position: fixed;top: 10%;left: 50%;margin-left: -350px;width: 700px;background: #39c;color: white;padding: 10px;color: #fff;text-shadow: 0 1px 0 rgba(0,0,0,.3);background-image: -webkit-linear-gradient(-45deg, rgba(255,255,255,0), rgba(255,255,255,.1) 60%, rgba(255,255,255,0) 60%);border-radius: 5px;border: 1px solid #17a;box-shadow: inset 0 0 0 1px rgba(255,255,255,.3);}#cssert-style-modal ul,#cssert-style-modal li {margin:0;padding:0;font-size:11px;list-style:none;}#cssert-style-modal>ul>li {float:left;width:140px;font-size:13px;}#cssert-style-modal ul {margin-bottom:10px;}#cssert-pre {position:fixed;top:10px;right:10px;background:white;border:1px solid gray;width:200px;height:200px;overflow:auto;}#cssert-drag {position:fixed;top:210px;right:10px;background:white;border:1px solid gray;width:200px;height:20px;overflow:auto;}</style></head><body><div class="container"><div class="hero-unit"><p>Vestibulum id ligula porta felis euismod semper. Integer posuere erat a ante venenatis dapibus posuere velit aliquet. Duis mollis, est non commodo luctus, nisi erat porttitor ligula, eget lacinia odio sem nec elit.</p></div></div></body></html>

The CSS selector identifying the element to verify:

html body div div p

And the styles we wish to verify:

{"font-family":"'Helvetica Neue', Helvetica, Arial, sans-serif","font-weight":"200","font-style":"normal","color":"rgb(64, 64, 64)","text-transform":"none","text-decoration":"none","letter-spacing":"normal","word-spacing":"0px","line-height":"27px","text-align":"-webkit-auto","vertical-align":"baseline","direction":"ltr"}

A test file can contain as many test units as you like. At the very end is the close of the test file structure

*/</script><script src="../lib/cssert.js"></script></body></html>

You'll probably notice the crucial bit in the test structure is the base element. The CSS available from the location specified here is the thing we are actually testing. In typical test lingo, the structure we have in our test file is the mock and our tests work by asserting the values ‘output’ from applying the CSS to this structure are as expected.

Running the tests

Running the tests in-browser

Open the test page in a browser. That's it. If it's green and says 'passed', the test passed. If it's red and says 'failed', the test failed. You can see the output by clicking on the title of the test.

This works by loading the test file, creating an iframe and injecting the test case into the iframe as source. It then looks into the iframe and measures the styles. If they match those specified in the test file, it passes, otherwise, it fails. Clicking on the test title simply removes the position:absolute which is hiding the iframe.

Running the tests on command-line

The exact same test page can also be used with the command-line interface. cssert uses PhantomJS to run the tests in a headless webkit instance. You'll need to install PhantomJS into your path after downloading it. Place your test case in the tests folder and run:

$ ./cssert testcase.html

To run all tests in the tests folder at once, simply run with no arguments:

$ ./cssert

This works by, again, loading the HTML from the test files. In this case, the structure is injected into a new headless browser window. The styles are measured and the output is redirected to stdout. Each browser window test is also rendered as a PNG so you can see what failed if any did.

Limitations

I'm not saying this is the ultimate solution to CSS testing. Declarative languages don't sit well with testing. This is as close as I can get for the moment. I'm also not going to be able to head off or counter all possible complaints or criticisms but I will cover a couple.

Firstly, most of the limitations you'll run into are caused by using the automatically generated tests. They're good for creating a starting point but at the moment, they need to be tweaked for many cases.

Sibling selectors

Because the test generates the DOM via following parents up the structure, sibling elements are ignored. These styles are still testable, though. Simply add the sibling element into your HTML test block.

Styles modified by JS

The styles are measured on the element as it is when the case is generated. The test compares this against the styles provided by the CSS. If the element contains JS-only styles not added by CSS, they will not be correctly matched. Modify your test case to allow for this.

Why not use Selenium?

This, essentially does the same as Selenium would do if you took the time to set up your test cases. This makes it much easier to set up test cases, though.

@font-face!

If your @font-face declaration contains a suggested 'local' source (as recommended in Paul Irish's bulletproof syntax), a bug in QTWebkit will prevent the test case from running correctly.

Installation

Just clone the git project from git@github.com:thingsinjars/cssert.git and you're good to go.

The tests directory comes with some sample tests generated using Twitter's Bootstrap project. Put your tests in that same place.

Copy-paste coding

This weekend, I had a spare couple of hours on Saturday night (as often happens when you have a kid) so I decided to pass the time with a bit of copy-paste coding.

I grabbed a bit of example code for a top-down racing game built using Box2d and GameJs. I then grabbed a Google Maps demo and a Nokia Maps demo and smooshed them together. I've seen a few racing games before so I know there's nothing innovative here, it's just a bit of silliness.

The results

• Driving on Nokia Maps
• Driving on Google Maps

Copy-Paste: Not a bad thing

There are many developers who will tell you that blindly copy-pasting is a terrible way to code. They make the valid points that you don't understand something if you don't read it properly and that you'll end up stuck down an algorithmic cul-de-sac with no way to three-point-turn your way out (although they may phrase it differently). These are valid points but...

If I'd sat down on Saturday night and thought "I want to build something in Box2D" then gone to the site, downloaded the library, read the docs and loaded the examples, by the time I had understood what I was doing, Sunday morning would have come round and I'd still have GameJS to read up on. There's absolutely no harm in blindly catapulting yourself a long way down the development track then looking round to see where you are. Sure, you'll end up somewhere unfamiliar and you may end up with absolutely no clue what the code around you does but at least you have something to look at while you figure it out. At least a few times, you'll find something almost works and a quick search of the docs later, you've figured it out and know what it's doing.

Basically, copy-pasting together a simple little demo and making it work is a lot more interesting than taking baby-steps through example code and beginner tutorials. Don't be too careful about trying to build something complicated.

An Idea

If you're in the mood for copy-paste coding, try taking the two demos above (Google and Nokia), grab the node.js server from the multi-user page experiment and figure out how to have multiple racers on the same map (by transmitting angle and geo-coords).

Sine

Another game concept prototype – Sine

Honestly, I have no idea what I was going for with this one. It started off last weekend with a vague idea about matching patterns of numbers and old-school graphics and I don't know what and ended up with this.

The idea is to make the bottom row match the top row, basically. There are several front-ends to this game so you can choose the style of play you prefer - numbers and letters, waves, colours or a generated sound wave (if you have a shiny new-fangled browser). It uses the nifty little Riffwave library to generate PCM data and push it into an audio element.

Further development

If I were to develop this further, I'd try and build it in a modular fashion so that front-ends could be generated really easily and open it to other people to see how many different ways this game could be played. It'd be an interesting social experiment to be able to play what is fundamentally the same game in a dozen different ways. You could find out if visual thinkers processed the information faster than numerical or audio-focused people. Leaderboards could allow different styles of player to compete on the same playing field but with a different ball (hmm, weak sports analogy). The rhythms of the game lend themselves well to generated drum tracks so there's probably something in that area for exploring as well.

At the moment, the code is written as the programming equivalent of stream-of-consciousness – global variables everywhere, some camel-case, some underscored, vague comments sprinkled around. There's some commented-out code for the wave mode that moves the waves closer together so that there's a time-limit set but I felt it didn't suit the game.

Warning

The audio mode is very annoying. Seriously. If I were still a full-time game designer, I would not release this due to public health concerns. As I'm now an enthusiastic amateur, here you go :D

Text Editors

Many years ago during the text-editor holy wars, I sided neither with the Vi Ninjas nor the Emacs Samurai but instead went a third way – Pico (short for Pine Composer). It was the text editing part of the e-mail program I used (Pine). For many years, this served me well. Even today, Pico's successor - Nano – is installed pretty much everywhere. It isn't , however, quite powerful enough for fast-paced web development. Serious, full-time development needs shortcuts and macros, syntax highlighting and snippets. When you spend 10 or more hours every day pressing buttons to change flashing lights, you need to optimise the way the lights flash.

After Pico, I found Crimson Editor which served me well for almost 10 years. I eventually started working on a Mac and became a TextMate user for most of the last 5 years.

In my new job, I find myself jumping from computer to computer to desktop to server quite a lot. The only constant editor available is Vi. Or Vim (Vi Improved). I've been trying to pick it up as a way to ensure I can always jump into a friendly text editor no matter where I am. Besides, these days Vim is the old-new-cool thing that all the cool-old-kids use, particularly MacVim so I thought it was worth giving it a go to see what the fuss was about.

Incidentally, the first thing that always jumped into my head whenever anybody mentioned Vi was the Vim Golf challenge two of my lecturers played while I was at Uni. It's a well-known and surprisingly popular game where two Seventh-dan Vi proficients try to accomplish an edit on a file using the fewest keystrokes. Each morning, one lecturer would lay down a challenge and the other had 24 hours to come up with a solution before presenting it and the next challenge the following day. The (possibly apocryphal) conclusion to the long-running game came when it was discovered that the lecturer who won most often was actually recompiling the network install of Vim every night to include his own custom keyboard shortcuts.

One of the biggest deciding factors in trying it out was actually fellow Edinburgh Web Dev/International Traveller Drew Neil (@nelstrom), creator and voice of the vimcasts.org series of screencasts who is actually writing a book on Vim this very second as I write this. Most people are evangelical about their choice of text editor to the point of rabid fundamentalist, frothing-at-the-mouth, intoning-keyboard-shortcuts craziness (hence my allusions to text-editor holy wars). When I mentioned to Drew that I used Pico instead, his response was long the lines of "Fair enough". This lack of confrontation actually inspired me to try it out. Well played, sir.

In an unrelated aside, I once did an illustration for the 'Collective Nouns Illustrated' project Drew ran.

Anyway, I'll give it a go and see what happens. If you're interested, I recommend reading Yehuda Katz' post 'Everyone Who Tried to Convince Me to use Vim was Wrong'. Don't worry, I'm definitely not going to try and convince anyone to use one code editor over another. You should probably stop using FrontPage, though.

Whitehat Syndication

When I refer to Google or Googlebot here, I could really be talking about any web search or web crawler. When I refer to Yahoo, I actually mean Yahoo. Don't get confused.

I recently ran into an interesting SEO problem on a project which has led to a question I just don't know the answer to:

How do you display syndicated content without triggering Google's duplicate content flag?

Hmm... intriguing.

Background

To explain the problem more fully (without giving out any project specifics), imagine you have a website. You probably already do so that shouldn't be hard. Now, imagine you fill that website full of original content. Again, that shouldn't be hard. For the sake of example, let's assume you run a news blog where you comment on the important stories of the day.

Next, you figure that your readers also want to read important related facts about the news story. Associated Press (AP) syndicates its content and through the API, you can pull in independently-checked related facts about whatever your original content deals with. So far, so good.

Unfortunately, a thousand other news blogs also pull in the same AP content alongside their original (and some not-so-original) content. Now, when the Googlebot crawls your site, it finds the same content there as it does in a thousand other places. Suddenly, you're marked with a 'duplicate content' black flag and all the lovely google juice you got from writing original articles has been taken away. Boo.

Your first thought might be to reach for the rel="canonical" attribute but that really only applies to entire pages. We need something that only affects a portion of the page.

Solution

What you need to do is find a way to include the content in your page when a visitor views it (providing extra value for readers) but prevent Google from reading it (hurting your search ranking). Fortunately, there are some methods for doing this. One involves having the content in an external JS file which is listed in your robots.txt to prevent Google from reading it. Another similar method involves having the content in an external HTML and including it as an iframe, again, preventing crawling via robots.txt. When the reader visits the page, the content is there, when Google visits, it isn't.

The Problem with the Solution

Both of the techniques mentioned here involve an extra HTTP request. You are including an external file so the visitor's browser has to go to your server, grab the file and include it. This isn't a huge problem for most sites but when you're dealing with high-traffic, highly-optimised websites, every file transferred counts. You go to all the trouble of turning all background images into sprites, why waste extra unnecessary connections on content?

There is an extra problem with the JS solution in that the content isn't available to visitors that don't have JS enabled but as I've mentioned before, that's only a problem if any of your visitors actually have JS disabled. Only you know that for sure.

Yahoo's Solution

Yahoo have a nice solution to this problem. If you include the attribute class="robots-nocontent" on any element, the Yahoo spider (called 'slurp') will ignore the content. Perfect. This does, however, only work for Yahoo. Not perfect.

My solution

My attempt at solving this problem which is a combination of SEO and high front-end performance was inspired by the technique GMail uses to deliver JS to mobile devices. In their article, Google delivers JS that they don't want run immediately in the initial payload. They figure that the cost of serving a single slightly larger HTTP request is less than the delay in retrieving data on demand.

I use HTML embedded in a JS comment in the original page which is then processed on DOMReady to strip out the comments and inject it into wherever it is supposed to be (identified by the data-destination attribute). I'm doing this on a page which already loads jQuery so this can all be accomplished with a simple bit of code.

<script type="text/html" class="norobot" data-destination=".content-destination">
 /*!
  <p>This content is hidden on load but displayed afterwards using javascript.</p>
 */
</script>
<div class="content-destination"></div>
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.5/jquery.min.js"></script>
<script>
 $('.norobot').each(function() {
  _this = $(this);
  $(_this.data('destination')).html(_this.html().replace('/*!','').replace('*/',''));
 });
</script>

Notes on the code

You may have noticed the type="text/html" attribute on the script block. That's to get around the fact that jQuery parses and executes any script blocks it finds when moving elements around (in an appendTo() or an html(), for example. Adding this attribute tells jQuery to process this as template code.

Also, the opening JS comment here begins /*!. The exclamation mark in this is a directive to any minifiers you might use on the code to tell them not to remove this comment block.

This is also available in a standalone page.

This is all a very long setup for my initial question. Does Google read this and, if so, does this affect duplicate content rankings?

Plus and Minus

• Minus: The duplicate content is definitely in the page.
• Plus: It's hidden in JavaScript
• Minus: we're using JavaScript to serve different content to users and to google.
• Plus: we're showing less to google than users. Spam techniques show more to increase keyword matches.
• Plus: faster response due to a single http request (Google likes fast pages)

Obviously, we could add an extra step of obfuscating the 'hidden' content by reversing it or encoding it. This would definitely hide it from google and it would be trivial to undo the process before showing it to the user but is this step necessary? Part of my reasoning for concluding that Google ignores JS comments is that thousands of sites include the same standard licences bundled with their JS library of choice and don't get penalised. This may, of course, be specific to licences, though.

I can find no definitive answer anywhere on this subject. If you have any good references, please let me know. Alternatively, if you happen to know Matt Cutts, ask him for me. If I get any conclusive answer, I'll update here.

A collection of bookmarklets

Bookmarklets are the handiest little things. In case you don't know (which I'm sure you do), they're small chunks of JS that you store in your browser's Bookmarks or Favourites section which you can launch while looking at any web page. I write bookmarklets for all kinds of different tasks – navigating quickly around the build monitor at work, filling in tricky forms that my browser autocomplete doesn't handle, etc.

Here are a few of bookmarklets I find extremely useful for web development. To add them to your browser, simply drag the link on the title into your bookmark bar, hold shift down and drag or right-click and 'Add to Favourites', depending on what browser you're in.

Add JS to the page by URL

This will allow you to add any script you like to the current page. This can be particularly useful if you want to add a certain library or plugin to a page to investigate it further.

javascript:(function(){document.body.appendChild(document.createElement('script')).src=prompt('Script to add');})();

Add the latest CDN jQuery to the page

A variation on the above bookmarklet, this simply adds the latest version of jQuery. If you want to be able to play with a page and are familiar with jQuery, this will ensure that it is loaded and attached.

javascript:(function(){document.body.appendChild(document.createElement('script')).src='http://code.jquery.com/jquery-latest.min.js'})();

Add CSS to the page by URL

Add any stylesheet to the current page with a particular URL. This is handy if you want to demo variations to clients, I find. particularly if you predefine the CSS URL.

javascript:(function(d){if(d.createStyleSheet)%20{d.createStyleSheet(%20u%20);}%20else%20{var%20css=d.createElement(%27style%27),u=prompt(%27CSS%20to%20add%27);css.setAttribute(%22type%22,%22text/css%22);css.appendChild(document.createTextNode(%22@import%20url(%22+u+%22)%22));d.getElementsByTagName(%22head%22)[0].appendChild(css);}}(document))

Submit this page to the webdev subreddit

This isn't so much a web dev helper, more a general helper. I use this (or a variation thereof) to submit posts to specific reddits with fields prefilled. This bookmarklet defaults to the webdev subreddit.

javascript:window.location=%22http://www.reddit.com/r/webdev/submit?url=%22+encodeURIComponent(window.location)+'&title='+encodeURIComponent(document.title);void(0);

Add a CSS rule to the page

I can't remember whether I wrote this one or if I found it somewhere. The 'x<style>' is something from the middle of jQuery, though. Anyhow.

This allows you to add a style block directly into the page. This is useful for small CSS injections where the browser's web inspector is unavailable or incapable. Even though it's a single prompt that pops up, there's no reason why you can't past an entire stylesheet in there. I sometimes find myself doing that when I'm previewing designs on a production server.

javascript:(function(){var div = document.createElement('div'); div.innerHTML ='x<style>' +prompt('Style to add')+ '</style>';document.body.appendChild(div.lastChild);})();

Workflow. The flow. Of work.

How to actually work

Don’t tell anyone (particularly not your clients and especially not my clients) but making websites is really very easy. Don't make it harder by deliberately hindering yourself. You should always try to travel the shortest distance between “amends needs done” and “amend is done”. Again, I hear a “Pah!” in the distance, “Well, that’s obvious.”, but is it? Are you sure you’re ‘Being all you can be’?

The shortest distance

Okay, picture the scene: you've run out of socks. You need to put on a load of washing. The washing is in a pile in the corner of the room. You hunch over, pick up all the socks, pants, t-shirts and walk, waddle and totter to the washing machine. As you walk, you drop a sock and bend down to pick it up, dropping another. You recover that one and another falls. Eventually, you make it to the washing machine after a few minutes, put everything in, throw in some powder and set it going. As you head back to your bedroom, you spot three more socks that you missed. Darn.

Okay, picture the scene: you've run out of socks. You need to put on a load of washing. Fortunately, every time you took your socks off, you put them straight in the washing machine. You wander to the kitchen, put in the powder and switch it on. Done.

Any time you try to do anything, the more steps involved between intention and completion, the more likely it is to go wrong. Whether the intention is 'wear clean socks' or 'update this website', if you can do it in a handful of steps, you'll make fewer mistakes than if you have to do it in a hunched-over armful.

Workflow. The flow. Of Work.

The next time you're making an amend somewhere, watch yourself. Watch which applications you jump between. Don't just make your changes, pay attention to how you do them. Are you jumping between text editor and FTP client? Text editor and web browser? FTP and Subversion? Just take a few minutes to think about which steps you might be able to miss out. For instance, if you use Textmate and Transmit, you can set up DockSend so that pressing ctrl-shift-f then 2 will upload the file you currently have open to the same place on the server. You can now change the editor↔FTP↔browser cycle to editor↔browser.

However...

Seamless does not imply flawless. Don't be tempted to simplify to the stage where you don't need any interaction between making a change and the change being live. If you rely on the fact that your changes 'just happen', you might be tempted not to check. That's the point at which they don't 'just happen'.

This article is modified from a chapter in a book Andrew and I were writing a couple of years ago about web development practical advice. Seeing as we both got too busy to finish it, I'm publishing bits here and there. If you'd like to see these in book form, let me know.

Testing

The web is a visual medium. Well, mostly.

There's no better way to test a visual medium than by looking at it. Look at your site in as many browsers as you can. If you've already got as many browsers installed on your development computer as you can fit, get another computer and install some more. Either that or run a Virtual Machine.

Definition: Virtual Machine (VM)

Applications like VirtualBox, VMWare or Parallels allow you to run an entire computer within your computer. It is a self-contained system that doesn't interact with your own machine meaning you can have IE6 installed on one VM, IE7 on another and IE8 on a third. All running in a window on your iMac. Shiny.

If you can't do that easily, you could use one of the growing number of browser testing services. These are server rooms packed with computers running Virtual Machines and automated systems to which you supply a URL, wait a few moments and get shown an image (or several hundred images) showing your URL in different browsers on different platforms. Some of the more sophisticated services allow you to scroll down a long page or specify different actions, text entry or mouse events you want to see triggered. These services can be exceptionally useful when it comes to developing HTML e-mails as there are some rare and esoteric e-mail clients out in the wild. Litmus does an excellent job at virtualised testing for HTML e-mails. On that note, the Campaign Monitor library of free HTML e-mail templates is a great place to start, learn and possibly finish when working on an HTML e-mail.

There is also a place for automated testing for some things. Recently, there has been a bit of a movement away from validating code as the purpose of web development is not to make it 'check a box' on a merely technical level, it is to get the message across via the design however possible. However, validation is still the best and easiest way to check your syntax. Syntax errors are still the main cause for mistakes appearing in your web sites and are the easiest thing to fix. Don't assume IE is wrong. Again, if you're keen on HTML e-mails, here's a great post on the Litmus blog.

This article is modified from a chapter in a book Andrew and I were writing a couple of years ago about web development practical advice. Seeing as we both got too busy to finish it, I'm publishing bits here and there. If you'd like to see these in book form, let me know.

Torch

Concept

Another proof-of-concept game design prototype. This is kind of a puzzle game. Ish. It's mostly a simple maze game but one in which you can't see the walls. You can see the goal all the time but you are limited to only being able to see the immediate area and any items lit up by your torch. You control by touching or clicking near the torch. The character will walk towards as long as you hold down. You can move around and it'll follow.

Torch v0.1

The first 10 levels are very easy and take practically no time at all. After that, they get progressively harder for a while before reaching a limit (somewhere around level 50, I think). The levels are procedurally generated from a pre-determined seed so it would be possible to share high scores on progression or time taken without having to code hundreds of individual levels.

Items that could be found around the maze (but aren't included in this prototype) include:

• Spare torches which can be dropped in an area and left to cast a light
• Entrance/exit swap so that you can retrace your steps to complete the level
• Lightning which displays the entire maze for a few seconds (but removes any dropped torches)
• Maze flips which flip the maze horizontally or vertically to disorient you.

I worked on this for a few months (off and on) and found it to be particularly entertaining with background sound effects of dripping water, shuffling feet with every step, distant thunder rumbling. It can be very atmospheric and archaeological at times.

Slightly technical bit

The game loop and draw controls are lifted almost line-for-line from this Opera Dev article on building a simple ray-casting engine. I discarded the main focus of the article - building a 3D first-person view - and used the top-down mini map bit on its own. The rays of light emanating from the torch are actually the rays cast by the engine to determine visible surfaces. It's the same trick as used in Wolfenstein 3D but with fewer uniforms. It's all rendered on a canvas using basic drawing functionality.

The audio is an interesting, complicated and confusing thing. If I were starting again, I'd look at integrating an existing sound manager. In fact, I'd probably use an entire game engine (something like impact.js, most likely) but it was handy to remember how I used to do stuff back in the days when I actually made games for a living. Most of all, I'd recommend not looking too closely at the code and instead focusing on the concept.

Go, make.

As with Knot from my previous blog post, I'm not going to do anything with this concept as I'm about to start a big, new job in a big, new country and I wanted to empty out my ‘To do’ folder. The code's so scrappy, I'm not going to put it up on GitHub along with my other stuff, I seriously do recommend taking the concept on its own. If you are really keen on seeing the original code, it's all unminified on the site so you can just grab it from there.

The usual rules apply, if you make something from it that makes you a huge heap of neverending cash, a credit would be nice and, if you felt generous, you could always buy me a larger TV.

Torch v0.1

Knots

Game design

I had an idea around Christmas for a new game design. it was basically to be a single or multi-player game where you would tie knots in a piece of ‘string’. The single-player game would focus on discovering new patterns while the multi-player one would be a time-based challenge where each would tie a knot, pass it over to their opponent and the first to untie it would win.

I built a basic prototype and had some fun inventing a few interesting knots (my favourite being ‘Yoda’) but didn't get round to doing anything with it. As I'm tidying up all my projects before immigrating to Germany, I figured I should just go ahead and put this out into the world.

Knot v0.1

Slightly technical bit

The system is built using an invisible 3x3 grid and 4 quadratic curves. The curves' start, control and end points are the centre of a tile. When you move a tile, it basically swaps position with the tile nearest where you drop it. This can also be done with multiple tiles simultaneously if you're on a multi-touch device. You can see the tiles if you enable debug mode. You can also switch between the two colour schemes it has at the moment.

The only addition I've made to it since Christmas was to add on a system to allow players to save knots back into the system. I've only built in 22 patterns so if you make a nice, interesting or funny one, give it a recognisable name (nothing too rude, please) and save it, it will then be available to everybody who plays with it. You can also set whether the pattern is to count as ‘matched’ when rotated 90°, 180°, flipped vertically or flipped horizontally. Calculating the isomorphisms (when two knots look the same to the viewer but aren't necessarily the same to the computer) was probably the trickiest bit of the system.

Go, make.

If you're interested in taking the idea further, grab the code from GitHub and make something from it. The usual rules apply, if you make something from it that makes you a huge tonne of neverending cash, a credit would be nice and, if you felt generous, you could always buy me a larger TV.

Knot v0.1

HTML5 for Large Public Bodies

Your country needs you...

...to stand at the cutting edge of technology.

Sounds awfully impressive, don't you think?

There are quite a few regulations to bear in mind and comply with when developing a website for a Government organisation or any large public body. This has lead to a lot of sites being developed in a very defensive manner, ensuring safe compliance at the expense of a great and consistent user experience.

This video features a presentation about how large publicly-liable institutions should and can embrace the latest in web technologies without sacrificing standards. By embracing them, in fact.

The content of this was developed while planning and building the National Museums Scotland website which launched last November. The messages presented are applicable to museums, galleries, swimming pools, councils, anywhere, really.

If you're a techie person in the cultural or government sector, you might find this useful in convincing others to take advantage of the latest cool (and useful) technologies.

Video

HTML5 for Large Public Bodies from Simon Madine on Vimeo.

Links from the presentation

• PolyFills
• .NET magazine article
• (Almost) complete list of polyfills
• Dive into HTML5
• HTML5 Boilerplate
• Smashing Magazine
• HTML5 Doctor
• AlphaGov

Slideshow

The source for the slides is available online although it's mostly webkit-friendly. I realise the irony of a presentation about cross-platform HTML5 not being a great example itself of that but it does degrade adequately. If I get the time in the future, I'll tidy it up. An actual good (and relevant) example of cross-platform web technologies is the National Museums Scotland website itself which performs fantastically across all manner of platforms.

Psychopathic megalomaniac vs Obsequious sycophant?

- or -

Tabs vs Spaces?

Which are you? Is there any middle ground?

In any modern, code-friendly text editor, you can set tab size. This means that one tab can appear to be as wide as a single space character or - more commonly - 2 or 4 spaces. It's even possible in some to set any arbitrary size so there's nothing stopping you setting tabs to be 30 spaces and

format() {
                              your code() {
                                                            like;
                              }
                              this;
}

However you do it, the point is that using tabs allows the reader personal preference.

Indenting your code using spaces, however, is completely different. A space is always a space. There's actually a simple conversion chart:

• 1 = One
• 2 = Two
• 3 = Three
• and so on.

The fundamental difference here is that the person reading the code has no choice about how it's laid out. Tabs means you get to read the code how you want to, spaces means you read the code how I want you to. It's a subtle difference but an important one.

Space indenting is therefore perfectly suited to psychopathic megalomaniacs who insist their way is right while tab indenting is for obsequious sycophants who are putting the needs of others above themselves. Sure, there may be lots of grades in-between but why let moderation get in the way of a barely coherent point?

Curiously, neither of these extremes feels a great need to comment their code. One sees no need as their code is for themselves and they already understand it, the other assumes that if they can understand it, its purpose is so obvious as to be trivial. Both are wrong here.

There's unfortunately no real middle ground here. Teams must agree amongst themselves what side of the fence they fall down on. It is entirely possible to set many text editors to automatically convert tabs to spaces or spaces to tabs on file save but if you're doing that, you'd better hope your favourite diff algorithm ignores white space otherwise version control goes out the window.

This article is modified from a chapter in a book Andrew and I were writing a couple of years ago about web development practical advice. Seeing as we both got too busy to finish it, I'm publishing bits here and there. If you'd like to see these in book form, let me know.

Archiving Tweets

Archiving Tweets - Technical

Twitter has an amazing amount of data and there's no end to the number of ideas a coffee-fuelled mind can come up with to take advantage of that data. One of the greatest advantages of Twitter – its immediacy, its currency, its of-the-moment-ness – is, however, also one of the drawbacks. It is very easy to find out what people are thinking about now, yesterday or a few days ago but trying to find something from more than a week ago is surprisingly tricky. The standard, unauthenticated search only returns the last 7 days of results, beyond that, there's nothing. You can get around this by building custom applications which authenticate themselves correctly with Twitter and provide a level of traceability but that's quite tricky.

The easiest way to archive search results for your own personal use is actually surprisingly simple. Every search results page includes an RSS feed link1. This will be updated everytime there are new results. Simply subscribe to this feed with a feed reader (Google Reader is simple and free) and your results will be archived permanently. This is great if you're searching for personal interest stuff but doesn't work so well if you want to share the results with the public.

1 Note: The feed link only appears for searches performed on search.twitter.com, not for searches performed via the standard web interface at https://twitter.com/#!/search/

This was the problem I was presented with when I was asked to build a tweet-archiving mechanism for Museum140's Museum Memories (#MusMem) project. Jenni wanted some kind of system that would grab search results, keep them permanently and display them nicely. I didn't want to create a fully OAuth-enabled, authenticated search system simply because that seemed like more work than should really be necessary for such a simple job. Instead, I went down the RSS route, grabbing the results every 10 minutes and storing them in a database using RSSIngest by Daniel Iversen. The system stores the unique ID of each Tweet along with the time it was tweeted and the search term used to find it. The first time a tweet is displayed, a request is made to the Twitter interface to ask for all the details, not only of the tweet, but also of the user who tweeted it. These are then stored in the database as well. This way, we don't make too many calls to Twitter and we don't get blocked.

If you want your own Tweet Archive, I've put the code on GitHub for anyone to use. It requires PHP, MySQL and, ideally, a techie-type to set it up.

Archiving Tweets - Non-technical

With the technical side out of the way, we're left with the human issues to deal with. If you're automatically saving all results with a particular phrase, all a malicious person needs to do is include that phrase in a spam-filled tweet or one with inappropriate language and suddenly, it's on your site, too. If you aren't going to individually approve each tweet before it is saved, you must keep a vigilant eye on it.

The other thing which has turned out to be a problem is the Signal-to-Noise ratio. When we initially decided on the hashtag #MusMem, nobody else was using it. To use Social Media parlance, there was no hashtag collision. The idea was to encourage people to use it when they wanted their memories stored in the MemoryBank. Unfortunately, it is now being used by anyone tweeting anything related to Museums and Memories. This is particularly troublesome at the moment as this month is International Museum month, one of the themes of which is ‘Memory’ (which is why we built the MemoryBank in the first place). This means that the permanent memories we want stored (the Signal) are getting lost in the morass of generic Museum Memories (the Noise). There is no way to solve this problem algorithmically. If we are to thin it down, we actually need to manually edit the several thousand entries stored.

If anyone can think of a solution to this issue, please let everybody know – the world needs you.

Investigating IE's innerHTML

Be warned, this is a long and quite techy post that rambles a bit. It's about JS, jQuery, IE and XML namespaces. Be careful.

I'm currently working on a tool which uses JS to parse an XML document and output it as JSON. Straightforward enough, you'd think. The issue I'm fighting against crops up when the XML tags have an arbitrary namespace. True enough, some of the files I'll be processing have this namespace defined at the top of the document but the tool has to be robust enough to cope when they don't.

To cut a long story short, IE6, IE7 and IE8 have an interesting attitude to innerHTML when the tags you are trying to insert have a namespace. IE9 seems to do the job as you'd expect. I've created some jsFiddle pages which try and identify the issues. They both use QUnit and the test code is included below.

createElement nodeName vs get(0).nodeName

View this on jsFiddle.net

I started off using jQuery to help identify this as the tool uses jQuery for the XML heavy-lifting. The two tests in this demo create elements in two different ways. First, we create the element using document.createElement and grab the nodeName then we use jQuery's constructor and use get(0) to grab the bare DOM element's nodeName. Also, in this first set of tests, we're creating non-standard elements.

test("Compare elements without namespace", function() {
 var element1, element2;

 element1 = document.createElement('spud').nodeName;
 element2 = $('<spud/>').get(0).nodeName;

 equals(element1, element2, "We expect these to match");
});

The code above runs fine everywhere – IE, FireFox, Opera, Chrome, etc. etc. Good.

test("Compare elements with namespace", function() {
 var element1, element2;

 element1 = document.createElement('a:spud').nodeName;
 element2 = $('<a:spud/>').get(0).nodeName;

 equals(element1, element2, "We expect these to match");
});

This runs fine in non-IE browsers, they all report the nodeName as 'a:spud'. IE now reports the nodeName as 'spud'. Ah. I dug through the jQuery source, tracking down the bare roots of the constructor and eventually figured out that just looking at the element itself isn't going to provide any clues. The bit that does the actual string-to-elements work (somewhere around line 5619 in jQuery 1.5.2) creates a container div then injects the (slightly modified) code as innerHTML. The issue must be in IE's interpretation of innerHTML, I thought to myself. And then to you by writing it here.

.innerHTML aside

or ‘jQuery is clever’

Before we continue with this long and, ultimately, unnecessary investigation into namespaces, I have to take a small diversion to cover some smart stuff jQuery does. One thing in particular, in fact. Around that line I mentioned earlier (5619-ish), an extra bit of text is inserted into the innerHTML to cope with IE's oddity. If you are trying to create a non-standard element using innerHTML, IE will not complain but also just do pretty much nothing at all:

        var div = document.createElement('div');
        div.innerHTML = '<spud></spud>';
        alert(div.innerHTML);

The above code will alert '<spud></spud>' in most browsers but '' in IE. What jQuery does is firstly wrap your element in an extra <div></div> (producing '<DIV></DIV>') then prepends the word 'div' to that. The innerHTML reported by IE is now 'div<DIV><SPUD></SPUD></DIV>'! There it is! Next, the extra gubbins is removed by calling .lastChild and you're left with innerHTML = '<SPUD></SPUD>'. That's pretty darned clever.

.innerHTML vs document.appendChild

View this on jsFiddle.net

Back on track. Armed with this little trick, we can reliably test innerHTML in IE using non-standard elements.

module("Known elements (span)");
 test("Compare elements without namespace", function() {
  var div1, div2;

  div1 = document.createElement('div');
  div1.innerHTML = '<span></span>';

  div2 = document.createElement('div');
  div2.appendChild(document.createElement('span'));

  equals(div1.innerHTML.toLowerCase(), div2.innerHTML.toLowerCase(),
     "We expect these to match");
 });

 test("Compare elements with namespace", function() {
  var div1, div2;

  div1 = document.createElement('div');
  div1.innerHTML = '<u:span></u:span>';

  div2 = document.createElement('div');
  div2.appendChild(document.createElement('u:span'));

  equals(div1.innerHTML.toLowerCase(), div2.innerHTML.toLowerCase(),
     "We expect these to match");
 });

The first test in this pair runs fine everywhere exactly as we'd hope and expect. The second fails miserably in IE. Let us quickly run the same test with unknown elements just to make sure we're identifying the right problem:

module("Unknown elements (spud)");
 test("Compare elements without namespace", function() {
  var div1, div2;

  div1 = document.createElement('div');
  div1.innerHTML = 'div<div>' + '<spud></spud>' + '</div>';
  div1 = div1.lastChild;

  div2 = document.createElement('div');
  div2.appendChild(document.createElement('spud'));

  equals(div1.innerHTML.toLowerCase(), div2.innerHTML.toLowerCase(),
     "We expect these to match");
 });
 test("Compare elements with namespace", function() {
  var div1, div2;

  div1 = document.createElement('div');
  div1.innerHTML = 'div<div>' + '<u:spud></u:spud>' + '</div>';
  div1 = div1.lastChild;

  div2 = document.createElement('div');
  div2.appendChild(document.createElement('u:spud'));

  equals(div1.innerHTML.toLowerCase(), div2.innerHTML.toLowerCase(),
     "We expect these to match");
 });

As before, the first test in this pair works fine, the second fails. Cool. Or not. Either way, you can now see that it doesn't really matter whether the elements are standard or custom and that little diversion we took earlier really was unnecessary. Still, you know more now about some of the cleverness in jQuery than you did before.

It turns out the reason IE reports the nodeNames as the non-namespaced ones is because it has been busy behind the scenes and added an extra XML namespace prefix into our current context. The innerHTML of the div we filled up using innerHTML has been modified to:

  <?xml:namespace prefix = u />
  <u:span></u:span>

Where'd that namespace declaration come from?! Goshdarnit, IE. From its point of view, within that little context, u:span is equivalent to span

The most stripped-down example

View this on jsFiddle

Seriously, it does not get more fundamental than this.

element = document.createElement('div');
testHTML = '<div></div>';
element.innerHTML = testHTML;
element.innerHTML.toLowerCase() == testHTML.toLowerCase() 

The last line there is true for all browsers.

element = document.createElement('div');
testHTML = '<a:div></a:div>';
element.innerHTML = testHTML;
element.innerHTML.toLowerCase() == testHTML.toLowerCase() 

The last line there is true for all browsers except IE 6, 7 and 8!

In conclusion?

Ultimately, there are no winners here. Identifying the problem is quite different from fixing it. I've added a note to the relevant jQuery bug in the tracker but it's not so much a bug in jQuery as a humorous IE quirk. There's some talk of refactoring the .find() method to handle more complicated tagnames so this might get picked up then. The fix will probably be something along the lines of checking the outcome of the innerHTML doesn't have an unexpected namespace declaration when the selector has a colon in it:

div.replace( /]*>/g, '' )

I'd submit the patch myself but I'm having difficulty getting unmodified jQuery to build on any of my machines without failing most of the QUnit tests. I've probably typed something wrong.

Autogenerated Everything

After seeing this collection of the 892 different ways you can partition a 3 x 4 grid1, I was struck by a thought. If these were generated as HTML templates, they could be combined with a couple of other useful websites and become a nice, API-driven site builder2.

The process

• On the site-building webpage, you'd enter a few keywords describing the site you want and drag a slider along between 1 and 12 to specify how many content areas you want. The value from the slider would be used to pick a template randomly from the number available for that combination of panels.
• This template would be dropped into the middle of an HTML 5 boilerplate (possibly generated via Initializr)
• The keywords would be passed to ColorAPI to generate an asthetically pleasing colour-scheme
• The keywords would then be passed to FlickHoldr along with the dimensions of some of the areas from the template to get relevant imagery
• Grab some lorem ipsum of the right length from LoremIpscream to fill out the content areas of the site
• Done. Your asthetically pleasing, nicely designed site is ready to download within a few seconds.

Once this service has been created, I'm fairly sure me and the rest of the industry will be out of a job.

1 Technically, there are 3,164 ways you can partition the grid but most of them are vertical or horizontal mirrors of the 892 mentioned.

2 There are a few references to 3x4grid.com which is apparently going to be an API-accessible collection of these layouts but there doesn't seem to be anything there yet.

8 bit alpha

8-bit alpha

Shelvist

I've been a bit quiet for the last couple of weeks. Now I can reveal why!

All my spare time has been taken up building Shelvist (well, that and looking after a 4-month old). It's a way of keeping track of which books you have read, which ones you are currently reading and which ones you want to read. That's it.

Just over a year ago, I started listening to a lot of audiobooks and, 12 months after I subscribed to Audible, I decided I wanted to see just how many books I'd read*. All the existing book tracking sites (e.g. Shelfari, Goodreads) focus on recommending, rating, reviewing which just seemed like hard work.

*Listening to an audio book counts as reading. It just sounds weird if you tell people you listened to a book.

Building this was also a chance to learn some new tech. I've been wanting to play with CakePHP for a while so this seemed like the best opportunity. It's been a while since I used any kind of framework and I've never used a PHP MVC framework (although I did build one last year as an intellectual exercise).

I got over 90% of the site built in spare time over about 5 days and most of that was spent just reading the CakePHP docs. The reason for the lengthy delay between initial build and launch is down to that last 10%. As often happens with frameworks, nearly everything you'd ever need to do is bundled into some nice, easy to access functionality. That is, after all, kinda the point of a framework. It's the stuff that makes your product or website unique that proves trickier. I won't go into any details just now although I might come back to it in a later post.

More later, first of all, go shelve some books: Shelvist.

Scoped Style

1. Have a look at a demo
2. Check out the source on GitHub
3. Download it and start scoping some styles.

CSS Scoped

Update: Bam! Plugin'd

One of the things in HTML5 that caught my eye a while back was the new 'scoped' attribute for style tags. The idea of it is that you can include a style element mid-document, mark it as scoped and its declarations will only apply to the style elements parent element and its child elements. The styles won't affect anything outside this. The biggest problem with bringing in this attribute is that it's not backwards compatible. If you include a style block mid-page, its declarations will be applied to ever element whose selector matches, inside or outside scope. It is anti-progressively enhancable. This means that designers and developers can't start using it until there's enough support. What we need is another of those JS tricks to make it usable.

Possibly the idea of having a style block dropped into the middle of a semantic document strikes you as a distasteful but thinking practically, this could make CMS workflows a bit easier and keep global stylesheets clean of lots of one-off specific styles. For example, this post describing striped CSS backgrounds could have benefited from having the demo directly within the body of the post but, without going into the backend to add an extra id, this could have messed up the layouts for all other posts.

My first attempt at solving this problem with JS involved generating a unique class, adding it to the parent element and then parsing the style blocks using JSCSSP so that I could rewrite them with the new class to add specificity. This approach only worked for the most basic declarations, unfortunately. The parser worked perfectly but there's a lot of detail in CSS specificity that mean this would be a much larger problem than I thought.

My second attempt involved:

1. Allowing the style blocks to affect everything on the page (at which point, the elements in-scope look right, those out-of-scope look wrong)
2. Using JS to read the current computed styles of each element in-scope and copy them to a temporary array
3. Emptying out the scoped style element (in-scope look wrong, out-of-scope looks right)
4. Copying the styles back from the temporary array onto each element

The first couple of versions only work in webkit, the latest (below) works in mozilla, too.

This worked great unless you had more than one scoped block – step 1 allowed scoped styles to affect each other.

The current attempt involves temporarily emptying all other scoped styles before taking the computed styles from a block. I'm now just thinking that this method might not work if you have multiple scoped blocks within the same context. Oh well, there's something to fix in the future.

This is where I'm at just now.

Yes, it's a mess, yes the JS is scrappy and yes, it doesn't currently work in IE but I'll get round to that next. It took long enough to get it working in Firefox as there's no simple way to convert a ComputedCSSStyleDeclaration to a string in Mozilla unlike Webkit's implementation of cssText or IE's currentStyle. I might even make it into one of those new-fangled jQuery plugins everyone's using these days.

Testing CSS3 stuff

You may have seen Google's 'Watch this space' advertising appearing all over the place. They have quite eye-catching diagonally striped backgrounds in various colours. A couple of days ago, I was wondering how easy it would be to recreate this in CSS without images. Unfortunately, the state of CSS 3 is such that some things work wonderfully, some just plain don't (scoped attribute, I'm looking at you). The following code relies on vendor extensions and so, unless you're willing to tend it and correct it after the spec is finalised, don't use this on a production server.

The most obvious thing to notice from the following code, though, is the competing syntax for the repeating linear gradient style. Mozilla have separated it into a distinct style (-moz-repeating-linear-gradient) while Webkit have built it as an option to their general gradient style (-webkit-gradient).

body {
	background-image: -moz-repeating-linear-gradient(
		-45deg,
		transparent,
		transparent       25%,
		rgba(0,0,0,0.15)  25%,
		rgba(0,0,0,0.15)  50%,
		transparent       50%,
		transparent       75%,
		rgba(0,0,0,0.15)  75%,
		rgba(0,0,0,0.15)
	);
	background-image: -webkit-gradient(
		linear,
		0% 0%,
		100% 100%,
		from(transparent),
		color-stop(0.25, transparent),
		color-stop(0.25, rgba(0,0,0,0.15)),
		color-stop(0.50, rgba(0,0,0,0.15)),
		color-stop(0.50, transparent),
		color-stop(0.75, transparent),
		color-stop(0.75, rgba(0,0,0,0.15)),
		to(rgba(0,0,0,0.15))
	);
}

To get a better idea of what this does, view source on this demo page. This includes a button to change the class on the body (using JS) which simply changes the background colour – the stripes are semi-transparent on top of that. Remember, due to the vendor prefixes, this only works in -moz or -webkit browsers.

It's supposed to look like this:

Writing a Plex Plugin Part III

This is the final part of my walkthrough of the Plex Media Server Transmission plugin.

Right.

We've done the required built-in functionality (preference management, for instance) and the bits that talk to Transmission itself. Basically, we're done. Anything else added here is just extra. That is, of course, the best reason to add stuff here. As I have previously ranted at length, there's no point doing anything if you aren't trying to do it as well as it possibly can be done. In this particular instance, that manifests itself in the ability to browse, search for and download torrents all within the Plex client interface.

EZTV

I love EZTV. It makes things easy. Previous versions of this plugin included an EZTV search but after rooting around in the source of the µTorrent plugin, I found some nifty code which turned me onto the clever XML parsing Plex can do.

Note: although some of this stuff looks clever, all the cleverness was done by the Plex dev team and the author of the µTorrent plugin. I'm a good copy-and-paster.

This function grabs the H2s from the page http://ezrss.it/shows/. If you go there, you'll see that the page lists every TV show in EZTV. The original µTorrent function listed everything but there are a lot of shows there now so it was actually taking a long time just to get that list. As they've split the page up by section, we can just grab the bits we want. This is going to be a full page in Plex (not a popup) so we're using a MediaContainer.

def TVShowListFolders(sender):
  dir = MediaContainer()

Using the built-in XML module, we can simply pass in a URL and get back an object containing the hierarchical structure of the entire page. Seriously, how simple is this? As it's HTML, add in the option isHTML=True.

  showsPage = XML.ElementFromURL(
                'http://ezrss.it/shows/', 
                isHTML=True, 
                errors='ignore'
              )

Now that we have the whole page structure, take the chunks of the page we want. All the sections we want (and one we don't) are divs with the class 'block' so use that in xpath to pull them out.

  blocks = showsPage.xpath('//div[@class="block"]')

The first block is the one we don't want (if you look at the page, it's the one that lists all the letters) so we remove it.

  blocks.pop(0)

For each of the remaining blocks, find the text in the first H2. That is the letter title of the section ('A', 'B', 'C', etc). Add that to Plex as a menu item then return the entire list.

  for block in blocks:
    letter = block.xpath("h2")[0].text
    dir.Append(
      Function(
        DirectoryItem(
          TVShowListSubfolders,
          letter,
          subtitle=None,
          summary=None,
          thumb=R(ICON),
          art=R(ART)
        ),
        letter=letter
      )
    )
  return dir

I hope I'm not the only one impressed with that (although I have a feeling I might be). Using just a couple of lines from the XML module and a sprinkle of xpath and we've got another menu, dynamically generated from a third-party website. If EZTV ever change their layout, it should be a simple matter of changing the xpath to match and we're done. Again.

We can now do the same again but this time, we only pull out a single section based on the letter passed in.

def TVShowListSubfolders(sender, letter):
  dir = MediaContainer()
  showsPage = XML.ElementFromURL(
                'http://ezrss.it/shows/',
                isHTML=True,
                errors='ignore'
              )
  blocks = showsPage.xpath(
            '//div[@class="block" and h2 = "%s"]' % letter
           )

Remembering to ignore any 'back to top' links, write out a list of the shows in this section. These will call the TVEpisodeList method next.

 for block in blocks:
  for href in block.xpath('.//a'):
   if href.text != "# Top":
    requestUrl = "http://ezrss.it" + href.get("href") + "&mode=rss"
    dir.Append(
     Function(
      DirectoryItem(
       TVEpisodeList,
       href.text,
       subtitle=None,
       summary=None,
       thumb=R(ICON),
       art=R(ART)
      ),
      name=href.text,
      url=requestUrl
     )
    )
 return dir

This lists all available torrents for the chosen show. By this point, you should be familiar with how this works. We're using the XML module to grab the page at the URL (this time it's an RSS feed so we don't need to parse it as HTML); we use XPath to iterate through the items in the feed; we generate a menu item from the data which will call a function when selected; we append that to a MediaContainer then return the whole thing to Plex. Done. The AddTorrent function was defined higher up.

def TVEpisodeList(sender, name, url):
 dir = MediaContainer()
 feed = XML.ElementFromURL(url, isHTML=False, errors='ignore').xpath("//item")
 for element in feed:
  title = element.xpath("title")[0].text
  link = element.xpath("link")[0].text
  dir.Append(
   Function(
    DirectoryItem(
     AddTorrent,
     title,
     subtitle=None,
     summary=None,
     thumb=R(ICON),
     art=R(ART)
    ),
    torrentUrl=link
   )
  )
 return dir

Adult considerations...

There is currently a section in the plugin which will allow you to search IsoHunt. This might get dropped in future versions of the plugin as results from IsoHunt are almost exclusively...ahem...adult, regardless of search terms. Sure, that might be exactly what you were looking for but if you were actually looking for Desperate Housewives, you might be surprised when your file comes down and it's actual 'desperate housewives'...

Search EZTV

The final part is a straightforward search of EZTV. The interesting thing to note is that this uses a different type of menu item. Where normally, you'd use a DirectoryItem in a Function, this uses an InputDirectoryItem in a Function. This type of menu item will pop open an on-screen keyboard before calling the target function giving you the opportunity to grab some user input.

It's appended to the menu in the usual way:

 dir.Append(
  Function(
   InputDirectoryItem(
    SearchEZTV,
    L('MenuSearchTV'),
    "Search the TV shows directory",
    summary="This will use EZTV to search.",
    thumb=R(SEARCH),
    art=R(ART)
   )
  )
 )

By the way, I think there's a minor bug in the InputDirectoryItem in that it doesn't like it when subtitle is passed as a named argument. I should probably file that as a bug with Elan.

When the user has entered their input and submitted, the named Function SearchEZTV is called with the standard argument sender and the extra argument query containing the user's input.

This function was a lot longer in the previous version of the Framework. It was so much simpler this time round.

def SearchEZTV(sender, query=None):
  dir = MediaContainer()
  url = "http://ezrss.it/search/index.php?simple&mode=rss&show_name="
  if query != None:
   url += "%s" % query
  feed = XML.ElementFromURL(
          url, 
          isHTML=False, 
          errors='ignore'
         ).xpath("//item")
  if feed == None:
    return MessageContainer("Error", "Search failed")
  if len(feed) == 0:
    return MessageContainer("Error", "No results")
  for element in feed:
    title = element.xpath("title")[0].text
    category = element.xpath("category")[0].text
    link = element.find("enclosure").get("url")
    size = prettysize(int(element.find("enclosure").get("length")))
    dir.Append(
      Function(
       DirectoryItem(
        AddTorrent,
        title,
        subtitle=None,
        summary="Category: %s\nSize: %s" % (category,size),
        thumb=R(ICON),
        art=R(ART)
       ),
      torrentUrl=link
     )
    )
  return dir

Done

That's it. The only other little thing to mention is how handy it is to use the built-in Log function. The first argument is a standard Python string, the second is 'Should this only turn up in the console when in debug mode?' to which the answer will almost always be 'True'. There is a third argument but unless you're messing with character encodings, you don't need to worry about it.

Log("Message to log is: %s %d" % (errorString, errorCode), True)

Go, make...

If you made it to the end here, you're probably either keen to start making your own Plex plugins or my wife who I am going to get to proofread this. Assuming you're the former, here are some handy links:

• Online plugin development manual

  There are plenty of bits missing but it's still the best reference available for the framework.

• Plex forums

  Particularly the Media Server Plugins forum

• Plex Plugins Lighthouse

  This is where bugs are filed, suggestions made and final plugin submission happens. It's handy for picking little tips if someone else has had the same problem as you.

If you want to use the plugin, it's available in 'Plex Online' in Plex/Nine or 'Plex App Store' in Plex/Eight. If you'd like to read through the complete source, you can download the zipped .Bundle.

Transmission Plugin for Plex Media Server v1.0 [Zip - 1.1MB]

Writing a Plex Plugin Part II

This is part two of my walkthrough of the Plex Media Server Transmission plugin.

The previous part dealt with the basic required functions and preparing the main menu. This bit goes through the torrent control and the next will cover the built-in site-scraping functionality. To be honest, I'm not sure how much of this middle section will be of use to anyone but other torrent client plugin makers. The cool stuff really happens in the next one. Think of this as the difficult second album that you have to listen to before the return-to-form third.

Listing the torrents

This is the main interface to Transmission. Using the RTC method from before, this prepares a request to send via HTTP and reacts depending on the result (or error) we get back. First of all, we say we want information ('torrent-get') and we specify what info we want (the 'fields')

def TorrentList(sender):
  error, result  = RTC("torrent-get",
    { "fields": [
      "hashString","name","status",
      "eta","errorString",
      "totalSize","leftUntilDone","sizeWhenDone",
      "peersGettingFromUs",  "peersSendingToUs",  "peersConnected",
      "rateDownload",      "rateUpload",
      "downloadedEver",    "uploadedEver"
    ] }
  )

If we get an error back, we check what it was.

  if error != None:
    if error != "Connection refused":
      return MessageContainer(
          "Transmission unavailable",
          "There was an unknown error."
      )
    else:
      return MessageContainer(
          "Transmission unavailable",
          "Please make sure Transmission is installed and running."
      )

Now we have our information, we create a MediaContainer to display it. We'll be building these entries up as if they were standard library MediaItems although the final action will not be to play them.

  elif result["torrents"] != None:
    dir = MediaContainer()

For each set of torrent information we get back, we need to prepare the info and make it pretty.

    for torrent in result["torrents"]:
      progress    = 100;
      summary = ""
 
      if torrent["errorString"]:
        summary += "Error: %s\n" % (torrent["errorString"])

If we have some time until we're done and we're not seeding, display the progress as "12.3 MB of 45.6 GB (0%)". We add this to the MediaItem's summary field. This is where we use the prettysize and prettyduration functions we imported at the top. They take a computer-friendly value (1048576 bytes) and return a human-friendly one (1MB).

  if torrent["leftUntilDone"] > 0 and
    torrent["status"] != TRANSMISSION_SEEDING:
    progress = ((torrent["sizeWhenDone"] - torrent["leftUntilDone"]) /
          (torrent["sizeWhenDone"] / 100))
 
    summary += "%s of %s (%d%%)\n" % (
        prettysize(torrent["sizeWhenDone"] - torrent["leftUntilDone"]),
        prettysize(torrent["sizeWhenDone"]), progress
      )

Similarly, if there's an estimated time until the file is finished downloading, add that to the summary as "3 days remaining"

    if torrent["eta"] > 0 and torrent["status"] != TRANSMISSION_PAUSED:
      summary += prettyduration(torrent["eta"]) + " remaining\n"
    else:
      summary += "Remaining time unknown\n"

Display download status ("Downloading from 3 of 6 peers") and download and upload rates ("Downloading at 3KB/s, Uploading at 1KB/s").

  if torrent["status"] == TRANSMISSION_DOWNLOADING:
    summary += "Downloading from %d of %d peers\n" % (
      torrent["peersSendingToUs"],
      torrent["peersConnected"])
    summary += "Downloading at %s/s\nUploading at %s/s\n" % (
      prettysize(torrent["rateDownload"]),
      prettysize(torrent["rateUpload"]))

For all other downloading statuses, we don't need extended information so we just return a human-friendly version of the status we get back (we do this via another method below).

  else:
    summary += TorrentStatus(torrent)

If we're seeding (the torrent has finished downloading and we're just uploading now), write out information about the uploading.

  else:
    if torrent["status"] == TRANSMISSION_SEEDING:
      summary += "Complete\n"
      progress=100
      if torrent["downloadedEver"] == 0:
        torrent["downloadedEver"] = 1

"45.6GB, uploaded 22.8GB (Ratio 0.50)" and some detail about the people we're uploading to.

  summary += "%s, uploaded %s (Ratio %.2f)\n" % (
    prettysize(torrent["totalSize"]),
    prettysize(torrent["uploadedEver"]),
    float(torrent["uploadedEver"]) / float(torrent["downloadedEver"]))
  if torrent["status"] == TRANSMISSION_SEEDING:
    summary += "Seeding to %d of %d peers\n" % (
      torrent["peersGettingFromUs"],
      torrent["peersConnected"])
    summary += "Uploading at %s/s\n" % (
      prettysize(torrent["rateUpload"]))

Icon generation

The next addition was a bit of a tricky point for this version of the plugin. Previous versions generated the thumbnail icon dynamically using the Python Imaging Library (PIL). It would create a progress bar showing the exact percentage and write the name of the file on the icon. In order to be able to achieve this, PIL had to be imported which generated a whole bunch of deprecation warnings. There are rumours that a future version of the plugin framework will include some functionality to generate images on-the-fly (possibly a variation of PIL itself) but until then, I decided the best way forward would be to generate the images by hand and include them in the plugin. This meant that I could either generate 101 images (0% - 100%) or display the percentage rounded off. In order to save space, I went with rounding to the nearest 10%.

  nearest = int(round(progress/10)*10)

The last thing to do in this loop (remember, we're still looping through the torrent information we received all the way back up at the top of the page) is to actually add this item. It is added as a PopupDirectoryItem so that selecting it will display a context menu of action choices specified in the TorrentInfo method below. With that, we also add the summary we've spent so long crafting, the percentage icon as the thumb and a couple of extra bits of information to help later functions know what to do.

  dir.Append(
    Function(
      PopupDirectoryItem(
        TorrentInfo,
        torrent["name"],
        summary=summary,
        thumb=R("%s.png" % nearest)
      ),
      name = torrent["name"],
      status = torrent["status"],
      hash = torrent["hashString"]
    )
  )

To finish this menu, we add the same functions that are available to individual torrents but acting on all – 'Pause all' and 'Resume all' – then return the menu to Plex for display.

  dir.Append(
    Function(
      DirectoryItem(
        PauseTorrent,
        L('MenuPauseAll'),
        subtitle=None,
        summary=None,
        thumb=R(PAUSE),
        art=R(ART)
      ),
      hash='all'
    )
  )
  dir.Append(
    Function(
      DirectoryItem(
        ResumeTorrent,
        L('MenuResumeAll'),
        subtitle=None,
        summary=None,
        thumb=R(RESUME),
        art=R(ART)
      ),
      hash='all'
    )
  )
  return dir

Here's the TorrentStatus lookup. Again, this uses the built-in localisation function 'L' to display the text and, again, I still haven't actually translated any of it so there's still only english. I must get round to that eventually.

def TorrentStatus(torrent):
  if torrent == None or torrent["status"] == None:
    return L('TorrentStatusUnknown')
  elif torrent["status"] == TRANSMISSION_WAITING:
    return L('TorrentStatusWaiting')
  elif torrent["status"] == TRANSMISSION_CHECKING:
    return L('TorrentStatusVerifying')
  elif torrent["status"] == TRANSMISSION_PAUSED:
    return L('TorrentStatusPaused')
  elif torrent["status"] == TRANSMISSION_DOWNLOADING:
    return L('TorrentStatusDownloading')
  elif torrent["status"] == TRANSMISSION_SEEDING:
    return L('TorrentStatusSeeding')
  else:
    return L('TorrentStatusUnknown')

Torrent action menu

This is the popup menu displayed when you select one of the listed torrents. The only thing to notice from these is that the option to pause is only shown for active torrents and the option to resume is only shown for paused torrents. The hash mentioned here is the id of the torrent which will be needed later.

def TorrentInfo(sender, name, status, hash):
  dir = MediaContainer()
  dir.Append(
    Function(
      DirectoryItem(
        ViewFiles,
        L('MenuViewFiles'),
        subtitle=None,
        summary=None,
        thumb=R(ICON),
        art=R(ART)
      ),
      hash=hash
    )
  )
  if status == TRANSMISSION_PAUSED:
    dir.Append(
      Function(
        DirectoryItem(
          ResumeTorrent,
          L('MenuResume'),
          subtitle=None,
          summary=None,
          thumb=R(ICON),
          art=R(ART)
        ),
        hash=hash
      )
    )
  else:
    dir.Append(
      Function(
        DirectoryItem(
          PauseTorrent,  
          L('MenuPause'),      
          subtitle=None,
          summary=None,
          thumb=R(ICON),
          art=R(ART)
        ),
        hash=hash
      )
    )
    dir.Append(
      Function(
        DirectoryItem(
          RemoveTorrent,
          L('MenuRemove'),
          subtitle=None,
          summary=None,
          thumb=R(ICON),
          art=R(ART)
        ),
        hash=hash
      )
    )
    dir.Append(
      Function(
        DirectoryItem(
          DeleteTorrent,
          L('MenuDelete'),
          subtitle=None,
          summary=None,
          thumb=R(ICON),
          art=R(ART)
        ),
        hash=hash
      )
    )
  return dir

Torrent action functions

Each of the torrent action functions called (ViewFiles, ResumeTorrent, etc.) could have been references to a more generic function with an action option passed in but I decided to keep them distinct and separate so that any extra customisation that might be done later would be easier to do rather than hacking it in. This isn't so much a problem with this plugin but if this were to be adapted for another torrent client, there might be specific hoops that needed jumped through.

I won't go through each of them in detail as they are all very similar. Instead, I'll just describe one of them – RemoveTorrent.

Each Function menu item (pretty much every menu item in this plugin) takes at least one argument: sender. This tells Plex where it's supposed to return control after it's finished here. The second argument is the id of the torrent we want to act on.

def RemoveTorrent(sender, hash):

We define the action to perform and the arguments to pass to Transmission.

  action = "torrent-remove"
  arguments = { "ids" : hash }

Call Transmission's RPC via the RTC method defined earlier catching the results and any errors returned.

  error, result = RTC(action, arguments)

If there's an error, any error, display it. Otherwise, display a success method. Both of these are displayed as a popup MessageContainer.

  if error != None:
    return MessageContainer("Error", error)
  else:
    return MessageContainer("Transmission", L('ActionTorrentRemoved'))

Okay, so the middle section of the plugin might not be that interesting. Next time I'll cover the clever built-in XML parsing bits and everything'll be cool again. I promise.

Writing a Plex Plugin Part I

In my awesome home cinema setup (about which I'll need to blog sometime), I use many pieces of inter-connected software and hardware. The hub, though, is the OS X computer in the middle running the Plex Media Server and the Transmission BitTorrent client. This post describes the plugin which ties them together.

I've just released version 1.0 of the Transmission plugin for the Plex Media Server. As with all good software projects, there were actually many releases before 1.0 but I thought this was the right time to write up a walk-through of what the code looks like and does. I didn't write the initial version of the plugin but I've maintained it since v0.7 and now pretty much rewritten everything a couple of times. I'll post my walkthrough in a few installments because it's quite long.

The code as it currently stands (v1.0) is available for download below but within a few days you should also be able to download it within 'Plex Online'

Transmission Plugin for Plex Media Server v1.0

As Michael A. pointed out in the comments, I haven't actually mentioned anywhere below what it is that the plugin does. For a quick overview, check out the Plex wiki Transmission page.

This plugin is built to be compatible with Plex Plugin Framework v1 which initially looked like quite a major change from the previous version of the framework but isn't really that different. For Plex plugins, every time an action is performed, the system generates a URL and passes it down through its own menu structure until it gets to a plugin that handles that URL. The plugin then deals with the info in the URL however it likes. In the previous versions of this plugin, the URL was parsed manually and split into strings separated by '/'

For example the URL:

/video/transmission/id/status/action/

would first cause Plex to look in its 'video' menu item and find the plugin that said it could handle 'transmission' URLs. The plugin would take the rest of the string and separate it out. First contact the Transmission application, ask for all information about the torrent called 'id', check its 'status' and then perform the 'action' if possible.

The new plugin does pretty much the same except I no longer manually parse the URL. The plugin still registers with Plex to say it can handle URLs starting '/video/transmission' but then it passes functions through instead of URL-catching menu items. If you're familiar with JavaScript, it's like passing an anonymous function to handle something instead of catching the event manually.

Anyway, here's the python code with a running commentary:

Imports

First we import the Plex Media Server framework:

from PMS import *
from PMS.Objects import *
from PMS.Shortcuts import *

These are a couple of handy functions from the very first version of the plugin which make the outputs much more readable.

from texttime  import prettyduration
from textbytes  import prettysize

This next line actually causes Plex to issue a warning. These libraries won't all be available in the next version of the framework. Instead of urllib and urllib2, developers are to use the built-in HTTP module. Unfortunately, HTTP doesn't allow access to the headers of responses and the Transmission authentication system relies on exchanging a session ID via headers.

import urllib,urllib2,base64

Declarations

Set up some constants to save typing later on

PLUGIN_PREFIX = "/video/transmission"
PLUGIN_TITLE = "Transmission"

This is the first call to the Localisation module. Plex plugins allow for complete string localisation via a JSON file (I've only included English in this version because my torrent-related German and Japanese are poor). This will look in the JSON file for the key 'Title' and return whatever value is associated with it (or 'Title' if there is none).

NAME = L('Title')

More shorthand

ART        = 'art-default.jpg'
ICON      = 'icon-default.png'
SETTINGS  = 'settings-hi.png'
PAUSE      = 'pause-hi.png'
RESUME    = 'resume-hi.png'
SEARCH    = 'search-hi.png'
TV        = 'tv-hi.png'

TRANSMISSION_WAITING      = 1
TRANSMISSION_CHECKING      = 2
TRANSMISSION_DOWNLOADING  = 4
TRANSMISSION_SEEDING      = 8
TRANSMISSION_PAUSED        = 16

Definitions

Required

Here is where we start the plugin code. This is one of the standard functions which gets called when the plugin is initialised.

def Start():

Tell Plex we can handle '/video/transmission' URLs and that our main function is called 'MainMenu'

  Plugin.AddPrefixHandler(
    PLUGIN_PREFIX, 
    MainMenu, 
    PLUGIN_TITLE, 
    ICON, 
    ART)

  MediaContainer.art = R(ART)
  MediaContainer.title1 = NAME
  DirectoryItem.thumb = R(ICON)

Another standard function, this handles the preferences. To connect to Transmission, you need the URL and port it is running on (127.0.0.1:9091 if it's on the same machine as the Plex Media Server) and the username and password if you have set them.

def CreatePrefs():
    Prefs.Add(id='hostname', type='text', default='127.0.0.1:9091', label='Hostname')
    Prefs.Add(id='username', type='text', default='', label='Username')
    Prefs.Add(id='password', type='text', default='', label='Password', option='hidden')

This is called immediately after the preferences dialog is submitted. This is the most basic checking you can do but it could include a call to Transmission to verify the info provided.

def ValidatePrefs():
    u = Prefs.Get('username')
    p = Prefs.Get('password')
    h = Prefs.Get('hostname')
    if( h ):
        return MessageContainer(
            "Success",
            "Info provided is ok"
        )
    else:
        return MessageContainer(
            "Error",
            "You need to provide url, username, and password"
        )

You'll notice the return here is a MessageContainer. That's Plex's version of an alert. It doesn't generate a new page, just pops up a little window.

Custom

That was the end of the predefined functions, the plugin proper starts here. As Transmission requires a username, password and a short-lived session ID (since Transmission v1.53) to perform actions, we define a function which will attempt to make a connection with just username & password. Transmission will then send back a 409 Conflict response to basically say "Urk, that's not quite right. If you want to talk to me, you'll need this:" and give us our session ID in a header.

def GetSession():
  h = Prefs.Get('hostname')
  u = Prefs.Get('username')
  p = Prefs.Get('password')
  url = "http://%s/transmission/rpc/" % h
  request = { "method" : "session-get" }
  headers = {}
  if( u and p and h):
    headers["Authorization"] = "Basic %s" % 
      (base64.encodestring("%s:%s" % (u, p))[:-1])
    try:
      body = urllib2.urlopen(
        urllib2.Request(
          url, 
          JSON.StringFromObject(request), 
          headers
        )
      ).read()
    except urllib2.HTTPError, e:
      if e.code == 401 or e.code == 403:
        return L('ErrorInvalidUsername'), {}
      return e.hdrs['X-Transmission-Session-Id']
    except:
      return L('ErrorNotRunning'), {}

Once the HTTP module allows access to returned headers, we will be able to use something like this to set global authorisation once and forget about it:

response = HTTP.Request(
    url, 
    { "method" : "session-get" }, 
    headers={}, 
    cacheTime=None
    )
HTTP.SetPassword(h,u,p)
HTTP.SetHeader(
  'X-Transmission-Session-Id', 
  response.headers['X-Transmission-Session-Id']
  )

Remote Transmission Calls

This uses the RPC API of Transmission to do everything we need. We pass into the function 'What we want to do' and 'Who we want it done to' basically.

def RTC(method, arguments = {}, headers = {}):
  h = Prefs.Get('hostname')
  u = Prefs.Get('username')
  p = Prefs.Get('password')
  url = "http://%s/transmission/rpc/" % h

  session_id = GetSession()

  request = {
    "method":    method,
    "arguments":  arguments
  }

Setup authentication here because, even though we've already gotten the session ID, it's useless if we don't actually use it.

  if( u and p ):
    headers["Authorization"] = "Basic %s" %
      (base64.encodestring("%s:%s" % (u, p))[:-1])

  headers["X-Transmission-Session-Id"] = session_id

Now that we've built our instruction, throw it at Transmission and see what comes back.

  try:
    body = urllib2.urlopen(
      urllib2.Request(
        url, 
        JSON.StringFromObject(request), 
        headers)
      ).read()
  except urllib2.HTTPError, e:
    if e.code == 401 or e.code == 403:
      return L('ErrorInvalidUsername'), {}
    return "Error reading response from Transmission", {}
  except urllib2.URLError, e:
    return e.reason[1], {}

  result = JSON.ObjectFromString(body)

We don't do error handling here as we want this function to be as generic as possible so we send anything we receive straight back to the calling function.

  if result["result"] == "success":
    result["result"] = None

  if result["arguments"] == None:
    result["arguments"] = {}

  return result["result"], result["arguments"]

Menus

Right, we've got our helper methods set up, we're ready to make our first menu. This is the main one we mentioned earlier.

def MainMenu():

You can set your menu screen to be laid out as “List”, “InfoList”, “MediaPreview”, “Showcase”, “CoverFlow”, “PanelStream” or “WallStream”. I'm keeping it simple here. Also, there's an extra call to GetSession just to check everything's fine and wake the app up.

    dir = MediaContainer(viewGroup="List")
    GetSession()

Pretty much all the menu items throughout the rest of this plugin are added using the same code which boils down to:

  dir.Append(
    Function(
      DirectoryItem(
        FunctionName,
        "Pretty Menu Item Name",
        subtitle="Short subtitle",
        summary="Longer menu item summary and description",
        thumb=R(ICON),
        art=R(ART)
      )
    )
  )

Starting in the middle, this reads as:

• Create a DirectoryItem.
• When this item is selected, use the function FunctionName to handle it.
• Display the text "Pretty Menu Item Name" for this item
• Display the text "Short subtitle" underneath this item (or None)
• Display the text "Longer menu item summary and description" for this item if required (or None)
• Use the resource called ICON (mentioned above) as the icon for this item
• Use the resource ART as the background
• This is a Function menu item
• Finally, Append this to the current menu

The first two main menu items are built exactly like that:

  dir.Append(
    Function(
      DirectoryItem(
        TorrentList,
        "Torrents",
        subtitle=None,
        summary="View torrent progress and control your downloads.",
        thumb=R(ICON),
        art=R(ART)
      )
    )
  )
  dir.Append(
    Function(
      DirectoryItem(
        SearchTorrents,
          "Search for a torrent",
          subtitle=None,
          summary="Browse the TV shows directory or search for files to download.",
          thumb=R(SEARCH),
          art=R(ART)
        )
      )
    )

This is a special 'Preferences' item that will call the Prefs functions defined at the top.

  dir.Append(
    PrefsItem(
      title="Preferences",
      subtitle="Set Transmission access details",
      summary="Make sure Transmission is running and 'Remote access' is enabled then enter the access details here.",
      thumb=R(SETTINGS)
    )
  )

A quick note: the function 'R' here, much like the localisation one 'L' above is a built-in helper function. It handles resources such as images. If you're developing a plugin and can't understand why your icon images are caching so much, it might be because you're going through this function.

Send the directory (or Menu) back to Plex

    return dir

The rest of the code deals with torrent control and some clever built-in site scraping functionality which I'll cover later.

Maze 1k

• Random perfect maze generation, solution and rendering in 1011 bytes

Okay, this is the last one for a while. Really.

Unlike my previous 1k JavaScript demos, I really had to struggle to get this one into the 1024 byte limit. I'd already done all the magnification and optimization techniques I knew of so I had to bring in some things which were new to me from qfox.nl and Ben Alman and a few other places. This, combined with some major code refactoring, brought it down from 1.5k to just under 1. In the process, all possible readability was lost so here's a quick run through what it does and why.

First up, a bunch of declarations.

These are the colours used to draw the maze. Note, I used the shortest possible way to write each colour (red instead of #F00, 99 instead of 100). The value stored in the maze array (mentioned later) refers not only to the type of block it is but also to the index of the colour in this array, saving some space.

u = ["#eff", "red", "#122", "rgba(99,255,99,.1)", "#ff0"],

This is used for the number of blocks wide and high the maze is, the number of pixels per block, the size of the canvas and the redraw interval. Thanks to Ben Alman for pointing out in his article how to best use a constant.

 c = 21,

Here is the reference to the canvas. Mid-minification, I did have a bunch of function shortcuts here - r=Math.random, for instance - but I ended up refactoring them out of the code.

m = document.body.children[0];

For most of the time when working on this, the maze was wider than it was high because I thought that made a more interesting maze. When it came down to it, though, it was really a matter of bytesaving to drop the distinct values for width and height. After that, we grab the canvas context so we can actually draw stuff.

m.width = m.height = c * c;
h = m.getContext("2d");

The drawing function

The generation and solution algorithm is quite nice and all but without this to draw it on the screen, it's really just a mathematical curio. This takes a colour, x and y and draws a square.

l = function (i, e, f) {
  h.fillStyle = i;
  h.fillRect(e * c, f * c, c, c)
};

Designing a perfect maze

"You've got 1 minute to design a maze it takes 2 minutes to solve."
- Cobb, Inception.

Apologies for the unnecessary Inception quote. It's really not relevant.

Algorithmically, this is a fairly standard perfect maze generator. It starts at one point and randomly picks a direction to walk in then it stops picks another random direction and repeats. If it can't move, it goes back to the last choice it made and picks a different direction, if there are no more directions, all blocks have been covered and we're done. In a perfect maze, there is a path (and only one path) between any two randomly chosen points so we can make the maze then denote the top left as the start and the bottom right as the end. This particular algorithm takes 2 steps in every direction instead of 1 so that we effectively carve out rooms and leave walls. You can take single steps but it's actually more of a hassle.

For more on how this kind of maze-generation works, check out this article on Tile-based maze generation.

Blank canvas

This is a standard loop to create a blank maze full of walls with no corridors. The 2 represents the 'wall' block type and the colour #122. The only really odd thing about this is the code f-->0 which is not to be read 'as f tends to zero' but is instead 'decrement f by 1, is it greater than zero?'

g = function () {
  v = [];  //our stack of moves taken so we can retrace our steps.
  for (i = [], e = c; e-- > 0;) {
    i[e] = [];
    for (f = c; f-- > 0;) i[e][f] = 2
  }

By this point, we have a two-dimensional JavaScript array filled with 2s

Putting things in

  f = e = 1;    // our starting point, top left.
  i[e][f] = 0; // us, the zippy yellow thing

Carving out the walls

This is our first proper abuse of the standard for-loop convention. You don't need to use the three-part structure for 'initialize variable; until variable reaches a different value; change value of variable'. It's 'do something before we start; keep going until this is false; do this after every repetition' so here we push our first move onto the stack then repeat the loop while there's still a move left on the stack.

  for (v.push([e, f]); v.length;) {

P here is the list of potential moves from our current position. For every block, we have a look to see what neighbours are available then concatenate that cardinal direction onto the strong of potential moves. This was originally done with bitwise flags (the proper way) but it ended up longer. It's also a bit of a nasty hack to set p to be 0 instead of "" but, again, it's all about the bytes.

  p = 0;

Can we walk this way?

These are all basically the same and mean 'if we aren't at the edge of the board and we're looking at a wall, we can tunnel into it.'.

 if (e < 18 && i[e + 2][f] == 2) p += "S"
 if (e >= 2 && i[e - 2][f] == 2) p += "N";
 if (f >= 2 && i[e][f - 2] == 2) p += "W";
 if (i[e][f + 2] == 2) p += "E";

 if (p) { //    If we've found at least one move
  switch (p[~~ (Math.random() * p.length)]) { // randomly pick one

If there was anything to note from that last chunk, it would be that the operator ~~ can be used to floor the current value. It will return the integer below thye current value.

Take two steps

This is a nice little hack. Because we're moving two spaces, we need to set the block we're on and the next one to be 0 (empty). This takes advantage of the right-associative unary operator 'decrement before' and the right associativity of assignment operators. It subtracts 1 from e (to place us on the square immediately above) then sets that to equal 0 then subtracts 1 from the new e (to put us on the next square up again) and sets that to equal the same as the previous operation, i.e. 0.

  case "N":
      i[--e][f] = i[--e][f] = 0;
      break;

Do the same for s, w and e

    case "S":
      i[++e][f] = i[++e][f] = 0;
      break;
    case "W":
      i[e][--f] = i[e][--f] = 0;
      break;
    case "E":
      i[e][++f] = i[e][++f] = 0
    }

Whichever move we chose, stick that onto the stack.

    v.push([e, f])

If there were no possible moves, backtrack

  } else {
    b = v.pop(); //take the top move off the stack
    e = b[0]; // move there
    f = b[1]
  }
 }

End at the end

At the very end, set the bottom right block to be the goal then return the completed maze.

  i[19][19] = 1;
  return i
};

Solver

This is the solving function. Initially, it used the same algorithm as the generation function, namely 'collect the possible moves, randomly choose one' but this took too much space. So instead it looks for spaces north, then south, then west, then east. It follows the first one it finds.

s = function () {

Set the block type of the previous block as 'visited' (rgba(99,255,99,.1) the alpha value makes the yellow fade to green).

  n[o][y] = 3;

A bit of ternary background

This next bit looks worse than it is. It's the ternary operator nested several times. The ternary operator is a shorthand way of writing:

if ( statement A is true ) {
  Do Action 1
} else {
  Do Action 2
}

In shorthand, this is written as:

Statement A ? Action 1 : Action 2;

In this, however, I've replace Action 2 with another ternary operator:

Statement A ? Action 1 : ( Statement B ? Action 2 : Action 3 );

And again, and again. Each time, it checks a direction, if it's empty, mark it as visited and push the move onto our stack.

(n[o + 1][y] < 2) ?
  (n[++o][y] = 0, v.push([o, y])) :
    (n[o - 1][y] < 2) ?
      (n[--o][y] = 0, v.push([o, y])) :
        (n[o][y - 1] < 2) ?
          (n[o][--y] = 0, v.push([o, y])) :
            (n[o][y + 1] < 2) ?
              (n[o][++y] = 0, v.push([o, y])) :

If none of the neighbours are available, backtrack

                (b = v.pop(), o = b[0], y = b[1]);

Show where we are

Finally, set our new current block to be yellow

  n[o][y] = 4;

Are we there yet?

If we are at the bottom right square, we've completed the maze

  if (o == 19 && y == 19) {
  n = g();    //Generate a new maze
  o = y = 1; //Move us back to the top right
  s()     //Solve again

If we haven't completed the maze, call the solve function again to take the next step but delay it for 21 milliseconds so that it looks pretty and doesn't zip around the maze too fast.

  } else setTimeout(s, c);

Paint it black. And green. And yellow.

This is the code to render the maze. It starts at the top and works through the whole maze array calling the painting function with each block type (a.k.a. colour) and position.

    for (d in n) for (a in n[d]) l(u[n[d][a]], a, d)
  };

Start

This is the initial call to solve the maze. The function s doesn't take any arguments but by passing these in, they get called before s and save a byte that would have been used if they had been on a line themselves.

s(n = g(), o = y = 1)

Done

This little demo isn't as visually appealing as the Art Maker 1k or as interactive as the Spinny Circles 1k but it is quite nice mathematically. There are now some astounding pieces of work in the JS1K demo competition, though. I do recommend spending a good hour playing about with them all. Don't, however, open a bunch of them in the background at the same time. Small code isn't necessarily memory-efficient and you could quite easily grind your computer to a standstill.

Art Maker 1K

Even though the rules for js1k only let me make one submission, I couldn't stop myself making another. This one is inspired by those pseudo-romantic pictures that you get all over tumblr that get reblogged endlessly (actually, it was inspired by the blog That Isn't Art by someone with the same opinion as myself).

It randomly creates a sentence, adds some softly-moving almost bokeh coloured-circles and look, I made an art! Wait for the sentence to change or click (or touch) to change it yourself.

Art Maker 1k

And of course, don't forget the original spinny-circles 1k

The quest for Extreme JavaScript Minification

As described in detail previously, I've recently taken part in the JS1K competition where you have to squeeze something cool and clever into 1024 bytes of JavaScript. The quest to condense has become quite addictive and I found myself obsessing over every byte. This is the kind of stuff that the Closure Compiler does quite well automatically but there are some cases where you just need to get in there and manually tweak.

Here are some of the tricks I've picked up in my struggle for extreme minification:

Basic improvements

Use short variable names.

This one's fairly obvious. A more useful addition to this is:

Shorten long variable names.

If you're going to be accessing an element more than once, especially if it's a built-in element like 'document', you'll save a few bytes every time you reference it if you create a shortened reference to it.

  document.body.style.overflow="hidden"
  document.body.style.background="red"
  (74 characters)

can shorten to

  d=document;b=d.body;s=b.style;
  s.overflow="hidden";
  s.background="red"
  (69 characters)

and any references to s after are going to save 19 characters every time.

Remove whitespace

This one's so obvious, I don't need to mention it.

Set Interval

The extremely handy setInterval function can take either a function or a string. If you give it an anonymous function declaration:

  setInterval(function(){x++;y--},10);

You will use up more characters than if you give it just the inside of the function as a string:

  setInterval('x++;y--',10);

But the outcome will be the same.

Little-used aspects

Not many people use JavScript's scientific notation unless they're doing scientific stuff but it can be a great byte saver. The number 100 is equivalent to 1 * 10^2 which is represented in JavaScript as 1E2. That's not a great saving for 100 but 1000 is 1E3, 10000 is 1E4. Every time you go up a factor of 10, you save 1 byte.

Fight your good style tendencies

In the war against space, you have to bite the bullet and accept that you may need to sacrifice some of your hard-earned practices. But only this once. Don't get in to the habit, okay?

No zeroes

  0.5  = .5

Yeah, it looks ugly but it works and saves a byte.

Naked clauses

  if {
    :
    :
  } else y

The y looks so naked out there. No braces to keep it warm. But if you only have one statement in your else clause, you don't need them...

No semi-final. . . final-semi. . . Semi-colon. No final colon.

You don't need a semi-colon on your last line, even if it does make it look as though you've stunted its growth.

The final few bytes

Operator precedence

You don't need brackets. Brackets are handy for you as the programmer to remember what's going on when and to reduce ambiguity but if you plan correctly, most of the time you won't need brackets to get your arithmetic to work out.

  b.getMilliseconds()/(a*250) 
      is the same as
  b.getMilliseconds()/a/250 

Shorthand notation

  l=l+1;l=l%14;
  l++;l%=14;
  l=++l%14;

The three lines above are equivalent and in order of bytes saved.

Shorthand CSS

If you need to set some CSS values in your script, remember to pick the most appropriate short form. Instead of s.background='black', use s.background='#000' but instead of s.background='#F00', use s.background='red'. In the same vein, the statements margin="0px" and margin=0 mean the same but the latter saves bytes.

Don't be generic

One final thing to mention is that these little challenges are not the norm. If you find yourself trying to squeeze code down like this you're probably working on a very specific project. Use that to your advantage and see if there are any savings to be made by discarding your usual policies on code reuse. In the JS1K challenge, we're provided with a specific HTML page and an empty script tag. One good saving made here (and mentioned in my previous post) was the way I grabbed the reference to the canvas element. The standard method is to use the id assigned to the canvas.

  d.getElementById('c')

Which is a good generic solution. No matter what else was on the page, no matter what order stuff was in, this would return the canvas. However, we have a very specific case here and the canvas is always going to be in the same place - the first child of the body element. That means we can do this instead:

  b.children[0]

This makes use of the reference we grabbed to the body earlier. If the page were rearranged, this would stop working but as it won't, we've saved 8 bytes.

In conclusion

Yes, this is all quite silly but it's also fun and tricky. Attempting these kinds of challenges keep us developers mindful of what it is we actually do and that makes it an extremely productive silly hobby.

Elementally, my dear JavaScript

The Angry Robot Zombie Factory launched its second iPhone/iPad app this week. I haven't mentioned it much yet because I spotted a minor typo in the final version after it had been approved so I submitted an update immediately. To get an early copy (like those misprinted stamps where the plane is upside down), go check out The Elementals. It's free, too. It's a simple, cartoonish periodic table.

Yesterday, the 1k JavaScript demo contest (#js1k) caught my eye. The idea is to create something cool using 1024bytes of JavaScript or less. I rootled around in the middle of The Elementals, grabbed the drawing function and 20 minutes later had made my entry.

The code I submitted is quite minified but isn't obfuscated. When it's unfolded, you can follow the flow fairly easily.

var d = document,
b = d.body,
s = b.style,
w = innerWidth,
h = innerHeight,
v = b.children[0],
p = 2 * Math.PI,
Z = 3,
x = tx = w / 2,
y = ty = h / 2;

The above is a bunch of declarations. Using things like d = document and b = d.body allows reuse later on without having to resort to the full document.body.style and saves a bunch of characters. When you've got such a small amount of space to play with, every character counts (mind you, the ZX81 only had 1k of RAM and look what you could do with that). Now that I'm looking at it, I think I could have tidied this a bit more. Darn. The sneaky bit about this code is the way we grab the reference to the canvas. The code d.getElementById('c') uses 21 characters but if we look at the provided HTML, we can use the fact that the canvas is the first child of the body element. The code b.children[0] uses 13 characters instead.

s.margin = "0px";
s.background = "black";
s.overflow = "hidden";
v.width = w;
v.height = h;
t = v.getContext("2d");

This sets the provided canvas to be the full width and height of the window then grabs the drawing context of it so we can make pretty pictures.

zi = function () {
 Z++;
 Z %= 14
};
m = function (X) {
 return (X * 200) % 255
};

Functions to be reused later. zi increases the number of spinning circles and is used by onmousedown and ontouchstart (oh yes, it works on the iPad, too). m is a mapping of the index of the circle to a colour. The 200 is arbitrary. I played about a bit until I found some colour combinations I liked.

 d.ontouchstart = function (e) {
 zi();
 tx = e.touches[0].pageX;
 ty = e.touches[0].pageY
};
d.onmousemove = function (e) {
 tx = e.clientX;
 ty = e.clientY
};
d.onmousedown = zi;

Setting the event handlers.

function r() {
 t.globalCompositeOperation = 'lighter';

I played about with the various composite operations. Lighter seemed the nicest.

 t.clearRect(0, 0, w, h);
 t.save();
 x = x + (tx - x) / 20;
 y = y + (ty - y) / 20;
 t.translate(x, y);

Originally, the circles followed the mouse pointer exactly but it lacked any life. By adding in this bit where the movement is delayed as if pulling against friction, it suddenly became a lot more fun and dynamic.

 var c = new Date();
 for (var i = 1; i       
]]>
Wed, 04 Aug 2010 00:00:00 GMT
development


http://thingsinjars.com/post/291/phonegap---the-drupal-of-app-development/
http://thingsinjars.com/post/291/phonegap---the-drupal-of-app-development/

        
PhoneGap - The Drupal of App development
I'm a fan of Drupal even though I don't use it that often. I like that I can see exactly what's going on. I can easily follow the execution from URL request to page serve.
What I usually end up doing on any Drupal project is:

	
build the majority of the site in a few hours
	
find one small piece of functionality missing that's absolutely essential
	
dig into the core to make it happen
	
find a simpler way of doing it and step out of the core a bit
	
find an even simpler way and step back a bit more
	
figure out how to do it in a single module file and put the core back the way it was.

That probably seems utterly inefficient but it has served me well since Drupal 4 and it means I've got a really good picture in my head of the internal workflow.
This is in stark comparison to other systems, particularly some .NET CMSs where a request comes in, something happens and the page is served. There are even some PHP frameworks and CMSs where everything is so abstracted, the only way you can get an accurate picture of what is happening is to already have an accurate picture of what is happening.
I've used several different ones and I keep coming back to Drupal (also, recently, Perch, but that's besides the point here).
“What on earth does this have to do with PhoneGap?” I hear you ask. Quite rightly, too.
When I was planning Harmonious, I looked at various frameworks for turning a combination of HTML, CSS & JavaScript into an app - PhoneGap, Appcelerator Titanium, Rhomobile. Rhomobile (or the Rhodes Framework) is built on Ruby so I didn't investigate too far. That's not to say it's not a good framework, I couldn't say either way. The idea behind using one of these frameworks is to save you the time of having to learn Objective-C and seeing as I've only done very minimal amounts of Ruby, I'd be replacing 'learn Objective-C' with 'learn Ruby'. That said, I've always thought Ruby developers opinion of themselves was slightly too high.
The first framework I properly spent some time with was Appcelerator. It seemed quite shiny and I liked having single interface access to compilation for iPhone and Android but I wasn't so keen on having to sign up for an account with them for no obvious reason. Some further investigation suggested that this was so you could send your project to their servers for proprietary cross-platform compilation of your desktop app. This is less useful, however, if you're developing just for iPhone and Android as for both, you need the SDK installed locally and the compilation is done on your own machine.
The main thing that I wasn't comfortable with in Appcelerator was that there seemed to be a lot happening behind the scenes. This is not necessarily a bad thing, of course, but it started that little buzz in the back of my head that I get when working on .NET. When I press 'compile', I want to know exactly what it's doing. I want to know exactly how it takes my JavaScript and embeds it, when does it include its own API files and what do I change to make it do stuff it doesn't do by default?
After that, I moved to PhoneGap (version 0.8.3) and found myself immediately falling into my Drupal workflow. The app fell into place in less than an hour (with a liberal sprinkling of jQTouch and the Glyphish icons). I then needed to take a screenshot and couldn't see an obvious way to do it but, due to the nature of PhoneGap being completely open-source, it was easy to spot where to jump into the code. I hacked in a screenshot function in another hour, spent another half hour making it better and another making it simpler. Just to complete the cycle, I have now wrapped up all my code into a plugin and removed my hacking from the core. Hmmm... that all seemed eerily familiar.
That's not to say PhoneGap is perfect. All the benefits of a completely open-source project referred to previously also come with all the drawbacks. The current version (0.9.0) is fiendishly difficult to download and get started with. It has been split into one parent project and several child projects (one per mobile platform) and it's no longer obvious what you do. It's easy enough if you're already set up but actually getting there is tricky. The most common failing of any open-source project is also true: poor documentation. There's a wiki but it's mostly out-of-date. There's a section on phonegap.com called 'docs' but they're also out-of-date. There's an API reference but it's autogenerated from comments and is also out-of-date. The only place to get accurate information is the Google group but that's not documentation, that's solutions to problems.
There have also been some claims that PhoneGap is unstable and crashes but personally, I haven't seen that. It's possible that the crashes and performance issues are the result of memory leaks in the JavaScript. Appcelerator automatically runs JSLint on your code before compilation so it will highlight any problems. If you can fit that into your standard workflow, you might be able to avoid some of the instability.

Additional comments

It seems that Disqus (the commenting system I'm using below) has some problems with Safari 5 & Chrome so this comment was sent via gist (I knew I shouldn't have stopped using Noodle).
I'll respond later. I've just got back from the Apple store and have toys to play with.


Comment from Jeff Haynie (@jhaynie)
A few comments about Appcelerator.
1. We're completely open source and you can see all our active development every single commit on github.com/appcelerator.  We have plenty of outside core open source contributors.
2. Yeah, to do what we're doing, it's complicated - much more than Phonegap - so it does mean with complexity it's hard to grok.  however, the source is all there. Also, it's full extensible through our SDKs and we this SDK as the way we build Titanium itself.
3. For Desktop, we _only_ upload to our build servers as a convenience to cross-platform packaging.  Nothing mysterious and all the scripts we run are included (and open source) so you can run them on your own. Plenty of our customers do this behind the firewall.  When you're developing locally (say on a OSX machine), it's all local during dev. Only cross-platform packaging is done as a convenience to developers.  We have to pay for this bandwidth and storage and we do it to make it easier.  And it's free.
Hope this clarifies some of the above.   Phonegap's a great project and we love the team - but I think we're trying to do different things and come at it from different approaches. In the end, this is good for developers as it gives everyone more choice based on their needs.
      
]]>
Thu, 17 Jun 2010 00:00:00 GMT
development


http://thingsinjars.com/post/255/crowdsourced-weather---part-2/
http://thingsinjars.com/post/255/crowdsourced-weather---part-2/

        
Crowdsourced Weather - Part 2
So, I couldn't help myself. I had a niggling idea at the back of my head that I needed to get out. After coming up with this Twitter weather idea last week, I decided to spend a couple of hours this weekend building it. As if I didn't have other things I should have been doing instead...

It works pretty much exactly how the pseudocode I mentioned last time describes. Every few minutes, a script will search Twitter for mentions of any weather words from several different languages. It will then look up the location of the person who tweeted that and store it. Single reports might be wrong and users might not have stored their actual location but over a large enough sample, this system becomes more accurate. The script removes any matching twets older than 6 hours.

To display, I actually ended up using Geohashes instead of Geotudes because it is easier to simplify them when you zoom out just by cutting off the tail of the hash. For example, the physical area denoted by gcvwr3qvmh8vn (the geohash for Edinburgh) is contained within gcvwr3 which is itself contained within gcv. There are a few technical problems with geohashes but it seems the best fit for this purpose. If anyone knows of any better suggestion, please let  me know. I do realise that this is quite possibly the slowest, most inefficient JavaScript I've ever written because it makes an AJAX call for every graticule and it probably should just send the South-East and North-West bounds and get back an array of them but, like I said, there were other things I should have been doing. Because the overlaid grid changes resolution based on zoom level, there are a few places where it is either tragically slow (resolution too fine) or terribly inaccurate (resolution too rough). That's just a case of tweaking the algorithm. Similarly, it's set to display reports of weather if there are 2 or more matches but it could be tweaked to only show if a larger number have reported something.

So go, play with the Twitter-generated weather map. If someone can come up with a good, catchy name, or some better graphics, that'd be great, thanks.


Source code is available: twitter-weather-1.0.zip [Zip - 298KB].
You'll need your own Twitter login and database account to use it.
      
]]>
Mon, 21 Dec 2009 00:00:00 GMT
development


http://thingsinjars.com/post/254/crowdsourced-weather/
http://thingsinjars.com/post/254/crowdsourced-weather/

        
Crowdsourced Weather
This is a more general version of the #uksnow map idea. It's a crowd-sourced weather map which relies on the fact that any one individual tweet about the weather might be inaccurate but given a large enough sample, enough people will mention the weather in their area to make this a workable idea. It doesn't require people to tweet in a particular format.

To get info

Have an array of weather words in various languages (rain, hail, snow, schnee, ame, yuki)
every 5 minutes:
	foreach weatherword
		search twitter for that word
			http://search.twitter.com/search.atom?q=rain
		retrieve latest 100 tweets
		foreach
			get user info
				http://twitter.com/users/show.xml?screen_name=username
			get user.location if available
			geocode
			save:
				username, time, lat, long, geotude, weatherword
		Remove any tweets about this weatherword older than 6 hours.
			


To display info
Show a Google map
Based on current Zoom level, split the current map into about 100 geotudes
foreach geotude
	search database for any weather results for that block (probably using an ilike "1234%" on the geotude field)
	sort by weatherword count descending
	draw an icon on top of that block to show the most common weatherword
	
If the user zooms in, recalculate geotudes and repeat.


I quite like that this uses geotudes which I think are an excellent idea.

I built a very basic version of this. Read more about it in Part 2.
      
]]>
Thu, 17 Dec 2009 00:00:00 GMT
development


http://thingsinjars.com/post/250/building-an-objective-c-growlview/
http://thingsinjars.com/post/250/building-an-objective-c-growlview/

        
Building an Objective-C growlView
Wherein our intrepid hero learns some Objective-C and figures out the easy bits are actually quite hard.

  
This is a little guide on how to write a growl plugin to automatically send notifications as tweets. If you just want the finished thing: GrowlBird.zip [Zip - 228KB]
A couple of days ago, I decided to give myself a little software development task to write a Twitter Growl view. Growl is a centralised notification system for Mac OS X that lots of other applications can use so that there's one consistent way of showing notifications. 
Background
The idea behind this little experiment wasn't to have tweets appear as growl notifications, there are already plenty of apps that do this, the idea was to have growl notifications sent to Twitter. Some friends have started organising a crowdsourced Friday afternoon playlist via Spotify and I thought it'd be handy if Spotify tweeted each song as it started. The easiest way I could think of doing this was to tap into the fact that Spotify sends a Growl notification on track start and get the Growl display plugin to tweet it as well [1]. 
Build
I downloaded the Growl Display Plugin Sample 1.2 [Zip - 186 KB] from the developer downloads page and the MGTwitterEngine library. I then downloaded Xcode so I could do the development. I have to point out here that this was my first foray into Objective-C programming and, indeed, my first attempt at anything vaguely C-related since I wrote a command-line calculator about 12 years ago. If I do it wrong, please forgive me.
The first thing to do was open the sample project in Xcode, figure out what files do what, etc. There is very little documentation on how Growl views or display styles work so I pretty much just spend an hour reading all the source from top to bottom. Here's a quick summary:

  Sample_Prefix.pchPre-compiled header. Stuff that's included before every pre-compiled fileGrowl/Folder containing standard Growl stuff. Don't need to touch.GrowlSampleDisplay.hHeader file, didn't need to change anythingGrowlSampleDisplay.mClass for setting up things. Again, didn't touch [2].GrowlSamplePrefs.hDefining default preference values and naming functions to handle them. More on this later.GrowlSamplePrefs.mThe actual functions mentioned in the previous header fileGrowlSampleWindowController.hNot doing anything visual, really so I didn't need to mess around with thisGrowlSampleWindowController.mAs aboveGrowlSampleWindowView.hDeclaring objects needed for executionGrowlSampleWindowView.mInstantiating the objects then actually using them later on.
Again, I'm not used to doing this stuff so if I'm using the wrong terminology, just pretend I'm not.
I then dragged the MGTwitterEngine library into the project drawer, saved and built. At this point it successfully did nothing different which is what I was hoping it would do. Well, it popped up the 'This is a Preview of the Sample Display' message using the MusicVideo style which is what it does when you don't screw with it.

  
Testing growlView
  
It's not actually that easy to test growlView plugins. You can't simply add and remove them. To install, you double-click on it. You should be able to see it listed in the Display tab of the preference pane. To re-install, you need to delete ~/Library/Preferences/com.Growl.GrowlHelperApp, ~/Library/Application Support/Growl/Plugins/Sample.growlView and restart growl (click on the paw in the menu bar). It's a bit of a hassle.

The next thing was to include the MGTwitterEngine. In GrowlSampleWindowController.h, #import "MGTwitterEngine.h" and create a new object. I just followed the instructions in the README but be sure to follow all of them. If you get errors about LibXML not being installed or YAJL not working, don't worry, you just need to make sure you set USE_LIBXML to 0 in all the places you're supposed to. GrowlSampleWindowController.h now contains this:

#import "GrowlDisplayWindowController.h"
#import "MGTwitterEngine.h"

@class GrowlBirdWindowView;

@interface GrowlBirdWindowController : GrowlDisplayWindowController {
	CGFloat						frameHeight;
	NSInteger					priority;
	NSPoint						frameOrigin;
	MGTwitterEngine *twitterEngine;
}
@end
In GrowlSampleWindowController.m, I then instantiated the new object:

  @implementation GrowlBirdWindowController
  - (id) init {
  	  :
  	  :
      twitterEngine = [[MGTwitterEngine alloc] initWithDelegate:self];
	    [twitterEngine setUsername:@"growlbirdtest" password:@"testgrowlbird"];
	  }
	  :
And then modified the setNotification function to also send an update:

- (void) setNotification: (GrowlApplicationNotification *) theNotification {
    :
	[view setTitle:title];
	[view setText:text];
  NSLog(@"sendUpdate: connectionIdentifier = %@", [twitterEngine sendUpdate:[NSString stringWithFormat:@"%@, %@", title, text]]); // The new line
  :
  }
That was enough to get growl to send messages to appear on http://twitter.com/growlbirdtest but it doesn't make it that useful for anybody else, to be honest. The next thing to figure out was the preferences. 
Preferences
Without documentation, this took a bit longer that I expected. To start off changing the english version before worrying about localization, find the GrowlBirdPrefs.xib in resources/en.lproj/ and open it. Interface Builder will launch then you can double-click on 'Window' and see the layout of the preference pane. Search in the Library for 'text' and drag a text field into the window then spend about half and hour clicking round the interface. Open up the various inspectors (right-click on an object), look through the different tabs, click between the newly added text field and the sliders and drop-downs that are already there just to see what's different. Once I was a bit familiar, I opened the connections tab so that I could bind the value of the text field to the value 'twitterUsername' in my code. I checked 'value', Bind to 'File's Owner' and entered 'twitterUsername' in Model Key Path. I then repeated this for twitterPassword using a Secure Text Field from the Library. The option nextKeyView is used to say which item is tabbed to next when you're navigating with the keyboard so to keep things tidy, I dragged lines from nextKeyView from each of them to the right places in the layout.
Back in the code, I added new default preferences in GrowlSamplePrefs.h:

  #define Sample_USERNAME_PREF		@"Username"
  #define Sample_DEFAULT_USERNAME		@"growlbirdtest"

  #define Sample_PASSWORD_PREF		@"Password"
  #define Sample_DEFAULT_PASSWORD		@"testgrowlbird"
  :
  :
  @interface GrowlBirdPrefs : NSPreferencePane {
  	IBOutlet NSSlider *slider_opacity;
  	IBOutlet NSString *twitterUsername;
  	IBOutlet NSString *twitterPassword;
  }
and named some handlers for them:

  - (NSString *) twitterUsername;
  - (void) setTwitterUsername:(NSString *)value;
  - (NSString *) twitterPassword;
  - (void) setTwitterPassword:(NSString *)value;
Be careful here, I got confused and didn't have the same spelling here for twitterUsername and twitterPassword as I had put in the interface builder as I hadn't realised the two were directly connected. They are. Obviously. The next thing to do is to write the code for these handlers:

  - (NSString *) twitterUsername {
  	NSString *value = nil;
  	READ_GROWL_PREF_VALUE(Sample_USERNAME_PREF, SamplePrefDomain, NSString *, &value);
  	return value;
  }
  - (void) setTwitterUsername:(NSString *)value {
  	WRITE_GROWL_PREF_VALUE(Sample_USERNAME_PREF, value, SamplePrefDomain);
  	UPDATE_GROWL_PREFS();
  }
  - (NSString *) twitterPassword {
  	NSString *value = nil;
  	READ_GROWL_PREF_VALUE(Sample_PASSWORD_PREF, SamplePrefDomain, NSString *, &value);
  	return value;
  }
  - (void) setTwitterPassword:(NSString *)value {
  	WRITE_GROWL_PREF_VALUE(Sample_PASSWORD_PREF, value, SamplePrefDomain);
  	UPDATE_GROWL_PREFS();
  }


Build and reinstall and this will now show the same preference pane as before but with two new text fields which allow you to enter your username and password. In fact, build at several stages along the way. Every time you make a change, in fact. If something breaks, check the error log to see if it's something predictable that should have broken at that point or if you've done something wrong. Also, keep the OS X log app Console open in the background. It will spew out error messages if you do something wrong. It's also good to have your code write out console messages to keep a track on what your code is doing like so:

  - (NSString *) twitterPassword {
  	NSString *value = nil;
  	READ_GROWL_PREF_VALUE(Bird_PASSWORD_PREF, SamplePrefDomain, NSString *, &value);
  	NSLog(@"twitterPassword = %@", value);
  	return value;
  }


You'll notice we're still sending messages to the growlbirdtest account because, even though we are reading and saving the username and password, we're not doing anything with them. That's easily remedied by editing GrowlSampleWindowView.m again and replacing the hard-coded login details with a couple of lines to read from the preferences or fall back on the default:

  twitterEngine = [[MGTwitterEngine alloc] initWithDelegate:self];
	NSString *twitter_username = Bird_DEFAULT_USERNAME;
	NSString *twitter_password = Bird_DEFAULT_PASSWORD;
	READ_GROWL_PREF_VALUE(Bird_USERNAME_PREF, SamplePrefDomain, NSString *, &twitter_username);
	READ_GROWL_PREF_VALUE(Bird_PASSWORD_PREF, SamplePrefDomain, NSString *, &twitter_password);
	[twitterEngine setUsername:twitter_username password:twitter_password];
	NSLog(@"Twitter Login: username = %@", twitter_username);

And, hooray! It works and posts to the account for which you entered details. Sort of. Some apps double-post. I haven't figured out why yet.
Renaming
After all that, the final bit (which I thought would be the easiest) was to rename the growlView from 'Sample' to 'Bird'. I have read that in the latest version of Xcode (which presumably comes with Snow Leopard), there's a global 'Rename' which will do all the relevant stuff for you. If you don't have that, you'll need to read 'On the Renaming of Xcode Projects' and do everything there. If you're still finding your growlView is called Sample, manually open every Info.plist you can find, 'Get Info' on everything, scour through the settings for the different build environments (Debug and Release)... It took longer to rename the project than to actually build it.


A trivial aside, I have also added two fields to the preferences for tweet prefix and tweet postfix in exactly the same way the username and password were added. I leave the details to the interested reader.


You should now have a completed, installable growlView/Growl View/Growl Display/growlStyle/whatever it's actually called. You can export the current Git project to have a look around or you can just download the finished GrowlBird.zip [Zip - 228KB] if you like. Note, the Git project isn't guaranteed buildable at any moment in time, I might break it. The localisations still need done and the layout of the prefPane isn't the greatest, either.


	
[] Since writing this, I have discovered that someone else already made a growlView to do this, it just didn't show up on any of my Google searching.
	
[] It turns out that I did have to modify these in the end due to a silly bug. Like I said, this is my first attempt at this kind of thing.
      
]]>
Sun, 01 Nov 2009 00:00:00 GMT
development