Using WCF to power Facebook feedstory forms

UPDATE: I think Facebook has changed the API so that links in full stories have to be split into src and href parts.

Several parts of the Facebook developer API requires application developers to furnish JSON web services that feed data to various actions. These include feed forms (“I want to stick something on my wall”) and the publisher to name a few. This post will descripe how that JSON can be generated through Windows Communication Foundation web services. You can see this code in action in the simple Louisefy application. Note that alternatives certainly exists (I recommend Json.NET), but now that we have WCF, why not use it?

(As Rune points out in a comment below, you can use the object-hierachy in the code to generate compliant JSON even if you’re using Json.NET instead of WCF).

As far as feed forms go, the developes pre-register a range of templates for the kind of stories they want their users. A set of templates that get fed to Feed.registerTemplateBundle might look like the following:

List oneLiners = new List() {
	"{*actor*} is <a href=\"{*linkurl*}\">{*linktitle*}</a> {*target*}s Facebook",
};
List shortTmplts = new List() {
	new Dictionary() {
		{"template_title","{*actor*} is <a href=\"{*linkurl*}\">{*linktitle*}</a> {*target*}s Facebook"},
		{"template_body","."}
	},
};
Dictionary fullTmplts = new Dictionary() {
	{"template_title","{*actor*} is <a href=\"{*linkurl*}\">{*linktitle*}</a> {*target*}s Facebook"},
	{"template_body","{*swf*}"}
};

The FBML markup to get a feed form that lets users publish on their friends feeds would look like this:

<form fbtype="multiFeedStory" action="http://link.to/myfeedhandler.svc">
Start typing names of your friends:
<fb:multi-friend-input />
<input type="submit" label="Louisefy your friends feeds" />
</form>

The code for the actual feedhandler follows below, with the web.config stuff at the very bottom. I really like the terse, declarative syntax — there’s not a single hint that this all becomes JSON at some point. Note that if you need to pass arguments to your feedhandler, they’re very easy to map in the UriTemplate. I generally find WCF services to be fiendishly complex to get up and running because of the dearth of good tutorials and documentation — but once you get them going, it’s pure joy.

[ServiceContract]
[AspNetCompatibilityRequirements(RequirementsMode = AspNetCompatibilityRequirementsMode.Allowed)]
public class feedHandler
{
	[WebInvoke(ResponseFormat = WebMessageFormat.Json,
	BodyStyle = WebMessageBodyStyle.Bare,
	Method = "POST",
	UriTemplate = "/friendsfeed")
	]
	[OperationContract]
	public FeedResponse GetResponse()
	{
		return new FeedResponse
		{
			method = "multiFeedStory",
			content = new Content
			{
				feed = new Feed
				{
					template_id = "1234",
					template_data = new TemplData
					{
						swf = "",
						linktitle = "Louisefying",
						linkurl = "http://apps.facebook.com/louisefy/",
						images = new FeedImage[]
						{
							new FeedImage
							{
								src = "http://foo.bar/my.png",
								href = "http://apps.facebook.com/louisefy"
							}
						},
					}
				},
				next = "http://apps.facebook.com/louisefy",
			}
		};
	}
}

[DataContract]
public class FeedResponse
{
	[DataMember]
	internal string method;

	[DataMember]
	internal Content content;
}

[DataContract]
class Content
{
	[DataMember]
	internal string next;
	[DataMember]
	internal Feed feed;
}

[DataContract]
class Feed
{
	[DataMember]
	internal string template_id;

	[DataMember]
	internal TemplData template_data;
}

[DataContract]
class TemplData
{
	[DataMember]
	internal FeedImage[] images;

	[DataMember]
	internal string swf;

	[DataMember]
	internal string linkurl;

	[DataMember]
	internal string linktitle;
}

[DataContract]
class FeedImage
{
	[DataMember]
	internal string src;

	[DataMember]
	internal string href;

}

web.config:

<system.serviceModel>
	<behaviors>
	  <endpointBehaviors>
		<behavior name="Popcorn.feedHandlerAspNetAjaxBehavior">
		  <webHttp />
		</behavior>
	  </endpointBehaviors>
	</behaviors>
	<serviceHostingEnvironment aspNetCompatibilityEnabled="true" />
	<services>
	  <service name="Popcorn.feedHandler">
		<endpoint address="" behaviorConfiguration="Popcorn.feedHandlerAspNetAjaxBehavior"
		  binding="webHttpBinding" contract="Popcorn.feedHandler" />
	  </service>
	</services>
</system.serviceModel>

Server based real-time face detection in Flex

This post will demonstrate the feasability of face detection in web cam feeds grabbed by a flash/flex application. It’s inspired by a prototype of Martin Speelmans, informed by my work with Flash and web cams and my experiments with OpenCV. The basic premise is that a flex application running in a users browser grabs web cam shots, compresses them and sends them to a server. The server decompresses the picture, does face detection on it, and sends the result back to the client. All fast enough to give the user a relatively smooth experience. A fairly impressive Actionscript 3 face detection implementation exists, but my guess is that — barring a quantum leap in Adobe compiler technology — it’s gonna be a few years before fast, reliable Actionscript 3 face detection is feasible in Flash Player. Silverlight may be a different story, but it doesn’t support web cams yet.

The first prototype was implemented in Flash since I refused to program in XML (which you tend to to in Flex) as a matter of principle. When it turned out that Flash doesn’t support web services, I relented and changed to Flex. The parameter marshalling code that Flex Builder generates turned out to be too slow for low latency operation tough and was dumped in favour of a HttpService. The server part is implemented in ASP.Net, with C# calling EmguCV wrapped OpenCV. The code is at the bottom of the post.

The client goes through the following steps:

  1. Setting up the UI, getting the web cam running, setting up a timer to tick three times a second.
  2. On timer ticks, a shot is grabbed from the web cam.
  3. The picture is scaled to 320×240 pixels. The scaling is handled is by native library code and takes a few milliseconds.
  4. The picture is converted to grayscale. OpenCV facedetection only operates on one colour channel anyway and grayscale images compress better. The colour transformation is handled by more native library code and also takes just a few milliseconds.
  5. The picture is then PNG encoded. An encoder lives in the mx.graphics.codec namespace and at ~150ms for a 320×240 pic, it’s pretty fast. The resulting bytearray is around 60kb.
  6. The bytearray is base64 encoded which takes around 100ms.
  7. Finally the bytearray is sent to the server along with a sequence number. The server returns the sequence number so that the client can make sure it doesn’t use old, out-of-sequence results.

Upon receipt of a post, the server does the following:

  1. The request string is base64 decoded to a bytearray.
  2. A bitmap is created from the bytearray and made into a EmguCV grayscale image.
  3. The faces are indentified with a Haar Cascade detector
  4. The results, along with some performance information and the sequence number, are returned to the client as XML

Total server-side time is around 150ms with almost all of it spent in the face detection call. The client can do whatever it want’s with the face information — my demonstration projects a bitmap on top of detected  faces in the video feed. Total roundtrip time is less than half a second at three frames a second on a non-bandwith constrained connection in the same city as the server. This is probably enough to give relatively passive users the illusion of real-time face detection. I’ve also gotten positive reports from testers in the US where latency is higher.

I experimented with zlib compression instead of PNG encoding but the difference in both space and time was marginal (PNG uses zlib so that shouldn’t be too surprising). Instead of grayscaling the image, I also tried yanking out just one colour channel from the 32bit RGBA pixels, but iterating 320x240x4 bytearrays in Actionscript was dead-slow. Ditching http in favour of AMF and using AMF.Net or FluorineFx on the server may improve latency further.

What is point if all this you ask? A haven’t the foggiest idea really, but it’s good fun. You can certainly use it to create very silly facebook applications. You could conceivably use it to detect the mood of your users by determining whether they are smiling or not. Is this usable in a production scenario? Hard to say. The current server is a 3.5GHz Celeron D — a right riceboy machine. With one user hitting it, the CPU is at 20-30% utlization, suggesting it’ll tolerate a few concurrent users at most. Getting a proper machine and installing Intel Performance Primitives would probably help a lot. If you really meant it, you could run the OpenCV part on a PS3 or on a GPU.

Download the important parts of the code here.