Webcam face detection in C# using Emgu CV

Some time ago I wrote a post on how to do face detection in C# using OpenCV. I’ve since begun using the Emgu CV wrapper instead of opencvdotnet. Emgu CV is much better, in active development and it even runs on Mono. Two gotchas:

  1. You don’t have to install OpenCV, but instead have to copy the relevant dlls (included with the Emgu CV download) to the folder where you code executes.
  2. Open CV and X64 are not friends. If you’re running X64 Windows (and unless you are up to recompiling OpenCV) you have to make sure your app is compiled to X86, instead of the usual “Any CPU”.
  3. Remember to add PictureBox as per the original tutorial.

Here’s sample code:

using System;
using System.Windows.Forms;
using System.Drawing;
using Emgu.CV;
using Emgu.Util;
using Emgu.CV.Structure;
using Emgu.CV.CvEnum;

namespace opencvtut
{
    public partial class Form1 : Form
    {
		private Capture cap;
		private HaarCascade haar;

        public Form1()
        {
            InitializeComponent();
        }

        private void timer1_Tick(object sender, EventArgs e)
        {
		using (Image<Bgr, byte> nextFrame = cap.QueryFrame())
		{
			if (nextFrame != null)
			{
				// there's only one channel (greyscale), hence the zero index
				//var faces = nextFrame.DetectHaarCascade(haar)[0];
				Image<Gray, byte> grayframe = nextFrame.Convert<Gray, byte>();
				var faces =
					grayframe.DetectHaarCascade(
						haar, 1.4, 4,
						HAAR_DETECTION_TYPE.DO_CANNY_PRUNING,
						new Size(nextFrame.Width/8, nextFrame.Height/8)
						)[0];

				foreach (var face in faces)
				{
					nextFrame.Draw(face.rect, new Bgr(0,double.MaxValue,0), 3);
				}
				pictureBox1.Image = nextFrame.ToBitmap();
			}
		}
        }

        private void Form1_Load(object sender, EventArgs e)
        {
            // passing 0 gets zeroth webcam
			cap = new Capture(0);
            // adjust path to find your xml
			haar = new HaarCascade(
                "..\\..\\..\\..\\lib\\haarcascade_frontalface_alt2.xml");
        }
    }
}

Smuggling exceptions out of try-catches with IEnumerable

The code in this post is from a talk by Erik Meijer, I claim no credit.

Take a look at this screenshot:

Exception thrown from try-catch

Pretty wacky huh? Visual Studio has broken into debug mode even though the division is clearly wrapped in a try-catch. Take a guess at what’s going on before reading on.

Here’s the full code listing:

static void Main(string[] args)
{
    var xs = new[] { 1, 2, 3, 0 };
    IEnumerable q = null;

    try
    {
        q = from x in xs select 1 / x;
    }
    catch { }

    foreach (var z in q)
    {
        Console.WriteLine(z);
    }
}

The obvious expectation is that division-by-zero will be caught by the try-catch. Alas, the division does not happen in the scope of the try statement. Instead, due to the deferred execution paradigm of IEnumerable, it happens on a by-need basis in the for-loop. From there the exception can propogate unopposed to the top af the stack. Haskell, where lazy evaluation dominates, avoids exceptions (and other messy side effects) alltogether, partly for reasons demonstrated by this example.

UPDATE: As Wayne points out, fixing this is as easy as extending the try to cover the enumeration. The point of this post was mainly to poke fun at the Visual Studio debugger and to point out that lazy evaluation can bite you if you’re not careful.

CRMMetal and LINQtoCRM

As you may know, Microsoft introduced many-to-many relationships in version 4.0 of Dynamics CRM. Unfortunately, querying these relationships is not supported through the general web service entities as the intermediary entities are not exposed. Fetch XML works fine however (and don’t forget this great hack to generate FetchXML). The missing entities meant that LINQtoCRM didn’t support many-to-many relationships either, even though the underlying query-engine generates Fetch XML.

To remedy this deficiency, a simple tool is now bundled with experimental versions of LINQtoCRM. In the tradition of the other major query providers, it’s called “CRMMetal”. It works by asking the metadata web service for all the metadata, including relationships, and then filtering out the many-to-many ones:

RetrieveAllEntitiesRequest allEntitiesRequest = new RetrieveAllEntitiesRequest();
allEntitiesRequest.RetrieveAsIfPublished = false;
allEntitiesRequest.MetadataItems = MetadataItems.IncludeRelationships;

RetrieveAllEntitiesResponse allEntitiesResponse = 
	(RetrieveAllEntitiesResponse)service.Execute(allEntitiesRequest);

var mtom = allEntitiesResponse.CrmMetadata.OfType().
	SelectMany(e => e.ManyToManyRelationships, (e, d) =>
		new
		{
			intersectname = d.IntersectEntityName,
			schemaname = d.SchemaName.Replace("_association",""),
			ent1Name = d.Entity1LogicalName,
			ent2Name = d.Entity2LogicalName,
			ent1Att = d.Entity1IntersectAttribute,
			ent2Att = d.Entity2IntersectAttribute
		}
		).Distinct().OrderBy(s => s.schemaname);

It then uses the CodeDOM API to generate classes similar to the web service ones, although the property bodies are empty and there’s no XML serialization attributes. It seems CodeDOM has not been updated with recent .Net releases: You can’t generate automatic properties and the API doesn’t support declarative composition of code, a great shame I think. Other than that, generating the code is pretty straightforward:

CodeCompileUnit targetUnit = new CodeCompileUnit();
string nameSpace = "LinqtoCRMApplication.CRM";

CodeNamespace ns = new CodeNamespace(nameSpace);
ns.Imports.Add(new CodeNamespaceImport("System"));

// The stupid codedom API doesn't properly support declarative DOM building, fail.
CodeTypeDeclarationCollection classes = new CodeTypeDeclarationCollection(
	mtom.Select(_ =>
		new CodeTypeDeclaration()
		{
			//Name = _.name,
			Name = _.schemaname,
			IsClass = true,
			IsPartial = true,
		}
	)
	.ToArray());

foreach (CodeTypeDeclaration c in classes)
{
	int count = mtom.Where(_ => _.schemaname == c.Name).Count();
	if (count > 1)
	{
		Console.WriteLine("Ignoring {0} due to duplicality", c.Name);
		continue; // bad one, multiple with same name
	}
	
	c.Members.AddRange(new CodeTypeMember[]
		{
			new CodeMemberProperty() 
			{ 
				Name = mtom.Single(_ => _.schemaname == c.Name).ent1Att,
				Type = new CodeTypeReference(nameSpace + ".Key"),
				HasGet = true,
				HasSet = true,
				Attributes = MemberAttributes.Public,
			},
			new CodeMemberProperty() 
			{ 
				Name = mtom.Single(_ => _.schemaname == c.Name).ent2Att,
				Type = new CodeTypeReference(nameSpace + ".Key"),
				HasGet = true,
				HasSet = true,
				Attributes = MemberAttributes.Public,
				
			}
		}
	);

	// goddam codedom doesn't support automatic properties, 
	// have to add something to getters. fail.
	foreach (CodeMemberProperty p in c.Members)
	{
		// just have it return null
		p.GetStatements.Add(
			new CodeMethodReturnStatement(
				new CodePrimitiveExpression(null)));
	}
}

ns.Types.AddRange(classes);
targetUnit.Namespaces.Add(ns);

CodeDomProvider provider = CodeDomProvider.CreateProvider("CSharp");
CodeGeneratorOptions options = new CodeGeneratorOptions();
options.BracingStyle = "C";
using (StreamWriter sourceWriter = new StreamWriter("ManyToManyClasses.cs"))
{
	provider.GenerateCodeFromCompileUnit(
		targetUnit, sourceWriter, options);
}

Go get the code and give it a whirl. LINQtoCRM still has a few large wharts, the ugliest probably being the very limited selectors permitted. I’ll try to lift the selector implementation from LINQtoSharePoint soon.

Using WCF to power Facebook feedstory forms

UPDATE: I think Facebook has changed the API so that links in full stories have to be split into src and href parts.

Several parts of the Facebook developer API requires application developers to furnish JSON web services that feed data to various actions. These include feed forms (“I want to stick something on my wall”) and the publisher to name a few. This post will descripe how that JSON can be generated through Windows Communication Foundation web services. You can see this code in action in the simple Louisefy application. Note that alternatives certainly exists (I recommend Json.NET), but now that we have WCF, why not use it?

(As Rune points out in a comment below, you can use the object-hierachy in the code to generate compliant JSON even if you’re using Json.NET instead of WCF).

As far as feed forms go, the developes pre-register a range of templates for the kind of stories they want their users. A set of templates that get fed to Feed.registerTemplateBundle might look like the following:

List oneLiners = new List() {
	"{*actor*} is <a href=\"{*linkurl*}\">{*linktitle*}</a> {*target*}s Facebook",
};
List shortTmplts = new List() {
	new Dictionary() {
		{"template_title","{*actor*} is <a href=\"{*linkurl*}\">{*linktitle*}</a> {*target*}s Facebook"},
		{"template_body","."}
	},
};
Dictionary fullTmplts = new Dictionary() {
	{"template_title","{*actor*} is <a href=\"{*linkurl*}\">{*linktitle*}</a> {*target*}s Facebook"},
	{"template_body","{*swf*}"}
};

The FBML markup to get a feed form that lets users publish on their friends feeds would look like this:

<form fbtype="multiFeedStory" action="http://link.to/myfeedhandler.svc">
Start typing names of your friends:
<fb:multi-friend-input />
<input type="submit" label="Louisefy your friends feeds" />
</form>

The code for the actual feedhandler follows below, with the web.config stuff at the very bottom. I really like the terse, declarative syntax — there’s not a single hint that this all becomes JSON at some point. Note that if you need to pass arguments to your feedhandler, they’re very easy to map in the UriTemplate. I generally find WCF services to be fiendishly complex to get up and running because of the dearth of good tutorials and documentation — but once you get them going, it’s pure joy.

[ServiceContract]
[AspNetCompatibilityRequirements(RequirementsMode = AspNetCompatibilityRequirementsMode.Allowed)]
public class feedHandler
{
	[WebInvoke(ResponseFormat = WebMessageFormat.Json,
	BodyStyle = WebMessageBodyStyle.Bare,
	Method = "POST",
	UriTemplate = "/friendsfeed")
	]
	[OperationContract]
	public FeedResponse GetResponse()
	{
		return new FeedResponse
		{
			method = "multiFeedStory",
			content = new Content
			{
				feed = new Feed
				{
					template_id = "1234",
					template_data = new TemplData
					{
						swf = "",
						linktitle = "Louisefying",
						linkurl = "http://apps.facebook.com/louisefy/",
						images = new FeedImage[]
						{
							new FeedImage
							{
								src = "http://foo.bar/my.png",
								href = "http://apps.facebook.com/louisefy"
							}
						},
					}
				},
				next = "http://apps.facebook.com/louisefy",
			}
		};
	}
}

[DataContract]
public class FeedResponse
{
	[DataMember]
	internal string method;

	[DataMember]
	internal Content content;
}

[DataContract]
class Content
{
	[DataMember]
	internal string next;
	[DataMember]
	internal Feed feed;
}

[DataContract]
class Feed
{
	[DataMember]
	internal string template_id;

	[DataMember]
	internal TemplData template_data;
}

[DataContract]
class TemplData
{
	[DataMember]
	internal FeedImage[] images;

	[DataMember]
	internal string swf;

	[DataMember]
	internal string linkurl;

	[DataMember]
	internal string linktitle;
}

[DataContract]
class FeedImage
{
	[DataMember]
	internal string src;

	[DataMember]
	internal string href;

}

web.config:

<system.serviceModel>
	<behaviors>
	  <endpointBehaviors>
		<behavior name="Popcorn.feedHandlerAspNetAjaxBehavior">
		  <webHttp />
		</behavior>
	  </endpointBehaviors>
	</behaviors>
	<serviceHostingEnvironment aspNetCompatibilityEnabled="true" />
	<services>
	  <service name="Popcorn.feedHandler">
		<endpoint address="" behaviorConfiguration="Popcorn.feedHandlerAspNetAjaxBehavior"
		  binding="webHttpBinding" contract="Popcorn.feedHandler" />
	  </service>
	</services>
</system.serviceModel>

Facebook signature generation in .Net

While the Facebook Developer Toolkit handles most of the needs of your average Facebook application, there is not support for the Post-Remove stuff that happens when users decide to remove your application. When your application is removed by a user, Facebook submits a “Post” in the the general direction of a url you’ve specified. The post contains a “fb_sig_user” with the relevant userid and the naive developer would remove this user from their database. That — however — would make the application vulnerable to an attack where a bad guy would hammer the post-remove form with request containing random userid, causing you to falsely believe these users have removed your application.

Below is some code that computes the proper signature that you can use to compare against the one Facebook sends along with the request. The crypto-parts were lifted from the guts of the Facebook Developer Toolkit. The process of computing the signature is described here.

public static string GenerateSignature(NameValueCollection formParams, string secret)
{
	string[] keys = formParams.AllKeys;
	Array.Sort(keys);
	string prefix = "fb_sig_";

	Dictionary dict = new Dictionary();
	var dictInit = keys.Where(k => k.StartsWith(prefix))
		.Select(k =>
			new { key = k.Substring(prefix.Length), val = formParams.Get(k) });
	foreach (var p in dictInit) { dict.Add(p.key, p.val); }

	var signatureBuilder = new StringBuilder();

	// Sort the keys of the method call in alphabetical order
	List keyList = ParameterDictionaryToList(dict);
	keyList.Sort();

	// Append all the parameters to the signature input paramaters
	foreach (string key in keyList)
		signatureBuilder.Append(String.Format(CultureInfo.InvariantCulture,
			"{0}={1}", key, dict[key]));

	// Append the secret to the signature builder
	signatureBuilder.Append(secret);

	MD5 md5 = MD5.Create();
	// Compute the MD5 hash of the signature builder
	byte[] hash = md5.ComputeHash(
		Encoding.UTF8.GetBytes(signatureBuilder.ToString().Trim()));

	// Reinitialize the signature builder to store the actual signature
	signatureBuilder = new StringBuilder();

	// Append the hash to the signature
	foreach (byte hashByte in hash)
		signatureBuilder.Append(hashByte.ToString("x2", CultureInfo.InvariantCulture));

	return signatureBuilder.ToString();
}

Here’s some code that uses the method:

string sig = GenerateSignature(Request.Form,
	ConfigurationManager.AppSettings["FACEBOOK_SECRET"]);
if (sig == Request.Form["fb_sig"])
{
	string fbuid = Request["fb_sig_user"];
	if (fbuid != null && fbuid.Length > 0)
	{
		// Remove user
	}
}

Facedectection in C# with less than 50 LoC

I’ve been mucking around with face detection a bit lately. Here’s a short guide to getting face detection running in C# in a short amount of time.

UPDATE: Recommend EmguCV for C# wrapping OpenCV, read the updated guide.

I’ve identified two free computer vision libraries that do face detection:  Torch3vision and OpenCV(I’m sure there are plenty more, but these seem to be comprehensive, recently updated and freely available). Torch3vision claims to be better than OpenCV but on the other hand more people are building libraries and wrappers around OpenCV and there’s even a wrapper for .Net. While it doesn’t yet wrap the face detection components of OpenCV, it seems to be the most promising solution.

To get face detection in OpenCV to work with C#, do the following:

  1. Install OpenCV and OpenCVDotNet as per the instructions
  2. Get CVHaar.h from this discussion and place it in C:\Program Files\OpenCVDotNet\src\OpenCVDotNet UPDATE: Dud link, read the updated guide
  3. Open OpenCVDotNet.sln, add CVHaar.h to the solution and include it in OpenCVDotNet.cpp
  4. Rebuild the solution
  5. Create a Windows forms application as described in the tutorial, but do something like the code below
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;
using OpenCVDotNet;

namespace opencvtut
{
    public partial class Form1 : Form
    {
        private CVCapture cap;
        private CVHaar haar;

        public Form1()
        {
            InitializeComponent();
        }

        private void timer1_Tick(object sender, EventArgs e)
        {
            using (CVImage nextFrame = cap.QueryFrame())
            {
                Rectangle[] faces = haar.detectPatterns(1.3, nextFrame, 20, 20, 1.1, 2, 1);
                for (int i = 0; i < faces.Length; i++)
                {
                    nextFrame.DrawRectangle(faces[i], Color.Yellow, 3);
                }
                pictureBox1.Image = nextFrame.ToBitmap();
            }
        }

        private void Form1_Load(object sender, EventArgs e)
        {
            // passing 0 gets zeroth webcam
            cap = new CVCapture(0);
            haar = new CVHaar(
                "C:Program FilesOpenCVdatahaarcascadeshaarcascade_frontalface_alt2.xml");
        }
    }
}

Happy detecting 🙂

.Net/Firefox Screen Scraping

Need to scrape a website? I have two links for you:

Solvent from the MIT SIMILI project. In combination with Piggy Bank it’s a scraper on it’s own, but I only use for its superb XPath generator. Just activate the sprayer and click on an element you want and Solvent will generate an intelligent XPath expression to get at it. Solvent will also higlight other elements on the page that would be returned with by the query and you can even dynamically edit the expression to narrow or broaden the result. All visible right there in the browser window.

Now switch to Visual Studio and Html Agility Pack, a great project that lets you parse HTML documents and query them using XPath just like they where XML. Solvent and Html Agility Packs (HAP) perception of the the DOM may sometimes differ slightly but, with a bit of tweaking, the visually generated XPath from Solvent works just great with your HAP code.

Truly a match made in heaven…

Concise code

Yesterday when porting LinqtoCRM to VS 2008 Beta 2, I had to get an array of the types of the properties of some generic type. This array would be passed to Type.GetConstructor(). I had an array of PropertyInfos which contain the type of the property, and I could have new’ed up an array and looped the types into that. Instead I did this:

PropertyInfo[] props = typeof(T).GetProperties();
Type[] types = props.Select(p => p.PropertyType).ToArray();
ConstructorInfo cInf = typeof(T).GetConstructor(types);

Newer Posts