Judging the Judges

In the startup world, there are many occasions in which startups are judged and few (if any) occasions when the judges themselves are judged.

I want to quote my friend and startup advisor Kevin Dewalt, who in a blog post wrote “We don’t need to be judges – the customers are the only judge that matters.” Kevin’s post references startup events in Asia but now that I’ve been back in the US for a while, I see many similarities in the way we can improve judging of startups. Here are some things I’ve done to try to be a better “judge.”

• Before I judged at a Startup Weekend for the first time, I ran my own one-person Startup Weekend to try to understand what the teams went through. While I had mentored there before, I had never been a participant in the event. I find that if the judges have never been in the position of the people on stage presenting, they lack the ability to quickly understand what the presenters have achieved. That being said, I think that Startup Weekend (if followed correctly) has one of the best opportunities to produce good judge and judgment experiences.
• Since then, I’ve judged at different events. I see that presentation skills can trump anything else even when we’re on guard and looking at the businesses behind the presentations. Again, to quote Kevin’s post: “As an Angel investor I really don’t care much about “pitch” quality – I care whether you’re solving a real problem for customers and can make money doing it.” But being a good presenter almost always gives you an advantage over the others. As a judge, I stay alert when I encounter this this skill.
• I memorize the judging criteria. This is something that I have not yet seen many others do, which causes confusion during deliberation. There are two reasons behind this: laziness of judges and organizers who don’t provide judging criteria until right before the event. Excuse me if I sound upset, but I figure if people are going to work hard on their projects for two days straight (or in other situations for months or years), expecting to be judged by pre-set criteria, the least we should do is to know the rules and judge them according to expectations. Lack of clarity and familiarity leads to confusion during deliberations.
• Knowing how to give feedback after a brief few minutes. Something I often see experienced people struggle with is giving feedback while judging. This is tricky because there’s usually only minimal information to go by and our individual biases run strong. I often see experienced people judging say why the team is on the wrong track, because they’re doing something contrary to the judge’s experience. Here are my favorite useless comments I’ve heard while sitting on judging panels:
  • “Look, I know this industry very well, because [famous name] pitched me [years ago] and this is not the way this works…”
  • “I’ve worked in this market for years and you don’t understand that [insert random specific factoid]. You ignored this, so I am penalizing you.”
  • “My employees could build this in a weekend.” This was at a more developed startup pitch (not a hackathon or Startup Weekend) so the comment was even more insulting. The entrepreneur was polite and handled it well.
  • “I would never invest in this.” Truth is it’s outside of the judge’s industry and he’s not an investor anyway.
• When I interact with the teams, my hack to keep myself open-minded and available to learn is to try to only ask questions rather than make declarations. As in, “You didn’t mention talking to any potential customers when you built [your hack / project / prototype]. Can you tell us about what you did to validate that you have a real problem and a validated solution?” “What are the per user metrics?” “Why did you build this as a mobile app instead of a web service?”
• Don’t try to look smart or to be the Simon Crowell of the panel. One of the oddest judging experiences I had was when I was on a panel of three — me and two others competing with each other to be the mean judge.
• I never am tough on the teams with the explanation that “that’s the only way they will learn,” as I hear judges say. Too often, the toughness does not accompany real feedback that is actionable. Judges should preface (even tacitly) any feedback with the phrase “what I would do if I were you…”

I would love to see a reverse judging event, where the judges are scored on following the judging criteria, giving actionable feedback and their relevancy. And revisiting the startups after a year to see which judges were right would be interesting also.

If you like this post, you might like my book Startup Sacrilege.