GotoTech.com

My baby names web site is now live. Baby Namester has a powerful recommendation engine to help you find that perfect baby name. Just create a free account, vote on some baby names, get get some good recommendations. Please try it out and let me know what you think! BabyNamester.com

I’ve just started a new side project that will involve finding some creative solutions to tough problems. I thought it would be neat to share what I learn as I go, so here goes…

For my project I decided to use the Doctrine 2 ORM to manage my data layer. We also use this at work, so the biggest reason I chose this was to be able to learn more about Doctrine to help me in my job. But this decision also makes sense for my project because my entity relationships will likely be fairly straightforward for the most part and using an ORM will allow me to make a lot of progress very quickly without (I hope) causing me lots of trouble later on.

One of the odd little things about Doctrine is how it handles persistence. The rules are a bit complex and non-intuitive. Here’s a summary:

• Changes are saved when you call flush() on the entity manager, and at no other time.
• Objects that were already in the database that you made changes to will be updated at this time with no special instructions.
• New objects will not be saved by default.
• If you want a new object saved, you need to persist() it.
• If something would have saved and you don’t want it to, you need to detach() it.
• Objects that are linked to each other through association fields can cause problems when one part is created or deleted and one isn’t – this can cause you to error out nastily and close your entity manager.
• You can avoid this by creating cascade rules in your model classes, but this can lead to updates happening that you didn’t intend if you aren’t careful!

Clear as mud, right? We kind of have a love and hate (sometimes hate and hate) relationship with Doctrine at work because the hardest 20% of things we need to do with it tend to be very hard and annoying. One of the annoying things is that when a function needs to update an object, it’s not always obvious if we should flush. If we flush, maybe we’re committing a temporary change that shouldn’t be committed. Or maybe our data is still in an inconsistent state and the entity manager will crash. If we don’t flush, maybe we’re creating a burden on the controller layer to remember to do it for us (which isn’t what our controller layer should care about). We’ve sometimes resorted to having a flush flag parameter to a function to tell it whether to flush or not. This is complexity that we’re better off without!

It tends to be easier to solve these problems by adding way more flushes than we strictly need, which is not good for performance. The data in my side project is simple, but there’s a lot of it, so I wanted to mitigate this problem. This is my first stab at a solution:

First of all, I’m not using Doctrine’s entity manager directly. I’ve created an abstraction layer called the Data Manager which my code calls to perform all entity manager actions. This not only gives me better options to tame Doctrine’s rough spots, but gives me a fighting chance to yank Doctrine out and replace it if I decide it’s causing me significant problems.

<?php

class BB_Manager_Data
{
     static protected $instance = null;
     static protected $em = null;
     ...

If I want to go old school and just flush, I have a function for that. If I just call it with no preliminaries, it’s just a pass through. But I’ve borrowed the idea of SQL transactions to handle chunks of related changes. If I call startTransaction(), the flush() function will do nothing. When I call commit(), flush() is reenabled, and then called. This makes the idea of transactions more explicit and allows me to be more flexible about how I commit changes without cluttering my method signatures. Here’s what the first version of my code looked like:

static protected $disableFlush = false;

static public function startTransaction()
{
     self::$disableFlush = true;
}

static public function commit()
{
     self::$disableFlush = false;
     self::flush();
}

static public function flush()
{
     if (!self::$disableFlush) {
          $em = self::getEntityManager();
          $em->flush();
     }
}

Ah, but Doctrine supports transactions! It’s just a matter of calling $em->getConnection()->beginTransaction(); to start and $em->getConnection()->commit(); to complete or $em->getConnection()->rollback(); to cancel. Until commit() is called, the flushes won’t actually flush. So another version of the above would be:

static public function startTransaction()
{
     $em = self::getEntityManager();
     $em->getConnection()->beginTransaction();
}

static public function commit()
{
     $em = self::getEntityManager();
     $em->getConnection()->commit();
}

static public function flush()
{
     $em = self::getEntityManager();
     $em->flush();
}

Or you could just use Doctrine’s transactional functions outside of an abstraction layer.

Which of these versions is better depends on how Doctrine implements its transactional functionality (which I don’t know yet) and what your particular needs are. My initial version may actually be more performant in some cases if Doctrine’s transactions are just pass-throughs to your DBMS’s transactions. But it may be a micro-optimization or none at all. I may dig a little further or do some A/B tests to gather some information about that.

I think the transactional capabilities are an underutilized tool in Doctrine 2′s toolbag that can provide useful flexibility, group related changes, and minimize database writes. And the transactional paradigm (whatever the implementation) goes well with many use cases to help tame those flushes and optimize database writes.

Ingredients:

• 2lbs of ground beef
• 2 1/4 cubs of chopped onions
• 3 TBSP of ground cumin
• 3 TBSP of ground chipotle chili powder
• 1/2 tsp red cayenne pepper powder
• 3/4 tsp dried oregano
• 1 1/2 TBSP garlic powder
• 1 1/2 – 2 TBSP of minced canned chipotle chili in adobo sauce
• 2 1/2 cups of water (more as needed)
• salt & pepper seasoned to taste
• Shredded Cheese
• Sour cream

Directions:

• In a large dutch oven or chili pot, add a little oil and saute the onions
• add beef and cook through
• add all seasonings, mix thoroughly
• add water, let it boil, then cut the heat back to medium-low
• cook for at least one and a half hours
• serve w/ shredded chedder cheese & sour cream

So my friend Wes Walser, who works for Atlassian in Sydney, Australia, posted about this amazing thing they just installed at their offices on his Facebook page:

When Atlassian opened up a second office across the street they wanted an efficient way to communicate across locations. What better way than a portal inspired by the Valve video game Portal? Unfortunately, it only transmits light and sound at this point, but it’s still very cool! Check out Atlassian’s blog for more details.

My favorite story about innovation vs. execution starts with two writers in a web forum. Since it was a web forum, an argument was raging. One writer claimed that a novel needed to have an interesting and creative premise to be successful. A second writer clung to the opposite point of view: even a poor or overused premise can lead to a great novel if it is well enough written.

The argument got more and more contentious until the second writer, whose name was Jim Butcher, offered the first a challenge. He said, don’t just give me one bad idea, give me the TWO worst and most played out ideas for a novel that you can think of, and I’ll write a great novel from them. The first writer said, okay, here are my ideas:

• Lost Roman Legion
• Pokémon

Jim Butcher took these two ideas and wrote The Furies of Calderon, a very successful novel that grew into a seven book series.

Why was the novel successful? Well, he did find creative spins on those lame ideas. But mostly it was great characters, well-crafted plotting, spot on pacing, fun plot twists, and very readable prose. In other words, superb execution. I’ve read the whole series, and it’s one of my favorites.

But what about software products? Can you be successful without innovation? One great example is the game Angry Birds.

The Angry Birds app hit the iPhone in December of 2009 and now, a scant two years later, it’s hard to imagine a world where toddlers aren’t going crazy for them and the founding members of Rovio Mobile aren’t lighting their cigars with €100 euro bills. Was it innovation or execution that led to their success? Well, ask Liam Bowmer, who in September of 2008 released a web game with the exact same game play called Castle Clout. The game was renamed to Crush the Castle and released on the iPhone in September 2009, three months before Angry Birds. It would not be an exaggeration to call Angry Birds a reskin of Crush the Castle. And similar gameplay has been in various games for years, even before Crush the Castle (Scorched Earth, for example, first released in 1991).

The critical difference that caused Angry Birds to be a world beater and Crush the Castle to be a footnote was execution. Angry Birds’ successful execution can be seen in the details. The graphic design clicked with their audience, especially younger children. The music and sound effects are polished. Small touches like the structures shaking slightly when a new level loads (showing that they can be knocked down, tempting you) excites your interest in a way that Crush the Castle just doesn’t.  Adding to the four principles from my last article, this brings us to our fifth principle:

Principle 5 – Great execution can make up for lack of innovation.

But what about the big innovators? Think about iPhone from Apple. Or sticking with games, what about the success of Valve with the game Portal? Or what about Pixar’s digital revolution in animated movies?

Companies that successfully innovate and then reap the benefits of that innovation have one thing in common: amazingly polished products. The iPhone is a great example. It is innovative, but like Angry Birds, it also shines in the details. The interface is clean, minimal, and intuitive. A lot of thought and user testing was obviously spent on how to make a 3.5″ touchscreen be the window into a satisfying user experience. Valve’s products, besides being innovative, are bug-free exemplars of great pacing and level design. You see that quality in the details with Pixar as well. It’s true they created a revolution of 3D animated movies, but the pure movie making craft that Pixar demonstrates would have been successful in a more traditional type of animation (just not as successful). Watch the first ten minutes of Up or the first twenty minutes of Wall-E and try to disagree. This leads us to the sixth principle:

Principle 6 – Companies who successfully innovate absolutely nail execution.

What’s interesting is how at least one of these companies accomplishes this level of polish. Valve has a strongly empirical approach to game development. They keep games in development much longer than a lot of their competitors and put even early versions of their games through tremendous amounts of user testing.

Quoting Gabe Newell, co-founder and managing director of video game development at Valve:

At Valve, we see our game designs as hypotheses and our playtests as experiments to validate these hypotheses. In addition, we always want to iterate and improve on our work, and we are constantly seeking feedback – through playtests and other means – in order to do so. By making use of a wide array of user research techniques, we will make better decisions and as a consequence, better games.“

…

“We start playtesting as early as we can—as soon as we have something playable. We’ll start with internal folks and then bring in external folks soon after.

This leads us to…

Principle 7 – The best way to assess how well you are executing and to get better is through user testing.

Does an innovation have to be polished right out of the gate? Well no. One interesting example is another game, Minecraft. Released by Markus Persson initially as a public alpha, it certainly had some rough edges and limited gameplay which undercut its innovative game idea. But people saw the promise in it – enough to purchase the for-sale beta at a reduced cost (which included access to the final game when released) – to fund the development of the game. Playing the release version, for all its quirky graphics and sandbox environment, it feels like a polished and complete game. The initial alpha was released in May of 2009, and the 1.0 release became available November 2011. To date, it has sold over 4 million units. It’s interesting to note that Markus Persson had a huge base of players essentially playtesting his product and paying for the privilege! This certainly helped Minecraft become more successful.

Since I’ve been harping on execution being more valuable than innovation, let me step back affirm that innovation is good. It will basically do two things for you. First, it will get you noticed. You’ll have a chance to follow through on the promise of your idea. It will buy you time – not forever – but you’ll have a window. Companies get that chance and mess it up all the time, but it’s a tremendous opportunity that many others have made good on. Second, as long as your execution is sound, innovation will amplify your success – sometimes quite dramatically. Polished execution is the price to get in. But superb innovation can multiply your success.

I like to think of execution as the foundation that will keep the house standing. But innovation, combined with things like business drivers, will determine the heights of your potential success. Superbly execute something that isn’t very interesting at its core, and your potential for success will usually be quite limited.

My last principle, which I’ll allow to sum up this article:

Principle 8 – Innovation raises the ceiling on how successful a product can be, but solid execution is a bedrock requirement for realizing that potential.