Artima Developer Spotlight

Rapid Hiring and Firing to Build the Best Teams

Message from fivefilters.org: If you can, please donate to the full-text RSS service so we can continue developing it.

Summary
Apparently the founder of Kayak.com does this, in an attempt to not only prevent poisonous people from destroying teams, but to go in the other direction and create exceptional teams.

Last night I went to the Extreme Programming San Diego meeting (it should probably be called "Agile" but I suspect the group is old enough that XP was what started it). A reader knew that I was in San Diego visiting my folks so he suggested it. I thought I'd dip my toe in the water and see if programming was any more interesting to me than the last time I checked. Especially since Agile practices tend to be more focused on people issues, which is where I seem to continue to drift.

The first thing I noticed was the turnout. I'm guessing there are thousands of programmers in San Diego, but we got maybe 10 people to show up. That's probably a condemnation of the profession, where the majority don't read or learn new things. It could also be a number of other issues, the biggest of which is, I think, the fact that user groups cling to the same old approach that everyone else does: get a speaker to come in and give a lecture. The speaker was fine, and had even reduced bullet points in favor of more pictures. But there's something in my brain that says "nap time" whenever a projector gets turned on. The interaction is primarily one-way and scripted, but you can't fast-forward the lecture like you can on the web.

When asked, I related my interest in people issues, where I've come to believe lie the biggest possibilities for improvement. As an example, I talked about building teams, and how difficult it is to do it well within the business structures we currently have. One of the people there mentioned Kayak.com, and how the founder had hired and fired hundreds of people in order to get the 30 or so that he currently employs.

One way to look at the problem of company-building is that it should focus on creating great teams. I've heard numerous people say that the company could be in various kinds of bad shape but they could be happy in it as long as they were on a great team. If the team is indeed the fundamental component of the company, then it would be interesting to make an environment whose primary focus is to be a culture medium for great teams.

One of the biggest problems in team building can be thought of as a version of the "sunk cost" issue. Although it's better to think of lost investment as just that -- lost -- when making decisions, our brains tend to think in terms of what we've already invested. So if you've invested a lot of time and money in hiring someone, you'll tend to hang on to that person until the pain of doing so exceeds your imagination of what it took to get them in that position. This means that a poisonous person can easily be in a team long enough to destroy it before finally being ejected, at which point it's too late. You've thrown a bass into your team of goldfish, and while the bass is eating up your team, you're thinking "maybe he'll get full."

The other big factor is legal. Some people sued companies for wrongful termination, so every company adapted practices to prevent it. You can't just fire someone anymore, you have to give them a couple of 6-month evaluations to show due diligence.

I suspect the Kayak.com founder has everyone sign a contract that allows them to be fired easily. Producing company loyalty in a situation like that might be tough ... but if you get the right people then maybe that goes away. People might become vicariously loyal to their company because they really want to be in their team.

The real problem with hiring someone for a team is that you can't actually know how well they will work out until they're participating on a day-to-day basis. Nothing in an interview really tells you this. So apparently Kayak.com decided that most of the "interview" would go on by putting the person in the real situation and seeing how they worked out. Perhaps a little unsettling and brutal -- although looking for a job is never a great experience, and maybe having one, even if just for a short period, might be better than a long period out of work and looking -- but from the standpoint of the company it could produce some excellent results.

Does anyone have links to stories about Kayak.com's approach? What other types of practices might help solve the problem of team-building?

Five Filters featured article: Chilcot Inquiry. Available tools: PDF Newspaper, Full Text RSS, Term Extraction.

The Wonders of Cooperative Inheritance, or Using Super in Python 3

Message from fivefilters.org: If you can, please donate to the full-text RSS service so we can continue developing it.

Deep down, super in Python 3 is the same as in Python 2.X. However, on the surface - at the syntactic level, not at the semantic level - there is a big difference: Python 3 super is smart enough to figure out the class it is invoked from and the first argument of the containing method. Actually it is so smart that it works also for inner classes and even if the first argument is not called self. In Python 2.X super is dumber and you must tell the class and the argument explicitly: for instance our first example must be written

class A(object):
    def __init__(self):
        print('A.__init__')
        super(A, self).__init__()

By the way, this syntax works both in Python 3 and in Python 2, this is why I said that deep down super is the same. The new feature in Python 3 is that there is a shortcut notation super() for super(A, self). In Python 3 the (bytecode) compiler is smart enough to recognize that the supercall is performed inside the class A so that it inserts the reference to A automagically; moreover it inserts the reference to the first argument of the current method too. Typically the first argument of the current method is self, but it may be cls or any identifier: super will work fine in any case.

Since super() knows the class it is invoked from and the class of the original caller, it can walk the MRO correctly. Such information is stored in the attributes .__thisclass__ and .__self_class__ and you may understand how it works from the following example:

class Mother(object):
    def __init__(self):
        sup = super()
        print(sup.__thisclass__)
        print(sup.__self_class__)
        sup.__init__()

class Child(Mother):
    pass

>>> child = Child()
<class '__main__.Mother'>
<class '__main__.Child'>

Here .__self__class__ is just the class of the first argument (self) but this is not always the case. The exception is the case of classmethods and staticmethods taking a class as first argument, such as __new__. Specifically, super(cls, x) checks if x is an instance of cls and then sets .__self_class__ to x.__class__; otherwise (and that happens for classmethods and for __new__) it checks if x is a subclass of cls and then sets .__self_class__ to x directly. For instance, in the following example

class C0(object):
    @classmethod
    def c(cls):
        print('called classmethod C0.c')

class C1(C0):
    @classmethod
    def c(cls):
        sup = super()
        print('__thisclass__', sup.__thisclass__)
        print('__selfclass__', sup.__self_class__)
        sup.c()

class C2(C1):
    pass

the attribute .__self_class__ is not the class of the first argument (which would be type the metaclass of all classes) but simply the first argument:

>>> C2.c()
__thisclass__ <class '__main__.C1'>
__selfclass__ <class '__main__.C2'>
called classmethod C0.c

There is a lot of magic going on in Python 3 super, and even more. For instance, this is a syntax that cannot work:

def __init__(self):
    print('calling __init__')
    super().__init__()

class C(object):
    __init__ = __init__

if __name__ == '__main__':
    c = C()

If you try to run this code you will get a SystemError: super(): __class__ cell not found and the reason is obvious: since the __init__ method is external to the class the compiler cannot infer to which class it will be attached at runtime. On the other hand, if you are completely explicit and you use the full syntax, by writing the external method as

def __init__(self):
    print('calling __init__')
    super(C, self).__init__()

everything will work because you are explicitly telling than the method will be attached to the class C.

I will close this section by noticing a wart of super in Python 3, pointed out by Armin Ronacher and others: the fact that super should be a keyword but it is not. Therefore horrors like the following are possible:

def super():
    print("I am evil, you are NOT calling the supermethod!")

class C(object):
    def __init__(self):
        super().__init__()

if __name__ == '__main__':
    c = C() # prints "I am evil, you are NOT calling the supermethod!"

DON'T DO THAT! Here the called __init__ is the __init__ method of the object None!!

Of course, only an evil programmer would shadow super on purpose, but that may happen accidentally. Consider for instance this use case: you are refactoring an old code base written before the existence of super and using from mod import * (this is ugly but we know that there are code bases written this way), with mod defining a function super which has nothing to do with the super builtin. If in this code you replace Base.method(self, *args) with super().method(*args) you will introduce a bug. This is not common (it never happened to me), but still it is bug that could not happen if super were a keyword.

Moreover, super is special and it will not work if you change its name as in this example:

# from http://lucumr.pocoo.org/2010/1/7/pros-and-cons-about-python-3
_super = super
class Foo(Bar):
    def foo(self):
        _super().foo()

Here the bytecode compiler will not treat specially _super, only super. It is unfortunate that we missed the opportunity to make super a keyword in Python 3, without good reasons (Python 3 was expected to break compatibility anyway).

Five Filters featured article: Chilcot Inquiry. Available tools: PDF Newspaper, Full Text RSS, Term Extraction.

Pattern Matching in Scala For Expressions

Message from fivefilters.org: If you can, please donate to the full-text RSS service so we can continue developing it.

Summary
Scala pattern matchers with side effects can have unexpected results in for expressions. Scala has taken the traditional for-construct and put it on steroids. One of the interesting features is the ability to use pattern matching as part of a for expression. For example, the following:

case class Person(firstName:String, lastName: String); val people = List( Person("Jane", "Smith"), Person("John", "Doe"), Person("Jane", "Eyre")); for (Person("Jane", last) <- people) yield "Ms. " + last;

will (effectively) iterate over all people with a first name of Jane, bind last to each person's last name in turn, and then generate a list of the results of the yield block, thus returning

List("Ms. Smith", "Ms. Eyre")

In this example, we're using a case class to provide the pattern matching for us, but we can easily role our own. For example, consider the following matcher which will return the first element of an iterator, if there is one:

object First { def unapply[A](iter:Iterator[A]): Option[(A)] = { if (iter.hasNext) Some(iter.next) else None; } }

We can try to get the first element of some iterators:

for(First(x) <- iters) yield x;

However, this does not yield List(1,3) as one might expect. Instead, it yields List(2, 4)!

So what is going on here? According to Programming in Scala, a pattern match in a for loop gets converted into a filter followed by a map. Specifically,

for (pat <- expr₁) yield expr₂

is translated by the compiler to

expr₁ filter { case pat => true; case _ => false; } map { case pat => expr₂; }

In other words, first the input expression is filtered for those elements which match the pattern, and then in a separate pass, the pattern is applied for the purpose of evaluating expr₂. Unfortunately, there are a couple of issues with this. First, if the pattern match is computationally expensive, that work will be doubled for every match. Second, and more concerning, if the pattern match is not side effect free, then unexpected results like the one above can happen. In the example above, the pattern match, by calling next on an iterator, changes the state of the iterator, so that a different result comes out in the second pass.

Interestingly, Programming Scala states that "it's guarannteed that a pattern-matching generator will never throw a MatchError". In fact, this is not true. For example, executing

for (First(x) <- List(Iterator.single(1))) yield x;

will try to access a single-element iterator twice, resulting in:

scala.MatchError: empty iterator at $anonfun$2.apply(<console>:6) at $anonfun$2.apply(<console>:6) at scala.List.map(List.scala:812)

One way to view this situation is that Scala's for construct is inherently functional, and should not be mixed with non-functional constructs such as iterators. However, it turns out that there is a completely functional fix to this problem. Simply change the compile-time pattern-match translation to something like:

expr₁ map { case pat => Some(expr₂); case _ => None; } filter { case Some(_) => true; case None => false; } map { case Some(x) => x; }

or, more succinctly:

expr₁ map { pat.unapply(_); } filter { _.isDefined; } map { _.get; }

Of course, there is a cost to this approach: by introducing a third pass, extra an extra list of Option objects is created. Another fix would be to add new methods filterMap and filterForEach which cut down the work to a single pass. While less elegant, it would clearly provide a performance boost, and it would be largely hidden from view. In the mean time, one should use pattern matching in for expressions with care.

Five Filters featured article: Chilcot Inquiry. Available tools: PDF Newspaper, Full Text RSS, Term Extraction.

Software Development Has Stalled

Message from fivefilters.org: If you can, please donate to the full-text RSS service so we can continue developing it.

Summary
I think my subconscious mind has been figuring this out over the past number of years.

For the longest time -- most of my time in the field -- we've always been able to look forward and have at least an inkling of the fantastic jumps ahead, and the productivity that software development tools would enable.

But in recent years it has started to look like we're moving out of the revolutionary period of big innovation, and into a phase of relative stability. There are lots of reasons for this failure of imagination, but a large part of it might be that we've satisfied our initial needs. Software development has moved forward enough, and now we are just exploring what we can do with it. And we can do a lot with our current tools that we haven't yet fathomed.

These things, however, do not drive forward the basic concepts of software development. They will certainly improve the world, but the pressure to rethink the way that we program seems to be off.

Do I have any research or facts to support this idea? None, whatsoever. It's all based on my intuition, and the fact that I've been getting less and less interested in programming languages over the past few years. But I've discovered that, however unreasonable my intuition might seem, it always tells me something useful.

Even more important is my increasing awareness that, no matter how good and powerful our software tools get, we are only getting a fraction of the leverage out of them that we could get.

Programming tools are no longer where the greatest potential lies.

We will get the biggest leverage, not just in programming but in all our endeavors, by discovering better ways to work together. Those are the sessions I will either be creating or attending at the upcoming Java Posse Roundup, March 15-19 2010. It's not going to be just about programming.

Five Filters featured article: Chilcot Inquiry. Available tools: PDF Newspaper, Full Text RSS, Term Extraction.