Diversity by hiring on merit. Does it work? / Do we need affirmative action?
Today I would like to look at the sentiment that keeps popping up in social media lately — “We don’t need affirmative action. If you just hire the best people for the job, you will automatically get a fair outcome.” Sometimes it is coupled with “if you assume that that won’t create a fair representation of the population, you are racist.” (or sexist, or ageist, take your pick).
Let us first look, in very simple and general terms, at the math behind this, and the hidden assumptions we are making about the math, and discover whether this sentiment has merit.
We’ll begin by a very simple model example. We have two groups, “A” and “B”. We are starting a new company, and need 10 people for our job, and we don’t really care about “A” or “B” one way or another, but we expect them to be represented fairly in our company.
We get 100 letters, 70 from group “A” and 30 from group “B”. We invite the best 20 candidates for an interview, and it happens that 14 of them are “A” and 6 of them are “B”. Then we pick 10 randomly, by picking them out of an urn. It comes out as seven “A” and three “B”.
(It mostly will, but not always by far.)
So far, so good, our workforce representation of “A” and “B” is 70%/30%, all is well with the world. Right?
Now if I said, that group “A” is women, and group “B” is men, does something become less right with this picture?
The population statistic and common sense say that the spread should be 50/50. How the heck did we just end up with men massively under-represented? After all, we were completely unbiased: We got 100 applications. We selected the top 10% of the talent for an interview. The talent was uniformly distributed between A and B. We made sure to conduct a fair selection…
… but we selected from an unfairly pre-selected (biased) group. The pre-selection was outside of our control —70% of our original applicants happened to be female.
We were fair. But is the outcome of our “fair selection” fair to the men — or to the women?
Let’s switch back to “A” and “B”, for neutrality’s sake, and look what is likely to happen next.
Sooner or later, someone is going to ask the obvious question:
“Why are “B” underrepresented at our company? Let’s hire some more “B””.
Let’s imagine how one of the possible scenarios works out:
“No, we conducted a fair selection, that would be unfairly favoring “B”. There must be a good reason why “B” doesn’t apply for this position!”
What do you think, where does this go?
“There are so much less “B” in our company. I am sure they must be bad at something that our company does, or otherwise there would be more “B” at our company.”
“That is good to know — I have a company in the same field, I should make sure we don’t hire any “B”.”
20 years pass. We do another hiring round. We get 100 letters. Now, 85% of them are from group “A”, and 15% are from group “B”.
It happens that the current generation of “B” grew up hearing how unfit they are to enter our industry. Our insistence on “fair” selection, and some unknown bias in the original round of applicants has created an additional bias that affects the entire society.
(If you want to say that such a rapid shift in demographics is unrealistic — did you know that the number of women in tech dropped from almost 40% down to under 20% between 1985 and 2005? More fun facts below*.)
With the examples in mind, let’s look at some real data. Let’s try to hire an author in the US. Assuming that the percentages of “good enough” authors — people who are qualified to write for us, and are writing in the field and style that is relevant for us — are exactly the same in every demographic, who will we end up with? Let’s look at the data:
In around 86% of all cases, our author will be white. Notice how the percentage of white authors is 10% higher than the percentage of white people in the demographic, while the amount of African Americans only half the general population figure*? Now, remember “A” and “B”?…
Now, let’s see what happens if we are trying to hire for a TV show. Let’s say, we need to hire 5 authors at once. I am awful at combinatorics and probabilities, but I happen to be a programmer, so I wrote a little simulation that tried this hiring process out for me — ten thousand (10,000) times. Guess what: The most probable outcome is … 5 white authors — in close to half the attempts. One African American and 4 whites in 15%, 1 Asian and 4 Whites in 14%. All the other combinations together —all 249 of them — make up the remaining 20% or so. In the end, the chances to get a, say, Native American writer on your show “at random”, are only a couple of thousand times higher than winning the national lottery.
Yes, 5 is a small sample. But how often are you going to be hiring more than 5 authors at one time? How often, do you think, your sample is going to be large enough to actually represent the population statistics?
And this is precisely the point. If you want diversity in your company, it is unlikely to happen by itself because the world is somehow magically fair. And even if you manage to make it fair in one hiring round, just by the way of normal demographics you are likely to still be fighting an uphill battle — and I will say more about this further on.
So, what do we do?
Now, for an alternative take. Let’s say, we happen to know that “A” and “B” should be distributed 50/50. We receive 100 applications. 80 of them are “A” and 20 of them are “B”. We invite 20 best-fit candidates for an interview. 14 of them are still “A” and only 6 of them are “B”. Turns out, only 3 of the “B” are on par with the top 5 “A” candidates. But we want to create a fair selection. What do we do? Should we really hire people who are not qualified?
No. Why would you? But you can
- Invite more candidates
- Wait for more applications
- Opt to hire only 6 people first — 3 “A”, 3 “B”, and do another hiring round later.
- Or, if you don’t mind slight inequality in the beginning, hire 5 “A” and all the “B”s that are eligible, and fill the remaining two positions with “B” once you find the right ones.
- Or if you really, really desperately need 10 people, by all means hire 7+3, but do not forget to fill up your ranks to 7+7 when you grow. That even makes the job ad easier: “Looking for “B” …”
What is this? Ah yes, the dreaded affirmative action. (notice how we didn’t hire any unqualified candidates?) But why go to this length in the first place?
The actual point is to produce the opposite effect of what we have seen above. To see, 20 years from now, the applicant demographics shifting closer to 50/50 because everyone can see that the applicants are equal, in their job, every single day.
This, by the way, is by no means easy on the employers. Not everyone of them (by far) will want or be able to afford to take social corrective actions on this scale, or at all — leaving positions unfilled until the correct demographic with the correct qualification comes along. But is this the wrong thing to do if you have the opportunity? If you think so, can you explain why? You are welcome to leave a comment.
It is true, this is not going to help overnight. No, it is not the end-all-be-all solution, and not the only thing that we have to do. (For instance, if we want more female employees in tech, how about we stop sexually harassing them for a change?). But if we give up on diversity, it will only make the situation worse — it is the self-perpetuating discouragement cycle all the way down. And it can easily happen even with all the caution in place.
Let’s say, we are a large US tech company and manage to hire exactly the representative amounts of people — because our company has a 1000 people and we can even manage to find a Native Alaskan for a change, which makes her automatically 300% over-represented compared to the general population.
What is the perception? “We only have one Alaskan in the whole company. They must be awful at programming if we could only find one.”
- Did you know that secretaries used to be male until 1880s?
- And how about the programmers of the first electronic computers, the ENIAC — if asked to imagine what they look like, what do you see? You can check it right here.
About the data:
- There is no “Mexican” or “Hispanic” category in the data set from datausa.io. I suspect, it is because Hispanic people self-identify as white, as do many other nationalities that US-Americans tend to lump in together into “brown people” or “Asians”. Which makes the real picture even more complicated.
- Fortunately, as we have seen in the “A” vs. “B” argument, the exact absolute demographic data doesn’t matter as long as there is significantly more “A” than “B”, or there is an imbalance between the demographics of the general population and the demographics of the group we are selecting from.
- The Alaskan native authors in the data are 180 people. You could literally shake hands with every single one of them.