Friday 8 November 2013

Mathematical Inequality


//

Disclaimer: This one is potentially a little more controversial than some of the other topics discussed. You've been warned.

//

 I believe in Meritocracy - meaning appointing the person most able to fulfill the duties and obligations of a role to that role or position. Meritocracy is about equality of opportunity (anyone can apply for the role) and elitism of choice (only the best will succeed). I'm, unfortunately, in a fairly small minority. Why? Because most people don't actually believe in meritocracy - they believe in equality of outcomes. It doesn't matter if the best people are selected, just so long as those people meet the required criteria in terms of factors other than ability (such as race, religion and gender).

**

There is, for want of a better word, a zeitgeist at the moment towards equality in various fields (pay, FTSE directors, university admissions and so on), based purely on population characteristics. 50% of the population is female, so 50% of total pay must be earned by women, 2% of the UK population is black so we must have 13 black MPs and so on. The main point of these factors is they ignore actual ability. For proportional representation style of equality to work it is necessary to prove that actual suitability for roles, actual productivity, actual work-ethic is evenly distributed.

Note - I pointedly use the term actual here. This is vital. I make no attempt to even touch on the minefield like subject of whether the neurological and biochemical make up of the male brain is more or less likely to produce talented astrophysicists than the female or any comparable topic. Instead the query runs more like this: Pick 100 random people off the street and give them a maths test which starts relatively simple (2+2) and finishes with post-graduate level algebra. You then grade and rank and papers. Are you really going to be that surprised if the results come back that 6 of the top 10 scores are female, 18 of the top 25 are male, and 30 of the top 50 are female? Hopefully not - even if you state as a precondition that men and women are not inherently better or worse at maths that doesn't mean any given sample will not have a bias one way or the other.

Rather than try and illustrate my point with anecdotes or theory as usual I'm instead going to use the dark art of simple mathematics. This is to try and head off the inevitable counter arguments predicated on my own 'innate bias' or some other straw-man counter which avoids addressing the genuine issue.

To run this out I created an excel spreadsheet with 100 Sample Groups, each Sample group containing 50 people from group "A" and 50 from group "B". (For a total of 10,000 observations). Each observation was then allocated a random score between 1-100 using the Excel RandBetween function. Therefore, assuming Excel itself is fair, the score an individual received was not based on their Group - the precondition of potential equality holds.

For each Sample Group I then summed up the scores for Group A and Group B, and allocated the group winner as either A or B based on the total. I then reproduced this across the100 Sample Groups to produce 1 Test Result - displayed as a percentage (i.e. 45% A / 55% B).

To flesh out the test size I re-did this for 100 Tests, giving a total sample size of 1,000,000 observations (10,000 observations per Test and 100 Tests). Now the population/representation lobby would predict/require/demand that the outcome here is equality - 50/50 split between Group A and Group B.

The actual results were as follows;

68 Tests had a higher percentage of sample groups won by Group A (compared with 32 Tests for B)

The average difference between Test Wins was 7.5 (i.e. if 50/50 is 0 and 49/51 is 2, the average was 7.5).

The most skewed Test was 63/37 (i.e. out of the 100 Sample Groups in the Test, 63 of them were won by the same Group).

Now repeating this exercise would no doubt deliver different results, but it is unlikely to deliver an exact 50/50 split between the groups on a specific interation (Out of my 100 tests only 13 came back as 50/50 splits between Group A and Group B, meaning by extension 87% of Tests had a skew (For those interested the numerically most common skew was 8 points (54/46), occurring in 17 Tests). While 50/50 may be the statistical 'average outcome' any given iteration of the Test is more likely to deliver a result within +/- some margin.

The purpose of this is simply to highlight that even with a perfectly level playing field going in (everyone gets a random value between 1 and 100) its not unlikely, or unusual, or unfair, for the total aggregate results to show a skew towards certain groups, simply because of the randomness of the sample.

(A tangential observation is about poker players. Imagine a poker player plays 1,000,000 hands in their life, and on average should get dealt the winning hand  15% of the time. That means in your life you have an expected value of 150,000 winning hands. Yet by the same virtue of randomness as shown above it isn't really that unlikely that someone out there might get 175,000, or 200,000, and for 1 in a million people they might even get 300,000 or higher. Maybe successful poker players really do just get better hands more often. This isn't because of a rigged deck, or an unfair game or anything else, its just a case of actual results not exactly matching statistical expectation).

With the maths out of the way my final point is about what I think the above illustrates. Firstly, (as with many of the points I raise) we (pluralist national "we" now) need to make a decision about what we want; equality or meritocracy. They are probably mutually exclusive. If you want the House of Commons to be exactly 325 men and 325 women then you have to accept that your not picking the top 650 candidates.If you want the top 650 candidates then you have to accept that due to the quirks of actual ability, your going to get a 55/45 split or a 49/51 or even a 63/37.

If the conclusion is equality is more important than meritocracy then while I disagree, at least the objective is consistent with quotas, positive discrimination (a horrendous phrase that should be demised as quickly as possible. Discrimination is discrimination, "positive" discrimination is akin to stealing something and giving it to Oxfam, then calling it charity not theft), and all the other malarcky of the current zeitgeist.

If we pick meritocracy we still have our work cut out for us, but in a very different way. There really is inherent bias in many systems, whether its preconceptions about women, or race or sexual preference these prejudices do exist. The quest for the meritocrats should be to devise ways to prevent these biases from impacting on decision making processes, if a process can be cleansed of bias then the outcome whether it be 100% white middle class male, or 31% black under 21s, or anything else is equally acceptable.

On that final point a suggestion that came out of comparatively recent discussion was to remove face to face interviews in a recruitment process. Some companies already ask for 'age neutral' CVs which contain no dates or other age-identifying information, why not go one further and have name/address/age neutral CVs, and shortlist candidates who supply their answers to the interview questions in written form? Certainly this ships with its own problems (not in scope of this post), but its not an entirely ridiculous proposal.

Happy Trails,

/Z

EDIT: In response to a valid comment raised regarding this I feel I should clarify that the purpose of this is not to try and defend the current situation with regards to parliament/FTSE directors or anything else, its a rebuttal of the view that the outcome (i.e. exact 50/50 split by gender, or ethnic group or anything else) should be set in advance, and if necessary enforced through reserved seats, gender-fixed shortlists or some other approach.

2 comments:

  1. I don't quite follow the conclusion here. You are right to say that even with no systematic biases, there outcome of genuinely random tests will not always be the mean. However, for very large sample sizes, it will tend towards it. You did 100 test; if you did 1000 your variance would be smaller, for 1 million tests the variance would be miniscule.

    Applying this to the context. If you had a small pool of companies and electoral seats (let's say 100 as per your example), and there was a 63/37 M-F split (that you saw as the worst case skew), well, that wouldn't be outrageous. It could genuinely and fairly be the result of random chance.

    But then, there are about 750 MPs in Lords at present. There are (shockingly!) 250 CEOs in the FTSE250. Some other . Your sample is now 1000 - ten times larger than your test. In this case, getting the same ratio - 630/370 - would start becoming a lot more suspicious. I would strongly expect the variance to start reducing. If you include all the other places were some bias is allegedly existing, the sample size becomes quite large - tens of thousands. At this point, I would expect a trend towards the population's characteristics (even, obviously, if not exactly matching).

    My point is that your statistical analysis should reflect the true sample size to offer better conclusions. Your underlying point is true, but doesn't sufficiently explain the huge discrepancies that exist with minorities. In your 100-test run, the worse skew was 63-37; this is still much more even than the number of female MPs, Lords, CEOs in any index etc etc.

    That said; I agree that discrimination to fix the problem is not the correct process and worsens the problem by effectively hiding the problem (with the exception of when you're trying to jump-start a new cultural shift from zero). I accordingly also strongly agree that meritocracy is more important than equality as at the end of the day a CEO or MP is supposed to run a company/country, not serve as an example of good selection processes. These, being a sample of the population, will never match the population's characteristics. But they will tend towards it in a true meritocracy (assuming, as you do, no underlying reason when a minority should do worse). And that's not what happening - minorities are far below their expected values, and so bias does exist in selection of the samples. That's what people are getting angsty about, and rightly.

    ReplyDelete
  2. Hi Alexander, thanks for your comments.

    As you say the tendency towards the mean will increase as you increase the sample size, though I would still conjecture that specifically for the 650 MPs in the House of Commons the sample size is still far to small to attempt to capture all the variance in so complex a variable as "being a good MP". (I'd also clarify that my own numbers come from an underlying pool of 1,000,000 observations, and 1 "Test" result has 10,000 underlying observations).

    The purpose of the maths is not to conclude anything about the current representation within Parliament (or any other group), but merely to illustrate that the heavy-handed approach of a direct 50/50 quota is no guarantee of capturing the best people for the job. As you seem to agree in your final point, the superior approach is to correct the process in the first place, rather than have a broken process which is then overruled by an equally broken quota system to deliver a superficially acceptable outcome.

    With regards to what "people" are getting agitated about, I think this very much depends on which "people" your talking about - and the consensus is far from universal. The calls for a 50/50 quota in Parliament come not just from politicians looking to appeal to feminist/liberal/progressive voters, but also from writers, academics and other members of the intelligentsia whose stated positions clearly place "equality" above merit.

    As I stated in the article I don't have a problem with the view that equality is a good in-and-of itself, but I do have a problem with the egalitarians wanting to claim both that equality is the aim AND that equality will, by definition, deliver the best result. If there was less focus on forcing the outcome to a preset target, and more on understanding the bias within the system, then I feel we would be on the right path.

    ReplyDelete