Statistician Paul Marchant has kindly agreed to let me publish an email he sent to the council explaining to them that the likelihood of receiving no survey forms from 299 streets is less than 13 in a billion. Paul addressed the email to Councillor Ryk Downes as Councillor Downes is the Scrutiny Board councillor who dismissed earlier statistical evidence indicating that the council’s consultation exercise is flawed. Paul sent a copy of the email to the INWAC councillors and the leaders of the three parties.
Dear Ryk,
I was told shortly after my original emailing that the number of streets I used was wrong and re-did the calculation based on the 454 valid streets. The results are basically the same in suggesting that getting responses from only 155 streets is improbable.
You seem to have some misunderstandings:
One does not need to assume that all streets have the same number of residents. My calculation assumes that the streets which did not respond only contained one house, to maximise the probability of non-response which is taken to be 1- 587/9982 i.e. 1- 5.9%. = 0.9412. The probability of none of the 299 streets generating even one response is, using the rule for independent probabilities, 0.9412 times itself 299 times or 13 parts in a billion….a very small number. (I chose the assumption of one house streets to maximise this probability, as any other choice will give an even smaller number.) Therefore I judge that there is a prima facie case suggesting that the exercise is flawed. Indeed I understand that several concerned local citizens say they did not receive the forms …. Why not believe them rather than the organisation doing the survey? As I said in my email, “What evidence can be given that the survey was carried out properly in the light of this improbable result?”
Your assertion that streets with 17 houses would ‘guarantee’ a response is not quite correct. It would not guarantee it …the average response equals one but is of course subject to probability and getting zero would be quite common…about 1/3.
Politicians seem to bandy statistics with an unwarranted air of confidence. You say “I believe ALL members of scrutiny were satisfied that enough data had been obtained.”…If your belief is correct, this is worrying…
It is true when you say that a sample of 1000 will estimate a proportion of a large population; say the proportion in favour of one of the major parties within a ‘margin of error’ of a few percent. It relies on equal selection probabilities. (A cook does not need to taste the whole of a pan of soup in order to know what the soup is like, sip will do. So it is with sampling, but it must be a fair sample!) In the BBQ survey it seems the selection probabilities are in fact unknown and it appears indeed that there is ‘something funny going on’ from the result above.
The ‘What is a Survey’ guide (elementary material) available at the American Statistical Association’s web site http://www.amstat.org/sections/srms/
See http://www.whatisasurvey.info/ . (I recommend it in my work at Leeds Metropolitan University). It outlines problems in surveys such as that due to non-response. The book ‘Survey Research Methods’ by Floyd Fowler is fairly light and good.
I would encourage the council to engage proper professional statistical advice for statistical issues. The danger is that otherwise the ‘information’ obtained is likely to be bunkum.
Yours sincerely,
Paul Marchant