Facts & Myths
- Age Groups
- Average Speed
- Braking
- Cause of Death
- Cell Phones
- Hidden Camera Trial
- Independent Reviews
- Injury Trends
- Kinetic Energy
- Kloeden Critique
- Overtaking
- Overtaking Trucks
- Rigid Enforcement
- Sex Differences
- Silly Statistics
- Speed Limits
- Ticket Quotas
- Worldwide Trends

« Previous     Next »
« Go back?
Hidden Camera Trial Comments

LTSA Defence

Fast & Safe Response

 

Alan Wilkinson's analysis of data related to the Midland hidden camera trial comprises plots of the log of monthly crashes against time, and plots of the ratio of crashes inside and outside the Midland region against time. The plots are for the period December 1993 up to as close to the present as the data availability will allow.

He asserts that the majority of ratio points from the trial period were above his line. He further asserts that this means that there was no adequate evidence that the hidden speed cameras had a selective impact on crashes in the Midland area. Having carefully reviewed Dr Wilkinson's analysis, the LTSA agrees that his treatment of the data detected no such evidence. This is not due to the non-existence of such evidence, but rather to the inadequacy of Dr Wilkinson's methods.

Dr Wilkinson chose arbitrarily to fit a simple linear trend to all three datasets, and chose both the beginning and end arbitrarily. After drawing this arbitrary line, he used arbitrary grounds to decide whether or not something was, in his words, "significant". These arbitrary methods are inappropriate for the task and it is just such inadequacies that modern statistical methods are designed to correct.

Those comments are spurious and unprofessional – unkindly I might say simply clueless:

  • The trend line is linear on a log plot corresponding to an exponential curve on a normal plot. That is exactly the same model as used in the LTSA paper which used a linear regression against log functions of the data.
  • I did not choose the beginning and the end arbitrarily. I used the same beginning point as the LTSA. I used all the data available and the latest provided by LTSA determined the end point.
  • There is nothing arbitrary about observing that the deviations from the trend line are on the wrong side of the line to indicate a benefit from the trial and rather suggest its impact was negative. It is clear the trial failed to show a benefit on any definition of significance. I did not attempt to determine how significant its negative impact was.
  • The last sentence is just PR fluff.
Unlike Dr Wilkinson's analysis, the statistical model used by the LTSA accounted for both trend and seasonality. The analysis therefore accounts systematically for differences that are not detectable by a simple inspection of graphs of the data, such as that carried out by Dr Wilkinson. The methods employed by the LTSA are normal, internationally accepted statistical practice, and qualified and experienced statisticians are needed to be able to carry them out properly.
  • The 12-month rolling average does account for both trend and seasonality. Trend is shown by changes in direction. Seasonality is removed by using the 12-month period for the average.
  • The LTSA has consistently refused to state to me the qualifications and experience of its own "statisticians" - the principal one of whom I suspect wrote this nonsense.

    But I am happy to provide mine:

    • BSc Hons degree in Chemistry and Mathematics including Mathematics Hons Pt 1 and Statistical and Numerical Analysis II
    • Ph.D. in Physical Organic chemistry including extensive linear and non-linear least squares analyses of reaction decays and nuclear magnetic resonance spectroscopy parameters
    • Two years similar post-doctoral research at Victoria University, BC
    • Thirteen years at Computer Centre, University of Canterbury. Responsible for advising research students and staff on use of statistical analysis software. Author of linear and non-linear least squares analysis software.

The LTSA's reports on the Midland trial have been subjected to extensive international peer review as follows:

  • Reviewed by Australian road safety expert Max Cameron (currently Professor at Monash University Accident Research Centre)
  • Underwent the refereeing process of the Accident Analysis and Prevention journal, and subsequently accepted for publication in AA&P (widely recognised by academics and researchers world-wide as the most prestigious journal in the field)
  • Judged Best Paper at the 1999 Road Safety Research Education and Policing Conference in Canberra (judged by leading international road safety experts)
  • Reviewed by the Office of the Auditor-General of New Zealand.
  • Monash University Accident Research Centre is the high church of rigid enforcement policies.
  • The LTSA papers as published in AA&P did not reveal the source data which shows the weaknesses in the analyses. It appears the referees did a very poor job of investigating and assessing them.
  • That it should have been judged Best Paper by this industry gives a clear indication of the lack of professionalism of said industry.
  • There is no indication in Audit Office report that it did anything but accept the LTSA study at face value.  Who would normally go to an accountant to obtain a scientific endorsement?

...

The LTSA's review has found Dr Wilkinson's analysis to be seriously flawed, for the reasons outlined above and on the following page. 

...

 Dr Wilkinson has on several occasions - including letters published in the New Zealand Herald - stated that publication of statistical research in a reputable scientific journal is the standard by which such work should be measured. The LTSA's research has met this standard. Dr Wilkinson's criticisms of it have not - the only place they have been published are in the newspapers.

If Dr Wilkinson remains convinced that he is right and we are wrong, we challenge him to submit his analysis to the editorial board of Accident Analysis and Prevention or any other reputable scientific journal for expert review and comment.

I will probably do this at some stage, although currently lack of responsiveness from AA&P leaves me in no hurry to do so. 

In the meantime I have challenged the Minister of Transport to have any qualified statistician from a New Zealand University review my critique.  No response was forthcoming. 

Specific responses to Dr Wilkinson's claims:

1.  The control area was chosen unscientifically in order to make the hidden cameras appear effective.

The objective of the analysis was to adhere to scientific principles in order to assess any effects due to the camera trial - not to "make hidden cameras appear effective". This involved a selection of a control area according to carefully set criteria, as specified by Professor Max Cameron of Monash University.

  • The papers appear to be deliberately obscure about how the trial area was chosen. They make out it was a carefully matched choice. They do not say it was simply the rest of New Zealand.
  • My main concern about the control area however was not its selection, but that there were unexplained, marked pre-trial differences in trends between the trial area and the control area. Effectively it was these trends that were being extrapolated in order to produce the calculated benefit from the trial.
  • Let's see what stuff Professor Cameron is made of.  Commission him to review my critique.

2.  The claim that crashes and casualties fell is incorrect as there was a change in the control area crash and casualty rate, not in the trial area.

The analysis technique took into account relative differences in trend and seasonality. As the control area had been chosen to be well-matched to the trial area by various important criteria, the function of the control area crash and casualty data was to indicate behaviour of crash and casualty rates that would also have occurred in the trial area if the hidden camera trial had not been initiated. Cusum charts of the ratio of casualties in trial and control areas show a change in this ratio coincident with the start of publicity regarding the trial. It is very unlikely that this could be purely by chance, unrelated to the trial. Further, there were other criteria used to judge the impact of the hidden cameras (in comparison to the overtly used cameras). These included measurements of car speeds and higher percentiles (a measure of the speeds travelled by the fastest cars in the trial compared to the control area). Also the number of casualties occurring per crash fell in the trial area, a reliable sign that impact speeds were reduced. All this evidence points to reduced trauma associated with the hidden speed camera trial resulting from reduced speeds, particularly reductions in the speeds of the fastest drivers.

 

 

The evidence points to LTSA concluding what they wanted to conclude in order to confirm their own assumptions. The simplest interpretation of the results is that the trend in the Trial area was following belatedly a trend that was already well in progress in the rest of New Zealand.

Examination of time series analysis results and other indicators such as those used by myself suggest that the Trial actually had if anything a detrimental rather than a beneficial effect.  I have seen two other reviewers using different methods come to this same conclusion.