Some rules in research were not followed
Publishing the same report with the latest report being a later version of the former
Importance of methodology was not emphasized
Limited generalization of results due to a small sample
Differences in the formats of the two papers is contradicting
My critics take and my reaction to them
After a careful reading of the article by Garan (2001), it is clear to me that the required and set down rules were never followed in the research. The research in itself has so many flaws as indicated by the writer and I strongly agree with the criticism on it.
Publication of two reports, one of which is a later version of the other, did not follow rules of citation. This is a drawback that is reducing the credibility which will not meet its goal at the end of it all. If a research is rewritten to correct errors form the first publication, it should be indicated as a second volume and also indicate what is changed. No one will accept results that have been doctored; and so is any policy adopted as a result of a flawed research. A research report should also entail limitations of the research which evidently lacked in this report as stated by the writer.
Listing of studies that were used in the analysis has flows in it and which will make it difficult for anyone reading the report. The right methodology will give the correct data that at the end will give the correct results. To me this was not followed and as a result, wrong results were acquired. The small number of studies used added to the many flaws on the results of the research, 38 in number according to the writer and the NPR terming it as unfortunate, throwing the question of how credible are the results?. Administration of uniform research questions matters a lot in research for uniformity which according to me was not followed. Some tests were conducted to some of the respondents and others were accepted, which is wrong and will skew the results.
NRP also did not touch on normally developing readers above grade 1. Any normal school has all categories of students from high performing to those having problems in English. This is also wrong as it does not provide enough data for greater variability hence better results. The data taken from the control group about a decade ago will definitely differ from the sample being studied when the number of the sample changes. The writer points out that in some years, a sample of 22 only was taken. This throws the question of consistency and reliability that NRP tries to put across that it used it in coming up with the results of the study, yet they did not follow at all. This makes the results wanting.
Contradiction in the reports on different formats is also another setback for this report. To quote form the findings that “the meta-analysis revealed that systematic phonics instruction produces significant benefits for students in kindergarten through sixth grade and for children having difficulty learning to read” (Garan, 2001, p. 506) and then on the other hand saying “there were insufficient data to draw any conclusions about the effects of phonics instruction with normally developing readers above first grade” (Garan, 2001, p. 506). This is unacceptable and it is a major flaw in this paper.
I know my opponents may say that a good sample is enough to represent the population since it was well sampled. They may also point out that administering the questions to a clique of the sample was meant to come up with a true reflection of the problem. I would only counter them by telling them that a good report should follow accepted research policies and also the fact that credible report on research gives an out right report rather than conflict itself just like the two publication with different findings. Also anyone doing research should ensure enough data is used to avoid what NRP had in the report. If enough data was not there then the report is null and void.