Tilburg University
Analysis of simulation with common random numbers Kleijnen, J.P.C.
Publication date: 1979 Link to publication
Citation for published version (APA): Kleijnen, J. P. C. (1979). Analysis of simulation with common random numbers: A note on Heikes et al. (1976) (Version 2). (Ter discussie FEW; Vol. 79.095). Unknown Publisher.
General rights Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. • Users may download and print one copy of any publication from the public portal for the purpose of private study or research • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal Take down policy If you believe that this document breaches copyright, please contact us providing details, and we will remove access to the work immediately and investigate your claim.
Download date: 27. nov. 2015
CBM R
7627 1979 95
..
I IpINIII~ II IIIII hll'~ I III Iqll h~ II~ ~q faculteit der .,.,~~~~~~~~~cne wetenschappen
REEKS "TER DISCUSSIE"
TI?~SC?-?R?FTrNBUREAU BISLi~;THc:: K K ~,1': I :.,1:~IC'~ HO~r.S~ri: ~OL TILf3UkG
KATHOLIEKE HOGESCHOOL TILBURG REEKS "TER DISCUSSIE"
No.
79.095
november 1979
ANALYSIS OF SIMULATION WITH COMMON RANDOM NUMBERS: Version
A NOTE ON HEIKES et al.
(1976)
2. Jack P.C.
Kleijnen
FACULTEIT DER ECONOMISCHE WETENSCHAPPEN
CONTENTS
ABSTRACT
1
1 .
Introduct'ion
2.
Independent random numbers:
3.
Common
4.
Conclusion
Appendix: References
1
random numbers:
OLS
corrected analysis
simpler procedure
3 5 ~
and WLS
estimators
~ 9
ANALYSIS OF
SIMULATION WITH COMMON
A NOTE ON HEIKES
et
al.
RANDOM NUMBERS:
(1976)
Jack P.C.
Kleijnen
Department of Business and Economics Katholieke Hogeschool Tilburg Tilburg,
Netherlands.
ABSTRACT
If different random number streams are used then traditional Analysis of Variance can be applied, but the standard errors of the estimated effects should account for heterogeneity of variance.
If common random numbers are used then the quite complicated procedure pronosed by Heikes et al. ín 1976, may be replaced by the simnle Student t-statistic combined with the so-called Bonferroni - inequality. The statistical techniques are illustrated with a simnle case study.
1.
INTRODUCTION
In 1976 Heikes et al.[2]
discussed the analysis
of simulation experiments that use common random numbers. Because the independence of the observations is destroyed they proposed an analysis technique taken from Graybill [1]. To illustrate the Graybill procedure they first simulated an inventory system with independent random numbers, (i)
next with common random numbers. In the present note I would like
their analysis of case incorrect,
(i~
to show that
1,
indenendent random numbers,
2,
common
is
and
their analysis of case
random numbers,
does not
necessitate the Graybill procedure but can be based on the familiar Student t-statistic. It is my experience that practitioners hate to apply any but the simplest statistical techniques,
and therefore
the t-statistic may be
attractive. The paper's organization is shown in Figure 1.
yes
yes
different random number streams ? no
constan t variances Q2ii - Q2
?
ordinary ANOVA; see Eq. (1)
OLS criterion ?
correct standard errors ~((3) ; see Eq. (2)
no
WLS; see reference 4 N
LEGEND ANOVA OLS WLS m
no
: : : :
ANalysis Of VAriance Ordinary Least Squanes Weighted Least Squares number of individual tests
FIGURE 1: Summary
yes
compl i cated analysis ?~
Graybi ll's technique ; reference 2
t-test with qlm ; see Eq. (3)
2.
INDEPENDENT RANDOM NUMBERS:
The results in [2]
CORRECTED ANALYSIS.
of simulating
a particular in-
ventory system with independent random number
(r)
are
repeated in Table 1. Table 1 Simulated total inventory costs for experiment which did not use common random numbers.
Order
Run
2
Run 3
Cost
Cost
Cost
Cost
Cost
Cost
Cost
Cost
485
11,846
9,934
7,536
13,638
7,504
8,221
8,004
6,435
500
9,080
7,555
7,313
8,136
7,930
19,268
7,320
7,390
515
8,253
10,350
8,994
8,810
8,525
7,967
8,228
8,827
530
8,389
8,744
8,861
9,176
8,266
9,075
9,959
8,765
Level(R)
1
Run
As Heikes et al.
Run
4
themselves state
Run
"...
treatments..."
duced var
with
theless
Table
of
(yij) -~ii they analyzed
variance
(ANOVA),
i.e.,
i1
least
estimates
squares í would
Sts
-
(OLS)
p.
ri
from
and -j
-
they
Run
8
intro-
one-way
variance
~leveranalysis
Qii
estimated using the
Thè
standard errors of
-
Q2"
ordithe
(1)
Eq.
to derive their Table 2
treatment effects.
7
1,...,8.
(X'X)-iQ2
al.
Run
:
if a common variance aii - Q2 held. by Heikes et
Hence
"ordinary"
are
criterion.
follow
82].
1,...,4 using
6
i[order level R]
assuming a common
In ANOVA the treatment effects nary
[2,
Run
the variance of the'.ob-
servations associated with factor level may differ with
5
(1)
is
implicitly used
showing no
significant
Even if the variances are heterogenous
the OLS criterion can still be used to estimate (aii ~ Q2)~ effects. However, the standard errors of the estimated effects follow not from Eq.
(1)
but from Eq.
matrix with elements vàr(y)
- Qii
'
(2), where
S~y
is a diagonal
- 4
S2S -
Applying Eq.
-
{ (X~X)-1X~ }~y { (X'X)-1X~ } ~
(2)
I
estimated effects
(2)
found that in
the example of Table 1 the T remain insignificant; see Table 2. How-
ever,
it seems dangerous to assume that Eqs. ways yield the same conclusions.
(1)
and
(2)
al-
Table 2 Ordínary Least Squares estimates with corrected standard errors. Effect~)
Estimate
Standard error Correct (Eq.
~)
1)
Wrong (Eq,
u
9.01
1.87
T1
0.43
0.13
2.57
T2
0.75
0.24
5.34
T3
0.75
-0.27
1.88
0.75
2)
3 T4 - -E Ti 1
If the variances differ, OLS may be replaced by Weighted Least Squares (WLS), the weights wi being inversely proportional to the estimated variances dii; see [4] for details. It is interesting to note that in the example of Table 1- and in any one-way ANOVA - WLS yields identically the same estimated effects and standard errors as OLS does: This phenomenon is explained in the appendix.
3.
COMMON RANDOM NUMBERS:
SIMPLER PROCEDURE.
Using common random numbers
results in Table 3,
re-
produced from [2].
Table 3 Simulated total inventory costs
for experiment using common
random numbers. Order
Run
level(R)
1
Run
2
Run
3
Run
4
Run
5
Run
6
Run
7
Run
8
Cost
Cost
Cost
Cost
Cost
Cost
Cost
Cost
Mean
485
6,133
8,100
8,001
7,483
9,357
12,549
13,756
8,502
9,235
500
6,183
7,095
7,639
7,481
7,653
10,229
10,440
8,645
8,171
515
6,929
7,015
8,001
7,871
7,490
9,690
8,877
9,018 8,111
530
7,680
7,600
8,751
8,621
8,233
9,687
8,566
9,392 8,566
If only two order levels were to be compared, of [3]
could be used.
then the analysis
For more than two levels Heikes et al.
used the little known and rather complicated technique of Graybill [1] to test the hypothesis of no treatment effect. They found... a-
0.005
"order levels are significantly different at the level...
observations,
Because of the correlative structure of the
it is difficult to apply multiple-comparison
methods to investigate more complex relationships between the order levels"
[2,
pp.
84-85].
Next
I will show that a simpler
technique is possible which moreover permits multiple comparisons. Consider the simple Student t-statistic for testing the difference between two treatments,
tb-1
á ii' - {vár(dii,)}~
i and i':
(3)
-
6 -
where
- yi
dii~ d
-
ii'j
- yi,
b -
E dii,j~b j-1
(5)
yij
vár(dii~)
-
vár(dii,)
- vár(dii,)~b
Note that my Eq.
E J-1
(3)
(dii~j
- dii~)~~(b-1)
b- 2.
(6) (7)
is analogous to Eq.
to which Graybill's technique reduces levels,
(4)
(8)
in [2],
the formula
for the case of two factor
The t-test applied to all
six pairwise comparisons
yields Table 4.
Table 4 t-statistics
(i,i')
in pairwise comparisons
(i,i')
(1,2)
(1,3)
(1,4)
(2,3)
~vàr(d)
1.28
1.99
2.33
0.73
1.10
0.41
t
2.36
1.60
0.81
0.23
-1.02
-3.15
(2,4
(3,4)
Since there are six instead of a single t-test, the Bonferroni inequality is used, i.e., the standard a value is replaced by a~m where m denotes the number of individual tests, here m- 6; see [4].
An experimentwise error rate
jection of the null-hypothesis:
(aE)
of 10~
leads to re-
aE - 0.10
implies two-sided individual tests with error rate 0.10~(2X6) - 0.0083; the extreme t value in Table 4 is -3.15 and has a smaller than 8.3g chance of occurrence under the (composite) null-hypothesis of no effects. My analysis has procedure:
less power
(higher S
error) than the Graybill the latter procedure gives significance at aE - 0.005
-~instead of aE - 0.010. Nevertheless the simple t-test does detect treatment differences at reasonable aE. complex relationships" gated:
(multiple comparisons)
Moreover "more can be
investi-
the composite null-hypothesis is rejected because levels
3 and 4 are significantly different;
the differences among the
other levels cannot be detected without additional simulation runs.
4.
CONCLUSION.
As summarized by Figure
1,
different random number
streams permit the familiar OLS estimates of the treatment effects T, effects familiar from the traditional ANOVA model. However, in general the standard errors Q(r) should be computed from Eq.
(2)
instead of Eq.
(1).
Common random numbers permit the simple t-test provided the test is combined with the Bonferroni inequality, i.e.,
the standard a value is replaced by a~m.
APPENDIX:
OLS and WLS estimators.
The traditional linear model y-Xs t e means that in the example of Table 1
the following relations
hold: S~
-(u ~
Y'
-
X -
(yl,
T1 ~ y2,
T2,
T3 )
y3, y4)
(A.2) (A.3)
(A.4)
3 where the last row holds because in the ANOVA model T4- -E Ti. Since X-1 exists ( X'X)-1 becomes X-1 (X')-1, etc. Hence 1 S2 S in Eq. ( 2) reduces to Sts
-
(X'
~yl
X) -1
,the well-known formula for WLS;
(A.5) see [4].
The WLS estimators are identical to the OLS estimators 4 for the following reason. OLS minimizes E(yi -y)2. Since t:zere are 4 parameters and 4 ly zero:
WLS minim~.zes
realized when yi - yi
observations 1 this minimum is exact4 E wi(yiyi)2 and this minimum is again 1 (i - 1 , . . . ,4) .
ACKNOWLEDGMENT
C.B.
Programming for this paper was done by P.J. Rens and Mijderwijk, Computer Center, Katholieke Hogeschool Tilburg.
9 REFERENCES
1
Graybill, F.A.,
Introduction to Linear Statistical Models.
McGraw-Hill, New York. Vol. 2
Heikes,
R.G.,
D.C.
1
1961.
Montgomery and R.L.
Rardin,
Using common random numbers in simulation experiments an approach to statistical analysis. Simulation, 3
Kleijnen,
27,
no.
J.P.C.,
3.
Sept.
1976,
pp.
81-85.
Comparing means and variances of two
simulations. Simulation, 4
Kleijnen,
26,
no.
J.P.C.,
3, March
1976,
pp.
87-88.
Regression Analysis for Simulation
Practioners. FEW 85,
Dept.
Hogeschool,
of Business and Economics,
Tilburg,
(Neth.),
1979.
Katholieke
In de Reeks ter Discussie zijn verschenen:
1.H.H.
Tiágelasr
2.J.P.C.Kleijnen 3.J.J.
Kriens
4.L.R.J.
Westermann
S.W. van Hulst J.Th. fan Lieshout
6.M.H.C.Paardekooper 7.J.P.C. B.J.
Kleijnen
Kri~ns
y.L.R..T. 10.B.C.J.
Westermann van Velthoven
11.J.P.C.
Kleijnen
12.F.J.
Vandamme
1 3. A. van Schaik 14.J.vanLieshout J.Ritzen J.Roemen 15.J.P.C.Kleijnen
16.J.P.C.
Kleijnen
17.J.P.C.
Kleijnen
18.F.J.
Vandamme
19.J.P.C. 20.H.H.
Kleijnen
Tigelaar
21.J.P.C.
Kleijnen
22.W.Derks '23.B. Diederen Th. Reijs W. Derks 24.,T.P.C. Kleijnen
2~.B., van Velthoven
Spectraalanalyse er. stochastische lineaire differentievergelijkingen. De rol van simulatie in de algemene econometrie. A stratification procedure for typical auditing problems. On bounds for Eigenvalues
j uni ' 75
Investment~financial planning with endogenous lifetimes: a heuristic approach to mixed integer programming. Distribution of errors among input and output variables. Design and analysis of simulation Practical statistical techniques. Accountantscontrole met behulp van steekproeven. A note on the regula falsi Analoge simulatie van ekonomische modellen. Het ekonomisch nut van nauwkeurige informatie: simulatie van ondernemingsbeslissingen en informatie. Theory change, incompatibility and non-deductibility. De arbeidswaardeleer onderbouwd? Input-ouputanalyse en gelaagde planning.
juli '75
Robustness of multiple ranking procedures: a Monte Carlo experiment illustrating design and analysis techniques. Computers and operations research: a survey. Statistical problems in the simulation of computer systems. Towards a more natural deontic logic. Design ar.d analysis of simulation: practical, statistical techniques. Identifiability in models with lagged variables. Quantile estimation in regenerative simulation: a case study. Inleiding tot econometrische modellen van landen van de E.E.G. Econometrisch model van België.
februari
Principles of Economics for computers. Hybriede simulatie van ekonomische modellen.
j uni
' 75
juni '75 juni '75
augustus '75 augustus '75 september '75 september '75 november '75 november '75 december '75 januari '76 februari '76 '76
februari '76 april '76 mei J~1
'76 '76
juli '76 augustus '76 augustus '76 september '76 augustus
'7G
augus t us ' 7f ~.
26. F.
~'(.N.
Cole
1~'urerr~sting t,y smootlrinE,~, ttre
lieuts
2t~.W. Derks 2y.J. Frijns 30.P. Meulendijks 31.W. Derks
32.W.
Derks
33.
A.
van Schaik
j4.
W.
Derks
35. W,
Derks
36. J. Roemen
37.
W.
Derks
3f3. A.
vt~n Schaik
3'~. W.
Derks
40. .T. Kleijnen 41. M, v.d.
la2. 43.
Tillaart.
W. Groenendaal Th. Punnewijk R. Heuts
r~xponecrtiril lic~x rsrr~i Jenkins
j~roce~iurr r:ui~1 ;;pect.rul isrr~slysis. A sirnulatiurr :;Luci,y. ,~~,me re1'urrtrulr~tions arrcl extensions iri the univaria.te Box-Jenkins time series analysis. ' Vier econometrische modellen. Estimation methods for multivariate dynamic models. Keynesiasnse theorieën van handelsliberalisatie. Structuuranalyse van econometrische modellen met behulp van Grafentheorie. Deel I: inleiding in de Grafentheorie. Structuuranalyse van econometrische modellen met behulp van Grafentheo-
rie. Deel II: Formule van Mason.
september
juli
'76
"(6
oktober '76 oktober
'76
september
oktober
'76
'76
Een direct verband tussen economische veroudering en bezettingsgrasdverliezen. september '76 Structuuranalyse van Econometrische Modellen met behulp van Grafentheorie. Deel III.De graaf van dynamische modellen met één vertraging. oktober '76 Structuuranalyse van Econometrische Modellen met behulp van Grafentheorie. Dee1 IV. Formulé van Mason en dynamische modellen met éé,n vertraging. oktober '76 De ontwikkeling van de omvangsverdeling in de levensmiddelenindustrie in de D.D.R. oktober '"(f~ Structuuranalyse van Econometrische modellen met behiilp van grafentheorie. neel V. De graaf van dynamische modellen met meerdere vertragingc:n. oktoher ''(f~ H;en direkt verband tussen economische verouderirig en bezettingsgraadverliezen. Deel II: gevoeligheidsanalyse. december '7E, Utructuuranalyse van Econometrische modellen met behulp van Grafentheorie. Deel VI. Model I van Klein, statisch, dec~mber '7F, Information Economics: Inleiding en kritiek november '"(F; De spectrale representatie van mutivariate zwak-stationaire stochastische processen met ~iiscrete tijdparameter, nr,v~.rnt,~.r~ '"f~. Een econometrisch model van Engeland december ''(~. Capital market models for portfolio selection septeruber ''(~.
44. J. ïJ . 45. J. P.
Kl~ijnen en Ren3 Kleijnen er~ Rena
46.
Willematein
A.
47. W.
Derks
48. L. Westermann 49. W.
Derks
5G. W.v. Groenendaal en Th. Dunnewijk 51. J. iClei.jner. en P. Rens 52. J.~.A. Moors
53. R.M.J. ïieuts
54.
B.B. v.d. Genugten
55. P.A. Verheyen
56 . 'vl . v , den Bogaard en J .Kl-ei j nen 57. W. Derks
58. R. Heuts 59. A.P. Willemstein
60. Th. Dunnewijk W. van Groenendaal 6í. A. Plaisier A. Hempenius
A critical e.r~alysia of IBM's inventory package impact. december '76 Computerized ir.ver.tory management : A critical analysis of IBM's impact system. december '76 Evaluatie en fotitenanalyse van econometrische modellen. Deel I. Een identificatie methode voor een lineair discreet systeem met storingen op input, output en structuur. januari '77 Structuuranalyse van econometrische modellen met behulp van grafentheorie. Deel VII. Model I van Klein, dynamisch.februari '77 On sysnems of linear inequalities over 7R . Structuuranalyse van econometrische modellen met behulp van Grafentheorie. Deel VIII. Klein-Goldberger model. Een econometrisch m~del var~ het Verenigd Koninkrijk A critical analysis of IBM's inventory package "IMPACT" Estimation in truncated parameter-
Feïiruari
'77
februari
'77
februari
'77
februari
'77
spaces
maart
'77
december
'76
mei
'77
Dynamic transfer function-noise modelling (Some thecretical considerations) Limit theorems for LS-estimators in linear regression models with independent errors. Economische interpretatie in modellen betreffende levensduur van kapitaalgoederen
juni
'77
Minimizing wasting times using priority classes
juni
'77
Structuuranalyse van Econometrische Modellen met beriulp van Grafentheorie. Deel IX. D'fodel van landen van de E.E.G.
juni
'77
Capital market models for pcrtfolio selection (a revised version) Evaluatie en foute:~analyse van econometrische modellen. Deel II. Het Modei I van L.R. Kleir,.
An econometric Model of the Federal Republic of Germany 1953-1973 Slagen of zakken. Eer. intern rapport
jur~i
'77
au~r .
'77
aug.
'77
aug.
'77
over de studieresultaten propedeuse-
economie 1974~1975
f~~. A. Hempenius
63. R.M.J. Heuts
Fí4. R.M.J. Heuts
65. A. Hempenius en J. Frijns
66. H.H. Tigelaar
~7. J. P.
van Lieshout en Verheyen
68. Pieter J.F.G. Meulendijks
69.
70.
7"i .
R.M.J. Heuts en P.J. Rens
J.C.P. Kleijnen
Tri .
vatl de Klundert
7~. B.B. van der Genugten
73. A.P. Willemstein
7k.
Piater J.F.G.Meulendijks
75. J.H.J. Roemen
Over een maat voor de juistheid van voorspellingen
aug.
'77
~ome reformulations and extensions in the univariate Box-Jenkins time series analysis approach (a revised version) sept.
'77
Applications of univariate time series modelling of U.S. monetary and business indicator data sept.
'77
Soorten van prijsheteroskedasticiteit in marktvraagfunkties.
okt.
'77
ldentifiability in Multiple Time Series
okt.
'77
nov.
'77
Levensduur in een jaargangenmodel
"De macro-economische betekenis van geïnduceerde technische ontwikkeling; een meer-sektoren model met jaargangentheorie" nov.
'77
A Monte Carlo study to obtain the percentage points of some goodness of fit tests in testing normality, when observations satisfy a certain low order ARMA-scheme dec.
'77
Generalizing Simulation Results through Metamodels
dec.
'77
WinstmF~acimalisatie in het Jaargangenmodel met vaste technische coëfficiënten; een inz-entarisatie van de problematiek
dec.
'77
A central limit theorem with applications in regression analysis
jan.
'78
Evaluatie en foutenanalyse van econometrische modellen. DEEL III Stochastische fluctuaties op de parameters en heteroscedasticiteit in een lineair model jan.
'78
Some reflections on macro-economic planning, policy and development in the Netherlands: ~9~8-~978
apr.
'78
Qvereenkomsten en verschillen in uitgangspunten tussen log-
normale en Pareto-verdeling.
juli '78
76.
J.D. Sylwestrowicz
77. Pieter J.F.G. Meulendijks
78. B.B. van der Genugten
79. J.M.G.
Frijns
F'0. H.H. Tigelaar
Applications of Hybrid Computers in Econometrics - Part I.
juli
'78
On a disequilibrium analysis of the labour market. Review of and comments upon R.S.G. Lenderink and J.C. Siebrand, A disequilibrium analysis of the labour market, Rotterdam University Press, '76. aug.
'78
Asymptotic normality of 2SLSestimators in simultaneous equation systems.
aug.
'78
Imperfect competition of the labour market(s) and the adjustment okt. of factor inputs.
'78
The identifiability problem in dynamic simultaneous equations with
moving average errors.
nov.
'78
an evaluation.
dec.
'78.
82. J.M.G. Frijns
Instrumental Variable Estimators Applied on Pooled Time Series Cross Section Models.
maart '79
83. M.H.C. Paardekooper
Inverse perturbations for approxi- maart mations of least squares solutions.
81. Jack P.C. K].eijnen
84.
J.i-I.J.
8;. J.H.J.
Roemen
Roemen
Bayesian information economics:
Kostenverbijzondering op basis van een input-output-model voor een situatie met vaste en variabele kosten.
'79
maart '79
Een schattingsprocedure op basis van macro data voor de (stationaire) overgangswaarschijnlijkheden in een Markovmodel ter beschrijving van de ontwikkeling van de omvangsverdeling van ondernemingen. april '7y Marketing mix sensitivity.
apríl
'79
87. J. Kriens
Statistical Sampling in Auditing
april
'79
88. J.D.
Applications of hybrid computers in econometrics, Part II. An implementation of a medium size model for the Dutch economy.
mei
g6.
s.J. de Nooij Th. :~!.M. Verhallen
Sylwestrowicz
'79
~~9- pot, Kaper 9U.
~d.J.
Uomens
~) I.
13.b.
v.d.
Genugten
~)~'. J.J.A. Mours ~~3.
íí.íl.
7'igelaar
J~a.
G.J. de Nooij J. de Schouwer
Comparative Statics on noriliriear input-output models. juli '79 Vertragingsverdelingen voor het overloopeffect van marketinguitgaven. Aug, '79 Central limit theorems for the OLS-estimator in linear regression models including lagged variables. gept, '79 Momentproblems and truncated binomial estimation Sept. '79 A general approach to identificaiion, predictability and the problem of minimal informative sample size. Sept. '79 Positionering en segmentatie in de popmuziekmarkt.
Nov.
'79
V I I h ~II IIn II NII NI ~II II~IINI ~VI N
