Law of large numbers - Wikipedia

In probability theory, the law of large numbers (LLN) is a theorem that describes the result of performing the same experiment a large number of times. Lawoflargenumbers FromWikipedia,thefreeencyclopedia Jumptonavigation Jumptosearch Nottobeconfusedwithlawoftrulylargenumbers. Averagesofrepeatedtrialsconvergetotheexpectedvalue Thisarticleneedsadditionalcitationsforverification.Pleasehelpimprovethisarticlebyaddingcitationstoreliablesources.Unsourcedmaterialmaybechallengedandremoved.Findsources: "Lawoflargenumbers" – news ·newspapers ·books ·scholar ·JSTOR(March2015)(Learnhowandwhentoremovethistemplatemessage) Anillustrationofthelawoflargenumbersusingaparticularrunofrollsofasingledie.Asthenumberofrollsinthisrunincreases,theaverageofthevaluesofalltheresultsapproaches3.5.Althougheachrunwouldshowadistinctiveshapeoverasmallnumberofthrows(attheleft),overalargenumberofrolls(totheright)theshapeswouldbeextremelysimilar. PartofaseriesonstatisticsProbabilitytheory Probability Axioms Determinism System Indeterminism Randomness Probabilityspace Samplespace Event Collectivelyexhaustiveevents Elementaryevent Mutualexclusivity Outcome Singleton Experiment Bernoullitrial Probabilitydistribution Bernoullidistribution Binomialdistribution Normaldistribution Probabilitymeasure Randomvariable Bernoulliprocess Continuousordiscrete Expectedvalue Markovchain Observedvalue Randomwalk Stochasticprocess Complementaryevent Jointprobability Marginalprobability Conditionalprobability Independence Conditionalindependence Lawoftotalprobability Lawoflargenumbers Bayes'theorem Boole'sinequality Venndiagram Treediagram vte Inprobabilitytheory,thelawoflargenumbers(LLN)isatheoremthatdescribestheresultofperformingthesameexperimentalargenumberoftimes.Accordingtothelaw,theaverageoftheresultsobtainedfromalargenumberoftrialsshouldbeclosetotheexpectedvalueandtendstobecomeclosertotheexpectedvalueasmoretrialsareperformed.[1] TheLLNisimportantbecauseitguaranteesstablelong-termresultsfortheaveragesofsomerandomevents.[1][2]Forexample,whileacasinomaylosemoneyinasinglespinoftheroulettewheel,itsearningswilltendtowardsapredictablepercentageoveralargenumberofspins.Anywinningstreakbyaplayerwilleventuallybeovercomebytheparametersofthegame.Importantly,thelawonlyapplies(asthenameindicates)whenalargenumberofobservationsisconsidered.Thereisnoprinciplethatasmallnumberofobservationswillcoincidewiththeexpectedvalueorthatastreakofonevaluewillimmediatelybe"balanced"bytheothers(seethegambler'sfallacy). ItisalsoimportanttonotethattheLLNonlyappliestotheaverage.Therefore,while lim n → ∞ ∑ i = 1 n X i n − X ¯ = 0 {\displaystyle\lim_{n\to\infty}\sum_{i=1}^{n}{\frac{X_{i}}{n}}-{\overline{X}}=0} otherformulasthatlooksimilararenotverified,suchastherawdeviationfrom"theoreticalresults" : ∑ i = 1 n X i − n × X ¯ {\displaystyle\sum_{i=1}^{n}X_{i}-n\times{\overline{X}}} notonlydoesitnotconvergetowardzeroasnincreases,butittendstoincreaseinabsolutevalueasnincreases. Contents 1Examples 2Limitation 3History 4Forms 4.1Weaklaw 4.2Stronglaw 4.3Differencesbetweentheweaklawandthestronglaw 4.4Uniformlawoflargenumbers 4.5Borel'slawoflargenumbers 5Proofoftheweaklaw 5.1ProofusingChebyshev'sinequalityassumingfinitevariance 5.2Proofusingconvergenceofcharacteristicfunctions 6Consequences 7Seealso 8Notes 9References 10Externallinks Examples[edit] Forexample,asinglerollofafair,six-sideddiceproducesoneofthenumbers1,2,3,4,5,or6,eachwithequalprobability.Therefore,theexpectedvalueoftheaverageoftherollsis: 1 + 2 + 3 + 4 + 5 + 6 6 = 3.5 {\displaystyle{\frac{1+2+3+4+5+6}{6}}=3.5} Accordingtothelawoflargenumbers,ifalargenumberofsix-sideddicearerolled,theaverageoftheirvalues(sometimescalledthesamplemean)islikelytobecloseto3.5,withtheprecisionincreasingasmoredicearerolled. ItfollowsfromthelawoflargenumbersthattheempiricalprobabilityofsuccessinaseriesofBernoullitrialswillconvergetothetheoreticalprobability.ForaBernoullirandomvariable,theexpectedvalueisthetheoreticalprobabilityofsuccess,andtheaverageofnsuchvariables(assumingtheyareindependentandidenticallydistributed(i.i.d.))ispreciselytherelativefrequency. Forexample,afaircointossisaBernoullitrial.Whenafaircoinisflippedonce,thetheoreticalprobabilitythattheoutcomewillbeheadsisequalto1⁄2.Therefore,accordingtothelawoflargenumbers,theproportionofheadsina"large"numberofcoinflips"shouldbe"roughly1⁄2.Inparticular,theproportionofheadsafternflipswillalmostsurelyconvergeto1⁄2asnapproachesinfinity. Althoughtheproportionofheads(andtails)approaches1/2,almostsurelytheabsolutedifferenceinthenumberofheadsandtailswillbecomelargeasthenumberofflipsbecomeslarge.Thatis,theprobabilitythattheabsolutedifferenceisasmallnumberapproacheszeroasthenumberofflipsbecomeslarge.Also,almostsurelytheratiooftheabsolutedifferencetothenumberofflipswillapproachzero.Intuitively,theexpecteddifferencegrows,butataslowerratethanthenumberofflips. AnothergoodexampleoftheLLNistheMonteCarlomethod.Thesemethodsareabroadclassofcomputationalalgorithmsthatrelyonrepeatedrandomsamplingtoobtainnumericalresults.Thelargerthenumberofrepetitions,thebettertheapproximationtendstobe.Thereasonthatthismethodisimportantismainlythat,sometimes,itisdifficultorimpossibletouseotherapproaches.[3] Limitation[edit] Theaverageoftheresultsobtainedfromalargenumberoftrialsmayfailtoconvergeinsomecases.Forinstance,theaverageofnresultstakenfromtheCauchydistributionorsomeParetodistributions(α<1)willnotconvergeasnbecomeslarger;thereasonisheavytails.TheCauchydistributionandtheParetodistributionrepresenttwocases:theCauchydistributiondoesnothaveanexpectation,[4]whereastheexpectationoftheParetodistribution(α<1)isinfinite.[5]Anotherexampleiswheretherandomnumbersequalthetangentofanangleuniformlydistributedbetween−90°and+90°.Themedianiszero,buttheexpectedvaluedoesnotexist,andindeedtheaverageofnsuchvariableshavethesamedistributionasonesuchvariable.Itdoesnotconvergeinprobabilitytowardzero(oranyothervalue)asngoestoinfinity. Andifthetrialsembedaselectionbias,typicalinhumaneconomic/rationalbehaviour,theLawoflargenumbersdoesnothelpinsolvingthebias.Evenifthenumberoftrialsisincreasedtheselectionbiasremains. History[edit] Diffusionisanexampleofthelawoflargenumbers.Initially,therearesolutemoleculesontheleftsideofabarrier(magentaline)andnoneontheright.Thebarrierisremoved,andthesolutediffusestofillthewholecontainer.Top:Withasinglemolecule,themotionappearstobequiterandom.Middle:Withmoremolecules,thereisclearlyatrendwherethesolutefillsthecontainermoreandmoreuniformly,buttherearealsorandomfluctuations.Bottom:Withanenormousnumberofsolutemolecules(toomanytosee),therandomnessisessentiallygone:Thesoluteappearstomovesmoothlyandsystematicallyfromhigh-concentrationareastolow-concentrationareas.Inrealisticsituations,chemistscandescribediffusionasadeterministicmacroscopicphenomenon(seeFick'slaws),despiteitsunderlyingrandomnature. TheItalianmathematicianGerolamoCardano(1501–1576)statedwithoutproofthattheaccuraciesofempiricalstatisticstendtoimprovewiththenumberoftrials.[6]Thiswasthenformalizedasalawoflargenumbers.AspecialformoftheLLN(forabinaryrandomvariable)wasfirstprovedbyJacobBernoulli.[7]Ittookhimover20yearstodevelopasufficientlyrigorousmathematicalproofwhichwaspublishedinhisArsConjectandi(TheArtofConjecturing)in1713.Henamedthishis"GoldenTheorem"butitbecamegenerallyknownas"Bernoulli'sTheorem".ThisshouldnotbeconfusedwithBernoulli'sprinciple,namedafterJacobBernoulli'snephewDanielBernoulli.In1837,S.D.Poissonfurtherdescribeditunderthename"laloidesgrandsnombres"("thelawoflargenumbers").[8][9]Thereafter,itwasknownunderbothnames,butthe"lawoflargenumbers"ismostfrequentlyused. AfterBernoulliandPoissonpublishedtheirefforts,othermathematiciansalsocontributedtorefinementofthelaw,includingChebyshev,[10]Markov,Borel,CantelliandKolmogorovandKhinchin.Markovshowedthatthelawcanapplytoarandomvariablethatdoesnothaveafinitevarianceundersomeotherweakerassumption,andKhinchinshowedin1929thatiftheseriesconsistsofindependentidenticallydistributedrandomvariables,itsufficesthattheexpectedvalueexistsfortheweaklawoflargenumberstobetrue.[11][12]ThesefurtherstudieshavegivenrisetotwoprominentformsoftheLLN.Oneiscalledthe"weak"lawandtheotherthe"strong"law,inreferencetotwodifferentmodesofconvergenceofthecumulativesamplemeanstotheexpectedvalue;inparticular,asexplainedbelow,thestrongformimpliestheweak.[11] Forms[edit] Therearetwodifferentversionsofthelawoflargenumbersthataredescribedbelow.Theyarecalledthestronglawoflargenumbersandtheweaklawoflargenumbers.[13][1]StatedforthecasewhereX1,X2,...isaninfinitesequenceofindependentandidenticallydistributed(i.i.d.)LebesgueintegrablerandomvariableswithexpectedvalueE(X1)=E(X2)=...=µ,bothversionsofthelawstatethat–withvirtualcertainty–thesampleaverage X ¯ n = 1 n ( X 1 + ⋯ + X n ) {\displaystyle{\overline{X}}_{n}={\frac{1}{n}}(X_{1}+\cdots+X_{n})} convergestotheexpectedvalue: X ¯ n → μ as   n → ∞ . {\displaystyle{\overline{X}}_{n}\to\mu\quad{\textrm{as}}\n\to\infty.}         (law.1) (LebesgueintegrabilityofXjmeansthattheexpectedvalueE(Xj)existsaccordingtoLebesgueintegrationandisfinite.ItdoesnotmeanthattheassociatedprobabilitymeasureisabsolutelycontinuouswithrespecttoLebesguemeasure.) Basedonthe(unnecessary,seebelow)assumptionoffinitevariance Var ⁡ ( X i ) = σ 2 {\displaystyle\operatorname{Var}(X_{i})=\sigma^{2}} (forall i {\displaystylei} )andnocorrelationbetweenrandomvariables,thevarianceoftheaverageofnrandomvariables Var ⁡ ( X ¯ n ) = Var ⁡ ( 1 n ( X 1 + ⋯ + X n ) ) = 1 n 2 Var ⁡ ( X 1 + ⋯ + X n ) = n σ 2 n 2 = σ 2 n . {\displaystyle\operatorname{Var}({\overline{X}}_{n})=\operatorname{Var}({\tfrac{1}{n}}(X_{1}+\cdots+X_{n}))={\frac{1}{n^{2}}}\operatorname{Var}(X_{1}+\cdots+X_{n})={\frac{n\sigma^{2}}{n^{2}}}={\frac{\sigma^{2}}{n}}.} Pleasenotethisassumptionoffinitevariance Var ⁡ ( X 1 ) = Var ⁡ ( X 2 ) = … = σ 2 < ∞ {\displaystyle\operatorname{Var}(X_{1})=\operatorname{Var}(X_{2})=\ldots=\sigma^{2} ε {\displaystyle|{\overline{X}}_{n}-\mu|>\varepsilon} happensaninfinitenumberoftimes,althoughatinfrequentintervals.(Notnecessarily | X ¯ n − μ | ≠ 0 {\displaystyle|{\overline{X}}_{n}-\mu|\neq0} foralln). Thestronglawshowsthatthisalmostsurelywillnotoccur.Inparticular,itimpliesthatwithprobability1,wehavethatforanyε>0theinequality | X ¯ n − μ | < ε {\displaystyle|{\overline{X}}_{n}-\mu|0, Pr ( | X − μ | ≥ k σ ) ≤ 1 k 2 . {\displaystyle\Pr(|X-\mu|\geqk\sigma)\leq{\frac{1}{k^{2}}}.} Proofoftheweaklaw[edit] GivenX1,X2,...aninfinitesequenceofi.i.d.randomvariableswithfiniteexpectedvalueE(X1)=E(X2)=...=µ