Law of Large Numbers - Statistics By Jim

The law of large numbers states that as the number of trials increases, sample values tend to converge on the expected result. The two forms of this law lay ... SkiptosecondarymenuSkiptomaincontentSkiptoprimarysidebar Thelawoflargenumbersstatesthatasthenumberoftrialsincreases,samplevaluestendtoconvergeontheexpectedresult.Thetwoformsofthislawlaythefoundationforbothstatisticsandprobabilitytheory. Inthispost,Iexplainbothformsofthelaw,simulatetheminaction,andexplainwhythey’recrucialforstatisticsandprobability! WeakLawofLargeNumbers Therearetwoformsofthelawoflargenumbers,butthedifferencesareprimarilytheoretical.Theweakandstronglawsoflargenumbersbothapplytoasequenceofvaluesforindependentandidenticallydistributed(i.i.d.)randomvariables:X1, X2, …, Xn. Theweaklawoflargenumbersstatesthatasnincreases,thesamplestatisticofthesequenceconvergesinprobabilitytothepopulationvalue.TheweaklawoflargenumbersisalsoknownasKhinchin’slaw. Here’swhatthatmeans.Supposeyouspecifyanonzerodifferencebetweenthetheoreticalvalueandthesamplevalue.Forexample,youmightdefineadifferencebetweenthetheoreticalprobabilityforcointossresults(0.50)andtheactualproportionyouobtainovermultipletrials.Asthenumberoftrialsincreases,theprobabilitythattheactualdifferencewillbesmallerthanthispredefineddifferencealsoincreases.Thisprobabilityconvergeson1asthesamplesizeapproachesinfinity. Thisideaappliesevenwhenyoudefinetinydifferencesbetweentheactualandexpectedvalues.Youjustneedalargersample! StrongLawofLargeNumbers Thestronglawoflargenumbersdescribeshowasamplestatisticconvergesonthepopulationvalueasthesamplesizeorthenumberoftrialsincreases.Forexample,thesamplemeanwillconvergeonthepopulationmeanasthesamplesizeincreases.ThestronglawoflargenumbersisalsoknownasKolmogorov’sstronglaw. Bothlawsapplytovariouscharacteristics,rangingfromthemeansforcontinuousvariablestotheproportionsforBernoullitrials.I’llsimulatebothofthesescenariosnext! SimulationsfortheLawofLargeNumbers Whiletherearemathematicalproofsforbothlawsoflargenumbers,Iwillsimulatethemusingmyfavoriterandomsamplingprogram,Statistics101!Youcandownloaditforfree. HerearemyscriptsfortheIQexampleandthecointossexample.Youcanperformthesimulationsyourselfandseetheresults.IincludeexamplegraphsbelowthatIcreatedusingthesescripts.IexportedthedataintoExcelforprettiergraphs,butStatistics101producesgraphstoo.Yoursimulationswon’tmatchmine,buttheyshouldfollowthesameoverallpatternthatIdiscuss. IQExample Imaginethatwe’restudyingIQscores.Wearerandomlyselecting100participantsandmeasuringtheirIQs.Aswegathersubjects,we’llassesstheirIQandthenrecalculatethesamplemeanwitheachadditionalperson.Thisprocessproducesasequenceofsamplemeansasthesamplesizeincreasesfrom1to100.Ifthelawoflargenumbersholdstrue,we’dexpectthesamplemeanstoconvergeonthepopulationmeanasthesamplesizeincreases.Let’ssee! Forthispopulation,I’lldefinethepopulationdistributionofIQscoresasfollowinganormaldistributionwithameanof100andastandarddeviationof15. Asyoucansee,thesamplemeansconvergeonthepopulationmeanIQvalueof100.Atthebeginningofthesequence,they’remoreerratic,buttheystabilizeandconvergeonthecorrectvalueasthesamplesizeincreases. CoinFlippingExample Now,let’slookatcoinflips.ThisisaBernoulliTrialbecausetherearepreciselytwooutcomes,headsortails.Thedataarebinaryandfollowthebinomialdistributiondefinedbyaproportionofevents.Forthisscenario,we’lldefineaneventasheadsinthecointoss.Acointossisonetrial.Thelawoflargenumberspredictsthatasthenumberoftrialsincreases,theproportionwillconvergeontheexpectedvalueof0.50. Itworks!Thesampleproportionbecomemorestableandconvergesontheexpectedprobabilityvalueof0.50asthesamplesizeincreases. PracticalImplicationsoftheLawofLargeNumbers Thelawoflargenumbersisessentialtobothstatisticsandprobabilitytheory. Forstatistics,bothlawsoflargenumbersindicatethatlargersamplesproduceestimatesthatareconsistentlyclosertothepopulationvalue.Thesepropertiesbecomeimportantininferentialstatistics,whereyouusesamplestoestimatethepropertiesofpopulations.That’swhyyoualwayshearstatisticianssayingthatlargesamplesizesarebetter! Relatedpost:InferentialversusDescriptiveStatistics Inprobabilitytheory,asthenumberoftrialsincreases,therelativefrequencyofobservedeventswillconvergeontheexpectedprobabilityvalue.Ifyouflipacoinfourtimes,it’snotsurprisingtogetthreeheads(75%).However,after100coinflips,thepercentagewillbeextremelycloseto50%. Theselawsbringatypeofordertorandomevents.Forexample,ifyou’retalkingaboutflippingcoins,rollingdice,orgamesofchance,youaremorelikelytoobserveanunusualsequenceofeventsovertheshortrun.However,asthenumberoftrialsgrows,theoveralloutcomesconvergeontheexpectedprobability. Consequently,casinoswithalargevolumeoftrafficcanpredicttheirearningsforgamesofchance.Theirearningswillconvergeonapredictablepercentageoveralargenumberofgames.Youmightbeatthehousewithseveralluckyhands,butinthelongrun,thehousealwayswins! Relatedpost:FundamentalsofProbability WhentheLawsofLargeNumbersFail Therearespecificsituationswherethelawsoflargenumberscanfailtoconvergeontheexpectedvalueasthesamplesizeorthenumberoftrialsincrease.WhenthedatafollowtheCauchydistribution,thenumberscan’tconvergeonanexpectedvaluebecausetheCauchydistributiondoesnothaveanexpectedvalue. Similarly,thelawsdon’tapplytotheParetodistributionbecauseitsexpectedvalueisinfinite. Sharethis:Tweet Related ReaderInteractionsComments HelloJim, DoestheLawofLargenumbersalsoappliestodependentevents?I’mcuriousaboutthisbecauseofMarkovChains.MarkovChainsusessemi-dependenteventsorstateswhencalculatingtheprobabilityofmovingfromonestatetothenextstate.Anotherexamplecouldbethestandardsplayingcards,likewhat’stheprobabilityofgettingtwoAcesinarowoveralargenumberofsamples.Ireadsomewherethatthislawcouldalsoapplytodependentevents.Isthistrue?Thankyouandhaveagoodday. Sincerely, Emikel Reply HiEmikel, Ibelieveitcanapplyinthosecasesbutyouhavetobecarefulindoingso.Youneedtoknowexactlywhatthedependentconditionisthataffectsthenextoutcome.Andthenknowthatthelawoflargenumbersappliestothatprobability.Forexample,supposeeventBhasa60%chanceofoccurringifeventAoccurs,butonlya30%chanceifeventAdoesnotoccur.Withalargenumberofopportunities,you’dexpectthattheobservedfrequencieswillcloseinonthosetheoreticalfrequencies.But,you’llneedtobecarefultoknowwhetherAoccursornotandkeeptrackoftheresultsaccordingly.Bu,withalargenumberofoutcomesyou’dexpecttheobservedfrequenciestobecloseto60%and30%,respectively. Reply Ilovethesetopicsthanksforsharing! Reply WhatareCauchydistributionandParetoDistribution? Reply CommentsandQuestionsCancelreply PrimarySidebarMeetJim I’llhelpyouintuitivelyunderstandstatisticsbyfocusingonconceptsandusingplainEnglishsoyoucanconcentrateonunderstandingyourresults. ReadMore... Searchthiswebsite BuyMyIntroductiontoStatisticseBook! New!BuyMyHypothesisTestingeBook! BuyMyRegressioneBook! SubscribebyEmailEnteryouremailaddresstoreceivenotificationsofnewpostsbyemail.SubscribeIwon'tsendyouspam.Unsubscribeatanytime. FollowMe Facebook RSSFeed Twitter Popular Latest Popular HowToInterpretR-squaredinRegressionAnalysis HowtoInterpretP-valuesandCoefficientsinRegressionAnalysis MeasuresofCentralTendency:Mean,Median,andMode NormalDistributioninStatistics MulticollinearityinRegressionAnalysis:Problems,Detection,andSolutions HowtoInterprettheF-testofOverallSignificanceinRegressionAnalysis UnderstandingInteractionEffectsinStatistics Latest Five-NumberSummary SimpleRandomSampling ConvenienceSampling SystematicSampling LognormalDistribution AStatisticalThanksgiving:GlobalIncomeDistributions Variance RecentComments TadeleonUnderstandingInteractionEffectsinStatisticsJimFrostonCentralLimitTheoremExplainedJimFrostonChebyshev’sTheoreminStatisticsChrisGibbonsonCentralLimitTheoremExplainedImanonChebyshev’sTheoreminStatistics