A practical guide to anomaly detection using Anomalib

A short guide on unsupervised anomaly detection and how to apply it ... The idea of PatchCore is that if a single patch is anomalous the ... OpeninappHomeNotificationsListsStoriesWritePublishedinML6teamApracticalguidetoimage-basedanomalydetectionusingAnomalibAshortintroductiontounsupervisedanomalydetectionandhowtoapplyit.Humanintelligence-basedanomalydetectioninTheSimpsonsInindustrialmanufacturingprocesses,qualityassuranceisanimportanttopic.Therefore,smalldefectsduringproductionneedtobedetectedreliably.Thisiswhatanomalydetectionaimsfor,detectinganomalousanddefectivepatternswhicharedifferentfromthenormalsamples.Thisproblemfacesanumberofuniquechallenges:ItisoftendifficulttoobtainalargeamountofanomalousdataThedifferencebetweenanormalsampleandananomaloussamplecanbeverysmallThetypeofanomaliesisnotalwaysknownbeforehandThesechallengesmaketrainingatraditionalclassifierdifficultandrequirespecialmethodsinordertosolvethem.Unsupervisedanomalydetectionandlocalizationmethodscanbecategorizedasdiscriminativeandgenerativemethods.Discriminativemethodsattempttomodelthedecisionboundarybetweenanomaloussamplesandnominalsamples.Thesemethodsgenerallyextracttheembeddingsfromanimageandcomparethemtothereferenceembeddingsfromthe“good”images.Thedistanceisusedastheanomalyscore.Thesemethodsgivedecentresultsforanomalydetectionbutoftenlackinterpretabilityasyoudon’tknowwhichpartoftheimagecausedtheimagetobeanomalous.AnexampleofsuchamethodisSPADEwhichrunsK-nearestneighbor(K-NN)clusteringonthecompletesetofembeddingvectorsattesttime.Thismeansthattheinferencecomplexityscaleslinearlywiththetrainingsetsize.Highinferencespeedisoftenimportantinmanufacturingwhichreducestheusefulnessofthismethodgreatly.Generativemethodsattempttomodeltheactualdistributionofeachclassfromwhichcanthenbesamplede.g.togeneratenewimages.Anomalydetectionapproachesusingthesemodelsarebasedontheideathattheanomaliescannotbegeneratedsincetheydonotexistinthetrainingset.Autoencoder-basedapproachestrytodetectanomaliesbycomparingtheoutputofanautoencodertoitsinput.Ahighreconstructionerrorshouldindicateananomalousregion.GAN-basedapproachesassumethatonlypositivesamplescanbegenerated.Althoughthesegenerativemethodsareveryintuitiveandinterpretable,theirperformanceislimitedbythefactthattheysometimeshavegoodreconstructionresultsforanomalousimagestoo.StateoftheartmethodsThissectionwilldiscussthreestate-of-the-artmethodsmoreindepth.Twodiscriminativeapproaches,andonegenerativeapproacharedescribed.Thesemethodswerechosenastheyrepresentthestate-of-the-artinanomalydetectionwhilehavingapracticalimplementationavailable.PaDiMBeforePaDiM,severaldiscriminativeapproacheshadbeenproposedwhicheitherrequiredeepneuralnetworktrainingwhichcanbecumbersomeortheyuseK-NNonalargedatasetwhichreducestheinferencespeedgreatly.Thesetwochallengesmighthinderthedeploymentofthealgorithmsinanindustrialenvironment.PatchDistributionModeling(PaDiM)aimstosolvethesechallenges.Theyuseapre-trainedCNN(ResNet,Wide-ResNet,oranEfficientNet)forembeddingextractionbasedonImageNetclassification.Theimagegetsdividedintopatchesandembeddingsareextractedforeachpatch.PaDiMusesallofthelayersofthepre-trainedCNN.Thisisdoneinordertocapturebothglobalcontextsandfinegraineddetails.Astheremightbealotofredundantinformationintheretheysubsampletheembeddingsbyrandomselection.Interestingly,thisworkedasgoodasdimensionalityreductiontechniqueslikePCAwhilebeingfaster.TheassumptionisthatallembeddingvectorsaresampledfromamultivariateGaussiandistribution.Theyestimatethesamplemeanandsamplecovarianceparametersofthisdistributionforeverypatch.TheresultisthateachpatchinthesetoftrainingimagesisdescribedbyamultivariateGaussiandistribution.PaDiMarchitectureoverview.TheanomalyscoreduringinferencenowgetsassignedbasedontheMahalanobisdistancebetweentheembeddingofatestpatchandthelearneddistributionforthatpatchlocation.Thefinalanomalyscoreisthemaximumoftheanomalymap.TheresultisanalgorithmwhichdoesnothavethescalabilityissueoftheKNNbasedmethodsasthereisnoneedtosortalargeamountofdistancevaluestogettheanomalyscoreofapatch.PatchCoreSimilarlytoPaDiM,PatchCoredividestheimagesintopatches.TheideaofPatchCoreisthatifasinglepatchisanomalousthewholeimagecanbeclassifiedasanomalous.PatchCoretriestosolvethesamechallengesPaDiMfaces.ThegoalofPatchCoreisthreefold:Maximizenominalinformationavailableattesttime.PaDiMlimitspatchlevelanomalydetectiontoMahalanobisdistancemeasuresspecificforeachpatch.InPatchCore,thefeaturesextractedduringtrainingphasearestoredinamemorybankwhichisequallyavailabletoallpatchesattesttime.ReducingbiasestowardsImageNetclasses.SimilartoPaDiM,apre-trainedCNNisusedfortheembeddingextraction.AdownsideofthisisthebiasestowardsImageNetclasses.Toreducethisbiasonlymid-levelfeaturesareusedaslowerlevelfeaturesaregenerallytoobroadandhigherlevelfeaturesaretospecifictoImageNet.Retainhighinferencespeeds.PatchCoreintroducescoresetsubsamplingwhichapproximatesthestructureoftheoriginaldatasetwhilereducingthesizegreatly.Thisdecreasesthecostofaanearestneighborsearchresultinginincreasedinferencespeeds.Duringtraining,embeddingsareextractedusingapre-trainedCNN,sub-sampledusingcoresetsubsampling,andstoredinamemorybank.Duringinferenceanearestneighborsearchisperformedonthememorybank.Thisarchitectureisdepictedintheimagebelow.PatchCorearchitectureoverview.CFlow-ADSofarwehavetalkedaboutdiscriminativemodels.Thelastmodelinthiscomparisonisadifferenttype,itisagenerativemodel.Agenerativemodeltellsyouhowlikelytheoccurrenceofagivenexampleis.Forexample,modelsthatpredictthenextwordinasequencearetypicallygenerativemodelsbecausetheycanassignaprobabilitytoasequenceofwords.TypesofgenerativenetworksusedforanomalydetectionincludeVariationalAutoEncoders(VAE),GenerativeAdversarialNetworks(GANs),andnormalizedflows.CFlow-ADisbasedonthelasttypeofnetworks,normalizedflows.CFlow-ADisbasedonaconditionalnormalizingflownetwork.NormalizedflownetworkscanbecomparedtoVAEswithacoupleoffavorablemathematicalproperties.Foranexcellentexplanationofnormalizedflows,seethisblog.Similartothepreviousapproachesanencoderisusedwhichispre-trainedonImageNet.Theembeddingvectorsarethenencodedusingaconventionalpositionalencoding(PE)intoconditionalvectors,henceConditionalFlow.Thedecoderisanormalizedflowdecoderwhichestimatesthelikelihoodoftheencodedfeatures.Theestimatedmulti-scalelikelihoodsareupsampledtoinputsizeandsummedtoproducetheanomalymap.Thisprocessisdepictedbelow.CFlow-ADarchitectureoverview.PerformancetestsOfficialimplementationsforallofthesemethodsareavailableonGitHub.However,thereisanovelopen-sourcePythonlibrarycalledAnomalibwhichimplementsalloftheabovealgorithmsinaneasytoaccessmanner.Anomalibcontainsasetofanomalydetectionalgorithms,asubsetofwhichwaspresentedabove.Thelibraryaimstoprovidecomponentstodesigncustomalgorithmsforspecificneeds,experimenttrackers,visualizers,andhyperparameteroptimizersallaimedatanomalydetection.DatasetApopulardatasetforanomalydetectioninmanufacturingprocessesistheMVTecdatasetwithfactorydefects.Itcontainsover5000high-resolutionimagesdividedintotendifferentobjectandfivetexturecategories.Eachcategorycomprisesasetofdefect-freetrainingimagesandatestsetofimageswithvariouskindsofdefectsaswellasimageswithoutdefects.TheexperimentsbelowwillbeconductedontheScrewobjectandtheCarpettexturecategories.MetricThemetricusedforcomparisonisthetheAreaUnderReceiverOperatingCharacteristiccurve(AUROC)wherethetruepositiverateisthepercentageofpixelscorrectlyclassifiedasanomalous.GettingstartedwithAnomalibInordertouseAnomalibyouwillneedatleastPython3.8+andacloneoftherepository.Installtherequirementslocatedintherequirementsfolder.Itisalsopossibletoinstallthelibraryusingpipinstallanomalib,howeverduetotheactivedevelopmentofthelibrarythisisnotrecommendeduntilreleasev0.2.5.Themodelsarelocatedinanomalib\models\ModelNamewhereeachofthemodelsareimplementedandthereisanaccompanyingconfig.yamlThisconfigfilecontainsinformationaboutthedataset(bydefaultMVTec),modelparameters,andthetrain/testparameters.Fortheexperimentsbelowthedefaultmodel,train,andtestparameterswereused.BydefaultallmodelsexpecttheMVTecdatasetindatasets\MVTec.Youcandownloadthedatasethere.Afterinstallingtherequirements,settingupthedataset,andmodifyingtheconfigfileasdesiredyoucantrainaspecificmodelusing:pythontools/train.py--modelTheresultingweightsandtestimageswillbestoredinresults\.Ifyoualreadyhaveatrainedmodelyoucanruninferenceonanewimageusing:pythontools/inference.py\--model_config_path\--weight_path\--image_pathResultsThissectionwillcomparetheimplementationofthethreemodelsdiscussedearlierandcompareittotheresultsintheirrespectivepapers.TheMVTecdatasetcontains10objectand5textureclasses.ThecomparisonwillcomparetheAUROCofthethreemodelsandwillberunontheScrewobjectclass(320trainimages)andtheCarpettextureclass(245trainimages).AlltestsarerunonaGoogleColabwithaNvidiaK80,2threads,and13GbRAM.Theresultingtablesareshownbelow.AUROCcomparisonfortheScrewobjectclassAUROCcomparisonfortheCarpettextureclass*TheoriginalPaDiMpaperonlypublishedaverageresultsforallclassesontheimagelevelAsexpectedtheresultsoftheAnomalibimplementationareverysimilartotheimplementationsintheoriginalpapers.TwoexampleoutputsofPaDiMareshownbelow.PaDiMoutputoftheScrewobjectclassPaDiMoutputoftheCarpettextureclassBesidestheperformanceresults,speedisalsoanimportantfactorwhendeployingthemodelsinreal-lifescenarios.Thetablebelowcontainsboththetrainingtimeandtheinferencespeedonthetestset(Screws).NotethattheseresultswereobtainedusingthedefaultAnomalibconfigfileandcouldbeimprovede.g.bychangingtheCNNbackbone,batch-size,orsub-samplesize.TrainandinferencespeedsofPaDiM,PatchCore,andCFlow-ADAsignificantdifferencecanbeseeninthetrainingtimewhichcanbeexplainedusingthemodeldescriptionsabove.Allofthemodelsuseapre-trainedCNNasanencoderafterwhichPaDiMrandomlyselectsanumberoffeaturesandcreatesthemultivariateGaussiandistributions.PatchCorehassimilarfunctionalityhoweverusescoresetsubsamplingwhichrequiresmoretrainingtime.CFlow-ADwasagenerativemodelbasedonnormalizingflows.Thismeansthatthedecoderswillhavetobeadaptedtothetrainingsetwhichincreasesthetrainingtimesignificantly.Whencomparingtheinferencespeed,PaDiMisagainthequickestasforeachpatchyouonlyhavetocomputetheMahalanobisdistancetothelearneddistribution.PatchCorehasmoreinformationavailableinthememorybankandrunsnearestneighborswhichisslower.ForCFlow,similartoduringtraining,theuseofthegenerativenetworkrequiresmorefromtheGPUandhasalowerinferencespeed.Notethatthisspeedcomparisonwasdoneusingthedefaultconfigfilewhichmightnotbetheperfectconfigurationforallsituations.TheCFlow-ADpaperforexamplenotesthatwithalighterencoder(MobileNetV3LorResNet-18)theyobtained12fpsonaGTX1080.SummaryInthisblogpostwecomparedthreestate-of-the-artanomalydetectionmethods,PaDiM,PatchCore,andCFlow-AD.WherePaDiMandPatchCoretakethediscriminativeapproachCFlow-ADusesagenerativenormalizingflownetworkinordertodetecttheanomalies.Whencomparingtheresults,theperformanceofallthreeisverysimilar.InrecentliteraturePaDiMisconsideredasabaselineandbothPatchCoreandCFlow-ADtrytoimproveonthisandsucceedinmostareasexceptforthespeed.DuetoitssimplicityPaDiMtrainsquicklyand,usingthedefaultconfigs,hasthehighestinferencespeed.Becauseoftheimplementationitmightbemoresensitivetoorientation/rotationwhichissomethingPatchCoreforexampletriestosolve.Likealways,thedecisionoftheperfectmodeldependsonthesituation,howeverAnomalibprovideseasyaccesstothesemodelsallowingyoutomakethisdecision.FutureworkThecurrenthighestperformerontheMVTecdatasetisFastFlow.AtthetimeofwritingthisblogpostFastflowwasnotavailableinAnomalibhoweverabranchwithapreliminaryimplementationalreadyexistingsuggestingthatitiscomingsoon.FastflowusesNormalizingFlowssimilartoCFlow-ADandtriestoimproveonthis.GoogleCut-Pasteintroducesatwostageframework.ThealgorithmiscalledCutPastebecauseofasimpledataaugmentationstrategythatcutsanimagepatchandpastesthepatchatarandomlocationofalargeimagewhichservesasananomaly.ACNNistrainedusingthisaugmentationinaself-supervisedmanner.Inthesecondstagetheyadoptone-classclassificationalgorithmssuchasOneClassSVMusingtheembeddingsofthefirststage.ThankstoJanVanLooy,JulesTalloen,andMatthiasFeys--1MorefromML6teamFollowWeareateamofAIexpertsandthefastestgrowingAIcompanyinBelgium.WithofficesinGhent,Amsterdam,BerlinandLondon,webuildandimplementselflearningsystemsacrossdifferentsectorstohelpourclientsoperatemoreefficiently.Findoutmoreviawww.ml6.euReadmorefromML6teamRecommendedfromMediumAkulaHemanthKumarinTowardsAIMulti-resolutionImageProcessingandCompressionAINewsClipsbyMorrisLee:NewstohelpyourR&DTrainthermalimagetrackeronplentifulRGBimagesforbetterperformanceMarcoPeixeiroinTowardsDataScienceTheCompleteGuidetoSupportVectorMachine(SVM)SherryYanginTowardsDataScienceMakingTextAscentMichaelAvendiinrandomAIAsummaryofDeepReinforcementLearning(RL)Bootcamp:Lecture5SagnikKunduDecisionTreeClassificationKrisHuanginPymedixABriefHistoryofMedicalImageRegistration:Part2NaomiWeinbergerHateSpeechClassificationModelPart1AboutHelpTermsPrivacyGettheMediumappGetstartedSebastianWehkamp3FollowersFollowMorefromMediumBornaAhmadzadehinDataDrivenInvestorCodingtheVisionTransformerinPyTorch,Part1:Bird’s-EyeViewDuncanBlytheinPyTorchPADL:portablePyTorchpipelinesfacilitatingdeep-learningmodeluseCarlosMocholíinPyTorchLightningDeveloperBlogBagua:Anew,efficient,distributedtrainingstrategyavailableinPyTorchLightning1.6ThibautAnoverviewofPicsellia,anMLOpsplatformforcomputervision.HelpStatusWritersBlogCareersPrivacyTermsAboutKnowable