Classify Defects on Wafer Maps Using Deep Learning

This example shows how to classify eight types of manufacturing defects on wafer maps using a simple convolutional neural network (CNN). Skiptocontent HelpCenterHelpCenter SearchHelpCenter HelpCenter MathWorks SearchMathWorks.com MathWorks Support CloseMobileSearch OpenMobileSearch Off-CanvasNavigationMenuToggle DocumentationHome DeepLearningToolbox DeepLearningApplicationsImageProcessingUsingDeepLearning ClassifyDefectsonWaferMapsUsingDeepLearning Onthispage DownloadWM-811KWaferDefectMapDataPreprocessandAugmentDataCreateNetworkSpecifyTrainingOptionsTrainNetworkorDownloadPretrainedNetworkQuantifyNetworkPerformanceonTestDataVisualizeNetworkDecisionsUsingGradCAMReferencesSeeAlsoRelatedExamplesMoreAbout DocumentationExamplesFunctionsBlocksAppsVideosAnswers TrialSoftware TrialSoftware ProductUpdates ProductUpdates Resources DocumentationExamplesFunctionsBlocksAppsVideosAnswers MainContent ClassifyDefectsonWaferMapsUsingDeepLearningThisexampleuses:ImageProcessingToolboxImageProcessingToolboxDeepLearningToolboxDeepLearningToolboxStatisticsandMachineLearningToolboxStatisticsandMachineLearningToolboxOpenLiveScriptThisexampleshowshowtoclassifyeighttypesofmanufacturingdefectsonwafermapsusingasimpleconvolutionalneuralnetwork(CNN).Wafersarethindisksofsemiconductingmaterial,typicallysilicon,thatserveasthefoundationforintegratedcircuits.Eachwaferyieldsseveralindividualcircuits(ICs),separatedintodies.AutomatedinspectionmachinestesttheperformanceofICsonthewafer.Themachinesproduceimages,calledwafermaps,thatindicatewhichdiesperformcorrectly(pass)andwhichdiesdonotmeetperformancestandards(fail).Thespatialpatternofthepassingandfailingdiesonawafermapcanindicatespecificissuesinthemanufacturingprocess.Deeplearningapproachescanefficientlyclassifythedefectpatternonalargenumberofwafers.Therefore,byusingdeeplearning,youcanquicklyidentifymanufacturingissues,enablingpromptrepairofthemanufacturingprocessandreducingwaste.Thisexampleshowshowtotrainaclassificationnetworkthatdetectsandclassifieseighttypesofmanufacturingdefectpatterns.Theexamplealsoshowshowtoevaluatetheperformanceofthenetwork.DownloadWM-811KWaferDefectMapDataThisexampleusestheWM-811KWaferDefectMapdataset[1][2].Thedatasetconsistsof811,457wafermapsimages,including172,950labeledimages.Eachimagehasonlythreepixelvalues.Thevalue0indicatesthebackground,thevalue1representscorrectlybehavingdies,andthevalue2representsdefectivedies.Thelabeledimageshaveoneofninelabelsbasedonthespatialpatternofthedefectivedies.Thesizeofthedatasetis3.5GB.SetdataDirasthedesiredlocationofthedataset.DownloadthedatasetusingthedownloadWaferMapDatahelperfunction.Thisfunctionisattachedtotheexampleasasupportingfile.dataDir=fullfile(tempdir,"WaferDefects"); downloadWaferMapData(dataDir)PreprocessandAugmentDataThedataisstoredinaMATfileasanarrayofstructures.Loadthedatasetintotheworkspace.dataMatFile=fullfile(dataDir,"MIR-WM811K","WM811K.mat"); waferData=load(dataMatFile); waferData=waferData.data;Explorethedatabydisplayingthefirstelementofthestructure.ThewaferMapfieldcontainstheimagedata.ThefailureTypefieldcontainsthelabelofthedefect.disp(waferData(1))waferMap:[45×48uint8] dieSize:1683 lotName:'lot1' waferIndex:1 trainTestLabel:'Training' failureType:'none' ReformatDataThisexampleusesonlylabeledimages.Removetheunlabeledimagesfromthestructure.unlabeledImages=zeros(size(waferData),"logical"); foridx=1:size(unlabeledImages,1) unlabeledImages(idx)=isempty(waferData(idx).trainTestLabel); end waferData(unlabeledImages)=[];ThedieSize,lotName,andwaferIndexfieldsarenotrelevanttotheclassificationoftheimages.Theexamplepartitionsdataintotraining,validation,andtestsetsusingadifferentconventionthanspecifiedbytrainTestLabelfield.Removethesefieldsfromthestructureusingthermfieldfunction.fieldsToRemove=["dieSize","lotName","waferIndex","trainTestLabel"]; waferData=rmfield(waferData,fieldsToRemove);Specifytheimageclasses.defectClasses=["Center","Donut","Edge-Loc","Edge-Ring","Loc","Near-full","Random","Scratch","none"]; numClasses=numel(defectClasses);Toapplyadditionalpreprocessingoperationsonthedata,suchasresizingtheimagetomatchthenetworkinputsizeorapplyingrandomtrainthenetworkforclassification,youcanuseanaugmentedimagedatastore.Youcannotcreateanaugmentedimagedatastorefromdatainastructure,butyoucancreatethedatastorefromdatainatable.Convertthedataintoatablewithtwovariables:WaferImage-WaferdefectmapimagesFailureType-CategoricallabelforeachimagewaferData=struct2table(waferData); waferData.Properties.VariableNames=["WaferImage","FailureType"]; waferData.FailureType=categorical(waferData.FailureType,defectClasses);DisplayasampleimagefromeachinputimageclassusingthedisplaySampleWaferMapshelperfunction.Thisfunctionisattachedtotheexampleasasupportingfile.displaySampleWaferMaps(waferData)BalanceDataByOversamplingDisplaythenumberofimagesofeachclass.Thedatasetisheavilyunbalanced,withsignificantlyfewerimagesofeachdefectclassthanthenumberofimageswithoutdefects.summary(waferData.FailureType)Center4294 Donut555 Edge-Loc5189 Edge-Ring9680 Loc3593 Near-full149 Random866 Scratch1193 none147431 Toimprovetheclassbalancing,oversamplethedefectclassesusingtheoversampleWaferDefectClasseshelperfunction.Thisfunctionisattachedtotheexampleasasupportingfile.Thehelperfunctionappendsthedatasetwithfivemodifiedcopiesofeachdefectimage.Eachcopyhasoneofthesemodifications:horizontalreflection,verticalreflection,orrotationbyamultipleof90degrees.waferData=oversampleWaferDefectClasses(waferData);Displaythenumberofimagesofeachclassafterclassbalancing.summary(waferData.FailureType)Center25764 Donut3330 Edge-Loc31134 Edge-Ring58080 Loc21558 Near-full894 Random5196 Scratch7158 none147431 PartitionDataintoTraining,Validation,andTestSetsSplittheoversampleddatasetintotraining,validation,andtestsetsusingthesplitlabels(ComputerVisionToolbox)function.Approximately90%ofthedataisusedfortraining,5%isusedforvalidation,and5%isusedfortesting.labelIdx=splitlabels(waferData,[0.90.050.05],"randomized",TableVariable="FailureType"); trainingData=waferData(labelIdx{1},:); validationData=waferData(labelIdx{2},:); testingData=waferData(labelIdx{3},:);AugmentTrainingDataSpecifyasetofrandomaugmentationstoapplytothetrainingdatausinganimageDataAugmenterobject.Addingrandomaugmentationstothetrainingimagescanavoidthenetworkfromoverfittingtothetrainingdata.aug=imageDataAugmenter(FillValue=0,RandXReflection=true,RandYReflection=true,RandRotation=[0360]);Specifytheinputsizeforthenetwork.CreateanaugmentedImageDatastorethatreadsthetrainingdata,resizesthedatatothenetworkinputsize,andappliesrandomaugmentations.inputSize=[4848]; dsTrain=augmentedImageDatastore(inputSize,trainingData,"FailureType",DataAugmentation=aug);Createdatastoresthatreadvalidationandtestdataandresizethedatatothenetworkinputsize.Youdonotneedtoapplyrandomaugmentationstovalidationortestdata.dsVal=augmentedImageDatastore(inputSize,validationData,"FailureType"); dsVal.MiniBatchSize=64; dsTest=augmentedImageDatastore(inputSize,testingData,"FailureType");CreateNetworkDefinetheconvolutionalneuralnetworkarchitecture.Therangeoftheimageinputlayerreflectsthefactthatthewafermapshaveonlythreelevels.layers=[ imageInputLayer([inputSize1],... Normalization="rescale-zero-one",Min=0,Max=2); convolution2dLayer(3,8,Padding="same") batchNormalizationLayer reluLayer maxPooling2dLayer(2,Stride=2) convolution2dLayer(3,16,Padding="same") batchNormalizationLayer reluLayer maxPooling2dLayer(2,Stride=2) convolution2dLayer(3,32,Padding="same") batchNormalizationLayer reluLayer maxPooling2dLayer(2,Stride=2) convolution2dLayer(3,64,Padding="same") batchNormalizationLayer reluLayer dropoutLayer fullyConnectedLayer(numClasses) softmaxLayer classificationLayer];SpecifyTrainingOptionsSpecifythetrainingoptionsforAdamoptimization.Trainthenetworkfor30epochs.options=trainingOptions("adam",... ResetInputNormalization=true,... MaxEpochs=30,... InitialLearnRate=0.001,... L2Regularization=0.001,... MiniBatchSize=64,... Shuffle="every-epoch",... Verbose=false,... Plots="training-progress",... ValidationData=dsVal,... ValidationFrequency=20);TrainNetworkorDownloadPretrainedNetworkBydefault,theexampleloadsapretrainedwaferdefectclassificationnetwork.Thepretrainednetworkenablesyoutoruntheentireexamplewithoutwaitingfortrainingtocomplete.Totrainthenetwork,setthedoTrainingvariableinthefollowingcodetotrue.TrainthemodelusingthetrainNetworkfunction.TrainonaGPUifoneisavailable.UsingaGPUrequiresParallelComputingToolbox™andaCUDA®enabledNVIDIA®GPU.Formoreinformation,seeGPUSupportbyRelease(ParallelComputingToolbox).doTraining=false; ifdoTraining trainedNet=trainNetwork(dsTrain,layers,options); modelDateTime=string(datetime("now",Format="yyyy-MM-dd-HH-mm-ss")); save(fullfile(dataDir,"trained-WM811K-"+modelDateTime+".mat"),"trainedNet"); else downloadTrainedWaferNet(dataDir); trainedNet=load(fullfile(dataDir,"CNN-WM811K.mat")); trainedNet=trainedNet.preTrainedNetwork; endQuantifyNetworkPerformanceonTestDataClassifyeachoftestimageusingtheclassifyfunction.defectPredicted=classify(trainedNet,dsTest);Calculatetheperformanceofthenetworkcomparedtothegroundtruthclassificationsasaconfusionmatrixusingtheconfusionmatfunction.Visualizetheconfusionmatrixusingtheconfusionchartfunction.Thevaluesacrossthediagonalofthismatrixindicatecorrectclassifications.Theconfusionmatrixforaperfectclassifierhasvaluesonlyonthediagonal.defectTruth=testingData.FailureType; cmTest=confusionmat(defectTruth,defectPredicted); figure confusionchart(cmTest,categories(defectTruth),Normalization="row-normalized",... Title="TestDataConfusionMatrix");Precision,Recall,andF1ScoresThisexampleevaluatesthenetworkperformanceusingseveralmetrics:precision,recall,andF1scores.Thesemetricsaredefinedforabinaryclassification.Toovercomethelimitationforthismulticlassproblem,youcanconsiderthepredictionasasetofbinaryclassifications,oneforeachclass.Precisionistheproportionofimagesthatarecorrectlypredictedtobelongtoaclass.Giventhecountoftruepositive(TP)andfalsepositive(FP)classifications,youcancalculateprecisionas: precision=TP (TP+FP)Recallistheproportionofimagesbelongingtoaspecificclassthatwerepredictedtobelongtheclass.GiventhecountofTPandfalsenegative(FN)classifications,youcancalculaterecallas: recall=TP (TP+FN)F1scoresaretheharmonicmeanoftheprecisionandrecallvalues: F1=2*precision*recall (precision+recall)Foreachclass,calculatetheprecision,recall,andF1scoreusingthecountsofTP,FP,andFNresultsavailableintheconfusionmatrix.prTable=table(Size=[numClasses3],VariableTypes=["cell","cell","double"],... VariableNames=["Recall","Precision","F1"],RowNames=defectClasses); foridx=1:numClasses numTP=cmTest(idx,idx); numFP=sum(cmTest(:,idx))-numTP; numFN=sum(cmTest(idx,:),2)-numTP; precision=numTP/(numTP+numFP); recall=numTP/(numTP+numFN); defectClass=defectClasses(idx); prTable.Recall{defectClass}=recall; prTable.Precision{defectClass}=precision; prTable.F1(defectClass)=2*precision*recall/(precision+recall); endDisplaythemetricsforeachclass.Scorescloserto1indicatebetternetworkperformance.prTableprTable=9×3table RecallPrecisionF1 ___________________________ Center{[0.9169]}{[0.9547]}0.93545 Donut{[0.8253]}{[0.8839]}0.85358 Edge-Loc{[0.7951]}{[0.8188]}0.80678 Edge-Ring{[0.9876]}{[0.9005]}0.94203 Loc{[0.6642]}{[0.8939]}0.76211 Near-full{[0.7778]}{[1]}0.875 Random{[0.9462]}{[0.7523]}0.83816 Scratch{[0.4944]}{[0.9219]}0.64364 none{[0.9660]}{[0.9390]}0.95226 Precision-RecallCurvesandArea-Under-Curve(AUC)Inadditiontoreturningaclassificationofeachtestimage,thenetworkcanalsopredicttheprobabilitythatatestimageiseachofthedefectclasses.Inthiscase,precision-recallcurvesprovideanalternativewaytoevaluatethenetworkperformance.Tocalculateprecision-recallcurves,startbyperformingabinaryclassificationforeachdefectclassbycomparingtheprobabilityagainstanarbitrarythreshold.Whentheprobabilityexceedsthethreshold,youcanassigntheimagetothetargetclass.ThechoiceofthresholdimpactsthenumberofTP,FP,andFNresultsandtheprecisionandrecallscores.Toevaluatethenetworkperformance,youmustconsidertheperformanceatarangeofthresholds.Precision-recallcurvesplotthetradeoffbetweenprecisionandrecallvaluesasyouadjustthethresholdforthebinaryclassification.TheAUCmetricsummarizestheprecision-recallcurveforaclassasasinglenumberintherange[0,1],where1indicatesaperfectclassificationregardlessofthreshold.Calculatetheprobabilitythateachtestimagebelongstoeachofthedefectclassesusingthepredictfunction.defectProbabilities=predict(trainedNet,dsTest);Usetherocmetricsfunctiontocalculatetheprecision,recall,andAUCforeachclassoverarangeofthresholds.Plottheprecision-recallcurves.roc=rocmetrics(defectTruth,defectProbabilities,defectClasses,AdditionalMetrics="prec"); figure plot(roc,XAxisMetric="reca",YAxisMetric="prec"); xlabel("Recall") ylabel("Precision") gridon title("Precision-RecallCurvesforAllClasses")Theprecision-recallcurveforanidealclassifierpassesthroughthepoint(1,1).Theclassesthathaveprecision-recallcurvesthattendtowards(1,1),suchasEdge-RingandCenter,aretheclassesforwhichthenetworkhasthebestperformance.ThenetworkhastheworstperformancefortheScratchclass.ComputeanddisplaytheAUCvaluesoftheprecision/recallcurvesforeachclass.prAUC=zeros(numClasses,1); foridx=1:numClasses defectClass=defectClasses(idx); currClassIdx=strcmpi(roc.Metrics.ClassName,defectClass); reca=roc.Metrics.TruePositiveRate(currClassIdx); prec=roc.Metrics.PositivePredictiveValue(currClassIdx); prAUC(idx)=trapz(reca(2:end),prec(2:end));%prec(1)isalwaysNaN end prTable.AUC=prAUC; prTableprTable=9×4table RecallPrecisionF1AUC __________________________________ Center{[0.9169]}{[0.9547]}0.935450.97074 Donut{[0.8253]}{[0.8839]}0.853580.88382 Edge-Loc{[0.7951]}{[0.8188]}0.806780.87182 Edge-Ring{[0.9876]}{[0.9005]}0.942030.74137 Loc{[0.6642]}{[0.8939]}0.762110.82748 Near-full{[0.7778]}{[1]}0.8750.7956 Random{[0.9462]}{[0.7523]}0.838160.93951 Scratch{[0.4944]}{[0.9219]}0.643640.69262 none{[0.9660]}{[0.9390]}0.952260.99076 VisualizeNetworkDecisionsUsingGradCAMGradient-weightedclassactivationmapping(Grad-CAM)producesavisualexplanationofdecisionsmadebythenetwork.YoucanusethegradCAMfunctiontoidentifypartsoftheimagethatmostinfluencedthenetworkprediction.DonutDefectClassTheDonutdefectischaracterizedbyanimagehavingdefectivepixelsclusteredinaconcentriccirclearoundthecenterofthedie.MostimagesoftheDonutdefectclassdonothavedefectivepixelsaroundtheedgeofthedie.ThesetwoimagesbothshowdatawiththeDonutdefect.ThenetworkcorrectlyclassifiedtheimageontheleftasaDonutdefect.ThenetworkmisclassifiedtheimageontherightasanEdge-Ringdefect.TheimageshaveacoloroverlaythatcorrespondstotheoutputofthegradCAMfunction.Theregionsoftheimagethatmostinfluencedthenetworkclassificationappearwithbrightcolorsontheoverlay.FortheimageclassifiedasanEdge-Ringdefect,thedefectsattheboundaryatthedieweretreatedasimportant.ApossiblereasonforthiscouldbetherearefarmoreEdge-RingimagesinthetrainingsetascomparedtoDonutimages.LocDefectClassTheLocdefectischaracterizedbyanimagehavingdefectivepixelsclusteredinablobawayfromtheedgesofthedie.ThesetwoimagesbothshowdatawiththeLocdefect.ThenetworkcorrectlyclassifiedtheimageontheleftasaLocdefect.ThenetworkmisclassifiedtheimageontherightandclassifiedthedefectasanEdge-Locdefect.FortheimageclassifiedasanEdge-Locdefect,thedefectsattheboundaryatthediearemostinfluentialinthenetworkprediction.TheEdge-LocdefectdiffersfromtheLocdefectprimarilyinthelocationoftheclusterofdefects.CompareCorrectClassificationsandMisclassificationsYoucanexploreotherinstancesofcorrectlyclassifiedandmisclassifiedimages.Specifyaclasstoevaluate.defectClass=defectClasses(2);Findtheindexofallimageswiththespecifieddefecttypeasthegroundtruthorpredictedlabel.idxTrue=find(testingData.FailureType==defectClass); idxPred=find(defectPredicted==defectClass);Findtheindicesofcorrectlyclassifiedimages.Then,selectoneoftheimagestoevaluate.Bydefault,thisexampleevaluatesthefirstcorrectlyclassifiedimage.idxCorrect=intersect(idxTrue,idxPred); idxToEvaluateCorrect=1; imCorrect=testingData.WaferImage{idxCorrect(idxToEvaluateCorrect)};Findtheindicesofmisclassifiedimages.Then,selectoneoftheimagestoevaluateandgetthepredictedclassofthatimage.Bydefault,thisexampleevaluatesthefirstmisclassifiedimage.idxIncorrect=setdiff(idxTrue,idxPred); idxToEvaluateIncorrect=1; imIncorrect=testingData.WaferImage{idxIncorrect(idxToEvaluateIncorrect)}; labelIncorrect=defectPredicted(idxIncorrect(idxToEvaluateIncorrect));Resizethetestimagestomatchtheinputsizeofthenetwork.imCorrect=imresize(imCorrect,inputSize); imIncorrect=imresize(imIncorrect,inputSize);GeneratethescoremapsusingthegradCAMfunction.scoreCorrect=gradCAM(trainedNet,imCorrect,defectClass); scoreIncorrect=gradCAM(trainedNet,imIncorrect,labelIncorrect);DisplaythescoremapsovertheoriginalwafermapsusingthedisplayWaferScoreMaphelperfunction.Thisfunctionisattachedtotheexampleasasupportingfile.figure tiledlayout(1,2) t=nexttile; displayWaferScoreMap(imCorrect,scoreCorrect,t) title("CorrectClassification("+defectClass+")") t=nexttile; displayWaferScoreMap(imIncorrect,scoreIncorrect,t) title("Misclassification("+string(labelIncorrect)+")")References[1]Wu,Ming-Ju,Jyh-ShingR.Jang,andJui-LongChen.“WaferMapFailurePatternRecognitionandSimilarityRankingforLarge-ScaleDataSets.”IEEETransactionsonSemiconductorManufacturing28,no.1(February2015):1–12.https://doi.org/10.1109/TSM.2014.2364237.[2]Jang,Roger."MIRCorpora."http://mirlab.org/dataset/public/.[3]Selvaraju,RamprasaathR.,MichaelCogswell,AbhishekDas,RamakrishnaVedantam,DeviParikh,andDhruvBatra.“Grad-CAM:VisualExplanationsfromDeepNetworksviaGradient-BasedLocalization.”In2017IEEEInternationalConferenceonComputerVision(ICCV),618–26.Venice:IEEE,2017.https://doi.org/10.1109/ICCV.2017.74.[4]T.,Bex.“ComprehensiveGuideonMulticlassClassificationMetrics.”October14,2021.https://towardsdatascience.com/comprehensive-guide-on-multiclass-classification-metrics-af94cfb83fbd. SeeAlsotrainingOptions|trainNetwork|augmentedImageDatastore|imageDataAugmenter|imageDatastore|classify|predict|confusionmat|confusionchart RelatedExamplesDetectImageAnomaliesUsingExplainableOne-ClassClassificationNeuralNetworkDetectImageAnomaliesUsingPretrainedResNet-18FeatureEmbeddings MoreAboutDatastoresforDeepLearningPreprocessImagesforDeepLearningListofDeepLearningLayers × OpenExample Youhaveamodifiedversionofthisexample.Doyouwanttoopenthisexamplewithyouredits? No,overwritethemodifiedversion Yes × MATLABCommand YouclickedalinkthatcorrespondstothisMATLABcommand: RunthecommandbyenteringitintheMATLABCommandWindow. WebbrowsersdonotsupportMATLABcommands. Close × SelectaWebSite Chooseawebsitetogettranslatedcontentwhereavailableandseelocaleventsandoffers.Basedonyourlocation,werecommendthatyouselect:. Switzerland(English) Switzerland(Deutsch) Switzerland(Français) 中国(简体中文) 中国(English) Youcanalsoselectawebsitefromthefollowinglist: HowtoGetBestSitePerformance SelecttheChinasite(inChineseorEnglish)forbestsiteperformance.OtherMathWorkscountrysitesarenotoptimizedforvisitsfromyourlocation. Americas AméricaLatina(Español) Canada(English) UnitedStates(English) Europe Belgium(English) Denmark(English) Deutschland(Deutsch) España(Español) Finland(English) France(Français) Ireland(English) Italia(Italiano) Luxembourg(English) Netherlands(English) Norway(English) Österreich(Deutsch) Portugal(English) Sweden(English) Switzerland Deutsch English Français UnitedKingdom(English) AsiaPacific Australia(English) India(English) NewZealand(English) 中国 简体中文 English 日本(日本語) 한국(한국어) Contactyourlocaloffice TrialSoftware TrialSoftware ProductUpdates ProductUpdates