EM算法详细例子及推导
EM算法详细例子及推导数θ),那么对于上面的实验,我们可以计算出他们出现我们观察到的结果即0=(5,9,.8,4,7,20=(B,A,A,B,4)的概率函数P(X=x10),2z)⑨)就叫做θ的似然函数。我们将它对θ求偏导并令偏导数为0,就可以得到如的结果P(X=x0,=20))=(;P(z=A)3(1-P(z=A)2C10(1-64)10A(1-6C104(1-0(1-6B)C106n(1-6我们将这个问题稍微改变一下,我们将我们所观察到的结果修改一…下我们现在只知道每次试验有几次投掷出正面,但是不知道每次试验投掷的是哪个硬币,也就是说我们只知道表中第一列和第三列。这个时候我们就称Z为隐藏变量( Hidden variable),X称为观察变量( Observed variable)。这个时候再来估计参数θ4和θB,就没有那么多数据可供使用了,这个时侯的估计叫做不完整数据的参数估计。如果我们这个时候冇某种方法(比如,正确的猜到每次投掷硬币是A还是B),这样的话我们就可以将这个不完整的数据估计变为完整数据估计当然我们如果没有方法来获得更多的数据的话,那么下面提供了一种在这种不完整数据的情况下来估计参数θ的方法。我们用迭代的方式来进行:(1)我们先赋给θ一个初始值,这个值不管是经验也好猜的也好,反正我们给它一个初始值。在实际使用中往往这个初始值是有其他算法的结果给岀的,当然随机给他分配一个符合定义域的值也可以。这里我们就给定64=0.7,6B=0.4(2)然后我们根据这个来判断或者猜测每次投掷更像是哪枚硬币投掷的结果。比如对于试验1,如果投掷的是Δ,那么出现5个止面的概率为C10×0.75×(1-07)5≈0.1029:;如果投掷的是B,出现5个正面的概率为C105×0.43×(1-0.4)5≈0.2007;基于试验1的试验结果,可以判断这个试验投掷的是使币A的概率为0.10290.10290.2007)-0.389是B的概率为02007(0.1029+0.2007)06611。因此这个结果更可能是投掷B出现的结果(3)假设上一步猜测的结果为B,A,A,B,A,那么恨据这个猜测,可以像完整数据的参数仙计一样(公式2重新计算的值这样一次一次的迭代2-3步骤直到收敛,我们就得到了θ的估计。现在你可能有疑问,这个方法靠谱么?事实证明,它确实是靠谱的。期望最大化算法就是在这个想法上改进的。它在估计每次投掷的硬币的吋候,并不要确定住这次就是硬币A或者B,它计算岀来这次投掷的硬币是A的概率和是B的概率;然后在用这个概率(或者叫做Z的分布)来计算似然函数。期望最大化算法步骤总结如下:F步骤先利用旧的参数值〃计算隐藏变量Z的(条件)分布P(万=2|Xn2),然后计算logP(,X=m)的期望B(o(2,X=x)=∑∑P(Z=别X=)P(Z=X=x)其中θ是当前的值,而θ是上一次迭代得到的值。公式中已经只剩下θ一个变量了,θ是一个确定的值,这个公式或者函数常常叫做Q函数,用Q(6,6)来表示。M步骤极大化Q,往往这一步是求导,得到由旧的θ值′米计算新的θ值的公式aQ总结一下,期望最大化算法就是先根据参数初值估计隐藏变量的分布,然后根据隐藏变量的分布来计算观察变量的似然函数,估计参数的值。前者通常称为E步骤,后者称为M步骤3数学基础首先来明确一下我们的目标:我们的目标是在观察变量X和给定观察样本:1,x2,…,rn的情況下,极大化对数似然函数(=>nP(X2=x;)(5)其中只包含观察变量的概率密度函数P(X2=2)=∑P(X=n,=)这里因为参数θ的写法与条件概率的写法相同,因此将参数θ写到下标以更明确的表述其中Z为隐藏随机变量,{}是Z的所有可能的取值。那么6)=∑h∑P(X=x,z=2)∑h∑。Px=x这里我们引入了一组参数(不要怕多,我们后面会处理掉它的)a,它满足可能的;,0;∈(0,1和∑;a=1到这里,先介绍一个凸函数的性质,或者叫做凸函数的定义。∫(x)为凸函数,=1,2,…,m,A∈[0,1∑1A对∫(x)定义域中的任意n个m1,x2,…,xn有f(∑Aa)≤∑mf(xr)i=1对于严格凸函数,上面的等号只有在x1=2xn的时候成立。关于凸函数的其他性质不再赘述。对数函数是一个严格凸数。因而我们可以有下面这个结果0)=∑hn∑≥∑∑ah(X=2n,2=C现在我们根据等号成立的条件来确定a;即P(X=x,Z=2)C(10)其中c是一个与j无关的常数。因为∑,=1,稍作变换就可以得到P(X;=x;)现在来解释下我们得到了什么。c;就是Z=2;在X=x;下的条件概率戌者后验概率。求α就是求隐藏随机变量Z的条件分布。总结一下目前得到的公式就是)-∑∑P(Xi=i,Z(12)直接就极大值比较难求,EM算法就是按照下面这个过程来的。它就是大名鼎鼎的琴生( Jensen)不等式(1)根据上一步的θ来计算α,即隐藏变量的条件分布(2)极大化似然函数来得到当前的的估计3.1极大似然估计好吧,我觉得还是再说说极大似然估计吧。给定一个概率分布D,假设其概率密度函数为f,其中f带有一组参数6。为了估计这组参数6,我们可以从这个分布中抽出一个具有n个采样值的X1,X2,…,Yn,那么这个就是n个(假设独立)同分布随机变量,他们分别有取值x1,x2…,xn,那么我们就可以计算出出现这样一组观察值的概率密度为lI f(ai)(13)对于f是离散的情况,就计算出现这组观察值的概率10)注意,这个函数中是含有参数0的。0的极大似然估计就是求让上面似然函数取极大值的时候的参数O值。般来说,会将上面那个似然函数取自然对数,这样往往可以简化计算。记住,这样仅仅是为了简化计算。取了自然对数之后的函数叫做对数似然函数。ln()=∑lnf(n)因为对数是一个严格单调递增的凹函数,所以对似然函数取极人值与对对数似然函数取极大值是等价的。3取了对数之后还可以跟信息熵等概念联系起来4关于凸函数有很多种说法,上凸函数和下凸函数,凸函数和凹函数等等,这里指的是二阶导数大」(等」)0的一类函数,而凹函数是其相反数为凸数的一类函数32期望最大化算法收敛性如何保证算法收敛呢?我们只用证明l(04+1)≥1(00)就可以了l(0(t11)∑∑(+1)1PX=x;2=2)(+(t+1∑∑nf(X=x;,z=z;)(+1)(t)o(tn /(r=i,Z=2(t)≥∑∑ahn(t)7(0其中第一个人于等于号是因为只有当a取值合适(琴生不等式等号成立条件)的时候才有等号成立,第二个人于等于号正是M步骤的操作所致。这样我们就知道l(θ)是随着迭代次数的增加越来越人的,收敛条件是值不再变化或者变化幅度很小。4应用举例4.1参数估计很直接的应用就是参数估计,上面举的例子就是参数估计42聚类但是如果估计的参数可以表明类别的话,比如某个参数表示某个样本是否属于某个集合。这样的话其实聚类问题也就可以归结为参数估计问题。References[]最大似然估计[oNline].Availablehttp://zh.wikipediaorg/wiki.%E6%9c%80%E5%A4%A7%E4%BC%BC%E7%84%B6%E4%BC%B0%E8%AE%A1[2] Ceppellini, r, Siniscalco, M.& Smith, C.A. Ann. Hum. Genet. 20, 97-115(1955)3 Hartley, H. Biometrics 14, 174-194(1958)4 Baum, L.E., Petric, T, Soulcs, G.& Weiss, N. Ann. Math. Stat 41, 164-171(1970)[ 5] Dempster, A P, Laird, N.M., Rubin, D B.(1977). "Maximum Likelihoodfrom Incomplete Data via the em algorithm. Journal of the royal statis-tical Society Series B(Methodological)39(1): 1-38. JSTOR 2984875 MR0501537[6]Whatistheexpectationmaximizationalgorithm[oNline].Avaiable:http//ai. stanford. edu/-chuongdo/papers/em tutorial pdf[7TheEmAlgorithmOnline.Availablehttp://www.cnblogs.com,jerrylead/ archive/2011/04/06/2006936html
- 2020-12-07下载
- 积分:1
系统辨识大牛Ljung编写的MATLAB系统辨识使用手册
系统辨识大牛Ljung编写的MATLAB系统辨识使用手册,这本书详细地介绍了在MATLAB已经所属simulink环境下,系统辨识工具箱的一些使用办法,是一本非常经典的教材!Revision Historypril 1988First printingJuly 1991Second printingMay1995Third printingNovember 2000 Fourth printingRevised for Version 5.0(Release 12)pril 2001Fifth printingJuly 2002Online onlyRevised for Version 5.0.2 Release 13)June 2004Sixth printingRevised for Version 6.0.1(Release 14)March 2005Online onlyRevised for Version 6.1.1Release 14SP2)September 2005 Seventh printingRevised for Version 6.1.2(Release 14SP3)March 2006Online onlyRevised for Version 6.1.3(Release 2006a)September 2006 Online onlyRevised for Version 6.2 Release 2006b)March 2007Online onlyRevised for Version 7.0 ( Release 2007a)September 2007 Online onlyRevised for Version 7.1 (Release 2007bMarch 2008Online onlyRevised for Version 7.2(Release 2008a)October 2008Online onlyRevised for Version 7.2.1 Release 2008b)March 2009Online onlyRevised for Version 7.3(Release 2009a)September 2009 Online onlyRevised for Version 7.3.1(Release 2009b)March 2010Online onlyRevised for Version 7. 4 (Release 2010a)eptember2010 Online onlyRevised for Version 7.4.1(Release 2010b)pril 2011Online onlRevised for Version 7.4.2(Release 2011a)September 2011 Online onlyRevised for Version 7.4.3(Release 2011b)March 2012Online onlyRevised for Version 8.0( Release 2012aabout the DevelopersAbout the Developersystem Identification Toolbox software is developed in association with thefollowing leading researchers in the system identification fieldLennart Ljung. Professor Lennart Ljung is with the department ofElectrical Engineering at Linkoping University in Sweden. He is a recognizedleader in system identification and has published numerous papers and booksin this areaQinghua Zhang. Dr. Qinghua Zhang is a researcher at Institut Nationalde recherche en Informatique et en Automatique(INria) and at Institut deRecherche en Informatique et systemes Aleatoires (Irisa), both in rennesFrance. He conducts research in the areas of nonlinear system identificationfault diagnosis, and signal processing with applications in the fields of energyautomotive, and biomedical systemsPeter Lindskog. Dr. Peter Lindskog is employed by nira dynamiAB, Sweden. He conducts research in the areas of system identificationsignal processing, and automatic control with a focus on vehicle industryapplicationsAnatoli Juditsky. Professor Anatoli Juditsky is with the laboratoire JeanKuntzmann at the Universite Joseph Fourier, Grenoble, france. He conductsresearch in the areas of nonparametric statistics, system identification, andstochastic optimizationAbout the developersContentsChoosing Your System Identification ApproachLinear model structures1-2What Are Model objects?Model objects represent linear systemsAbout model data1-5Types of Model objectsDynamic System Models1-9Numeric Models1-11umeric Linear Time Invariant (LTD Models1-11Identified LTI modelsIdentified Nonlinear models1-12Nonlinear model structures1-13Recommended Model Estimation Sequence1-14Supported Models for Time- and Frequency-DomainData,,,,,,,1-16Supported Models for Time-Domain Data1-16Supported Models for Frequency-Domain Data1-17See also1-18Supported Continuous-and Discrete-Time Models1-19Model estimation commands1-21Creating Model Structures at the command Line ... 1-22about system Identification Toolbox Model Objects ... 1-22When to Construct a Model Structure Independently ofEstimation1-23Commands for Constructing Model Structures1-24Model Properties1-25See als1-27Modeling Multiple-Output Systems ......... 1-28About Modeling multiple-Output Systems1-28Modeling Multiple Outputs Directly1-29Modeling multiple outputs as a Combination ofSingle-Output Models.......1-29Improving Multiple-Output Estimation Results byWeighing Outputs During Estimation ....... 1-30Identified linear Time-Invariant models1-32IDLTI Models1-32Configuration of the Structure of Measured and Noise oRepresentation of the Measured and noise Components foVarious model Types1-33Components ....1-35Imposing Constraints on the Values of ModeParameters1-37Estimation of Linear models1-8Data Import and Processing2「Supported Data ...2-3Ways to Obtain Identification DataWays to Prepare Data for System Identification ... 2-6Requirements on Data SamplingRepresenting Data in MATLAB Workspace·····Time-Domain Data Representation2-9Time-Series Data Representation2-10ContentsFrequency-Domain Data Representation ....... 2-11Importing Data into the Gui2-17Types of Data You Can import into the GUi2-17Importing time-Domain Data into the GUI2-18Importing Frequency-Domain Data into the GUI2-22Importing Data Objects into the GUI ......... 2-30Specifying the data sampling interval2-34Specifying estimation and validation Data2-35Preping data Using Quick StartCreating Data Sets from a Subset of Signal Channelo2-362-37Creating multiexperiment Data Sets in the gUi2-39Managing data in the gui ............. 2-46Representing Time- and Frequency-Domain Data Usingiddata object2-55iddata constructor2-55iddata Properties.........2-58Creating Multiexperiment Data at the Command Line .. 2-61Select Data Channels, I/O Data and Experiments in iddataObjects2-63Increasing Number of Channels or Data Points of iddataObjects2-67Managing iddata Objects2-69Representing Frequency-Response Data Using idfrdObiec2-76idfrd Constructor2-76idfrd Properties2-77Select I/o Channels and Data in idfrd Objects ..... 2-79Adding Input or Output Channels in idfrd Objects2-80Managing idfrd Objects2-83Operations That Create idfrd Objects2-83Analyzing Data quality2-85Is your data ready for modeling?2-85Plotting Data in the guI Versus at the command line2-86How to plot data in the gui2-86How to plot data at the command line2-92How to Analyze Data Using the advice Command2-94Selecting Subsets of Data2-96IXWhy Select Subsets of Data?2-96Extract Subsets of Data Using the GUI2-97Extract Subsets of data at the Command Line2-99Handling Missing Data and outliers2-100Handling missing data2-100Handling outliers2-101Extract and Model Specific Data Segments2-102See also2-103Handling offsets and Trends in Data2-104When to detrend data2-104Alternatives for Detrending Data in GUi or at theCommand-Line2-105Next Steps After detrending2-107How to Detrend Data Using the Gui2-108How to detrend data at the Command line2-109Detrending Steady-State Dat109cending transient Dat2-109See also2-110Resampling Data2-111What Is resampling?...,,.,,,,,,,,,,,.2-111Resampling data without Aliasing Effects2-112See also2-116Resampling data Using the GUi.,,,,2-117Resampling Data at the Command line2-118Filtering Data2-120Supported Filters2-120Choosing to Prefilter Your Data2-120See also2-121How to Filter Data Using the gui2-122Filtering Time-Domain Data in the GuI........ 2-122Content
- 2020-12-11下载
- 积分:1