Spark开发指南
Spark开发指南.pdf本书参考Spark官方文档和源码，通过本书你将精通Spark的安装、配置、开发、监控和调优。Apache SparkSpark是伯克利 APMLab实验室精心打造的,力图在算法( Algorithms)、机器( Machines)、人( People)之间通过大规模集成,来展现大数据应用旳一个平台,其核心引擎就是 Spark,其计算基础是弹性分布式数据集,也就是RDD。通过Spark, MPLab运用大数据、云计算、通信等各种源,以及各种灵活的技术方案,对海量不透明的数据进行甄別并转化为有用的信息,以供人们更好的理解世界。 Spark已经涉及到机器学习、数据挖掘、数据库、信息检索、自然语言处理和语音识别等多个领域。Sparp ecological environment陡着 spark的日趋完善, Spark以其优异的性能正逐渐成为下一个业界和学术界的开源大数据处理平台。随着 Spark1.1.0的发布和 Spark生态圈的不断扩大,可以预见在今后的一段吋间内, Spark将越来越火热。spak生态圈以Spa为核心引擎,以HDFS、S3、 Tachyon为持久层读写原生数据,以 Mesos、YARN和自身携带的Standalone作为资源管理器调度job,来完成spak应用程序的计算;而这些spak应用程序可以来源于不同的组件,如 Spark的批处理应用、 Spark Streaming的实时处理应用、 Spark sρL的即席查询、 BlinkDB的权衝查询、MLib或 MLbase的机器学习、 GraphX的图处理等等。更多的新信息请参看伯克利 APMLab实验室的项目进展htps:/ mplab. cS. berkeley. edu/projects或者 Spark峰会信息htp:/ spark-summit org。Spark Spark MLlib GraphXSQL Streaming(machine(graph)learningApache SparkSparkSpark是一个快速的通用大规模数据丛理系统,和 Hadoop MapReduce相比更好的容锆性和内存计算高速,在内存中运算100倍速度于 MapReduce易用,相同的应用程序代码量要比 MapReduce少25倍提供了丰富的AP支持互动和迭代程序Spark大数据平台之所以能日渐红火,得益于 Spark内核架构的优秀·提供了支持DAG图的分布式并行计算框架,减少多次计算之间中间结果O开销·提供 Cache机制来支持多次迭代计算或者数据共享,减少开销*·RDD之间维护了血统关系,一旦 RDD fail掉了,能通过父RDD自动重建,保证了容错性·, RDD Partition可以就近读取分布式文件系统中的数据块到各个节点内存中进行计算使用多线程池模型来减少task启动开稍shuffle过程中避免不必要的sor操作采用容错的、高可伸缩性的aka作为通讯框架SparkStreamingSparkstreaming是一个对实时数据流进行高通量、容锴处理的流式处理系统,可以对多种数据源(如Kdka、Fume、Twitter、zero和TCP套接字)进行类似map、 reduce、join、 window等复杂操作,并捋结果保存到外部文件系统、数据库或应用到实时仪表盘Sparkstreaming流式必理系统特点有捋流式计算分解成一系列短小的批处理作业将失败或者执行校慢的任务在其它节点上并行执行较强的容错能力(基于RDD继承关系 Lineage)使用和RDD一样的语义Spark SQLSpark SQL是一个即席查询系统,可以通过SQL表达式、 HiveQL或者 Scala dsl在 Spark上执行查询。Spark SQL的特点·引人了新的RDD类型 SchemaRDD,可以象传统数据库定义表一样来定义 SchemaRDD, SchemaRDD由定义了列数据类型的行对象构成。· SchemaRDD可以从RDD转换过来,也可以从 Parquet文件读入,也可以使用 Hive QL从Hve中获取·在应用程序中可以混合使用不同来源的数据,如可以将来自 HiveQL的数据和来自sQL的数据进行jn操作。·内嵌 catalys优化器对用户查询语句进行自动优化MLlibMLib是Spak实现一些常见的机器学习算法和实用程序,包括分类,回归,聚类,协同过滤,降维,以及底层GraphXGraphX是基于 Spark的图处理和图并行计算AP。 GraphX定义了一个新的概念:弹性分布式属性图,一个每个顶点和边都带有属性的定向多重图;并引人了三种核心RDD: Vertices、 Edges、 Triplets;还开放了一组基本操作(如 subgraph,joinvertices, and mapReduce Triplets),并且在不断的扩展图形算法和图形构建工具来筒化图分析工作生态圈的应用Spark生态圈以 Spark为核心、以RDD为基础,打造了一个基于内存DAG计算的大数据平台,为人们提供了一栈式的数据处理方奚。人们可以根据不同的汤景使月主要应用场景用户曲像的建立用户异常行为的发现社交网络关系洞察用户定向商品、活动推荐spak运维相关安装配置、监控等,请求参考《 Spark运维实战》graphiteum install -y bitmap bitmap-fonts-compat Django django-tagging fontconfig cairo python-devel python-memcachedpython-twisted pycairo mod python python-Idap python-simplejson memcached python-zope-interface mod wsgipython-sqlite2Spark BaseSpark开发环境Spark本身是由 scala语言开发的,提供了三种语雷接口: Scala、Java、 Python。根据自己的喜好可以使用相应语言的开发工具。本书使用 scala语言做为开发Spak应用的语,采用 Eclipse为主要的开发工具主要介绍了两个流行的开发工貝: Eclipse、 Intell IDEA。JDK安装配置下载官方网址:htp/www.oracle.com/technetwork/javaljavase/downloads/jdk7-downloads-1880260hml选择好操作系统版本,32位操作采统选择带j586的安装文件;64位操作系统选择菅×64的安装文件。Linux操作系统推荐下载 tar. gz格式的安装文件, Window当然也只有exe格式的文件。Linux下安装解压tar -zxvf jdk-7ug-linux-1586. tar. gz-C/opt/In-/opt/jdk170_09 /opt/jdk设置环境变量用ⅵ编辑配置文件:/etc/ profileexport JAVA HOME=/ pt/jdkexport CLASSPATH=$JAVA HOME/lib/dt jar: SJAVA HOME/lib/tools. jarexport PATH= $JAVA HOME/bin: s PATH保存退出按Esc然后输入Wq使配置生效source /etc/profileWindows下安装选择好操作系统版本是32还是64,解压双击进行安装一路下一步,便可安装成功。设置环境变量测试是否成功命合行输人Java -versIon如果出现下面提示说明成功
- 2020-12-01下载
- 积分：1
Lectures on Stochastic Programming-Model
这是一本关于随机规划比较全面的书！比较难，不太容易啃，但是读了之后收获很大。这是高清版的！To Julia, Benjamin, Daniel, Nalan, and Yael;to Tsonka Konstatin and Marekand to the memory of feliks, Maria, and dentcho2009/8/20pagContentsList of notationserace1 Stochastic Programming ModelsIntroduction1.2 Invento1.2.1The news vendor problem1.2.2Constraints12.3Multistage modelsMultiproduct assembl1.3.1Two-Stage Model1.3.2Chance Constrained ModeMultistage modelPortfolio selection131.4.1Static model14.2Multistage Portfolio selection14.3Decision rule211.5 Supply Chain Network Design22Exercises2 Two-Stage Problems272.1 Linear Two-Stage Problems2.1.1Basic pi272.1.2The Expected Recourse Cost for Discrete Distributions 302.1.3The Expected Recourse Cost for General Distributions.. 322.1.4Optimality Conditions垂Polyhedral Two-Stage Problems422.2.1General Properties422.2.2Expected recourse CostOptimality conditions2.3 General Two-Stage Problems82.3.1Problem Formulation, Interchangeability482.3.2Convex Two-Stage Problems2.4 Nonanticipativity2009/8/20page villContents2.4.1Scenario formulation2.4.2Dualization of Nonanticipativity Constraints2.4.3Nonanticipativity duality for general Distributions2.4.4Value of perfect infExercises3 Multistage problems3. 1 Problem Formulation633.1.1The general setting3.1The Linear case653.1.3Scenario trees3.1.4Algebraic Formulation of nonanticipativity constraints 7lDuality....763.2.1Convex multistage problems·763.2.2Optimality Conditions3.2.3Dualization of Feasibility Constraints3.2.4Dualization of nonanticipativity ConstraintsExercises4 Optimization models with Probabilistic Constraints874.1 Introduction874.2 Convexity in Probabilistic Optimization4.2Generalized Concavity of Functions and measures4.2.2Convexity of probabilistically constrained sets1064.2.3Connectedness of Probabilistically Constrained Sets... 113Separable probabilistic Constraints.1144.3Continuity and Differentiability Properties ofDistribution functions4.3.2p-Efficient Points.1154.3.3Optimality Conditions and Duality Theory1224 Optimization Problems with Nonseparable Probabilistic Constraints.. 1324.4Differentiability of Probability Functions and OptimalityConditions13344.2Approximations of Nonseparable ProbabilisticConstraints134.5 Semi-infinite Probabilistic Problems144E1505 Statistical Inference155Statistical Properties of Sample Average Approximation Estimators.. 1555.1.1Consistency of SAA estimators1575.1.2Asymptotics of the saa Optimal value1635.1.3Second order asStochastic Programs5.2 Stoch1745.2.1Consistency of solutions of the SAA GeneralizedEquatio1752009/8/20pContents5.2.2Atotics of saa generalized equations estimators 1775.3 Monte Carlo Sampling Methods180Exponential Rates of Convergence and Sample sizeEstimates in the Case of a finite Feasible se1815.3.2Sample size estimates in the General Case1855.3.3Finite Exponential Convergence1915.4 Quasi-Monte Carlo Methods1935.Variance-Reduction Techniques198Latin hmpling1985.5.2Linear Control random variables method200ng and likelihood ratio methods 205.6 Validation analysis5.6.1Estimation of the optimality g2025.6.2Statistical Testing of Optimality Conditions2075.7Constrained Probler5.7.1Monte Carlo Sampling Approach2105.7.2Validation of an Optimal solution5.8 SAA Method Applied to Multistage Stochastic Programmin205.8.1Statistical Properties of Multistage SAA Estimators22l5.8.2Complexity estimates of Multistage Programs2265.9 Stochastic Approximation Method2305.9Classical Approach5.9.2Robust sA approach..23359.3Mirror Descent sa method235.9.4Accuracy Certificates for Mirror Descent Sa Solutions.. 244Exercis6 Risk Averse Optimi2536.1 Introductio6.2 Mean-Risk models.2546.2.1Main ideas of mean -Risk analysis546.2.2Semideviation6.2.3Weighted Mean Deviations from Quantiles.2566.2.4Average value-at-Risk2576.3 Coherent risk measures2616.3.1Differentiability Properties of Risk Measures2656.3.2Examples of risk Measures..2696.3.3Law invariant risk measures and Stochastic orders2796.3.4Relation to Ambiguous Chance Constraints2856.4 Optimization of risk measures.2886.4.1Dualization of Nonanticipativity Constraints2916.4.2Examples...2956.5 Statistical Properties of Risk measures6.5.IAverage value-at-Ris6.52Absolute semideviation risk measure301Von mises statistical functionals3046.6The problem of moments306中2009/8/20page xContents6.7 Multistage Risk Averse Optimization3086.7.1Scenario tree formulation3086.7.2Conditional risk mappings3156.7.3Risk Averse multistage Stochastic Programming318Exercises3287 Background material3337.1 Optimization and Convex Analysis..334Directional Differentiability3347.1.2Elements of Convex Analysis3367.1.3Optimization and duality3397.1.4Optimality Conditions.............3467.1.5Perturbation analysis3517.1.6Epiconvergence3572 Probability3597.2.1Probability spaces and random variables7.2.2Conditional Probability and Conditional Expectation... 36372.3Measurable multifunctions and random functions3657.2.4Expectation Functions.3687.2.5Uniform Laws of Large Numbers...,,3747.2.6Law of Large Numbers for Random Sets andSubdifferentials3797.2.7Delta method7.2.8Exponential Bounds of the Large Deviations Theory3877.2.9Uniform Exponential Bounds7.3 Elements of Functional analysis3997.3Conjugate duality and differentiability.......... 4017.3.2Lattice structure4034058 Bibliographical remarks407Biibliography415Index4312009/8/20pageList of Notationsequal by definition, 333IR", n-dimensional space, 333A, transpose of matrix(vector)A, 3336I, domain of the conjugate of risk mea-C(X) space of continuous functions, 165sure p, 262CK, polar of cone C, 337Cn, the space of nonempty compact sub-C(v,R"), space of continuously differ-sets of r 379entiable mappings,176set of probability density functions,I Fr influence function. 3042L, orthogonal of (linear) space L, 41Sz, set of contact points, 3990(1), generic constant, 188b(k; a, N), cdf of binomial distribution,Op(), term, 382214S, the set of &-optimal solutions of theo, distance generating function, 236true problem, 18g(x), right-hand-side derivative, 297Va(a), Lebesgue measure of set A C RdCl(A), topological closure of set A, 334195conv(C), convex hull of set C, 337W,(U), space of Lipschitz continuousCorr(X, Y), correlation of X and Y 200functions. 166. 353CoV(X, Y, covariance of X and y, 180[a]+=max{a,0},2ga, weighted mean deviation, 256IA(, indicator function of set A, 334Sc(, support function of set C, 337n(n.f. p). space. 399A(x), set ofdist(x, A), distance from point x to set Ae multipliers vectors334348dom f, domain of function f, 333N(μ,∑), nonmal distribution,16Nc, normal cone to set C, 337dom 9, domain of multifunction 9, 365IR, set of extended real numbers. 333o(z), cdf of standard normal distribution,epif, epigraph of function f, 333IIx, metric projection onto set X, 231epiconvergence, 377convergence in distribution, 163SN, the set of optimal solutions of the0(x,h)d order tangent set 348SAA problem. 156AVOR. Average value-at-Risk. 258Sa, the set of 8-optimal solutions of thef, set of probability measures, 306SAA problem. 181ID(A, B), deviation of set A from set Bn,N, optimal value of the Saa problem,334156IDIZ], dispersion measure of random vari-N(x), sample average function, 155able 7. 2541A(, characteristic function of set A, 334吧, expectation,361int(C), interior of set C, 336TH(A, B), Hausdorff distance between setsLa」, integer part of a∈R,219A and B. 334Isc f, lower semicontinuous hull of funcN, set of positive integers, 359tion f, 3332009/8/20pageList of notationsRc, radial cone to set C, 337C, tangent cone to set C, 337V-f(r), Hessian matrix of second orderpartial derivatives, 179a. subdifferential. 338a, Clarke generalized gradient, 336as, epsilon subdifferential, 380pos w, positive hull of matrix W, 29Pr(A), probability of event A, 360ri relative interior. 337upper semideviation, 255Le, lower semideviation, 255@R. Value-at-Risk. 25Var[X], variance of X, 149, optimal value of the true problem, 1565=(51,……,5), history of the process,{a,b},186r, conjugate of function/, 338f(x, d), generalized directional deriva-g(x, h), directional derivative, 334O,(, term, 382p-efficient point, 116lid, independently identically distributed,1562009/8/20page xlllPrefaceThe main topic of this book is optimization problems involving uncertain parametersfor which stochastic models are available. Although many ways have been proposed tomodel uncertain quantities stochastic models have proved their flexibility and usefulnessin diverse areas of science. This is mainly due to solid mathematical foundations andtheoretical richness of the theory of probabilitystochastic processes, and to soundstatistical techniques of using real dataOptimization problems involving stochastic models occur in almost all areas of scienceand engineering, from telecommunication and medicine to finance This stimulates interestin rigorous ways of formulating, analyzing, and solving such problems. Due to the presenceof random parameters in the model, the theory combines concepts of the optimization theory,the theory of probability and statistics, and functional analysis. Moreover, in recent years thetheory and methods of stochastic programming have undergone major advances. all thesefactors motivated us to present in an accessible and rigorous form contemporary models andideas of stochastic programming. We hope that the book will encourage other researchersto apply stochastic programming models and to undertake further studies of this fascinatinand rapidly developing areaWe do not try to provide a comprehensive presentation of all aspects of stochasticprogramming, but we rather concentrate on theoretical foundations and recent advances inselected areas. The book is organized into seven chapters The first chapter addresses modeling issues. The basic concepts, such as recourse actions, chance(probabilistic)constraintsand the nonanticipativity principle, are introduced in the context of specific models. Thediscussion is aimed at providing motivation for the theoretical developments in the book,rather than practical recommendationsChapters 2 and 3 present detailed development of the theory of two-stage and multistage stochastic programming problems. We analyze properties of the models and developoptimality conditions and duality theory in a rather general setting. Our analysis coversgeneral distributions of uncertain parameters and provides special results for discrete distributions, which are relevant for numerical methods. Due to specific properties of two- andmultistage stochastic programming problems, we were able to derive many of these resultswithout resorting to methods of functional analvsisThe basic assumption in the modeling and technical developments is that the proba-bility distribution of the random data is not influenced by our actions(decisions). In someapplications, this assumption could be unjustified. However, dependence of probability dis-tribution on decisions typically destroys the convex structure of the optimization problemsconsidered, and our analysis exploits convexity in a significant way
- 2020-12-09下载
- 积分：1
