Python大数据分析文档

说明：  这是使用Python实现的DBScan。像Numpy、熊猫这样的图书馆也被使用过。DBScan算法已经在两个变色龙数据集t4.8k和t5.8k上进行了测试。然后利用matplotlib将得到的结果可视化。为了便于比较，本文将所得到的输出结果与DBScan实现的skLearning库的结果进行了比较。计算每个数据集的同质性和分离度，以观察簇间的相似性和不同的度量。epsilon和min值分别为8.5和16.5。(This is a DBScan implemented using Python. Libraries like Numpy and Panda have also been used. DBScan algorithm has been tested on two chameleon datasets t4.8k and t5.8k. Then the results are visualized by matplotlib. In order to facilitate comparison, the output results are compared with those of skLearning library implemented by DBScan. The homogeneity and segregation of each data set are calculated to observe the similarity and different measures between clusters. Epsilon and min were 8.5 and 16.5 respectively.)

说明：  利用python实现神经网络算法预测信用卡违约情况，使用的是海豚大数据大数据分析赛的数据(Using Python neural network algorithm to predict credit card default, using dolphin big data big data analysis game data)

说明：  图形自动编码器的pytorch实现，参考论文vgae(Implementation of graph self encoder gae with Python)

说明：  MTCNN，Multi-task convolutional neural network（多任务卷积神经网络），将人脸区域检测与人脸关键点检测放在了一起，它的主题框架类似于cascade。总体可分为P-Net、R-Net、和O-Net三层网络结构。(MTCNN, Multi-task convolutional neural network (multi-task convolutional neural network), puts face region detection and key point detection together, and its subject framework is similar to cascade. Overall, it can be divided into three layers: P-Net, R-Net and O-Net.)

使用tflearn框架搭建cnn网络，对DNS隐蔽通道进行检测，模型已经过训练，在程序所包含的测试集上进行预测，准确率可达0.99以上(an cnn deep network which can detect DNS tunnle, the model has already been trained and the accuracy can reach to 0.99)

说明：  利用Q学习进行迷宫路径选择,完成简单的强化学习任务(use Q_learning to play maze game)

一款基于PYTHON语言，用VIZARD软件写的小游戏 操作上类似于打地鼠 实际上是赚钱开农场的剧情 有排行榜 表妹说..挺好玩的(A small game, based on Python, written by Vizard.)

样本数据不均衡的数据的解决方案： 过采样：生成样本少的样本，和样本多的样本一样多（同样多） 下采样：两种样本一样小，从多的样本中删除一部分（同样少） 特征缩放：对Amount列进行特征缩放，使其分布范围和其他特征属性的分布范围保持一致 机器学习误区：会认为数值大的数据重要程度较大，但是我们拿到数据，其实每个属性的重要程度是一样的，所以需要进行特征缩放 所以可以进行：归一化（缩放到0~1）或者标准化(缩放到-1~1)

使用python进行信号EMD分析，然后LSTM输入信号特征，然后分类(The signal EMD is analyzed by python, then LSTM input signal characteristics are classified.)