spark | 做基本计算和批量提交

2019-11-08 18:23:35

字体：大中小

来源：转载

供稿：网友

初学spark，就按照书上的来学习

１、首先，在ubunu下登录到拥有spark权限的用户下。

#以我的为例，我的用户名是hadoopsu hadoop #这里会提示输入密码＃切换到spark目录下＃cd /home/hadoop/spark

2、启动python环境

./bin/pyspark

3、以一个案例为例子来介绍，数据见底部的百度云链接

# -*- coding:utf-8 -*- from pyspark import SparkContext＃定义SparkContext对象,2个线程，命名为First Spark Appsc = SparkContext("local[2]", "First Spark App")＃读数据，放在spark/data下data = sc.textFile("data/UserPurchaseHistory.csv").map(lambda line: line.split(",")).map(lambda record: (record[0], record[1], record[2]))＃计算总购买次数numPurchases = data.count()＃计算有多少不同客户购买过商品uniqueUsers = data.map(lambda record: record[0]).distinct().count()＃计算总收入totalRevenue = data.map(lambda record: float(record[2])).sum()＃计算最畅销的产品PRoducts = data.map(lambda record: (record[1], 1.0)).reduceByKey(lambda a, b: a + b).collect()mostPopular = sorted(products, key=lambda x: x[1], reverse=True)[0]＃结果打印出来print ("Total purchases: %d" % numPurchases)print ("Unique users: %d" % uniqueUsers)print ("Total revenue: %2.2f" % totalRevenue)print ("Most popular product: %s with %d purchases" % (mostPopular[0], mostPopular[1]))

结果： Total purchases: 5 Unique users: 4 Total revenue: 39.91 Most popular product: iphone Cover with 2 purchases

４、停止这个计算

sc.stop()

批量计算，不需要进入python环境，直接在spark目录下：将上面的py脚本代码放在spark目录下执行：

/home/hadoop/spark/bin/spark-submit pythonapp.py

这里写图片描述