第一次看到Spark崩溃：Spark Shell内存OOM的现象！

第一次看到Spark崩溃

Spark Shell内存OOM的现象

要搞Spark图计算，所以用了Google的web-Google.txt，大小71.8MB。

以命令：

val graph = GraphLoader.edgeListFile(sc,"hdfs://192.168.0.10:9000/input/graph/web-Google.txt")

建立图的时候，运算了半天后直接退回了控制台。

界面xian

scala> val graph = GraphLoader.edgeListFile(sc,"hdfs://192.168.0.10:9000/input/graph/web-Google.txt")

[Stage 0:> (0 + 2) / 2]./bin/spark-shell: line 44: 3592 Killed "${SPARK_HOME}"/bin/spark-submit --class org.apache.spark.repl.Main --name "Spark shell" "$@"

第二次Spark崩溃！

执行pagerank算法测试，使用的是google的数据web-Google.txt，大小71.8MB。

val rank = graph.pageRank(0.01).vertices

Spark是一个master，2个worker的集群，结果slave2挂了

16/11/14 09:51:44 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20161114084026-0000/1333 on hostPort 192.168.0.5:42898 with 4 cores, 1024.0 MB RAM

16/11/14 09:51:44 INFO client.AppClient$ClientEndpoint: Executor updated: app-20161114084026-0000/1333 is now RUNNING

16/11/14 09:51:44 INFO client.AppClient$ClientEndpoint: Executor updated: app-20161114084026-0000/1333 is now EXITED (Command exited with code 1)

16/11/14 09:51:44 INFO cluster.SparkDeploySchedulerBackend: Executor app-20161114084026-0000/1333 removed: Command exited with code 1

16/11/14 09:51:44 INFO cluster.SparkDeploySchedulerBackend: Asked to remove non-existent executor 1333

16/11/14 09:51:44 INFO client.AppClient$ClientEndpoint: Executor added: app-20161114084026-0000/1334 on worker-20161114083607-192.168.0.5-42898 (192.168.0.5:42898) with 4 cores

结果一直尝试连接slave2，呵呵