1, 对Universal Recommender进行pio build成功,但是却提示No engine found
Building and delpoying model[INFO] [Engine$] Using command '/home/vagrant/pio_elastic1/PredictionIO-0.11.1-SNAPSHOT/sbt/sbt' at /home/vagrant/workspace/universal-recommender to build.[INFO] [Engine$] If the path above is incorrect, this process will fail.[INFO] [Engine$] Uber JAR disabled. Making sure lib/pio-assembly-0.11.1-SNAPSHOT.jar is absent.[INFO] [Engine$] Going to run: /home/vagrant/pio_elastic1/PredictionIO-0.11.1-SNAPSHOT/sbt/sbt package assemblyPackageDependency in /home/vagrant/workspace/universal-recommender[INFO] [Engine$] Compilation finished successfully.[INFO] [Engine$] Looking for an engine...[ERROR] [Engine$] No engine found. Your build might have failed. Aborting.
而我们的PredictionIO在make时指定版本是Scala2.11,所以会去scala-2.11目录下去寻找engine的jar包,自然会出现No engine found
这里有个临时的解决方案,就是直接把scala-2.10改名或者拷贝为scala-2.11,就可以让PredictionIO可以正常执行。
[vagrant@master universal-recommender]$ cd /home/vagrant/workspace/universal-recommender/target[vagrant@master target]$ lsresolution-cache scala-2.10 streams[vagrant@master target]$ cp -r scala-2.10 scala-2.11
2,解决Universal Recommender的Scala版本问题
上面的办法只是个临时解决办法,还是需要统一PredictionIO和Universal Recommender的Scala版本。
我们可以通过修改build.sbt来指定Universal Recommender的Scala版本
name := "universal-recommender" scalaVersion := "2.11.8"
但是,最终会发现出现编译错误。原因是Universal Recommender的一些依赖包,没有Scala2.11的版本。比如mahout包。
[vagrant@master universal-recommender]$ pio build[INFO] [Engine$] Using command '/home/vagrant/pio_elastic1/PredictionIO-0.11.1-SNAPSHOT/sbt/sbt' at /home/vagrant/workspace/universal-recommender to build.[INFO] [Engine$] If the path above is incorrect, this process will fail.[INFO] [Engine$] Uber JAR disabled. Making sure lib/pio-assembly-0.11.1-SNAPSHOT.jar is absent.[INFO] [Engine$] Going to run: /home/vagrant/pio_elastic1/PredictionIO-0.11.1-SNAPSHOT/sbt/sbt package assemblyPackageDependency in /home/vagrant/workspace/universal-recommender[ERROR] [Engine$] [error] (*:update) sbt.ResolveException: unresolved dependency: org.apache.mahout#mahout-math-scala_2.11;0.13.0: not found[ERROR] [Engine$] [error] unresolved dependency: org.apache.mahout#mahout-spark_2.11;0.13.0: not found[ERROR] [Engine$] [error] Total time: 69 s, completed Sep 8, 2017 10:06:41 AM[ERROR] [Engine$] Return code of build command: /home/vagrant/pio_elastic1/PredictionIO-0.11.1-SNAPSHOT/sbt/sbt package assemblyPackageDependency is 1. Aborting.
最终只好对build.sbt动了一下大手术,基本原则是:
1),能够升级到Scala2.11的依赖包,升级;
2),没有2.11的包,比如mahout,强制指定包版本为2.10
3),依赖中出现2.10和2.11并存冲突的包,exclude掉2.10版本
最后修改的样子如下:
import scalariform.formatter.preferences._import com.typesafe.sbt.SbtScalariformimport com.typesafe.sbt.SbtScalariform.ScalariformKeysimport sbt.Keys.scalaVersionname := "universal-recommender"version := "0.6.1-SNAPSHOT"organization := "com.actionml"scalaVersion := "2.11.8"val mahoutVersion = "0.13.0"val pioVersion = "0.11.0-incubating"val elasticsearch1Version = "1.7.6"//val elasticsearch5Version = "5.1.2"libraryDependencies ++= Seq( "org.apache.predictionio" %% "apache-predictionio-core" % pioVersion % "provided", "org.apache.predictionio" %% "apache-predictionio-data-elasticsearch1" % pioVersion % "provided", "org.apache.spark" % "spark-core_2.11" % "2.1.0" % "provided", "org.apache.spark" % "spark-mllib_2.11" % "1.4.0" % "provided", "org.xerial.snappy" % "snappy-java" % "1.1.1.7", // Mahout's Spark libs "org.apache.mahout" % "mahout-math-scala_2.10" % mahoutVersion exclude("com.github.scopt", "scopt_2.10") exclude("org.spire-math", "spire_2.10") exclude("org.scalanlp", "breeze_2.10") exclude("org.spire-math", "spire-macros_2.10") exclude("org.apache.spark", "spark-mllib_2.10") exclude("org.json4s", "json4s-ast_2.10") exclude("org.json4s", "json4s-core_2.10") exclude("org.json4s", "json4s-native_2.10") exclude("org.scalanlp", "breeze-macros_2.10") exclude("com.esotericsoftware.kryo", "kryo") exclude("com.twitter", "chill_2.10"), "org.apache.mahout" % "mahout-spark_2.10" % mahoutVersion exclude("com.github.scopt", "scopt_2.10") exclude("org.spire-math", "spire_2.10") exclude("org.scalanlp", "breeze_2.10") exclude("org.spire-math", "spire-macros_2.10") exclude("org.apache.spark", "spark-mllib_2.10") exclude("org.json4s", "json4s-ast_2.10") exclude("org.json4s", "json4s-core_2.10") exclude("org.json4s", "json4s-native_2.10") exclude("com.twitter", "chill_2.10") exclude("org.scalanlp", "breeze-macros_2.10") exclude("com.esotericsoftware.kryo", "kryo") exclude("org.apache.spark", "spark-launcher_2.10") exclude("org.apache.spark", "spark-unsafe_2.10") exclude("org.apache.spark", "spark-tags_2.10") exclude("org.apache.spark", "spark-core_2.10") exclude("org.apache.spark", "spark-network-common_2.10") exclude("org.apache.spark", "spark-streaming_2.10") exclude("org.apache.spark", "spark-graphx_2.10") exclude("org.apache.spark", "spark-catalyst_2.10") exclude("org.apache.spark", "spark-sql_2.10"), "org.apache.mahout" % "mahout-math" % mahoutVersion, "org.apache.mahout" % "mahout-hdfs" % mahoutVersion exclude("com.thoughtworks.xstream", "xstream") exclude("org.apache.hadoop", "hadoop-client"), //"org.apache.hbase" % "hbase-client" % "0.98.5-hadoop2" % "provided", // exclude("org.apache.zookeeper", "zookeeper"), // other external libs "com.thoughtworks.xstream" % "xstream" % "1.4.4" exclude("xmlpull", "xmlpull"), // possible build for es5 //"org.elasticsearch" %% "elasticsearch-spark-13" % elasticsearch5Version % "provided", "org.elasticsearch" % "elasticsearch" % "1.7.5" % "provided", "org.elasticsearch" % "elasticsearch-spark-20_2.11" % "5.5.1",// exclude("org.apache.spark", "spark-launcher_2.11")// exclude("org.apache.spark", "spark-unsafe_2.11")// exclude("org.apache.spark", "spark-tags_2.11")// exclude("org.apache.spark", "spark-core_2.11")// exclude("org.apache.spark", "spark-network-common_2.11")// exclude("org.apache.spark", "spark-streaming_2.11")// exclude("org.apache.spark", "spark-catalyst_2.11")// exclude("org.apache.spark", "spark-sql_2.11"), "org.json4s" % "json4s-native_2.11" % "3.2.10") .map(_.exclude("org.apache.lucene","lucene-core")).map(_.exclude("org.apache.lucene","lucene-analyzers-common"))resolvers += Resolver.mavenLocalSbtScalariform.scalariformSettingsScalariformKeys.preferences := ScalariformKeys.preferences.value .setPreference(AlignSingleLineCaseStatements, true) .setPreference(DoubleIndentClassDeclaration, true) .setPreference(DanglingCloseParenthesis, Prevent) .setPreference(MultilineScaladocCommentsStartOnFirstLine, true)assemblyMergeStrategy in assembly := { case "plugin.properties" => MergeStrategy.discard case PathList(ps @ _*) if ps.last endsWith "package-info.class" => MergeStrategy.first case PathList(ps @ _*) if ps.last endsWith "UnusedStubClass.class" => MergeStrategy.first case x => val oldStrategy = (assemblyMergeStrategy in assembly).value oldStrategy(x)}
PredictionIO和Universal Recommender这样的开源产品,确实存在着官方文档不太完整或者更新不太及时的问题,按照官方手册一次成功的概率很低,需要多次的试验和调查,从其官网,邮件组,以及其他互联网渠道搜索各种线索,才能最终解决问题。
但PredictionIO的社区活跃度很好,Universal Recommender的开发者本人是PredictionIO的重要开发者,还对自己的产品有运营的意愿和行动,邮件组中的技术支持比较到位。