博客
关于我
强烈建议你试试无所不能的chatGPT,快点击我
PredictionIO+Universal Recommender快速开发部署推荐引擎的问题总结(2)
阅读量:5899 次
发布时间:2019-06-19

本文共 7420 字,大约阅读时间需要 24 分钟。

1, 对Universal Recommender进行pio build成功,但是却提示No engine found

Building and delpoying model[INFO] [Engine$] Using command '/home/vagrant/pio_elastic1/PredictionIO-0.11.1-SNAPSHOT/sbt/sbt' at /home/vagrant/workspace/universal-recommender to build.[INFO] [Engine$] If the path above is incorrect, this process will fail.[INFO] [Engine$] Uber JAR disabled. Making sure lib/pio-assembly-0.11.1-SNAPSHOT.jar is absent.[INFO] [Engine$] Going to run: /home/vagrant/pio_elastic1/PredictionIO-0.11.1-SNAPSHOT/sbt/sbt  package assemblyPackageDependency in /home/vagrant/workspace/universal-recommender[INFO] [Engine$] Compilation finished successfully.[INFO] [Engine$] Looking for an engine...[ERROR] [Engine$] No engine found. Your build might have failed. Aborting.

这是Scala版本导致的问题。进入到universal-recommender的打包目录target中查看,会发现一个叫做scala-2.10的目录。

而我们的PredictionIO在make时指定版本是Scala2.11,所以会去scala-2.11目录下去寻找engine的jar包,自然会出现No engine found

这里有个临时的解决方案,就是直接把scala-2.10改名或者拷贝为scala-2.11,就可以让PredictionIO可以正常执行。

[vagrant@master universal-recommender]$ cd /home/vagrant/workspace/universal-recommender/target[vagrant@master target]$ lsresolution-cache  scala-2.10  streams[vagrant@master target]$ cp -r scala-2.10 scala-2.11

 

2,解决Universal Recommender的Scala版本问题

上面的办法只是个临时解决办法,还是需要统一PredictionIO和Universal Recommender的Scala版本。

我们可以通过修改build.sbt来指定Universal Recommender的Scala版本

name := "universal-recommender"                                  scalaVersion := "2.11.8"

但是,最终会发现出现编译错误。原因是Universal Recommender的一些依赖包,没有Scala2.11的版本。比如mahout包。

[vagrant@master universal-recommender]$ pio build[INFO] [Engine$] Using command '/home/vagrant/pio_elastic1/PredictionIO-0.11.1-SNAPSHOT/sbt/sbt' at /home/vagrant/workspace/universal-recommender to build.[INFO] [Engine$] If the path above is incorrect, this process will fail.[INFO] [Engine$] Uber JAR disabled. Making sure lib/pio-assembly-0.11.1-SNAPSHOT.jar is absent.[INFO] [Engine$] Going to run: /home/vagrant/pio_elastic1/PredictionIO-0.11.1-SNAPSHOT/sbt/sbt  package assemblyPackageDependency in /home/vagrant/workspace/universal-recommender[ERROR] [Engine$] [error] (*:update) sbt.ResolveException: unresolved dependency: org.apache.mahout#mahout-math-scala_2.11;0.13.0: not found[ERROR] [Engine$] [error] unresolved dependency: org.apache.mahout#mahout-spark_2.11;0.13.0: not found[ERROR] [Engine$] [error] Total time: 69 s, completed Sep 8, 2017 10:06:41 AM[ERROR] [Engine$] Return code of build command: /home/vagrant/pio_elastic1/PredictionIO-0.11.1-SNAPSHOT/sbt/sbt  package assemblyPackageDependency is 1. Aborting.

最终只好对build.sbt动了一下大手术,基本原则是:

 1),能够升级到Scala2.11的依赖包,升级;

 2),没有2.11的包,比如mahout,强制指定包版本为2.10

 3),依赖中出现2.10和2.11并存冲突的包,exclude掉2.10版本

最后修改的样子如下:

import scalariform.formatter.preferences._import com.typesafe.sbt.SbtScalariformimport com.typesafe.sbt.SbtScalariform.ScalariformKeysimport sbt.Keys.scalaVersionname := "universal-recommender"version := "0.6.1-SNAPSHOT"organization := "com.actionml"scalaVersion := "2.11.8"val mahoutVersion = "0.13.0"val pioVersion = "0.11.0-incubating"val elasticsearch1Version = "1.7.6"//val elasticsearch5Version = "5.1.2"libraryDependencies ++= Seq(  "org.apache.predictionio" %% "apache-predictionio-core" % pioVersion % "provided",  "org.apache.predictionio" %% "apache-predictionio-data-elasticsearch1" % pioVersion % "provided",  "org.apache.spark" % "spark-core_2.11" % "2.1.0" % "provided",  "org.apache.spark" % "spark-mllib_2.11" % "1.4.0" % "provided",  "org.xerial.snappy" % "snappy-java" % "1.1.1.7",  // Mahout's Spark libs  "org.apache.mahout" % "mahout-math-scala_2.10" % mahoutVersion    exclude("com.github.scopt", "scopt_2.10")    exclude("org.spire-math", "spire_2.10")    exclude("org.scalanlp", "breeze_2.10")    exclude("org.spire-math", "spire-macros_2.10")    exclude("org.apache.spark", "spark-mllib_2.10")    exclude("org.json4s", "json4s-ast_2.10")    exclude("org.json4s", "json4s-core_2.10")    exclude("org.json4s", "json4s-native_2.10")    exclude("org.scalanlp", "breeze-macros_2.10")    exclude("com.esotericsoftware.kryo", "kryo")    exclude("com.twitter", "chill_2.10"),  "org.apache.mahout" % "mahout-spark_2.10" % mahoutVersion    exclude("com.github.scopt", "scopt_2.10")    exclude("org.spire-math", "spire_2.10")    exclude("org.scalanlp", "breeze_2.10")    exclude("org.spire-math", "spire-macros_2.10")    exclude("org.apache.spark", "spark-mllib_2.10")    exclude("org.json4s", "json4s-ast_2.10")    exclude("org.json4s", "json4s-core_2.10")    exclude("org.json4s", "json4s-native_2.10")    exclude("com.twitter", "chill_2.10")    exclude("org.scalanlp", "breeze-macros_2.10")    exclude("com.esotericsoftware.kryo", "kryo")    exclude("org.apache.spark", "spark-launcher_2.10")    exclude("org.apache.spark", "spark-unsafe_2.10")    exclude("org.apache.spark", "spark-tags_2.10")    exclude("org.apache.spark", "spark-core_2.10")    exclude("org.apache.spark", "spark-network-common_2.10")    exclude("org.apache.spark", "spark-streaming_2.10")    exclude("org.apache.spark", "spark-graphx_2.10")    exclude("org.apache.spark", "spark-catalyst_2.10")    exclude("org.apache.spark", "spark-sql_2.10"),  "org.apache.mahout"  % "mahout-math" % mahoutVersion,  "org.apache.mahout"  % "mahout-hdfs" % mahoutVersion    exclude("com.thoughtworks.xstream", "xstream")    exclude("org.apache.hadoop", "hadoop-client"),  //"org.apache.hbase"        % "hbase-client"   % "0.98.5-hadoop2" % "provided",  //  exclude("org.apache.zookeeper", "zookeeper"),  // other external libs  "com.thoughtworks.xstream" % "xstream" % "1.4.4"    exclude("xmlpull", "xmlpull"),  // possible build for es5   //"org.elasticsearch"       %% "elasticsearch-spark-13" % elasticsearch5Version % "provided",  "org.elasticsearch" % "elasticsearch" % "1.7.5" % "provided",  "org.elasticsearch" % "elasticsearch-spark-20_2.11" % "5.5.1",//    exclude("org.apache.spark", "spark-launcher_2.11")//    exclude("org.apache.spark", "spark-unsafe_2.11")//    exclude("org.apache.spark", "spark-tags_2.11")//    exclude("org.apache.spark", "spark-core_2.11")//    exclude("org.apache.spark", "spark-network-common_2.11")//    exclude("org.apache.spark", "spark-streaming_2.11")//    exclude("org.apache.spark", "spark-catalyst_2.11")//    exclude("org.apache.spark", "spark-sql_2.11"),  "org.json4s" % "json4s-native_2.11" % "3.2.10")  .map(_.exclude("org.apache.lucene","lucene-core")).map(_.exclude("org.apache.lucene","lucene-analyzers-common"))resolvers += Resolver.mavenLocalSbtScalariform.scalariformSettingsScalariformKeys.preferences := ScalariformKeys.preferences.value  .setPreference(AlignSingleLineCaseStatements, true)  .setPreference(DoubleIndentClassDeclaration, true)  .setPreference(DanglingCloseParenthesis, Prevent)  .setPreference(MultilineScaladocCommentsStartOnFirstLine, true)assemblyMergeStrategy in assembly := {  case "plugin.properties" => MergeStrategy.discard  case PathList(ps @ _*) if ps.last endsWith "package-info.class" =>    MergeStrategy.first  case PathList(ps @ _*) if ps.last endsWith "UnusedStubClass.class" =>    MergeStrategy.first  case x =>    val oldStrategy = (assemblyMergeStrategy in assembly).value    oldStrategy(x)}

 

PredictionIO和Universal Recommender这样的开源产品,确实存在着官方文档不太完整或者更新不太及时的问题,按照官方手册一次成功的概率很低,需要多次的试验和调查,从其官网,邮件组,以及其他互联网渠道搜索各种线索,才能最终解决问题。

但PredictionIO的社区活跃度很好,Universal Recommender的开发者本人是PredictionIO的重要开发者,还对自己的产品有运营的意愿和行动,邮件组中的技术支持比较到位。

 

转载于:https://www.cnblogs.com/csliwei/p/8093083.html

你可能感兴趣的文章
生命是一场旅程,人生是一个过程。走得越远,去的地方越多,越清楚自己要什么生活。...
查看>>
【转】Android异常:that was originally added here
查看>>
Camera中对焦模式总结
查看>>
字符数组、字符串及常量字符串具体解释
查看>>
python代码格式检查工具部署pre_commit
查看>>
WIN32_FIND_DATA 循环获取文件大小BUG
查看>>
GIT用服务器上的版本强制覆盖掉本地的版本(转)
查看>>
jQuery实现鼠标经过图片预览大图效果
查看>>
Linux编辑器的选择使用
查看>>
从零开始做UI-静电的sketch设计教室 视频教程
查看>>
log4cplus使用(二)-自定义日志等级
查看>>
[Android Pro] 小心ReleaseByteArrayElements 中的参数问题
查看>>
asp.net MVC 应用程序的生命周期
查看>>
innobackupex 备份实验
查看>>
什么是目标框架别名(What are the Target Framework Monikers (TFMs))?
查看>>
java -Xmx3550m -Xms3550m -Xmn2g -Xss128k -XX:+UseParallelGC -XX:MaxGCPauseMillis=100/虚拟机调优...
查看>>
【网络爬虫】Httpclient4.X中使用HTTPS的方法采集12306网站
查看>>
资质平庸的人该如何在职场上做出一番成就
查看>>
linux--每日一个命令
查看>>
win7如何快速设置开机启动项?
查看>>