amazon emr - Spark streaming 1.6.1 is not working with Kinesis asl 1.6.1 and asl 2.0.0-preview -

-

i trying run spark streaming job on emr kinesis. spark 1.6.1 kinesis asl 1.6.1. writing plain sample wordcount example.

        <dependency>         <groupid>org.apache.spark</groupid>         <artifactid>spark-streaming-kinesis-asl_2.10</artifactid>         <version>1.6.1</version>     </dependency>       <dependency>         <groupid>com.amazonaws</groupid>         <artifactid>amazon-kinesis-client</artifactid>         <version>1.6.3</version>     </dependency>     <dependency>         <groupid>com.amazonaws</groupid>         <artifactid>amazon-kinesis-producer</artifactid>         <version>0.10.2</version>     </dependency>

this throws following exception 

java.lang.runtimeexception: java.util.concurrent.executionexception: java.lang.noclassdeffounderror: com/google/protobuf/protocolstringlist     @ com.amazonaws.services.kinesis.clientlibrary.lib.worker.shardconsumer.checkandsubmitnexttask(shardconsumer.java:157)     @ com.amazonaws.services.kinesis.clientlibrary.lib.worker.shardconsumer.consumeshard(shardconsumer.java:126)

upgrading 2.0.0-preview

        <dependency>         <groupid>org.apache.spark</groupid>         <artifactid>spark-streaming-kinesis-asl_2.10</artifactid>         <version>2.0.0-preview</version>     </dependency>

gives following exception

java.lang.noclassdeffounderror: org/apache/spark/internal/logging

at org.apache.spark.streaming.kinesis.kinesisutils$$anonfun$createstream$1.apply(kinesisutils.scala:74)

i have similar problem, mentioned in several places like:

when try run spark streaming application on aws emr still results in:

java.lang.nosuchmethoderror: com.google.protobuf.lazystringlist.getunmodifiableview()lcom/google/protobuf/lazystringlist;

the getunmodifiableview() not available in of versions of protobuf guess still loads wrong version of protobuf. tried several combinations of dependencies versions still same. works fine locally on dev machine, once try submit application on master node error. last tried version of pom file is:

    <dependencies>     <!-- https://mvnrepository.com/artifact/com.google.protobuf/protobuf-java -->     <dependency>         <groupid>com.google.protobuf</groupid>         <artifactid>protobuf-java</artifactid>         <version>2.6.1</version>     </dependency>      <dependency>         <groupid>com.amazonaws</groupid>         <artifactid>amazon-kinesis-client</artifactid>         <version>1.6.1</version>     </dependency>     <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.10 -->     <dependency>         <groupid>org.apache.spark</groupid>         <artifactid>spark-core_2.10</artifactid>         <version>2.1.0</version>         <scope>provided</scope>     </dependency>     <!-- https://mvnrepository.com/artifact/com.google.protobuf/protobuf-java -->        <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client -->     <dependency>         <groupid>org.apache.hadoop</groupid>         <artifactid>hadoop-client</artifactid>         <version>2.7.3</version>         <scope>provided</scope>     </dependency>     <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-mllib_2.10 -->     <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-mllib_2.10 -->     <dependency>         <groupid>org.apache.spark</groupid>         <artifactid>spark-mllib_2.10</artifactid>         <version>2.1.0</version>         <scope>provided</scope>     </dependency>     <dependency>         <groupid>org.apache.spark</groupid>         <artifactid>spark-sql_2.10</artifactid>         <version>2.1.0</version>         <scope>provided</scope>     </dependency>     <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-hive_2.10 -->     <dependency>         <groupid>org.apache.spark</groupid>         <artifactid>spark-hive_2.10</artifactid>         <version>2.1.0</version>         <scope>provided</scope>     </dependency>     <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kinesis-asl_2.11 -->     <dependency>         <groupid>org.apache.spark</groupid>         <artifactid>spark-streaming-kinesis-asl_2.11</artifactid>         <version>2.0.0</version>     </dependency>       <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws -->     <dependency>         <groupid>org.apache.hadoop</groupid>         <artifactid>hadoop-aws</artifactid>         <version>2.7.3</version>         <scope>provided</scope>     </dependency>       <!--https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-->     <dependency>         <groupid>com.amazonaws</groupid>         <artifactid>aws-java-sdk</artifactid>         <version>1.10.77</version>         <exclusions>             <exclusion>                 <artifactid>jackson-core</artifactid>                 <groupid>com.fasterxml.jackson.core</groupid>             </exclusion>             <exclusion>                 <artifactid>jackson-databind</artifactid>                 <groupid>com.fasterxml.jackson.core</groupid>             </exclusion>             <exclusion>                 <artifactid>jackson-annotations</artifactid>                 <groupid>com.fasterxml.jackson.core</groupid>             </exclusion>         </exclusions>         <scope>provided</scope>     </dependency>     <!-- https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-annotations -->     <dependency>         <groupid>com.fasterxml.jackson.core</groupid>         <artifactid>jackson-annotations</artifactid>         <version>2.6.7</version>     </dependency>     <!-- https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-core -->     <dependency>         <groupid>com.fasterxml.jackson.core</groupid>         <artifactid>jackson-core</artifactid>         <version>2.6.7</version>     </dependency>     <!-- https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-databind -->     <dependency>         <groupid>com.fasterxml.jackson.core</groupid>         <artifactid>jackson-databind</artifactid>         <version>2.6.7</version>     </dependency>     <!-- https://mvnrepository.com/artifact/net.java.dev.jets3t/jets3t -->     <dependency>         <groupid>net.java.dev.jets3t</groupid>         <artifactid>jets3t</artifactid>         <version>0.9.4</version>     </dependency>  </dependencies>

Comments

Post a Comment

Popular posts from this blog

matlab - error with cyclic autocorrelation function -

-
Image
i'm trying code cyclic autocorrelation function in matlab follows: t=0:(n-1); t=t*te; i_alpha=0; tau=-n2*te:te:n2*te; alpha=-1/2:1/n:1/2; ryy_cl=zeros(length(alpha),length(tau)); alpha=alpha/te ind_tau=0; i_alpha=i_alpha+1; k=tau ind_tau=ind_tau+1; if k>0 % ryy_cl(i_alpha,ind_tau)=1/length(sig_bin_syl_mod(1:end- k))*sum((sig_bin_syl_mod(1:end-k)).*sig_bin_syl_mod(k+1:end)).*exp(1i*2*pi*alpha*t(1+k:length(sig_bin_syl_mod(1:end)))); ryy_cl(i_alpha,ind_tau)=(1/n)*sum((sig_bin_syl_mod(1:end-k)).*sig_bin_syl_mod(k+1:end)).*exp(-1i*2*pi*alpha*t(1:end-k)); else ryy_cl(i_alpha,ind_tau)=(1/n)*sum((sig_bin_syl_mod(1-k:end)).*sig_bin_syl_mod(1:end+k)).*exp(-1i*2*pi*alpha*t(1-k:end)); end end end i'm getting errors, , doesn't show expecting. according below formulae, how can fix it? the code provided incomplete, difficult confidently reproduce error. in particular, haven't de...
Read more

django - (fields.E300) Field defines a relation with model 'AbstractEmailUser' which is either not installed, or is abstract -

-
i trying create customuser django project email username , add radio button truck , company. in registration process email-id registered per truck or company. mentioned radio button 'tag' , add manytomany field tag in emailuser model. when makemigrations, raising error : (fields.e300) field defines relation model 'abstractemailuser' either not installed, or abstract. i quite new django , not sure whether have created correct code wants. please me solving this. here code, models.py: import django django.contrib.auth.models import ( abstractbaseuser, baseusermanager, permissionsmixin) django.core.mail import send_mail django.db import models django.utils import timezone django.utils.translation import ugettext_lazy _ class emailusermanager(baseusermanager): """custom manager emailuser.""" def _create_user(self, email, password, is_staff, is_superuser, **extra_fields): """create , save emailus...
Read more

c# - What is a good .Net RefEdit control to use with ExcelDna? -

-
i using exceldna , c# build suite of tools. part of forms i'd want user select cells/ranges workbook. i've got application.inputbox route working...but doesn't cool... i haven't read many things excel's refedit control, , see many have in past written workarounds, in (shudder) vba or vb.net. however, none of these seem recent... best approach/control refedit functionality on form run via exceldna (if applicable)? the refedit sample project of github explores different ways of dealing threading when showing form excel add-in, main issue making refedit control .net add-in. check page links various other implementations: https://github.com/excel-dna/exceldna/wiki/links-about-refedit
Read more