hadoop - Spark submit, spark shell or pyspark not taking default environment values -

-

i have installed hadoop , spark on google cloud using click deploy. trying run spark shell , spark submit test installation. when try

spark-shell --master yarn-client

i error caused by: org.apache.hadoop.yarn.exceptions.invalidresourcerequestexception: invalid resource request, requested memory < 0, or requested memory > max configured, requestedmemory=6383, maxmemory=5999

the problem when don't provide --executor-memory 1g or value , doesn't pick default 1g value , don't know why yarn allocates max memory executor. commands arguments , without arguments

pyspark --master yarn-client --executor-memory 1g --num-executors 1 --verbose     parsed arguments:       master                  yarn-client       deploymode              null       executormemory          1g       numexecutors            1  pyspark --master yarn --verbose parsed arguments:   master                  yarn   deploymode              null   executormemory          null   numexecutors            null

is spark bug or google cloud configuration issue? there anyway can set default values.


Comments

Post a Comment

Popular posts from this blog

django - (fields.E300) Field defines a relation with model 'AbstractEmailUser' which is either not installed, or is abstract -

-
i trying create customuser django project email username , add radio button truck , company. in registration process email-id registered per truck or company. mentioned radio button 'tag' , add manytomany field tag in emailuser model. when makemigrations, raising error : (fields.e300) field defines relation model 'abstractemailuser' either not installed, or abstract. i quite new django , not sure whether have created correct code wants. please me solving this. here code, models.py: import django django.contrib.auth.models import ( abstractbaseuser, baseusermanager, permissionsmixin) django.core.mail import send_mail django.db import models django.utils import timezone django.utils.translation import ugettext_lazy _ class emailusermanager(baseusermanager): """custom manager emailuser.""" def _create_user(self, email, password, is_staff, is_superuser, **extra_fields): """create , save emailus...
Read more

matlab - error with cyclic autocorrelation function -

-
Image
i'm trying code cyclic autocorrelation function in matlab follows: t=0:(n-1); t=t*te; i_alpha=0; tau=-n2*te:te:n2*te; alpha=-1/2:1/n:1/2; ryy_cl=zeros(length(alpha),length(tau)); alpha=alpha/te ind_tau=0; i_alpha=i_alpha+1; k=tau ind_tau=ind_tau+1; if k>0 % ryy_cl(i_alpha,ind_tau)=1/length(sig_bin_syl_mod(1:end- k))*sum((sig_bin_syl_mod(1:end-k)).*sig_bin_syl_mod(k+1:end)).*exp(1i*2*pi*alpha*t(1+k:length(sig_bin_syl_mod(1:end)))); ryy_cl(i_alpha,ind_tau)=(1/n)*sum((sig_bin_syl_mod(1:end-k)).*sig_bin_syl_mod(k+1:end)).*exp(-1i*2*pi*alpha*t(1:end-k)); else ryy_cl(i_alpha,ind_tau)=(1/n)*sum((sig_bin_syl_mod(1-k:end)).*sig_bin_syl_mod(1:end+k)).*exp(-1i*2*pi*alpha*t(1-k:end)); end end end i'm getting errors, , doesn't show expecting. according below formulae, how can fix it? the code provided incomplete, difficult confidently reproduce error. in particular, haven't de...
Read more

php - Using grpc in Laravel, "Class 'Grpc\ChannelCredentials' not found." -

-
i trying use grpc in laravel project. this part of composer.json file: "require": { ... "datto/protobuf-php": "dev-master", "google/auth": "^0.7.0", "grpc/grpc": "dev-release-0_13", ... } i have generated php file proto file. placed in app/services/ , included in composer.json autoload.files section. have run composer install without issues. i have created route testing. has following code: $client = new vat_service\vatserviceclient('localhost:50051', [ 'credentials' => grpc\channelcredentials::createinsecure() ]); but receive response class 'grpc\channelcredentials' not found. when visit route. the laravel app running in homestead vagrant box. i don't know how proceed. missing dependency? did miss install something? i have solved problem, not sure of below steps solution. things tried: i installed grpc-beta in homestead box (...
Read more