Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

I cannot read the output of a mapreduce job #20

Open
Sergiomtz100074 opened this issue Apr 10, 2017 · 1 comment
Open

I cannot read the output of a mapreduce job #20

Sergiomtz100074 opened this issue Apr 10, 2017 · 1 comment

Comments

@Sergiomtz100074
Copy link

Sergiomtz100074 commented Apr 10, 2017

Hello I could help with this error does not take me the data of the mapreduce and I mark error some idea of ​​which may be the fault
This is the code I'm using
Sys.setenv(HADOOP_HOME="/usr/local/hadoop")
Sys.setenv(HADOOP_CMD="/usr/local/hadoop/bin/hadoop")
Sys.setenv(JAVA_HOME="/usr/lib/jvm/java-7-openjdk-i386")
Sys.setenv(HADOOP_STREAMING="/usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.6.0.jar")
library('rmr2')
data=to.dfs(1:10)
res = mapreduce(input = data, map = function(k, v) cbind(v, 2*v))
from.dfs(res)

This is what appears to me on the console

Sys.setenv(HADOOP_CMD="/usr/local/hadoop/bin/hadoop")

Sys.setenv(JAVA_HOME="/usr/lib/jvm/java-7-openjdk-i386")
Sys.setenv(HADOOP_STREAMING="/usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.6.0.jar")
library('rmr2')
data=to.dfs(1:10)
OpenJDK Server VM warning: You have loaded library /usr/local/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c ', or link it with '-z noexecstack'.
17/04/10 18:12:44 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/04/10 18:12:46 INFO compress.CodecPool: Got brand-new compressor [.deflate]
res = mapreduce(input = data, map = function(k, v) cbind(v, 2*v))
OpenJDK Server VM warning: You have loaded library /usr/local/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c ', or link it with '-z noexecstack'.
17/04/10 18:12:50 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/04/10 18:12:50 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
packageJobJar: [/tmp/hadoop-unjar5133688999707817678/] [] /tmp/streamjob6112579330814301418.jar tmpDir=null
17/04/10 18:12:51 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.0.24:8050
17/04/10 18:12:52 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.0.24:8050
17/04/10 18:12:53 INFO mapred.FileInputFormat: Total input paths to process : 1
17/04/10 18:12:53 INFO mapreduce.JobSubmitter: number of splits:2
17/04/10 18:12:54 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1491851865431_0014
17/04/10 18:12:54 INFO impl.YarnClientImpl: Submitted application application_1491851865431_0014
17/04/10 18:12:54 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1491851865431_0014/
17/04/10 18:12:54 INFO mapreduce.Job: Running job: job_1491851865431_0014
17/04/10 18:13:02 INFO mapreduce.Job: Job job_1491851865431_0014 running in uber mode : false
17/04/10 18:13:02 INFO mapreduce.Job: map 0% reduce 0%
17/04/10 18:13:10 INFO mapreduce.Job: map 50% reduce 0%
17/04/10 18:13:11 INFO mapreduce.Job: map 100% reduce 0%
17/04/10 18:13:12 INFO mapreduce.Job: Job job_1491851865431_0014 completed successfully
17/04/10 18:13:13 INFO mapreduce.Job: Counters: 30
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=220440
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=973
HDFS: Number of bytes written=244
HDFS: Number of read operations=14
HDFS: Number of large read operations=0
HDFS: Number of write operations=4
Job Counters
Launched map tasks=2
Data-local map tasks=2
Total time spent by all maps in occupied slots (ms)=13405
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=13405
Total vcore-seconds taken by all map tasks=13405
Total megabyte-seconds taken by all map tasks=13726720
Map-Reduce Framework
Map input records=3
Map output records=0
Input split bytes=180
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=124
CPU time spent (ms)=2450
Physical memory (bytes) snapshot=296972288
Virtual memory (bytes) snapshot=1424506880
Total committed heap usage (bytes)=217579520
File Input Format Counters
Bytes Read=793
File Output Format Counters
Bytes Written=244
17/04/10 18:13:13 INFO streaming.StreamJob: Output directory: /tmp/file649a194b8a0e
from.dfs(res)
OpenJDK Server VM warning: You have loaded library /usr/local/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c ', or link it with '-z noexecstack'.
17/04/10 18:13:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/04/10 18:13:21 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Error in scan(file = file, what = what, sep = sep, quote = quote, dec = dec, :
Line 1 does not have 8 elements

@sfweller
Copy link

This error message seems to indicate that their is a problem with the configuration of your Hadoop cluster, rather than a problem in the R code/script. My suggestion would be to direct this question to either HortonWorks support or Cloudera support for help with fixing the issue, if you are running on a HortonWorks or Cloudera cluster.

For some information on this error message, you can consult the information listed on this URL:
http://oracler12applicationfunctional.blogspot.fr/2014/09/java-hotspottm-server-vm-warning-you.html

Hope this helps.

Stephen Weller
Microsoft R Product Team

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants