Announcement

Collapse
No announcement yet.

Hadoop Streaming API error

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Hadoop Streaming API error

    I'm getting a file not found error when running the Hadoop Streaming API with a Python job. From the Stackoverflow forums, I gather that it isn't really a file not found problem and that it is some other sort of problem. Any advice would be appreciated. Here is my call to the API and my stacktrace:

    Script to Call to the API:
    1#!/bin/sh
    2
    3 # grab the current working directory
    4 BASE=$(pwd)
    5
    6 # create the latest deployable package
    7 sbin/deploy.sh
    8
    9 # change directory to where Hadoop lives
    10 cd $HADOOP_HOME
    11
    12 # (potentially optional): turn off safe mode
    13 bin/hdfs dfsadmin -safemode leave
    14
    15 # remove the previous output directory
    16 bin/hdfs dfs -rm -r /user/ubuntu/ukbench/output
    17
    18 # define the set of local files that need to be present to run the Hadoop
    19 # job -- comma separate each file path
    20 FILES="${BASE}/feature_extractor_mapper.py,\
    21 ${BASE}/deploy/pyimagesearch.zip"
    22
    23 # run the job on Hadoop
    24 bin/hadoop jar share/hadoop/tools/lib/hadoop-streaming-*.jar \
    25 -D mapreduce.job.reduces=0 \
    26 -files ${FILES} \
    27 -mapper ${BASE}/feature_extractor_mapper.py \
    28 -input /user/ubuntu/ukbench/input/ukbench_dataset.txt \
    29 -output /user/ubuntu/ukbench/output


    Stacktrace:
    Caused by: java.io.IOException: Cannot run program "/home/ubuntu/high_throughput_feature_extraction/feature_extractor_mapper.py": error=2, No such file or directory
    at java.lang.ProcessBuilder.start(ProcessBuilder.java :1048)
    at org.apache.hadoop.streaming.PipeMapRed.configure(P ipeMapRed.java:209)
    ... 23 more
    Caused by: java.io.IOException: error=2, No such file or directory
    at java.lang.UNIXProcess.forkAndExec(Native Method)
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
    at java.lang.ProcessImpl.start(ProcessImpl.java:134)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java :1029)
    ... 24 more
Working...
X