I am writing an app to automate some benchmarking and I am having a couple of odd issues with SpringHadoop I was hoping someone could help me with.
I have each benchmark defined as it own app context XML file. If I run them in succession everything runs fine, but if I try to run the second one alone It will only use the local job runner. I am not sure why it won't pick up the job tracker address.
The version below doesn't seem to work on its own. my DFSIO xml looks exactly like this except it pulls from dfsio.properties and is a tool-runner. I also tried tool-runner with teragen and I get a weird error:
"org.springframework.beans.factory.BeanCreationExc eption: Error creating bean with name 'TeragenJob': Invocation of init method failed; nested exception is java.lang.ClassCastException: org.apache.hadoop.conf.Configuration cannot be cast to org.apache.hadoop.mapred.JobConf"
------------
<context:property-placeholder
location="classpath*:./conf/hadoop.properties,classpath*:./.properties/teragen.properties" />
<hdp:configuration>
fs.default.name=${hadoop.namenode}
mapred.job.tracker=${hadoop.jobtracker}
</hdp:configuration>
<hdp:jar-runner id="TeragenJob" jar="file:///usr/lib/hadoop/hadoop-examples.jar"
run-at-startup="true"
main-class="org.apache.hadoop.examples.terasort.TeraGen "
configuration-ref="hadoopConfiguration">
<!-- <hdp:arg value="teragen" /> -->
<hdp:arg value="${teragen.numrows}"
/> <hdp:arg value="${teragen.outputdir}" />
</hdp:jar-runner>
<!-- <hdp:tool-runner id="job1"
tool-class="org.apache.hadoop.examples.terasort.TeraGen " jar="file:///usr/lib/hadoop/hadoop-examples.jar"
run-at-startup="true"
configuration-ref="hadoopConfiguration">
<hdp:arg value="teragen" />
<hdp:arg value="${teragen.numrows}" />
<hdp:arg value="${teragen.outputdir}" />
</hdp:tool-runner> -->
I have each benchmark defined as it own app context XML file. If I run them in succession everything runs fine, but if I try to run the second one alone It will only use the local job runner. I am not sure why it won't pick up the job tracker address.
The version below doesn't seem to work on its own. my DFSIO xml looks exactly like this except it pulls from dfsio.properties and is a tool-runner. I also tried tool-runner with teragen and I get a weird error:
"org.springframework.beans.factory.BeanCreationExc eption: Error creating bean with name 'TeragenJob': Invocation of init method failed; nested exception is java.lang.ClassCastException: org.apache.hadoop.conf.Configuration cannot be cast to org.apache.hadoop.mapred.JobConf"
------------
<context:property-placeholder
location="classpath*:./conf/hadoop.properties,classpath*:./.properties/teragen.properties" />
<hdp:configuration>
fs.default.name=${hadoop.namenode}
mapred.job.tracker=${hadoop.jobtracker}
</hdp:configuration>
<hdp:jar-runner id="TeragenJob" jar="file:///usr/lib/hadoop/hadoop-examples.jar"
run-at-startup="true"
main-class="org.apache.hadoop.examples.terasort.TeraGen "
configuration-ref="hadoopConfiguration">
<!-- <hdp:arg value="teragen" /> -->
<hdp:arg value="${teragen.numrows}"
/> <hdp:arg value="${teragen.outputdir}" />
</hdp:jar-runner>
<!-- <hdp:tool-runner id="job1"
tool-class="org.apache.hadoop.examples.terasort.TeraGen " jar="file:///usr/lib/hadoop/hadoop-examples.jar"
run-at-startup="true"
configuration-ref="hadoopConfiguration">
<hdp:arg value="teragen" />
<hdp:arg value="${teragen.numrows}" />
<hdp:arg value="${teragen.outputdir}" />
</hdp:tool-runner> -->