Hi,
I'm trying to using Spring Hadoop to execute MRv1 jobs on a 2.0.2-alpha cluster, but am having some challenges. I'm using the basic WordCount type example which works just fine as a regular job. It looks like something is expecting a MRv2 job, however, as I get the following error when I run my code:
Exception in thread "main" org.springframework.beans.factory.BeanCreationExce ption: Error creating bean with name 'mr-job': Invocation of init method failed; nested exception is java.lang.RuntimeException: class mr.v1.spring.warnpeace.StartsWithCountMapper not org.apache.hadoop.mapreduce.Mapper
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.initializeBean(Abstract AutowireCapableBeanFactory.java:1422)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.doCreateBean(AbstractAu towireCapableBeanFactory.java:518)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.createBean(AbstractAuto wireCapableBeanFactory.java:455)
at org.springframework.beans.factory.support.Abstract BeanFactory$1.getObject(AbstractBeanFactory.java:2 93)
at org.springframework.beans.factory.support.DefaultS ingletonBeanRegistry.getSingleton(DefaultSingleton BeanRegistry.java:222)
at org.springframework.beans.factory.support.Abstract BeanFactory.doGetBean(AbstractBeanFactory.java:290 )
at org.springframework.beans.factory.support.Abstract BeanFactory.getBean(AbstractBeanFactory.java:192)
at org.springframework.beans.factory.support.DefaultL istableBeanFactory.preInstantiateSingletons(Defaul tListableBeanFactory.java:567)
at org.springframework.context.support.AbstractApplic ationContext.finishBeanFactoryInitialization(Abstr actApplicationContext.java:895)
at org.springframework.context.support.AbstractApplic ationContext.refresh(AbstractApplicationContext.ja va:425)
at org.springframework.context.support.ClassPathXmlAp plicationContext.<init>(ClassPathXmlApplicationCon text.java:139)
at org.springframework.context.support.ClassPathXmlAp plicationContext.<init>(ClassPathXmlApplicationCon text.java:83)
at mr.v1.spring.Main.main(Main.java:8)
Caused by: java.lang.RuntimeException: class mr.v1.spring.warnpeace.StartsWithCountMapper not org.apache.hadoop.mapreduce.Mapper
at org.apache.hadoop.conf.Configuration.setClass(Conf iguration.java:1661)
at org.apache.hadoop.mapreduce.Job.setMapperClass(Job .java:179)
at org.springframework.data.hadoop.mapreduce.JobFacto ryBean.afterPropertiesSet(JobFactoryBean.java:153)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.invokeInitMethods(Abstr actAutowireCapableBeanFactory.java:1479)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.initializeBean(Abstract AutowireCapableBeanFactory.java:1419)
... 12 more
Here is my spring-config.xml
And here is my pom.xml
Thanks for any pointers.
--Jens
I'm trying to using Spring Hadoop to execute MRv1 jobs on a 2.0.2-alpha cluster, but am having some challenges. I'm using the basic WordCount type example which works just fine as a regular job. It looks like something is expecting a MRv2 job, however, as I get the following error when I run my code:
Quote:
Exception in thread "main" org.springframework.beans.factory.BeanCreationExce ption: Error creating bean with name 'mr-job': Invocation of init method failed; nested exception is java.lang.RuntimeException: class mr.v1.spring.warnpeace.StartsWithCountMapper not org.apache.hadoop.mapreduce.Mapper
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.initializeBean(Abstract AutowireCapableBeanFactory.java:1422)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.doCreateBean(AbstractAu towireCapableBeanFactory.java:518)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.createBean(AbstractAuto wireCapableBeanFactory.java:455)
at org.springframework.beans.factory.support.Abstract BeanFactory$1.getObject(AbstractBeanFactory.java:2 93)
at org.springframework.beans.factory.support.DefaultS ingletonBeanRegistry.getSingleton(DefaultSingleton BeanRegistry.java:222)
at org.springframework.beans.factory.support.Abstract BeanFactory.doGetBean(AbstractBeanFactory.java:290 )
at org.springframework.beans.factory.support.Abstract BeanFactory.getBean(AbstractBeanFactory.java:192)
at org.springframework.beans.factory.support.DefaultL istableBeanFactory.preInstantiateSingletons(Defaul tListableBeanFactory.java:567)
at org.springframework.context.support.AbstractApplic ationContext.finishBeanFactoryInitialization(Abstr actApplicationContext.java:895)
at org.springframework.context.support.AbstractApplic ationContext.refresh(AbstractApplicationContext.ja va:425)
at org.springframework.context.support.ClassPathXmlAp plicationContext.<init>(ClassPathXmlApplicationCon text.java:139)
at org.springframework.context.support.ClassPathXmlAp plicationContext.<init>(ClassPathXmlApplicationCon text.java:83)
at mr.v1.spring.Main.main(Main.java:8)
Caused by: java.lang.RuntimeException: class mr.v1.spring.warnpeace.StartsWithCountMapper not org.apache.hadoop.mapreduce.Mapper
at org.apache.hadoop.conf.Configuration.setClass(Conf iguration.java:1661)
at org.apache.hadoop.mapreduce.Job.setMapperClass(Job .java:179)
at org.springframework.data.hadoop.mapreduce.JobFacto ryBean.afterPropertiesSet(JobFactoryBean.java:153)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.invokeInitMethods(Abstr actAutowireCapableBeanFactory.java:1479)
at org.springframework.beans.factory.support.Abstract AutowireCapableBeanFactory.initializeBean(Abstract AutowireCapableBeanFactory.java:1419)
... 12 more
Code:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:hdp="http://www.springframework.org/schema/hadoop"
xmlns:context="http://www.springframework.org/schema/context"
xsi:schemaLocation="
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
http://www.springframework.org/schema/hadoop http://www.springframework.org/schema/hadoop/spring-hadoop-1.0.xsd">
<context:property-placeholder location="classpath:hadoop.properties"/>
<hdp:configuration resources="classpath:/core-site.xml"/>
<hdp:job id="mr-job"
input-path="/training/data/war_and_peace.txt"
output-path="/training/playArea/wordCount/"
mapper="mr.v1.spring.warnpeace.StartsWithCountMapper"
reducer="mr.v1.spring.warnpeace.StartsWithCountReducer"/>
<!--
<hdp:job-runner id="myjob-runner" job="mr-job" run-at-startup="true"/>
<hdp:tool-runner id="myTool"
tool-class="mr.v1.spring.warnpeace.StartsWithCountJob"
run-at-startup="true">
<hdp:arg value="/training/data/war_and_peace.txt"/>
<hdp:arg value="/training/playArea/wordCount/"/>
</hdp:tool-runner>
-->
</beans>Code:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>hadoop-spring</groupId>
<artifactId>hadoop-spring</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<name>hadoop-spring</name>
<url>http://maven.apache.org</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-hadoop</artifactId>
<version>1.0.1.RC1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>2.0.0-mr1-cdh4.3.1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.0.0-cdh4.3.1</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
</dependencies>
<repositories>
<repository>
<id>cloudera-release</id>
<url>http://repository.cloudera.com/artifactory/cloudera-repos</url>
</repository>
<repository>
<id>spring-maven-milestone</id>
<name>Springframework Maven Milestone Repository</name>
<url>http://repo.springsource.org/milestone</url>
</repository>
</repositories>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>META-INF/spring.handlers</resource>
</transformer>
<transformer
implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>META-INF/spring.schemas</resource>
</transformer>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>mr.v1.spring.Main</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>--Jens