Building Spark 1.0.1 against HDP 2.1.3

1. You will need SCALA_HOME & JAVA_HOME set.
2. Get Spark 1.0.1 source
3. Add following to the root Spark pom.xml
<repository> <id>horton-repo</id> <name>Hortonworks Repository</name> <url>http://repo.hortonworks.com/content/repositories/releases/</url> <releases> <enabled>true</enabled> </releases> <snapshots> <enabled>false</enabled> </snapshots> </repository> <protobuf.version>2.5.0</protobuf.version> 4. .Build spark tarball using

export SCALA_HOME=~/tools/scala-2.10.4 (whatever is appropriate for your env)

export MAVEN_OPTS=”-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m”
$MAVEN_HOME/bin/mvn -q versions:set -DnewVersion=1.0.1.2.1.3.0-563
$MAVEN_HOME/bin/mvn -Pyarn -Phadoop-2.4 -DskipTests clean package
./make-distribution.sh –tgz –hadoop 2.4.0.2.1.3.0-563 –with-yarn