Synchronize Subversion to CA AllFusion Harvest

For a long time I’ve been searching how to synchronize code from a Subversion code repository to a CA AllFusion Harvest code repository.  Finally, I decided to just write a script to do it myself.  The script takes one argument, which is the Harvest package name.  There are other constants you’ll need to set within the script, such as the Subversion repository and Harvest username and password.  You can easily change those to be a script parameter, too.

We’re using this script right now with our Jenkins (formerly Hudson) CI server so a new Harvest package is created (named by the %JOB_NAME% variable set by Jenkins) with each Production build is run.

@echo off

REM ###########################################################################
REM #
REM # Script file to move changes from Subversion to Harvest
REM #
REM ###########################################################################

if "%1" == "" goto usage

setlocal

set PROJECT_STAGE=-b "" -en "" -st ""
set VIEW=-vp ""
set CREDENTIALS=-usr "" -pw ""
set SUBVERSION_REPO=svn:////trunk/

REM Clean up any build artifacts if present
call ant clean

REM Create Harvest package
hcp %1 %PROJECT_STAGE% %CREDENTIALS%

REM Delete the checked-out Subversion code
REM Note: You will need to remove everything (except this file of course), so more rmdir or del statements may be required below
rmdir /S /Q project

REM Check out the files in Harvest to modify
hco * %PROJECT_STAGE% %CREDENTIALS% %VIEW% -p "%1" -pn "Check Out Files for Update" -up -r -s * -op pc -cp %CD%

REM Delete the checked-out Harvest code
REM Note: You will need to remove everything (except this file of course), so more rmdir or del statements may be required below
rmdir /S /Q project

REM Replace with the latest code from Subversion repository
svn co %SUBVERSION_REPO% .

REM Delete the .svn directories
for /f "tokens=* delims=" %%i in ('dir /s /b /a:d *svn') do (
  rd /s /q "%%i"
)

REM What are the updates for Harvest?  Check them into Harvest
hci * %PROJECT_STAGE% %CREDENTIALS% %VIEW% -p "%1" -pn "Check In Modified Files" -s * -ur -de "Added in Subversion" -if ne -op pc -cp %CD%

REM Remove the log files from Harvest
REM Note: You may need to change or remove this statement depending on whether you want the Harvest logs checked in
hdv *.log %PROJECT_STAGE% %CREDENTIALS% %VIEW% -pn "Delete Versions"

REM What removals from Harvest do we need to process? (What files were deleted in Subversion?)
hsv %PROJECT_STAGE% %CREDENTIALS% %VIEW% -p "%1" -it r -s "*"

REM This will not work if the file path has spaces in it.  You can use %%j %%k ... in the -vp switch for one space in your project name (For example, if you have 2 spaces, it should be -vp %%j %%k %%l)
for /f "tokens=1-5 skip=3" %%i in (hsv.log) do (
	if not "%%i"=="hsv" (
		hci "%%i" %PROJECT_STAGE% %CREDENTIALS% -vp "%%j" -p "%1" -pn "Check In Modified Files" -ro -cp %CD%
		hri "%%i" %PROJECT_STAGE% %CREDENTIALS% -vp "%%j" -p "%1"
	)
)

REM remove read-only attribute from all files
attrib -r -h * /s

REM delete all harvest.sig files
del /f /s /q harvest.sig

endlocal

goto end

:usage

echo USAGE:
echo -----------------------------------------------------------------------------
echo "svn2harvest {package_name}"
echo -----------------------------------------------------------------------------

:end

As you can see, I’m using the command line utilities for both Subversion and Harvest to make this happen so you’ll need both installed.

Let me know if you see anything that could be improved or have any other questions about it.

Building Axis2 Java Classes From WSDLs Using Ant Foreach

I’ve had a few requests to get an example of the build script I referenced in my post, Ant Foreach Properties, so here it is.

In our ant build.xml file we have a task called buildAllWsdls which gets all of the WSDL files from the web service URL, then feeds each of them into Axis2 to convert them to Java classes.

Here are the basic steps of what this build does:

  1. Finds each property which ends with “Service” and concatenates them into a comma delimited string (ex. ATestService, BTestService, CTestService…)
    <propertyselector property="services.list" delimiter="," match="(w)*Service" />
  2. Loop through each of those Service properties using the foreach task and get the WSDL file from the WSDL end point URL.
    <foreach list="${services.list}" inheritall="true" target="-getSingleWsdl" delimiter="," param="prettyName" />
  3. Translate the namespace to package properties file into a comma delimited list (as required by Axis 2).
    <loadfile srcfile="${conf.dir}/NStoPkg.properties" property="ns2p.all">
         <filterchain>
             <striplinecomments>
                 <comment value="//" />
             </striplinecomments>
             <prefixlines prefix="," />
             <striplinebreaks />
             <tokenfilter>
                 <trim />
                 <ignoreblank />
             </tokenfilter>
         </filterchain>
     </loadfile>
  4. Run each WSDL in the download directory through the Axis2 Java class creation Ant task.
     <foreach target="-createStubs" param="file" inheritall="true">
         <path>
             <fileset dir="${wsdldir}" />
         </path>
     </foreach>
    
    <target name="-createStubs">
        <basename property="wsdl" file="${file}" />
        <echo message="Generating client code for service: ${wsdl}" />
        <java fork="true" classname="org.apache.axis2.wsdl.WSDL2Java" failonerror="true">
            <arg line="-ns2p ${ns2p.all} -s -u -uri ${wsdldir}/${wsdl} -o ${stubsdir} --noBuildXML" />
            <classpath refid="cp.axis" />
        </java>
    </target>

If you don’t mind your web services being in a Java package that corresponds with the web service namespace, you can skip step 3 and leave out the -ns2p command line argument in the java ant task in step 4.

Here is the full example Ant build.xml file for doing this:

<project name="Axis2WSDLBuild" default="buildAllWsdls" basedir=".">
 <tstamp>
 <format property="TODAY" pattern="MM/dd/yyyy HH:mm:ss"/>
 </tstamp>

 <property name="ws-domain" value="http://services.test.com:8080" />
 <property name="webservices.dir" value="${app.dir}/axisStubsProject" />
 <property name="wsdldir" value="${webservices.dir}/wsdls" />
 <property name="stubsdir" value="${webservices.dir}" />

 <!-- patternsets -->
 <patternset id="jar.files">
 <include name="**/*.jar"/>
 </patternset>

 <path id="cp.axis">
 <fileset dir="${lib.dir}/axis">
 <patternset refid="jar.files"/>
 </fileset>
 </path>
 <path id="cp.antcontrib">
 <fileset dir="${lib.dir}/antcontrib">
 <patternset refid="jar.files"/>
 </fileset>
 </path>

 <!-- Allow usage of ant-contrib defined tasks -->
 <taskdef resource="net/sf/antcontrib/antlib.xml">
 <classpath refid="cp.antcontrib"/>
 </taskdef>

 <!-- =================================================================== -->
 <!-- Project initialization                                              -->
 <!-- =================================================================== -->
 <target name="init">
 <property name="version" value="build ${TODAY}"/>
 <property file="${conf.dir}/soa-addresses.properties"/>
 <filter token="version" value="${version}"/>
 <filter token="today" value="${TODAY}"/>
 </target>

 <!--=======================================================-->
 <!-- TARGET [buildAllWsdls]                                -->
 <!-- Target which gets and creates the web service stubs necessary for client invocation. -->
 <!--=======================================================-->
 <target name="buildAllWsdls" depends="init">
 <antcall target="wsdlGet" />
 <antcall target="buildWsdls" />
 </target>

 <!--=======================================================-->
 <!-- TARGET [buildWsdls]                                -->
 <!-- Target which creates the web service stubs necessary for client invocation. -->
 <!--=======================================================-->
 <target name="buildWsdls">
 <!-- create the download directory -->
 <mkdir dir="${stubsdir}" />
 <!--
 Translate the namespace to package properties file (as typically used in
 an Axis 1 implementation) into a comma delimited list (as required by
 Axis 2).
 -->
 <loadfile srcfile="${conf.dir}/NStoPkg.properties" property="ns2p.all">
 <filterchain>
 <striplinecomments>
 <comment value="//" />
 </striplinecomments>
 <prefixlines prefix="," />
 <striplinebreaks />
 <tokenfilter>
 <trim />
 <ignoreblank />
 </tokenfilter>
 </filterchain>
 </loadfile>
 <!-- run client generation for each WSDL in the download directory -->
 <foreach target="-createStubs" param="file" inheritall="true">
 <path>
 <fileset dir="${wsdldir}" />
 </path>
 </foreach>
 </target>

 <!--=======================================================-->
 <!-- TARGET [wsdlGet]                                -->
 <!--  Target retrieving all the WSDL files necessary for generating client code. -->
 <!--=======================================================-->
 <target name="wsdlGet">
 <!-- create the download directory -->
 <mkdir dir="${wsdldir}" />

 <!-- find the properties related to WSDL locations -->
 <propertyselector property="services.list" delimiter="," match="(w)*Service" />

 <!-- for each property found, retrieve the WSDL -->
 <foreach list="${services.list}" inheritall="true" target="-getSingleWsdl" delimiter="," param="prettyName" />

 </target>

 <!--=======================================================-->
 <!-- TARGET [-getSingleWsdl]                                -->
 <!-- Hidden Target to retrieve a single WSDL. -->
 <!--=======================================================-->
 <target name="-getSingleWsdl">
 <!-- get the property value -->
 <propertycopy property="fullPath" from="${prettyName}" />
 <!-- download the WSDL file -->
 <get src="${ws-domain}/${fullPath}?wsdl" dest="${wsdldir}/${prettyName}.wsdl" verbose="true" usetimestamp="true" />
 </target>

 <!--=======================================================-->
 <!-- TARGET [-createStubs]                                -->
 <!-- Hidden target to run WSDL2Java on an input file. -->
 <!--=======================================================-->
 <target name="-createStubs">
 <!-- strip the full path from the file -->
 <basename property="wsdl" file="${file}" />
 <echo message="Generating client code for service: ${wsdl}" />
 <java fork="true" classname="org.apache.axis2.wsdl.WSDL2Java" failonerror="true">
 <arg line="-ns2p ${ns2p.all} -s -u -uri ${wsdldir}/${wsdl} -o ${stubsdir} --noBuildXML" />
 <classpath refid="cp.axis" />
 </java>
 </target>

</project>

the soa-addresses.properties file, which contains the different web service URLs:

ATestService=services/ATestService
BTestService=services/BTestService
CTestService=services/CTestService
DTestService=services/DTestService
ETestService=services/ETestService

and the NStoPkg.properties file which converts the default web service namespace based off the WSDL URL to a specified java package:

http://typens.test.com/common=com.test.axis.common.types
http://servicens.test.com/common/ATestService=com.test.axis.common.services.atestservice
http://servicens.test.com/common/BTestService=com.test.axis.common.services.btestservice
http://servicens.test.com/common/CTestService=com.test.axis.common.services.ctestservice
http://servicens.test.com/common/DTestService=com.test.axis.common.services.dtestservice
http://servicens.test.com/common/ETestService=com.test.axis.common.services.etestservice

Ant Foreach Properties

Update 10/23/2009: An example of this build has been posted.

I’ve been working with Apache Ant a bit at work the past couple weeks. We’ve been setting up a Hudson continuous integration server to automate some of our builds and deployments. We’re also hoping to setup more automated tests using tools like Selenium, HTTPUnit, JUnit, etc., but those are going to be gradually added with the coming releases.

I’ve worked with Ant a lot before, but hadn’t really used the Ant-Contrib Tasks until modifying this build script. The foreach task is useful for running a list of things through a certain ant target. We have a properties file that has the webservice name and the associated WSDL URL. We feed that list to an Ant target through the foreach task (for each URL, go get the WSDL file/definition), then we generate Axis2 stubs from each WSDL file. To automate the generation/retrieving of these WSDL files I was moving the base URL (ex: http://services.test.com/) to the environment property file, since this is different in each environment (Test, QA, Production), but the relative URL (ex: /services/…) stays the same. So to summarize, the webservices properties file now has ExampleService=/services/ExampleService and the environment properties file has SERVICES-DOMAIN=http://services.test.com (SERVICES-DOMAIN is assigned to the ws-domain after the environment properties file is loaded in the ant build file).

Here’s where I ran into a little issue that I wanted to share so no one else has to spend a half hour debugging it. Well, also so I can find the answer here the next time I run into it. The property for the base URL wasn’t evaluating when I was running the WSDL list through the foreach call to get/download the WSDL file. It kept erroring saying ${ws-domain}/services/ExampleService?wsdl couldn’t be found. I spent a few minutes putting echo’s in the ant script to see where it was failing until I saw ${ws-domain} was set up until the ant target that gets the WSDL file. It was then I saw this foreach task attribute:

Attribute Description
inheritall If true, pass all properties to the called target. Defaults to false.

Now I’m not sure why you wouldn’t want properties passed, but I’m sure there is some good reason why this is false by default. Maybe. So of course, adding this attribute set as true to the foreach ant task made everything work without a hitch.