Detail
Advanced JavaCC Task for Ant
This case relates specifically to Ant 1.6.2 and JavaCC 3.2.
The source code for this new task is included in the text of this case, AND attached as a file at the end (no need to copy and past from the browser window if you don't wish to). For more information, contact support at jmatrix dot net or bemowski at yahoo dot com.
The Problem:
Ant contains an optional JavaCC task. The Task is functional, however it has several drawbacks:
It cannot scan a directory, and generate parsers for all .jj files. Each file must be specified individually to the task.
It cannot specify a relative path for generated parser output. The path must be absolute, including the package name of the generated parser. It would be nice to specify a base directory (like .../generated) and allow the task to determine the fully qualified package name based on the package specified in the .jj file (as in package foo.bar; relating to an absolute output diretory of .../generated/foo/bar).
The Task included with Ant 1.6.2 (and earlier) will only correctly check if the file is up to date if the name of the parser defined with then .jj file is the same as the basename of the .jj file. This is minor, however it could cause problems in some situations.
The Task included with Ant 1.6.2 internally uses the Java command line execution utilities of Ant. So for each javac task in a buildfile, it will spawn a new Java Virtual Machine to generate that parser. This is expensive, time consuming, and un-necessary (if we do not require backward compatability before JavaCC 3.2). It can be especially expensive if the up to date method does not function for reasons in the previous bullet point.
All of these reasons would probably add up to a minor nuisence for most smaller projects. However with more sophisticated build systems, where the build files are independent of the source they are building (as in a modular build system), then we cannot explicitly specify the .jj file and its output directory without severely compromising the elegance of the build system.
The Solution
The straightforward solution is to create a new Ant task that corrects the problems stated above. If we lock ourselves to JavaCC 3.2, we can eliminate some of the hacks to deal with backward compatability.
The new Task, called JavaCCRecursive (because it can use filesets to recurse into subdirectories) uses the JavaCC code to parse the .jj file twice. First, in a 'pre-parse' step, we extract the package and parser class name informaiton. Then, check if the file is up-to-date. Finally, if it is not, then we generate the parser to the output directory by invoking the JavaCC Main.mainProgram method directly (ie, within the context of the current JVM).
/*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/
package com.jetools.build.tasks.javacc;
import java.io.*;
import java.util.Vector;
import java.util.Date;
import java.text.SimpleDateFormat;
// Ant imports
import org.apache.tools.ant.*; // Project, Task
import org.apache.tools.ant.types.FileSet;
// JavaCC 3.2 imports
import org.javacc.parser.*;
import org.javacc.parser.Main; //explicit import because of conflict with ant
/*
* $Log: JavaCCRecursive.java,v $
* Revision 1.1 2005/01/04 20:57:28 bemocvs
* *** empty log message ***
*
*/
/**
* Searches a nested fileset element for javacc files, running javacc
* on each element in turn. <p>
*
* The motivation for this task is simple: the current (ant 1.6.2) javacc
* task requires the user to <u>explicitly</u> define the location of
* javacc, and the name of a <u>single</u> javacc source file (commonly
* suffixed with .jj). This is unacceptable for more sophisticated
* build systems, where generic build files do not have specific knowledge
* of each individual target file. <p>
*
* For example, when we use the <code>javac</code> task, we do not
* specifically define each and every .java file, and the javac class
* to be used to compile them. We define a fileset, and take javac from
* the local environment. This task is designed to do exactly the same
* with javacc. <p>
*
* As nearly all methods in JavaCC 3.2 are static and stateful, this
* task is not threadsafe. I do not know of many instances where multiple
* threads are accessing Ant concurrently, however it is worth mentioning. <p>
*
* <b>This task was developed to work with JavaCC 3.2. There are no
* guarntees it will work with earlier or later versions of JavaCC without
* maintenance work.</b>
*
* @author Paul Bemowski (bemowski at yahoo dot com)
*/
public class JavaCCRecursive extends Task {
// GLOBAL TODO (FIXME):
// * Where available, translate javacc's file, line and column
// number information into Location objects so that Ant reports
// JavaCC errors with file/line/column accuracy.
/** The directory root where javacc files will be placed. Files
* will be place in a directory heirarchy below this directory
* corresponding to the package naming of the .jj parser. */
File outputdirectory=null;
/** Storage for filesets. */
Vector filesets=new Vector();
/** The name of the main class file. This is used on init() to
* confirm that javacc is in the classpath. (see the init() method). */
String javaCCMain="org.javacc.parser.Main";
/** */
public void init()
throws BuildException {
project.log("Initializing JavaCC task.", project.MSG_DEBUG);
// First, check to make sure the user has the correct
// javacc Main class in their classpath
try {
Class.forName(javaCCMain);
} catch (ClassNotFoundException ex) {
throw new BuildException("ERROR: To run JavaCC you must have the "+
"javacc.jar class file in your ant "+
"classpath.", ex);
}
project.log("Done initializing.", project.MSG_DEBUG);
}
public void setOutputdirectory(File f) {
outputdirectory=f;
}
/**
* Adds a set of files.
*/
public void addFileset(FileSet set) {
filesets.addElement(set);
}
/** */
public void execute()
throws BuildException {
project.log("Executing JavaCCRecursive.", project.MSG_WARN);
if (outputdirectory == null) {
throw new
BuildException("JavaCC Recursive requires a base output directory.");
}
//System.out.println ("Have "+filesets.size()+" filesets");
for (int i=0; i<filesets.size(); i++) {
FileSet fileset=(FileSet)filesets.get(i);
project.log(" Fileset: "+fileset, project.MSG_DEBUG);
DirectoryScanner ds=fileset.getDirectoryScanner(this.project);
// ds.scan(); // don't do this, it was already done inside fileset
String files[]=ds.getIncludedFiles();
for (int j=0; j<files.length; j++) {
File javaccFile=new File(ds.getBasedir(), files[j]);
ParserInfo parserInfo=getParserInfo(javaccFile);
String javaccPackageDir=
parserInfo.getPackageName().replace('.', '/');
project.log("Found package: "+parserInfo.getPackageName(),
project.MSG_DEBUG);
String outputDir=
(new File(outputdirectory, javaccPackageDir)).getAbsolutePath();
project.log("Output dir is "+outputDir, project.MSG_DEBUG);
// JavaCC main does a File.mkdir() instead of File.mkdirs().
// mkdir() will not create parent directories as necessary.
// So, we're going to help javacc out a bit by explicitly
// making the output directory if necessary.
createOutputDir(outputDir);
if (upToDate(javaccFile, outputDir, parserInfo.getParserName())) {
// upToDate will log some debug level stuff.
return;
}
// Ok. Now, we've gleaned the package directory relative
// to the output directory. We are ready to execute javacc.
// To do this, we'll hit its frontend method, the same
// as if we called it from the command line. So we need
// to form the args[] array. For info see the command
// line help from javacc 3.2, or view the org.javacc.parser.Main
// class source for detail.
// FIXME: Handle additional command line arguments
Vector commandLineArgs=new Vector();
commandLineArgs.add("-OUTPUT_DIRECTORY="+outputDir);
commandLineArgs.add(javaccFile.getAbsolutePath());
String args[]=(String[])commandLineArgs.toArray(new String[0]);
int result=-1;
try {
// pre-parse data will be reset when the internal reInit()
// method is called from Main.mainProgram().
result=Main.mainProgram(args);
} catch (Exception ex) {
throw new
BuildException("JavaCC main threw exception.", ex);
}
project.log("javacc exit result is "+result, project.MSG_WARN);
if (result != 0) {
throw new
BuildException("JavaCC failed with error "+result+
" on "+files[i], getLocation());
}
}
}
project.log("JavaCCRecursive complete.", project.MSG_WARN);
}
/**
* Creates a fully qualified directory name, if it does not exist.
* Throws a build exception if the directory cannot be created.
*/
void createOutputDir(String absDir)
throws BuildException {
File dir=new File(absDir);
if (!dir.exists()) {
project.log("Creating output directory: "+absDir, project.MSG_WARN);
boolean success=dir.mkdirs();
if (!success) {
throw new BuildException("Cannot create output directory: "+
absDir);
}
}
if (dir.exists()) {
if (!dir.isDirectory()) {
// the file exists, but it is not a directory.
throw new BuildException("There appears to be a file with the "+
"same name as the output directory. "+
"Remove it. "+
absDir);
}
if (!dir.canWrite()) {
throw new BuildException("Cannot write to output dir: "+
absDir);
}
}
}
/** Runs a 'pre-parse' on the javacc .jj input file to determine its
* package name. The package name is then used to build the
* fully qualified output directory name, which is used as input
* to the actual parse operation. <p>
*
* This code was adapted from the driver code in JavaCC(3.2)'s
* Main.mainProgram(args[]) method.
*
* @return Package name of parser to be generated.
*/
ParserInfo getParserInfo(File jjInputFile)
throws BuildException {
String filename=jjInputFile.getAbsolutePath();
// Re-initialize all elements within the parser.
Main.reInitAll();
JavaCCGlobals.bannerLine("Parser Generator", "");
Options.JavaCCInit();
JavaCCParser parser = null;
try {
if (!jjInputFile.exists()) {
throw new BuildException("Parser input file '"+
jjInputFile.getAbsolutePath()+
"' does not exist.");
}
if (jjInputFile.isDirectory()) {
throw new BuildException("Parser input file '"+
jjInputFile.getAbsolutePath()+
"' is a directory.");
}
parser = new JavaCCParser(new java.io.FileReader(jjInputFile));
} catch (NullPointerException ne) { // Should never happen
throw new
BuildException("Caught null pointer constructing parser", ne);
} catch (SecurityException se) {
throw new
BuildException("Security voilation while trying to open " +
filename, se);
} catch (java.io.FileNotFoundException e) {
throw new BuildException("FileNotFoundException reading " +
filename, e);
}
try {
project.log("Reading from file " +
jjInputFile.getAbsolutePath() + " . . .",
project.MSG_DEBUG);
JavaCCGlobals.fileName = JavaCCGlobals.origFileName = filename;
// JavaCCGlobals.jjtreeGenerated =
// JavaCCGlobals.isGeneratedBy("JJTree", filename);
// JavaCCGlobals.jjcovGenerated =
// JavaCCGlobals.isGeneratedBy("JJCov", filename);
JavaCCGlobals.toolNames = JavaCCGlobals.getToolNames(filename);
parser.javacc_input();
} catch (MetaParseException e) {
throw new
BuildException("Detected "+JavaCCErrors.get_error_count() +
" errors and "+JavaCCErrors.get_warning_count() +
" warnings.", e);
} catch (ParseException e) {
throw new
BuildException("Detected " + JavaCCErrors.get_error_count() +
" errors and "+JavaCCErrors.get_warning_count() +
" warnings.", e);
}
// Ok, by the time we get here, we should have parsed the file.
// this give the necessary state to the ParserInfoFinder via static
// superclass variables in JavaCCGlobals
return ParserInfoFinder.getParserInfo();
}
/**
* Everything in the parser is static, so by extending JavaCCGlobals
* we have access to the superclass static variables that were
* set by parser.javacc_input() (see above). Since the only method
* in this class is stateless, there is no work to be done on a reInit().
*/
static class ParserInfoFinder
extends JavaCCGlobals
implements JavaCCParserConstants {
// For reference on what's going on in here, see the internal
// classes in the org.javacc.parser. especially subclasses
// of JavaCCGlobals (JavaFiles, etc).
/** */
public static ParserInfo getParserInfo() {
return new ParserInfo(getPackage(), getParserClassName());
}
/** Returns the package defined in a JavaCC .jj input file. */
public static String getPackage() {
StringBuffer pack=new StringBuffer();
if (cu_to_insertion_point_1.size() != 0 &&
((Token)cu_to_insertion_point_1.elementAt(0)).kind == PACKAGE)
{
for (int i = 1; i < cu_to_insertion_point_1.size(); i++) {
Token token=(Token)cu_to_insertion_point_1.elementAt(i);
if (token.kind == SEMICOLON) {
// Bemo: element(0) is 'package'. Changing index to 1
// element(i) is ';'. Changing index to i-1.
for (int j = 1; j <= i-1; j++) {
Token anotherToken=
(Token)cu_to_insertion_point_1.elementAt(j);
pack.append(anotherToken.toString());
}
break;
}
}
}
return pack.toString();
}
/** Returns the Parser class name, for date time comparison.
* This is the parser name, sans the '.java' part.
*/
public static String getParserClassName() {
return cu_name; // from the superclass
}
} // end class PackageFinder
/**
* A small utility class to hold 2 pieces of data from the
* pre-parse step. It stores the packagename used to generate
* the fully qualified output directory name, and the ParserName
* which becomes the parser class file name, for up-to-date comparision.
*/
static class ParserInfo {
String packageName=null;
String parserName=null;
public ParserInfo(String packName, String parName) {
this.packageName=packName;
this.parserName=parName;
}
public String getPackageName() {return packageName;}
public String getParserName() {return parserName;}
}
/**
* Checks if the generated parser file is up to date
* relative to the input file. Based on File.lastModified().
*/
boolean upToDate(File inputFile, String outputDir, String parserName) {
File outputFile=new File(outputDir, parserName+".java");
if (outputFile.exists() &&
outputFile.lastModified() > inputFile.lastModified()) {
SimpleDateFormat df=new SimpleDateFormat("MM/dd/yyyy HH:mm:ss");
String message=
"upToDate: \n"+
" output ("+df.format(new Date(outputFile.lastModified()))+"): "+
outputFile.getAbsolutePath()+"\n"+
" input ("+df.format(new Date(inputFile.lastModified()))+"): "+
inputFile.getAbsolutePath()+"\n"+
" output newer than input, javacc up to date.";
project.log(message, project.MSG_DEBUG);
project.log("Parser is up to date. "+
"JavaCCRecursive Task not regenerating parser.",
project.MSG_WARN);
return true;
}
return false;
}
}
|
