I am fond of IDEs. They make my life simpler, mainly because I am not fond of memorizing every thing. Code completion, syntax highlighting, run configurations, automatic code generation… Well, to be honest, ‘automated’ things is where I usually start getting aggravated. Not because they don’t work, but because they usually hide complex configurations and use ‘sensible defaults’ that are strongly opinionated and that blow everything up when you try to do things your way. In today’s post the culprit is Xtext.
Hiding behind curtain number one
There are two Xtext features that do tons of work, and one is particularly heavy on sensible defaults. The first one is the use of Xtend; although to be honest Xtext has tried to distance itself from Xtend as the Java API has made Xtend a bit redundant. The second one is the language generation via MWE2.
For Xtend, you have not probably realized that Xtend code has to be actually compiled into Java code. You probably have not realized it, because it is automatically done by the Xtend editor when you save your code. If curios, go ahead and delete the files in your xtend-gen folders. Because it is usually frowned upon to commit generated code, you will happily run your Tycho build just to find out that your build fails, if you are lucky, do to missing classes; if you are out of luck, your build will fail because ‘no tests where found’ and you would spend 2 days looking at your Xtend test, asking why you see the test in your editor and your build refuses to find it. Well, it can’t find any tests because your Xtend test classes have not been compiled.
For Xtext, you know that every time you change your language you need to run your ‘Generate XXX (yyy) Language Infrastructure’ run configuration. Unless you are using imported ecore metamodels or complex language mixins, you have probably never looked at the MWE2 file that Xtext creates for you. And even if you did, you probably made changes to the parts that do not affect the ‘sensible defaults’ that will ruin your week… OK, I am over reacting.
So the MWE2 file is responsible for generating the language infrastructure. This means it generates the ECore metamodel for your language, and all the required java classes for your language to work. If curios, go ahead and delete the files in your src-gen folders. Again, all those src-gen files should not be committed, so your Tycho build will fail do to missing classes. Thus, your Tycho build must run the MWE2 configuration… and here is where the sensible defaults will test your deduction skills.
Solving the Xtend compilation
There are some considerations and assumptions we need to get straight before you understand what we need to do to make it work. First things are simpler for the source bundles than for the tests. I have a feeling that the tycho-source-plugin has something to do with this; I assume it is responsible for the tweaks I had to do to make the tests work. Second, your test bundles SHOULD ONLY USE xTend sources. If you try to mix java with xtend tests, then you will have another problem in your hands as maven does not like projects with multiple source folders. There is an official guide on how to make it work; perhaps you can adapt his post to that scenario.
In both cases all you need to do is use the Xtend maven plugin. Easy right? Just need to add it to the poms of all the projects that use xtend. I am not a mojo expert, but from the source code of the xtend-maven-plugin, I think it is designed to work on a per-project basis. If your project is pom-less then you need to add poms to all the projects with xtend sources. In the pom, you need to add the xtend-maven-plugin. We are setting to paramters in the configuration, the encoding and the output dir. To keep it in sync with Eclipse, we will use the xtend-gen folder.
<project>
...
<build>
...
<pluginManagement>
<plugins>
<plugin>
<groupId>org.eclipse.xtend</groupId>
<artifactId>xtend-maven-plugin</artifactId>
<version>${xtext-version}</version>
<configuration>
<encoding>${project.build.sourceEncoding}</encoding>
<!-- need to prefix by basedir to generate to currently built module -->
<outputDirectory>${basedir}/xtend-gen</outputDirectory>
</configuration>
</plugin>
...
So for the source plugins that, is all there it’s to it. Now for the test plugins…
First, the xtend-maven-plugin runs during the testCompile phase for the test plugins. In that phase, maven will look for classes to compile (e.g. xtend) in the testCompileSourceRoots folder. In a typical Xtext project, that would be your src folder. So we could add the src folder to the project’s testCompileSourceRoots. However, not all projects have sources that need compilation in the src folder, so how can we avoid this? I found a helpful plugin called init-sources-maven-plugin. This plugin allows you to add folders to the testCompileSourceRoots on a per-project basis. Sweet!
The second part of the tweaks is that we genrated our java test files in the xtend-gen folder, so we need to tell maven where to find them. Fianaly, we need to make suer that the tycho-compiler-plugin is called after the xtend files are generated, and that the tycho-surefire-plugin finds the compiled test classes in the correct location. Additionally, I also cleaned the xtend-gen folder, to make sure remanent classes don’t hide compilation problems. The resulting pom for the TEST projects would be:
<build>
<testSourceDirectory>${project.basedir}/xtend-gen</testSourceDirectory>
<plugins>
...
<plugin>
<groupId>org.eclipse.xtend</groupId>
<artifactId>xtend-maven-plugin</artifactId>
<version>${xtext-version}</version>
<executions>
<execution>
<goals>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
<configuration>
<encoding>${project.build.sourceEncoding}</encoding>
<!-- need to prefix by basedir to generate to currently built module -->
<testOutputDirectory>${basedir}/xtend-gen</testOutputDirectory>
</configuration>
</plugin>
<plugin>
<groupId>io.github.lengors</groupId>
<artifactId>init-sources-maven-plugin</artifactId>
<version>1.0.0</version>
<executions>
<execution>
<goals>
<goal>init-sources</goal>
</goals>
<configuration>
<testCompileSourceRoots>
<testCompileSourceRoot>${project.basedir}/src</testCompileSourceRoot>
</testCompileSourceRoots>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-clean-plugin</artifactId>
<version>3.0.0</version>
<executions>
<execution>
<id>auto-clean</id>
<phase>clean</phase>
<goals>
<goal>clean</goal>
</goals>
<configuration>
<filesets>
<fileset>
<directory>${basedir}</directory>
<includes>**/xtend-gen/**</includes>
</fileset>
</filesets>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.eclipse.tycho</groupId>
<artifactId>tycho-compiler-plugin</artifactId>
<version>${tycho.version}</version>
<executions>
<execution>
<goals>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.eclipse.tycho</groupId>
<artifactId>tycho-surefire-plugin</artifactId>
<version>${tycho-version}</version>
<configuration>
<testClassesDirectory>target/test-classes</testClassesDirectory>
</configuration>
</plugin>
...
Solving the Xtext generation
TL;DR Execute the MWE2 from Tycho (maven).
But for this to work, we need a bit more work and some special considerations. First, credit where credit’s due. The Xtext project has a post on how to do this (which also includes information about Xtend). However, their guide assumes that you don’t deviate from the ‘sensible defaults’. So this are my considerations, and why the sensible defaults wont work:
- I want my project to use the ‘standard’ eclipse project layout, which is also required for Tycho pom-less to work. In this layout the bundles/plugins are in a separate folder than the tests.
- I should still be able to run the MWE2 generator from Eclipse.
Both require some modifications to the Xtext guide, because the guide assumes that the language bundles reside side-by-side with the tests.
Modifying the core language project’s POM
The first is a small modification and addition to the Tycho pom-less structured build layout:
- root folder – contains parent pom
- bundles (or plugins)
- bundle1
- bundle2
- xtextlanguange-bundle – language core project contains your grammar’s xtext file and the mwe2
- pom.xml – needed to run the MWE2 build
- …
- features
- sites
- products
- tests – contains tests bundles, including your xtext grammar and ui tests
- releng – contains a bundle with the target platform
- bundles (or plugins)
The changes are the inclusion of the tests folder, and the addition of a pom to the Xtext grammar plugin. The pom.xml for the language project contains information about how Maven should run the Xtext’s code generator. The first plug-in invokes the MWE2 file through a standard Java process:
<project>
...
<parent>
<relativePath>../../pom.xml</relativePath>
...
</parent>
<build>
...
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>3.5.0</version>
<executions>
<execution>
<id>mwe2Launcher</id>
<phase>generate-sources</phase>
<goals>
<goal>java</goal>
</goals>
</execution>
</executions>
<configuration>
<mainClass>org.eclipse.emf.mwe2.launch.runtime.Mwe2Launcher</mainClass>
<arguments>
<argument>/${project.basedir}/src/some/path/GenerateXXX.mwe2</argument>
<argument>-p</argument>
<argument>rootPath=/${project.basedir}/..</argument>
</arguments>
<classpathScope>compile</classpathScope>
<includePluginDependencies>true</includePluginDependencies>
<cleanupDaemonThreads>false</cleanupDaemonThreads><!-- see https://bugs.eclipse.org/bugs/show_bug.cgi?id=475098#c3 -->
</configuration>
<dependencies>
<dependency>
<groupId>org.eclipse.emf</groupId>
<artifactId>org.eclipse.emf.mwe2.launch</artifactId>
<version>${mwe-version}</version>
</dependency>
<dependency>
<groupId>org.eclipse.emf</groupId>
<artifactId>org.eclipse.emf.common</artifactId>
<version>${emf-version}</version>
</dependency>
<dependency>
<groupId>org.eclipse.xtext</groupId>
<artifactId>org.eclipse.xtext.common.types</artifactId>
<version>${xtext-version}</version>
</dependency>
<dependency>
<groupId>org.eclipse.xtext</groupId>
<artifactId>org.eclipse.xtext.xtext.generator</artifactId>
<version>${xtext-version}</version>
</dependency>
<dependency>
<groupId>org.eclipse.xtext</groupId>
<artifactId>org.eclipse.xtext.xbase</artifactId>
<version>${xtext-version}</version>
</dependency>
<dependency>
<groupId>org.eclipse.xtext</groupId>
<artifactId>xtext-antlr-generator</artifactId>
<version>[2.1.1, 3)</version>
</dependency>
</dependencies>
</plugin>
...
</plugins>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.eclipse.m2e</groupId>
<artifactId>lifecycle-mapping</artifactId>
<version>1.0.0</version>
<configuration>
<lifecycleMappingMetadata>
<pluginExecutions>
<pluginExecution>
<pluginExecutionFilter>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<versionRange>[1.2.1,)</versionRange>
<goals>
<goal>java</goal>
</goals>
</pluginExecutionFilter>
<action>
<ignore></ignore>
</action>
</pluginExecution>
</pluginExecutions>
</lifecycleMappingMetadata>
</configuration>
</plugin>
</plugins>
</pluginManagement>
...
It’s basically the same as in the Xtext guide, with two small additions. One, the referencing the root pom as a parent, as we are using a pom-less build so there are no other poms. Make sure to use your correct root coordinates (groupId, artifactId and version). The second addition is done to map the exec-maven-plugin (used for generating the grammar) to the java goal. This ensures that the required test classes are generated before the testing phase. Another task that we wanted to have is cleaning the generated source and model folders. We used the maven-clean-plugin for this (replace your plugin paths accordingly):
<project>
...
<plugins>
...
<!-- Cleans the directories containing generated resources during the clean phase -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-clean-plugin</artifactId>
<version>3.4.0</version>
<configuration>
<filesets>
<fileset>
<directory>plugins/core.language/src-gen/</directory>
<includes>
<include>**/*</include>
</includes>
</fileset>
<fileset>
<directory>plugins/core.language.ide/src-gen/</directory>
<includes>
<include>**/*</include>
</includes>
</fileset>
<fileset>
<directory>plugins/core.language.ui/src-gen/</directory>
<includes>
<include>**/*</include>
</includes>
</fileset>
<fileset>
<directory>tests/core.language.tests/src-gen/</directory>
<includes>
<include>**/*</include>
</includes>
</fileset>
<fileset>
<directory>tests/core.language.ui.tests/src-gen/</directory>
<includes>
<include>**/*</include>
</includes>
</fileset>
<fileset>
<directory>plugins/core.language/model/generated/</directory>
<includes>
<include>**/*</include>
</includes>
</fileset>
</filesets>
</configuration>
</plugin>
...
The final part of the changes will allow the MWE2 to work with the new folder structure both from the Tycho build and from within Eclipse.
Tweaking the Language Generator
The language generation section of the Xtext documentation, states that the default MWE configuration “uses two convenience classes StandardProjectConfig and StandardLanguage, both of which are designed to apply default configurations that work for the majority of language projects.” So basically that if you are not using a standard project, they will not work. In our case, what fails is that the StandardProjectConfig assumes that the language and tests projects are in the same folder. Since we moved the test folders, then it wont be able to find them during the build.
Looking at the StandardProjectConfig, the assumption of the projects in the same location is used to generate the ‘root’ value for the nested SubProjectConfig. And since I assume you did not understand any of that, let me explain with an example MWE:
...
Workflow {
component = XtextGenerator {
configuration = {
project = StandardProjectConfig {
baseName = "my.text.language"
rootPath = rootPath
runtimeTest = {
enabled = true
}
...
What that configuration does is use the baseName to look for all the Xtext bundles of your language; your core bundle will use that name. Next, the runtimeTest part instructs the generator that you have a test plugin for your language. Looking at the StandardProjectConfig implementation, it will use baseName + ".tests" to find the test bundle. Since that value actually translates to a path, it will effectively try to find the my.text.language.tests bundle in the same folder as your core bundle. Looking a bit more into the source code, I found that the runtimeTest (and other nested configurations) are instantiated as SubProjectConfigs. And, SubProjectConfigs have a root attribute we can specify, to change the location/path where the bundle is located. So, we can modify the MWE workflow to make the paths explicit.
...
Workflow {
component = XtextGenerator {
configuration = {
project = StandardProjectConfig {
baseName = "my.text.language"
rootPath = rootPath
runtimeTest = {
root = "tests/my.text.language.tests"
}
eclipsePlugin = {
enabled = true
}
eclipsePluginTest = {
enabled = true
root = "tests/my.text.language.ui.tests"
}
...
Finally, note that the paths we provide are relative to the repository root. This is required because when running the generator from Tycho, the working directory is the root folder. In order to make the modified MWE also work from eclipse, we need to make sure the Run Configuration also runs from the root folder. In the Arguments tab of the run configuration, you can change the working directory, as shown below:
Those pesky Eclipse dependencies
The final piece of the code is access to eclipse plugins from Tycho (maven). From my experience, it is not always easy to find Eclipse projects’ plugins in the maven repositories (it takes some effort to separate the pure java from the plugin code, and to configure your build to publish your plugins to a maven repo). In particular, the Ecore ones, AKAIK, are not officially published by the EMF project. Non the less, you can find them in the sonatype repositories. So in my root pom, I added the snapshot and releases repos, just in case:
...
<repositories>
<repository>
<id>sonatype-snapshots</id>
<url>https://oss.sonatype.org/content/repositories/snapshots</url>
<releases><enabled>false</enabled></releases>
<snapshots><enabled>true</enabled></snapshots>
</repository>
<repository>
<id>sonatype-releases</id>
<url>https://oss.sonatype.org/content/repositories/releases</url>
<releases><enabled>true</enabled></releases>
<snapshots><enabled>false</enabled></snapshots>
</repository>
</repositories>
An that should get you going! Happy building :).