If you know where to look you can step through pretty much every line of code that runs on an AEM instance. This article contains details on where to look and how to look there.
Figuring out how stuff works is hard. When you’re working on AEM customizations, sometimes you need to know how something works at a level that is much deeper than the information provided by the documentation from your peers or even the source code itself. In this article I offer multiple multiple strategies to get to that deeper level. With this you can observe your code from multiple angles as it executes and determine exactly why certain behaviors are occurring.
Tailing the logs
Before we begin getting our hands dirty, a quick recommendation: When I’m developing AEM code I am always tailing the error logs for my active instances on a screen that is visible while I work. With this you can often catch exceptions as they occur by the unique indentation they have. You can see if an infinite loop has been created when the logs ramp up speed unexpectedly. With experience you can catch other issues too.
Again I recommend tailing the error logs at all times during development. For most instances the logs are in crx-quickstart/logs with error.log being the most important. Note that all of the loggers open a new file each day. If you happen to be tailing when this happens, most apps will not correctly make the switch to the new file. You will need to halt the tail and restart it.
For my setup I run 3 monitors – one with my browser, one with my IDE, and one with iTerm2. On iTerm2 I’m running tmux with a pane to run builds and git commands, and one pane each for the error.logs of the running AEM instances I have. I use less +F rather than tail so that I can ^c to halt the follow and go back to lines that have passed by. I start everything (including the AEM instances!) using a tmuxp profile each day (I recommend turning your computer off completely at the end of the day). You may find another setup that works for you.
Tracing a Response to its Servlet
If you are checking behavior for which you do not already have a stack trace, the first thing you need to do is find the handler for that behavior. In AEM the quickest and easiest way to do that for most behaviors is to know your Sling resolution basics. Adobe’s documentation has a great cheat sheet. There is a great deal of more detailed information in the Sling documentation.
Tracking down the correct jsp, HTL template, or other Servlet manually can still be a lot of work. Luckily the OSGi console offers a special utility that can take a request path and tell you what is responding. That is located at /system/console/servletresolver. Note that if you are looking at a page the responding script will almost always be /libs/cq/Page/Page.jsp. That script generally just rewrites the request to use the underlying jcr:content node to respond. So to short cut this change /content/page/path.html to /content/page/path/_jcr_content.html when using this form. However in most cases you can pass the direct path to a component to get narrower results, such as /content/page/path/_jcr_content/header.html.
Another important thing to know when tracing a response back to its servlet is the various locations where the request can be rewritten. I recommend bypassing the Dispatcher when performing work like this, but it is important to know that the Dispatcher is one source of rewrites. There are several others within AEM (nodes in the jcr, an OSGi config, and request filters off the top of my head). Luckily you can test the majority of these using another page in the OSGi console – /system/console/jcrresolver. For the purposes of debugging, you will want to use the “Resolve” button (which tests rewrites for incoming requests) rather than the “Map” button (which tests rewrites of links on outgoing content.
Getting Java Source Code
When debugging, the first thing you need is the source code. You can get pretty much all the source for AEM through decompiling. However before we get to decompiling I recommend that you check to see if the source is already up online. Many of the modules used in AEM are open source and their code is on GitHub, or another accessible repo.
Note: If you can’t find the source but you’re using IntelliJ, you may be able to get some direct decompilation during debugging without finding the OSGi bundle. I still recommend knowing where everything is and how to do this in case IntelliJ fails.
For the code that you can’t find online, you can grab the OSGi bundle for it and decompile it. In many cases you can find a specific class by searching for it, or its interface on the OSGi console’s Package Dependencies page at /system/console/depfinder. However sometimes this doesn’t work. I recommend skipping that step and doing a ^f on /system/console/status-Bundles instead.
Once you know the OSGi bundle that contains your target class, grab that OSGi bundle’s bundle number. The bundle number actually corresponds to the bundle’s storage folder on your file system. On most systems the bundle with that number will be located at crx-quickstart/launchpad/felix/bundle<discovered bundle id>. Your jar will be under a folder named version#.# (pick the highest version number) and named bundle.jar. Copy it to second location before moving on. I recommend renaming the copy after the jar name listed in the bundle location field on /system/console/bundles.
Note that some bundles are also in the jcr. The bundle location will then list jcrinstall:/apps… for these bundles you can also download them directly via that path.
Once you have your appropriately named jar, you can decompile it directly. For this I recommend using JD-GUI. Note that JD-GUI is getting a little old. So it does some funny stuff when decompiling more functional Java. It will attempt to decompile classes to Java even if they were originally written in Groovy.
JD-GUI can save the decompiled source code as a zip file that IDEs can use for debugging. To do this go to the File menu and then select “Save All Sources”.
Debugging Java (and Groovy) Source Code
Enabling Debug Mode on an AEM Instance
Before you start adding breakpoints and running tests you will need to restart your AEM instance in debug mode. To do this you will need to modify the start script and then start your instance using the commandline instead of by double-clicking the jar.
To do this open the start script in your favorite text editor. On most instances the start script is at crx-quickstart/bin/start. Then paste in the following content just above the comment that says “do not configure below this point”. In AEM 6.4 this is around line 88.
Note: if you are using Windows for development then there is a similar change you can make to start.bat but I’m unfamiliar with that setup. These instructions are for OS X but should work on any unix based system with Bash available.
From now on, you should use the following command to start your instance when you want debug mode enabled:
Note that this assumes you are currently in the same folder as the jar. I actually have a custom alias command that I run that uses the start script’s absolute path.
A couple of caveats here:
- The port in the lines you add to your start script must be unique and not in use. I recommend switching this to 40404 for the publish instance.
- The start script no longer is correctly updated on initial build if your instance is a publish instance. You will need to ensure that the CQ_RUNMODE is set to publish for that. To do that you can find the setting toward the top of the start script. In AEM 6.4 it is at line 24.
- The start script also no longer contains an updated port. So if you are using a custom port number or you are on a publish instance you will need to update the CQ_PORT variable as appropriate. In AEM 6.5 it is at line 13.
Attaching a Debugger and Setting Breakpoints
The big 3 Java IDEs all have built in debuggers that should work for debugging AEM. I recommend using either Netbeans or IntelliJ. In both cases you want to connect to a remote running JVM. Provide the port that you added in the start script (30303 by default) and keep localhost as the host if you are working locally. Once you connect your IDE to your running instance, you can set breakpoints by clicking to the far left of a line or by using your IDE’s debug configs to add method or class level breakpoints.
If you need to use decompiled source from JD-GUI then add it to your debug sources using your IDE’s debug settings.
Once breakpoints are set, run the corresponding behavior that you are testing and wait for your IDE to pause execution. Depending on what you are testing your browser may get stuck loading. As you debug on a regular basis you will get better at creating more targeted breakpoints and come up with a debugging style that works for you.
A couple of caveats here:
- Do not run a build while your debugger is attached. Detach the debugger, run the build, and reattach it again to repeat your test. An attached debugger may prevent the build from completing or cause other issues.
- In some cases in IntelliJ the debugger will trip a breakpoint but the IDE won’t catch it. If execution pauses but the IDE isn’t working you can try pausing the debugger manually using its pause button and then hit play again to resume.
- In some cases your debugger will not detach properly. If this occurs you may see halted execution or other weird behavior. Try reattaching to fix this. Otherwise you may need to restart your instance. This is pretty rare.
- In some cases an AEM instance will not shut down properly after a debugger has been used. To stop it you will need to find it’s process id from the commandline and manually kill it. There are easy instructions for that in the Odds and Ends section below.
Debugging AEM JSP and HTL (Sightly) Servlets
When debugging sometimes I’ve come across a particularly complex JSP or HTL template. It is not possible to set breakpoints on these as-is. So it can be difficult to figure out what is going on. Fortunately these files actually get transformed into Java that is then used as Servlets for the execution of requests. AEM offers a way to retrieve the generated Java source! To do this, go to /system/console/fsclassloader in the OSGi console and click view. You can then hit download below Java in the table at the top. From there you can debug this source like any other Java source!
Note: If you are trying to retrieve the source for a jsp or htl template in this way, make sure that you’ve actually run a page that requires that template on this server, otherwise it will not appear in the list.
Decompiling and Debugging the MVN Build
Occasionally I’ve been working on custom build changes for our AEM projects. Most projects use Maven which also happens to be a Java based tool. Maven offers the ability to debug its execution just like AEM. To do this, use mvndebug instead of mvn as your build command in your terminal. Execution will pause until a debugger is attached. You should set breakpoints in the Maven plugin code before attaching and then attach your debugger. By default the debug port will be 8000 (with localhost as the domain). Once your debugger is attached, execution will continue until the first breakpoint is hit. From there you can debug normally.
When looking for plugin source code, note that almost all of this is open source. Try to look around online for the actual source. If you can’t find it, you can also find the jar in your .m2 folder (usually under ~ on most systems) and decompile it.
De-Minifying and Debugging AEM JS
When working with JavaScript it can be difficult to follow the code due to optimizations like minification, which consolidate the code by shortening variable names and removing whitespace. AEM has a built in feature that does this which can be disabled during debugging. If your js file ends in .min.js but is not stored in the JCR like that, then it is likely that this setting is on. To disable it, go to /system/console/configMgr and find the Adobe Granite HTML Library Manager. Uncheck the Minify checkbox. You may also want to check the Debug checkbox, which will stop clientlib concatenation and keep the js files separated. That way you can more easily track down the exact source file that contains the code you need to modify.
All of the major browsers offer a developer console that you can use. The quick and dirty way to open it is to right click on something and choose inspect element. There are also other menus and key commands that you can learn. Once you have it open there are a lot of different items that I find very useful.
The biggest feature is the sources tab (name varies). This allows you to explore all of the js that is running on the page. You can set breakpoints and step through the code line by line just like with Java debugging. The network tab allows you to see requests and many details about them as they happen. Finally the console tab allows you to run individual lines of JavaScript and see logged errors as they come in. There are a lot of other useful features as well. So look around and try everything!
If you can’t disable minification for any reason, there is another option. Most of the major browser dev tools offer a “prettify” option that will take a minified js file and add basic indentation to make it more readable. Often this is also the easiest way to set meaningful breakpoints on a one line minified js file.
Another nifty feature that you may want to take advantage of are DOM event breakpoints. These are only available in Chrome right now as far as I know, but they allow you to break execution whenever code makes a change to the DOM. This can be useful if you don’t know what JavaScript is causing an element to transform.
Note that the developer consoles also have tabs where you can manipulate the css which can be vital in tracking down visual defects.
Debugging Groovy Console Scripts
Our company offers an open source solution called the AEM Groovy Console which allows administrators to write quick Groovy scripts that run against the live AEM instance. With it you can do things like gather reports, find and replace for a specific property, or even build an entire site from scratch. The scripts are written in a browser window and submitted to the backend for processing. It took some digging to figure out how to add breakpoints to these scripts so that I can debug them while they are running.
I found the secret by debugging the console itself using the techniques above. You can see the method that actually runs the script at com.icfolson.aem.groovy.console.impl.DefaultGroovyConsoleService.runScript(). Once you know that, you can step through and check the class name of the script variable.
With the classname, in this case it is almost always Script1.groovy (fully qualified – no package), you can trick your IDE into using the source of your script to back that class. In IntelliJ and Netbeans, just putting the script in a file named Script1.groovy in the default class path was enough (for most AEM projects this would be in /<name of project>/<name of project>-core/src/main/java/Script1.groovy). From there you can set a breakpoint on that file and it will trigger for the same line as the Groovy Console script executes. Hooray debugging!
Odds and Ends – Misc Good Stuff
Here’s a few quick notes on other things I think you should know when decompiling and debugging AEM.
Killing your instance
If your instance becomes stuck or won’t respond, you may need to manually kill it and restart it. To do this I recommend the commandline utility lsof. AEM connects to it’s main port and also connects to the specified debug port when debugging. So you can quickly find the process by searching for programs that are connected to those ports.
I run the following to find the author port listeners
I also check
in case the instance is partially shut down and only connected to the debug port. The results of this should give you a process id that you can then kill with the following:
Note that if you are doing this it probably means that the -kill is necessary (force quit) but you can try the command without it for graceful shutdown if you want.
Grabbing /libs
A quick bit of advice that came to me from Mark Daugherty: package up /libs using the package manager and unzip it locally to peruse that source code. I’ve always explored this in /crx/de/index.jsp but this gives you access to unix commandline tools and other programs. You can search with grep. Plus if you do this with multiple versions, you can even use diff utilities to see exactly what’s changed between versions!
Documentation
One important part of debugging is knowing what each piece is supposed to do. Documentation, especially JavaDoc is a huge help here. So you should keep track of where all of your documentation is! For instance Sightly Spec is at https://github.com/adobe/htl-spec/blob/master/SPECIFICATION.md, and the AEM 6.4 JavaDoc is currently at https://helpx.adobe.com/experience-manager/6-4/sites/developing/using/reference-materials/javadoc/index.html
Never forget to look up the stack
In the debugger and in log stack traces, we often focus down on the currently running function or method, but you can gain a lot of insight by looking up the stack. All of the major debuggers offer a pane where you can see the method chain that called the method you are currently in. Take a look in there and you can get an idea of where your input values came from.
Conclusion
I hope that this article has provided you with some new strategies for working with your code in AEM. Programming AEM features can be complicated and time consuming, but, if you can trace through execution, you can get there a lot faster. If there’s one thing I hope you remember from this article, it’s that there are many different avenues available for gaining information about your code. So you should keep digging until you find what you need.
I had originally intended for this article to include step by step instructions for each debugging task. Unfortunately I’ve simply run out of time to put those instructions together. I hope that this article can help you get 90% of the way there anyway. If you get stuck, poke around and try things, you might even find an improvement on my process!
“Take chances, make mistakes, and get messy!” – The Magic School Bus
References
While working in AEM I found a large number of articles and posts that helped me arrive at the processes above. Unfortunately many of those have since been lost. In some cases I derived the methods above directly and later found an article describing them. The references I do have noted are the following:
No comments:
Post a Comment
If you have any doubts or questions, please let us know.