Pre-Upgrade Maintenance Tasks

Before beginning your upgrade, it is important to follow these maintenance tasks to ensure that the system is ready and can be rolled back should an issue occur:

• Ensure Sufficient Disk Space
• Fully Back Up AEM
• Back Up Changes to /etc
• Generate The quickstart.properties File
• Configure Workflow and Audit Log Purging
• Install, Configure, and Run The Pre-Upgrade Tasks
• Disable Custom Login Modules
• Remove Updates From The /install Directory
• Stop Any Cold Standby Instances
• Disable Custom Scheduled Jobs
• Execute Offline Revision Cleanup
• Execute Datastore Garbage Collection
• Upgrade the Database Schema If Needed
• Delete Users that Might Hinder the Upgrade
• Rotate Log Files

Ensure Sufficient Disk Space
When executing the upgrade, in addition to the content and code upgrade activities, a repository migration will need to be performed. The migration will create a copy of the repository in the new Segment Tar format. As a result, you will need enough disk space to retain a second, potentially larger, version of your repository.

Fully Back Up AEM
AEM should be fully backed up before beginning the upgrade. Make sure to back up your repository, application installation, datastore, and Mongo instances if applicable. For more information on backing up and restoring an AEM instance, see Backup and Restore.

Back Up Changes to /etc
The upgrade process does a good job of maintaining and merging existing content and configurations from under the /apps and /libs paths in the repository. For changes made to the /etc path, including Context Hub configurations, it is often necessary to re-apply these changes after the upgrade. While the upgrade will make a backup copy of any changes that it cannot merge under /var, we recommend backing up these changes manually before beginning the upgrade.

Generate The quickstart.properties File
When starting AEM from the jar file, a quickstart.properties file will be generated under crx-quickstart/conf. If AEM has only been started with the start script in the past, this file will not be present and the upgrade will fail. Make sure to check for the existence of this file and restart AEM from the jar file if it is not present.

Configure Workflow and Audit Log Purging
The WorkflowPurgeTask and com.day.cq.audit.impl.AuditLogMaintenanceTask tasks require separate OSGi configurations and will not work without them. If they fail during pre-upgrade task execution, missing configurations is the most likely reason. Therefore, make sure to add OSGi configurations for these tasks or remove them altogether from the pre-upgrade optimization tasks list if you do not wish to run them. Documentation for configuring workflow purging tasks can be found at Administering Workflow Instances and audit log maintenance task configuration can be found at Audit Log Maintenance in AEM 6.

For workflow and audit log purging on CQ 5.6 as well as audit log purging on AEM 6.0, see Purge workflow and audit nodes.

Install, Configure, and Run The Pre-Upgrade Tasks
Because of the level of customization AEM allows, environments usually do not adhere to a uniform way of performing upgrades. This makes creating a standardized procedure for upgrades a difficult process.

In previous versions, it was also difficult for AEM upgrades that were stopped or that have failed to be safely resumed. This led to situations where restarting the full upgrade procedure was necessary or where defective upgrades were carried out without triggering any warnings.

In order to address these issues, Adobe has added several enhancements to the upgrade process, making it more resilient and user friendly. Pre-upgrade maintenance tasks that before had to be performed manually are being optimized and automated. Also, post-upgrade reports have been added so that the process can be fully scrutinized in the hope that any issues are found more easily.

Pre-upgrade maintenance tasks are currently spread over various interfaces which are partially or completely performed manually. The pre-upgrade maintenance optimization introduced in AEM 6.3 enables a unified way to trigger these tasks and be able to inspect their result on demand.

All tasks included in the pre-upgrade optimization step are compatible with all versions from AEM 6.0 onwards.

How to Set It Up
In AEM 6.3 and later, the pre-upgrade maintenance optimization tasks come included in the quickstart jar. If you are upgrading from an older version of AEM 6, they are made available through separate packages that you can download from the Package Manager. You can find the packages at these locations:

• For upgrading from AEM 6.0
• For upgrading from AEM 6.1
• For upgrading from AEM 6.2

How to Use It

The PreUpgradeTasksMBean OSGI component comes preconfigured with a list of pre-upgrade maintenance tasks that can be run all at once. You can configure the tasks by following the below procedure:

Go to the Web Console by browsing to https://serveraddress:serverport/system/console/configMgr

Search for “preupgradetasks”, then click on the first matching component. The full name of the component is com.adobe.aem.upgrade.prechecks.mbean.impl.PreUpgradeTasksMBeanImpl

Modify the list of maintenance tasks that need to run as shown below:


The task list differs depending on the run mode that is being used to start the instance. Below is a description of the run mode each maintenance task is designed for.

Task	Run Mode	Notes
TarIndexMergeTask	crx2
DataStoreGarbageCollectionTask	crx2	Will run mark and sweep. For shared datastores, remove this step and run manually or properly prepare instances before executing.
ConsistencyCheckTask	crx2
WorkflowPurgeTask	crx2/crx3	Must configure the Adobe Granite Workflow Purge Configuration OSGi before running.
GenerateBundlesListFileTask	crx2/crx3
RevisionCleanupTask	crx3	For TarMK instances on AEM 6.0 to 6.2, manually run Offline Revision Cleanup instead.
com.day.cq.audit.impl.AuditLogMaintenanceTask	crx3	Must configure the Audit Log Purge Scheduler OSGi configuration before running.

CAUTION
DataStoreGarbageCollectionTask is calling a Datastore Garbage Collection operation with the mark and sweep phase if used. For deployments that use a shared datastore make sure to either reconfigure it or properly or prepare the instance to avoid deletion of items referenced by another instance. This might require running the mark phase manually on all instances before triggering this pre-upgrade task.

Default Configuration of the Pre-Upgrade Health Checks
The PreUpgradeTasksMBeanImpl OSGI component comes pre-configured with a list of pre-upgrade health check tags to execute when the runAllPreUpgradeHealthChecks method is called:

system - the tag used by the granite maintenance health checks

pre-upgrade - this is a custom tag that could be added to all the health checks that you can set to run before an upgrade

The list is editable. You can use the plus (+) and minus (-) buttons besides the tags to add more custom tags, or remove the default ones.

MBean Methods
The managed bean functionality can be accessed using the JMX Console.

You can access the MBeans by:
Going to the JMX Console at https://serveraddress:serverport/system/console/jmx

Search for PreUpgradeTasks and click the result

Select any method from the Operations section and select Invoke in the following window.

Below is a list of all the available methods that the PreUpgradeTasksMBeanImpl exposes:

Method Name	Type	Description
getAvailablePreUpgradeTasksNames()	INFO	Displays the list of available pre-upgrade maintenance tasks names.
getAvailablePreUpgradeHealthChecksTagNames()	INFO	Displays the list of pre-upgrade health checks tag names.
runAllPreUpgradeTasks()	ACTION	Runs all the pre-upgrade maintenance tasks in the list.
runPreUpgradeTask(preUpgradeTaskName)	ACTION	Runs the pre-upgrade maintenance task with the name given as the parameter.
isRunAllPreUpgradeTaskRunning()	ACTION_INFO	Checks if the runAllPreUpgradeTasksmaintenance task is currently running.
getAnyPreUpgradeTaskRunning()	ACTION_INFO	Checks if any pre-upgrade maintenance task is currently running and returns an array containing the names of currently running tasks.
getPreUpgradeTaskLastRunTime(preUpgradeTaskName)	ACTION	Displays the exact running time of the pre-upgrade maintenance task with the name given as the parameter.
getPreUpgradeTaskLastRunState(preUpgradeTaskName)	ACTION	Displays the last running state of the pre-upgrade maintenance task with the name given as the parameter.
runAllPreUpgradeHealthChecks(shutDownOnSuccess)	ACTION	Runs all the pre-upgrade health checks and saves their status in a file named preUpgradeHCStatus.properties that is located in the sling home path. If the shutDownOnSuccess parameter is set to true, the AEM instance will be shut down, but only if all the pre-upgrade health checks have an OK status. The properties file will be used as a precondition for any future upgrade and the upgrade process will be stopped if the pre-upgrade health check execution failed. If you want to ignore the result of the pre-upgrade health checks and launch the upgrade anyway, you can delete the file.
detectUsageOfUnavailableAPI(aemVersion)	ACTION	Lists all the imported packages that will no longer be satisfied when upgrading to the specified AEM version. The target AEM version must be given as parameter.

NOTE
The MBean methods can be invoked via:
The JMX Console
Any external application that connects to JMX
cURL

Disable Custom Login Modules
NOTE
This step is only required if you are upgrading from an AEM 5 version. It can be skipped entirely for upgrades from older AEM 6 versions.

The way custom LoginModules are configured for authentication at the repository level has fundamentally changed in Apache Oak.

In AEM versions that used CRX2 configuration was placed in the repository.xml file, while from AEM 6 onwards it is done in the Apache Felix JAAS Configuration Factory service via the Web Console.

Therefore, any existing configurations will have to be disabled and re-created for Apache Oak after the upgrade.

To disable the custom modules defined in the JAAS configuration of repository.xml, you need to modify the configuration to make use of default LoginModule, as in this example:

<Security >
             ....
          <!--
                 Use LoginModule authenticating against repository itself
                 -->
                 <LoginModule class = "com.day.crx.core.CRXLoginModule" >
                     <param name = "anonymousId" value = "anonymous" />
                     <param name = "adminId" value ="admin" />
                     <param name = "disableNTLMAuth" value = "true" />
                     <param name = "tokenExpiration" value = "43200000" />
                     <!-- param name="trust_credentials_attribute" value="d5b9167e95dad6e7d3b5d6fa8df48af8"/
                -->
                 </LoginModule >
         </ Security>

NOTE
For more information, see Authentication with the External Login Module.
For an example of LoginModule configuration in AEM 6, see Configuring LDAP with AEM 6.

Remove Updates From The /install Directory

NOTE
Only remove packages from the crx-quickstart/install directory AFTER shutting down the AEM instance. This will be one of the last steps before starting the in-place upgrade procedure.

Remove any service packs, feature packs or hotfixes that have been deployed through the crx-quickstart/install directory on the local file system. This will prevent the inadvertent installation of old hotfixes and service packs on top of the new AEM version after the update has completed.

Stop Any Cold Standby Instances
If using TarMK cold standby, stop any cold standby instances. These will guarantee an efficient way to come back online in case of issues in the upgrade. After the upgrade has completed successfully, the cold standby instances will need to be rebuilt from the upgraded primary instances.

Disable Custom Scheduled Jobs
Disable any OSGi scheduled jobs that are included in your application code.

Execute Offline Revision Cleanup

NOTE
This step is only necessary for TarMK installations

If using TarMK, you should execute Offline Revision Cleanup before upgrading. This will make the repository migration step and subsequent upgrade tasks execute much faster and will help to ensure that Online Revision Cleanup can execute successfully after the upgrade has completed. For information on running Offline Revision Cleanup, see Performing Offline Revision Cleanup.

Execute Datastore Garbage Collection
NOTE
This step is only necessary for instances running crx3

After running revision cleanup on CRX3 instances, you should run Datastore Garbage Collection to remove any unreferenced blobs in the data store. For instructions, see the documentation on Data Store Garbage Collection.

Upgrade the Database Schema If Needed
Usually, the underlying Apache Oak stack AEM uses for persistence will take care of upgrading the database schema if needed.

However, cases might arise when the schema cannot be upgraded automatically. These are mostly high security environments where the database is running under a user with very limited priviledges. If this happens, AEM will continue to use the old schema.

In order to prevent this from happening, you need to upgrade the schema by following the below procedure:

Shut down the AEM instance that needs to be upgraded.
Upgrade the database schema. Please consult the documentation for your database type in order to see what is the tooling you need to use in order to achieve this.

For more information on how Oak handles schema upgrades, see this page on the Apache website.

Proceed with upgrading AEM.
Delete Users that Might Hinder the Upgrade

NOTE
This pre-upgrade maintenance task is only necessary if:
You are upgrading from AEM versions older than AEM 6.3
You encounter any of the errors mentioned below during the upgrade.

There are exceptional cases when service users might end up in an older AEM versions being improperly tagged as regular users.

If this happens, the upgrade will fail with a message like this one:ERROR [Apache Sling Repository Startup Thread] com.adobe.granite.repository.impl.SlingRepositoryManager Exception in a SlingRepositoryInitializer, SlingRepository service registration aborted java.lang.RuntimeException: Unable to create service user [communities-utility-reader]:java.lang.RuntimeException: Existing user communities-utility-reader is not a service user.

In order to work around this issue, make sure you do the following:

1. Detach the instance from production traffic
2. Create a backup of the user(s) causing the problem. You can do this via Package Manager. For more information, see How to Work with Packages.
3. Delete the user(s) causing the problem. Below is a list of users that might fall under this category:

• dynamic-media-replication
• communities-ugc-writer
• communities-utility-reader
• communities-user-admin
• oauthservice
• sling-scripting

Rotate Log Files
We recommend archiving your current log files before beginning your upgrade. This will make it easier to monitor and scan your log files during and after the upgrade to identify and resolve any issues that may occur.