Reverting updates not fully supported by the revert-update tool

Reverting updates not fully supported by the revert-update tool

Published: 12/18/2018

Problem Description:

A directory administrator may wish to revert to a previous version of the Directory Server even though the revert-update cannot perform the process completely.

Running the update tool from a specific software distribution allows you to upgrade the software and configuration of an existing PingDirectory installation to the distribution's version. After an update, you can often use revert-update to undo that update.

revert-update uses a log of file system operations to return the file system back to the previous state. For some updates, revert-update cannot completely undo the update and additional steps must be performed to restore the Directory Server. Examples include:

1. When a change has been made to the data storage implementation such as the backend library version, changes in the implementation of composite indexes, and the compaction of JSON objects
2. The introduction of the topology registry in PingDirectory 7

When revert-update might require additional actions on the part of the administrator, the update tool will warn the administrator before the update is applied. For example:

$ ./update --serverRoot /prod/ds6/ds1 --licenseKeyFile $HOME/Downloads/PingDirectory7.2.lic

Warning:  The updated server version includes database changes that are not compatible with the current server version. If you wish to later revert to an older version, then you will need to export the data to LDIF before performing the reversion, and then re-import the data after the revert process has completed. In addition, the changelogDb/ and db/changelog/ directories in the reverted server root must be deleted after the revert has completed.

This warning can be ignored when upgrading from version 7.0.x.  If you cancel now, the server will be left in its original state.  Continue? (yes / no) [no]: yes

This document provides a walk-through of an update from 6.0 to 7.2 and the additional steps necessary to revert the update.
 

Solution:

In the following walk-through, we will upgrade a service consisting of two Directory Servers from 6.0 to 7.2 and then show how to revert them back to 6.0. The nodes are ds1 and ds2 and reside on the same server.

Prepare the two servers for update by giving them each an instance-name (also note that instance names cannot be changed after being set):
ds1/bin/dsconfig set-global-configuration-prop --set instance-name:ds1
ds2/bin/dsconfig set-global-configuration-prop --set instance-name:ds2

Update the first server:
$ ./update --serverRoot /prod/ds6/ds1 --licenseKeyFile $HOME/Downloads/PingDirectory7.2.lic

Warning:  The updated server version includes database changes that are not compatible with the current server version. If you wish to later revert to an older version, then you will need to export the data to LDIF before performing the reversion, and then re-import the data after the revert process has completed. In addition, the changelogDb/ and db/changelog/ directories in the reverted server root must be deleted after the revert has completed.

This warning can be ignored when upgrading from version 7.0.x.  If you cancel now, the server will be left in its original state.  Continue? (yes / no) [no]: yes

Warning:  The server will now store the values of certain JSON fields in a more compact manner, in order to reduce the on-disk and in-memory footprint required to store that data.  This compaction support is enabled by default and will automatically be used for any writes processed after the update.  However, the compacted representations of the data cannot be interpreted by older versions of the server, and if you wish to later revert to an older version, then you will need to export the data to LDIF before performing the reversion, and then re-import the data after the revert process has completed.  If you cancel now, the server will be left in its original state.  Continue? (yes / no) [no]: yes

Warning:  There are restrictions for reverting this update. Topology information such as server instances, instance and secret keys, server groups, and administrator users have moved to the topology portion of the configuration from the admin backend. As long as new servers are not added to the topology after this update has occurred, then the revert-update command can be used to return to the previous version. However, if new servers are added, then the restored admin backend of this server will not contain information about the new servers, and this local server will not be able to communicate with all other servers in the topology. New servers must not be added to the topology while reverting this update is still a possibility.  If you cancel now, the server will be left in its original state.  Continue? (yes / no) [no]: yes

Warning:  The format for Local DB Composite Indexes has changed. In order to update a server with composite indexes in the old format, delete these indexes and re-run update. After update is complete, recreate the indexes and use rebuild-index to rebuild the indexes. The command for recreating an index will be in the "Undo" portion of the logs/config-audit.log file. If you wish to later revert to an older version, then you will need to delete and recreate those composite indexes again.  If you cancel now, the server will be left in its original state.  Continue? (yes / no) [no]: yes

Warning:  The updated server version includes index changes that are not compatible with the current server version. If you wish to later revert to an older version, then you will need to export the data to LDIF before performing the reversion, and then re-import the data after the revert process has completed.  If you cancel now, the server will be left in its original state.  Continue? (yes / no) [no]: yes

The modified default value(s) for entry 'cn=Access Control Handler,cn=config' attribute 'ds-cfg-global-aci' will be merged into the server's current configuration by adding the new default value(s) [(targetcontrol="1.3.6.1.4.1.30221.2.5.55")(version 3.0;acl "Anonymous access to the UnboundID-proprietary permit unindexed search request control"; allow (read) userdn="ldap:///anyone";), (targetattr="ds-changelog-encrypted-password")(version 3.0; acl "When users change their own password, ensure that the Changelog Password Encryption Plugin can write the encrypted password to the changelog."; allow (write) userdn="ldap:///self";), (targetcontrol="1.3.6.1.4.1.30221.2.5.54")(version 3.0; acl "Anonymous access to the UnboundID-proprietary reject unindexed search request control"; allow (read) userdn="ldap:///anyone";)]. To skip merging in default values, cancel and re-run update with the --noMergeDefaults option.  If you cancel now, the server will be left in its original state.  Continue? (yes / no) [yes]:

The server will need to be stopped in order to proceed with the update.  Continue? (yes / no) [yes]:

Initializing ..... Done
Stopping Directory Server .....

Updating ..... Done
Updating Configuration ..... Done
Updating Topology Registry ..... Done
Updating Java Properties ..... Done
Verifying ..... Done
Starting Directory Server .....

[14/Dec/2018:14:26:54.112 -0800] category=CORE severity=NOTICE msgID=458886 msg="Ping Identity Directory Server 7.2.0.0 (build 20181207174011Z, R29497) starting up"
...
[14/Dec/2018:14:27:17.884 -0800] category=EXTENSIONS severity=NOTICE msgID=1880555611 msg="Administrative alert type=server-started id=d08f94ce-d386-4a48-bd57-9bef7d7d6f40 class=com.unboundid.directory.server.core.DirectoryServer msg='The Directory Server (Ping Identity Directory Server 7.2.0.0 build 20181207174011Z, R29497) has started successfully'"
Starting Directory Server ..... Done

Update Completed Successfully.  The installation at /prod/ds6/ds1 has now been updated to version 7.2.0.0.29497 20181207174011Z.

See /prod/ds6/ds1/history/1544826119824-6.0.0.1.24644/update.log for a detailed log of this operation.

Update the second server:
$ ./update --serverRoot /prod/ds6/ds2 --licenseKeyFile $HOME/Downloads/PingDirectory7.2.lic

Enter password for the root user:

The cluster-wide configuration was found to be out-of-sync among servers in the topology. Refer to the update.log file for details and instructions on how to rectify the problems. If there are no customizations to cluster-wide configuration and the differences are just in the out-of-the-box cluster-wide configuration, then this warning may be safely ignored.  If you cancel now, the server will be left in its original state.  Continue? (yes / no) [yes]:
...
The server will need to be stopped in order to proceed with the update.  Continue? (yes / no) [yes]:

Initializing ..... Done
Stopping Directory Server ..... Done
Updating ..... Done
Updating Configuration ..... Done
Updating Topology Registry ..... Done
Updating Java Properties ..... Done
Verifying ..... Done
Starting Directory Server .....

[14/Dec/2018:14:57:00.933 -0800] category=CORE severity=NOTICE msgID=458886 msg="Ping Identity Directory Server 7.2.0.0 (build 20181207174011Z, R29497) starting up"
...
[14/Dec/2018:14:57:14.352 -0800] category=CORE severity=NOTICE msgID=458887 msg="The Directory Server (Ping Identity Directory Server 7.2.0.0 build 20181207174011Z, R29497) has started successfully"
[14/Dec/2018:14:57:14.356 -0800] category=EXTENSIONS severity=NOTICE msgID=1880555611 msg="Administrative alert type=server-started id=5a3905d6-d06b-46a1-a3fc-c9f54a417a87 class=com.unboundid.directory.server.core.DirectoryServer msg='The Directory Server (Ping Identity Directory Server 7.2.0.0 build 20181207174011Z, R29497) has started successfully'"

Starting Directory Server ..... Done
Updating server version in topology registry ..... Done
Update Completed Successfully.  The installation at /prod/ds6/ds2 has now been updated to version 7.2.0.0.29497 20181207174011Z.

See /prod/ds6/ds2/history/1544828017484-6.0.0.1.24644/update.log for a detailed log of this operation.

We see that the update tool warns both of steps that must be taken post-update (such as recreating composite indexes) and additional steps that might be required if you run revert-update.

For our example, the message "The updated server version includes index changes that are not compatible with the current server version." will certainly require additional steps to revert, but as we will see, a revert might also require changes in the remaining 7.x topology configuration.

For this example, reverting requires that we first export the local-db backends. Determine the local-db backends that need to be exported:
ds1/bin/dsconfig list-backends

Backend             : Type                : enabled : base-dn
--------------------:---------------------:---------:-----------------------
ads-truststore      : trust-store         : true    : cn=ads-truststore
alarms              : alarm               : true    : cn=alarms
alerts              : alert               : true    : cn=alerts
backup              : backup              : true    : cn=backups
changelog           : changelog           : false   : cn=changelog
config              : config-file-handler : true    : cn=config
encryption-settings : encryption-settings : true    : cn=encryption settings
metrics             : metrics             : true    : cn=metrics
monitor             : monitor             : true    : cn=monitor
replicationChanges  : custom              : true    : dc=replicationChanges
schema              : schema              : true    : cn=schema
tasks               : task                : true    : cn=tasks
userRoot            : local-db            : true    : "dc=example,dc=com"

So we will need to export the userRoot backend before reverting and import it after reverting.

Stop the server:
$ ds1/bin/stop-server

Export the local-db backends:
ds1/bin/export-ldif --backendID userRoot --ldifFile ds1-userRoot.ldif

Run revert-update:
$ ds1/revert-update

Warning:  The updated server version includes database changes that are not compatible with the older server version. You will need to export any Local DB backends to LDIF before performing the reversion, and then re-import the data after the revert process has completed. In addition, the changelogDb/ and db/changelog/ directories in the reverted server root must be deleted.

This warning can be ignored when reverting to version 7.0.x.  If you cancel now, the server will be left in its original state.  Continue? (yes / no) [no]:

Warning:  The older version of the server does not provide support for automatic JSON value compaction, and will not be able to interpret any entries stored with compacted values.  You should export the data to LDIF before performing the reversion, and then import the data from that LDIF file after the revert process has completed.  If you cancel now, the server will be left in its original state.  Continue? (yes / no) [no]: yes

Warning:  There are restrictions for reverting this update. Topology information such as server instances, instance and secret keys, server groups, and administrator users was moved to the topology portion of the configuration from the admin backend. As long as new servers have not been added to the topology since this server was updated, then the revert-update command can be used to return to the previous version. However, if new servers were added, then the restored admin backend of this server will not contain information about the new servers, and this local server will not be able to communicate with all other servers in the topology. In this case, the new servers must be temporarily removed from the topology until all servers have been reverted to the previous version.

Each time a server is reverted, any servers in the topology using the topology portion of the configuration (rather than the admin backend) will need to be told that the reverted server was downgraded to the admin backend. This is done by running the following dsconfig command on one of the servers that has not been reverted:

    dsconfig set-server-instance-prop \
        --instance-name <Reverted server instance name> \
        --set server-version:<Version to which server is reverted>

If the topology no longer has a master server when this command is run, it will not succeed. In this case one of the remaining updated servers in the topology must be made master with the following command:

    dsconfig set-global-configuration-prop \
        --set force-as-master-for-mirrored-data:true

This will allow the chosen instance to run the first command successfully.  If you cancel now, the server will be left in its original state.  Continue? (yes / no) [no]: yes

Warning:  The older version of the server uses an old, incompatible format Local DB Composite Indexes. Any composite indexes will have to be deleted, recreated and rebuilt using the rebuild-index command after the revert has completed.  If you cancel now, the server will be left in its original state.  Continue? (yes / no) [no]:

Action Required:  
* Before starting the operation you should export the entire data set for this server to LDIF format.  If you have not completed this step you should cancel this operation now.
* Continue with this operation until this tool has finished.
* When this operation is complete, manually delete the files in the 'db' directory.
* Re-import that data from the LDIF file that you had created in the first step.
.  If you cancel now, the server will be left in its original state.  Continue? (yes / no) [no]:

Following this operation the server will be left in a non-running state so that you can perform the required administrative actions before restarting the server  If you cancel now, the server will be left in its original state.  Continue? (yes / no) [yes]:

Initializing .....

This installation will be reverted to version 6.0.0.1.24644 20161107215309Z using the files in /prod/ds6/ds1/history/1544826119824-6.0.0.1.24644.  Continue? (yes / no) [yes]:

Initializing ..... Done
Reverting Components ..... Done
Cleaning Up ..... Done

Reversion Completed Successfully.  The installation at /Users/rkogelheide/pd/ds6/PingDirectory6001 has now been reverted to version 6.0.0.1.24644 20161107215309Z.

After running revert-update, we must perform the actions indicated by the utility before we can start the old Directory Server.

Since there is remaining 7.x topology, we let it know there is now an admin backend (pre-7.x) DS in the topology. Since we're also down to a single node in the 7.x topology, we must first "force-as-master":
ds2/bin/dsconfig set-global-configuration-prop --set force-as-master-for-mirrored-data:true
ds2/bin/dsconfig set-server-instance-prop --instance-name ds1 --set server-version:6.0.0.1

 
We then clear the local DB, changelog, and replication changelog directories:

rm -rf ds1/db/*
rm -rf ds1/changelogDb/

 
We can then import the userRoot backend:

ds1/bin/import-ldif --backendID userRoot --ldifFile ds1-userRoot.ldif
...
[15:39:24]  Import Summary:
   Entries Processed:          1,008
            Imported:          1,008 (100.0%)
             Skipped:              0 (  0.0%)
            Rejected:              0 (  0.0%)

          Total Time:  1 second (1 sec)
        Average Rate:  784 / second
[15:39:24]  Closing the database to flush all imported entries
[15:39:24]  Closing database complete, duration = 9 ms

Note: Before restarting the server, review the "Known issues" section below concerning DS-37480. You'll need to ensure backlog is processed before regular operations commence.

Start the server and check the status:
ds1/bin/start-ds
ds1/bin/status
ds1/bin/dsreplication status --displayServerTable --showAll

Reverting the second server is similar, except we do not have to force-as-master as there's no remaining topology:
ds2/bin/stop-server
ds2/bin/export-ldif --backendID userRoot --ldifFile ds2-userRoot.ldif
ds2/revert-update

Warning:  The updated server version includes database changes that are not compatible with the older server version. You will need to export any Local DB backends to LDIF before performing the reversion, and then re-import the data after the revert process has completed. In addition, the changelogDb/ and db/changelog/ directories in the reverted server root must be deleted.
This warning can be ignored when reverting to version 7.0.x.  If you cancel now, the server will be left in its original state.  Continue? (yes / no) [no]: yes

Warning:  The older version of the server does not provide support for automatic JSON value compaction, and will not be able to interpret any entries stored with compacted values.  You should export the data to LDIF before performing the reversion, and then import the data from that LDIF file after the revert process has completed.  If you cancel now, the server will be left in its original state.  Continue? (yes / no) [no]: yes

Warning:  There are restrictions for reverting this update. Topology information such as server instances, instance and secret keys, server groups, and administrator users was moved to the topology portion of the configuration from the admin backend. As long as new servers have not been added to the topology since this server was updated, then the revert-update command can be used to return to the previous version. However, if new servers were added, then the restored admin backend of this server will not contain information about the new servers, and this local server will not be able to communicate with all other servers in the topology. In this case, the new servers must be temporarily removed from the topology until all servers have been reverted to the previous version.

Each time a server is reverted, any servers in the topology using the topology portion of the configuration (rather than the admin backend) will need to be told that the reverted server was downgraded to the admin backend. This is done by running the following dsconfig command on one of the servers that has not been reverted:

    dsconfig set-server-instance-prop \
        --instance-name <Reverted server instance name> \
        --set server-version:<Version to which server is reverted>

If the topology no longer has a master server when this command is run, it will not succeed. In this case one of the remaining updated servers in the topology must be made master with the following command:

    dsconfig set-global-configuration-prop \
        --set force-as-master-for-mirrored-data:true

This will allow the chosen instance to run the first command successfully.  If you cancel now, the server will be left in its original state.  Continue? (yes / no) [no]: yes

Warning:  The older version of the server uses an old, incompatible format Local DB Composite Indexes. Any composite indexes will have to be deleted, recreated and rebuilt using the rebuild-index command after the revert has completed.  If you cancel now, the server will be left in its original state.  Continue? (yes / no) [no]: yes

Action Required:  
* Before starting the operation you should export the entire data set for this server to LDIF format.  If you have not completed this step you should cancel this operation now.
* Continue with this operation until this tool has finished.
* When this operation is complete, manually delete the files in the 'db' directory.
* Re-import that data from the LDIF file that you had created in the first step.
.  If you cancel now, the server will be left in its original state.  Continue? (yes / no) [no]: yes

Following this operation the server will be left in a non-running state so that you can perform the required administrative actions before restarting the server  If you cancel now, the server will be left in its original state.  Continue? (yes / no) [yes]:

Initializing .....

This installation will be reverted to version 6.0.0.1.24644 20161107215309Z using the files in /prod/ds6/ds2/history/1544828017484-6.0.0.1.24644.  Continue? (yes / no) [yes]:

Initializing ..... Done
Reverting Components ..... Done
Cleaning Up ..... Done

Reversion Completed Successfully.  The installation at /prod/ds6/ds2 has now been reverted to version 6.0.0.1.24644 20161107215309Z.

Perform the required post-revert tasks:
rm -rf ds2/db/*
rm -rf ds2/changelogDb/

ds2/bin/import-ldif --backendID userRoot --ldifFile ds2-userRoot.ldif
...
[15:55:13]  Processed 1008 entries, imported 1007, skipped 0, and rejected 1 in 1 seconds (average rate 837.8/sec)
[15:55:13]  Number of index values that exceeded the entry limit: 0
[15:55:13]  Import Summary:
   Entries Processed:          1,008
            Imported:          1,007 ( 99.9%)
             Skipped:              0 (  0.0%)
            Rejected:              1 (  0.1%)

          Total Time:  1 second (1 sec)
        Average Rate:  838 / second
[15:55:13]  Closing the database to flush all imported entries
[15:55:13]  Closing database complete, duration = 11 ms

Start the server and check the status:
ds2/bin/start-ds
ds2/bin/status
ds2/bin/dsreplication status --displayServerTable --showAll

Troubleshooting:

If you see the error
The Directory Server Instance could not be modified due to a communications problem: A communication problem occurred while contacting the server: Node ds2 cannot process operation ModifyOperation(connID=134, opID=19, dn=cn=ds1,cn=Server Instances,cn=topology,cn=config) (controls: Control(oid=1.3.6.1.4.1.30221.2.5.19, isCritical=false, value={byte[186]})) because it does not know which server is its master. This may be an indication of a bigger problem such as a network partition preventing the node from identifying a server that has a quorum majority of servers it can reach to act as master and should be addressed immediately. The problem may temporarily be fixed by setting the 'force-as-master-for-mirrored-data' global configuration setting on one of the servers, which will force that server to assume the master role

You must force the remaining 7.x server as master first:
dsconfig set-global-configuration-prop --set force-as-master-for-mirrored-data:true

Known issues:

If a topology has had new Directory Servers added to it after an update, those servers must first be removed from the topology before a revert can be performed.

Due to DS-37480, it is possible that servers that have been reverted and are accepting operations while still processing backlogs might get those changes before the backlog. To avoid this, administrators should ensure that newly reverted nodes can process any backlog from other nodes before they start processing regular operations.
 
Category:
KB or other URL: