Error-specific Database Recovery Procedures

The previous chapter described the general process to follow to recover a DBAM database. This chapter describes specific recovery processes for the main types of inconsistencies that can occur in a DBAM database. Each section in this chapter describes one of these inconsistencies, including what it means, how to detect it, additional data that should be collected for analysis, and how to recover from it.

Attribute Index Inconsistency

Attribute index inconsistency means that there is an error in the stored attribute indexes, either a structural inconsistency in the stored indexes, or misinformation stored in the attribute index; for example, an entry is present in the index but shouldn’t be there based on its data.This type of inconsistency can lead to incorrect search results or to rejected modifications in the case of a structural inconsistency.

How to Detect It

The dbamverify command writes the result of attribute index checks into the aidxcheck*dbamverify_PID.txt* file in the $DIRX_INST_PATH/tools/log directory.At the end of this file, there is a table with a line “attribute index(es) inconsistent”.If this line starts with a number other than zero, there is an attribute index inconsistency in the database.

Additional Data to be Collected for Analysis

The dbamverify command writes detailed logs about attribute index checks into the aidxcheck*dbamverify_PID.txt* file it creates in the $DIRX_INST_PATH/tools/log directory.Be sure to provide this file for analysis in addition to the files described in the section “Collecting Data for Analysis” in the chapter “General Database Error Recovery Process”.

Specific Recovery Steps

Attribute indexes can be easily repaired or rebuilt using dirxadm. Most attribute index problems can be resolved by running the db check operation with the -repair option or by rebuilding the attribute index if the repair operation doesn’t fix the problem. To perform this recovery procedure:

  • Run the following dirxadm operation to repair the index(es):

    db check -bs ATTRIBUTE [-attribute attribute_type] -repair

    If the problem affects only one attribute index or just a few of them, specify the attribute type to avoid checking all the attribute indices.

  • The db check operation produces the same aidxcheck*PID.txt* file as dbamverify, but in the $DIRX_INST_PATH/server/log directory.After the db check operation has finished, check this generated aidxcheck file.At the end of the file, there are two lines: “attribute index(es) inconsistent” and “attribute index(es) repaired”.If the numbers at the beginning of the lines are equal, the inconsistency was repaired, and you can proceed to the next step in the general database recovery process.

  • If the numbers are different, run the following dirxadm operation to rebuild the index:

    db attrconfig attribute_type_abbreviation -index BUILD index_type(s)

    If index_type(s) is not specified, the command re-creates an index as it was originally configured.

  • After the index is rebuilt, run the dirxadm db check command again without the -repair option to check whether the inconsistencies have been resolved.If the “attribute index(es) inconsistent” line at the end of the aidxcheck file shows zero inconsistencies, the database recovery was successful, and you can proceed to the next step in the general recovery process.

  • If the inconsistency cannot be resolved with the dirxadm repair or rebuild operations, use one of the general recovery methods described in the next chapter to restore the database.

Subordinate Bit String Inconsistency

Subordinate bit string inconsistency means that there is an error in the stored subordinate bit strings, either a structural inconsistency in the stored bit strings or misinformation stored in it; for example, a subordinate entry is present in the index, but it shouldn’t be there based on its data.This type of inconsistency can lead to incorrect search results and rejected modifications.

How to Detect It

The dbamverify command writes the result of subordinate bit string check into the subcheck*dbamverify_PID.txt* file in the $DIRX_INST_PATH/tools/log directory.At the end of the subcheck file, there is a line like “Subordinate bit string check finished base=base n errors”.If n is non-zero, there is a subordinate bit string inconsistency in the database.

Additional Data to be Collected for Analysis

The dbamverify command writes detailed logs about subordinate bit string checks into the subcheck*dbamverify_PID.txt* file in the $DIRX_INST_PATH/tools/log directory.Be sure to provide this file for analysis in addition to the files described in the section “Collecting Data for Analysis” in the chapter “General Database Error Recovery Process”.

Specific Recovery Steps

Subordinate indexes can be repaired using dirxadm. Some problems can be resolved by running the db check operation with the -repair option. To perform this procedure:

  • Run the following dirxadm operation:

    db check -bs SUBORDINATE [-base distinguished_name] -repair

    If the inconsistency affects a specific branch of the tree or just a few branches, use the
    -base option to specify the base to avoid checking all the subordinate bit strings.

  • The db check operation produces the same subcheck file as dbamverify, but in the $DIRX_INST_PATH/server/log directory.Examine this generated subcheck file.At the end of the file, there is a line like “Subordinate bit string check finished base=base n errors”.If n is zero in this line, the repair operation was successful, and you can proceed to the next step in the general recovery process.

  • If n is non-zero, the inconsistency cannot be resolved with the db check operation.Use one of the general recovery methods described in the next chapter to restore the database.

Tree Inconsistency

Tree inconsistency means that one of the structural building blocks of the DIT is damaged.It can result in several kind of problems, like rejected operations, failed purge operation, etc.

How to Detect It

The dbamverify command writes tree verification problems into its stderr output.If this output contains one or more references to pseudo or tree blocks, there is a tree inconsistency in the database.

Additional Data to Collect for Analysis

A tree inconsistency is related to the database objects that define the structure of the tree, so a tree dump must be generated on 1) the last error-free backup and 2) on the backup or the database in which the inconsistency was detected.Provide these tree dumps for analysis in addition to the files described in the section “Collecting Data for Analysis” in the chapter “General Database Recovery Process”.

Use the the following dbamdump command to generate the tree dump:

dbamdump -Tfl

The dbamdump command writes the tree dump to stdout, so we recommend redirecting stdout to a file.

Note: the dbamdump command is an internal tool used primarily by DirX Directory support.It is not described in the DirX documentation.

Specific Recovery Steps

There are no specific recovery steps for a tree inconsistency.Use one of the recovery methods described in the chapter “General Methods for Database Recovery” to restore the database.Note that the backup archive file is the dump of the database blocks and can thus be in a state that is just a few modifications away from the same inconsistency.So, for a tree inconsistency, we recommend rebuilding the database from an LDIF dump file.If the total update by media feature is enabled for the system, you can use the LDIF file with the "Restore by Initiating a Total Update" method.Otherwise, use the "Restore Using an LDIF Dump" method.

Real Object Inconsistency

Real object inconsistency means that the data stored in the real object blocks or in the follow blocks of the database is inconsistent.Any data stored in the database can be corrupt according to the rules checked during the verification, including the entries of the database, the database schema, configuration subentries, etc.This type of inconsistency has several sub types.Most of them are briefly described in the heading of the entrycheck log files.

How to Detect It

The dbamverify command writes the result of data consistency checks into the entrycheck*dbamverify_PID.txt* log file in the $DIRX_INST_PATH/tools/log directory.If this file contains a line like “Timestamp Error in REAL block/DSE-ID DSE_ID”, the entry with the specified DSE_ID has an inconsistency.

Additional Data to be Collected for Analysis

The dbamverify command writes detailed logs about data consistency checks into the entrycheck*dbamverifyPID.txt* file in the $DIRX_INST_PATH/tools/log directory.Be sure to provide this file for analysis in addition to the files described in the section “Collecting Data for Analysis” in the chapter “General Database Recovery Procedures”.

A dump of the erroneous entries should also be created and attached to the ticket. If there are a lot of errors of the same kind, the block dump of at least three entries and their follow blocks should be collected. Use the following dbamdump command to create a complete block dump of the DSE:

dbamdump -B -b DSE_ID -t 128 -z real_object_block_size

where real_object_block_size is the number that corresponds to the real object block size in use: 0=1kB, 1=4kB, 2=16kB, 3=64kB. The dbamdump command writes the DSE dump to stdout, so we recommend redirecting stdout to a file.

Note: the dbamdump command is an internal tool used primarily by DirX Directory support. It is not described in the DirX documentation.

Specific Recovery Steps

To recover from real object inconsistencies, follow this procedure:

  • Examine the error message itself. If it contains a line starting with “Recommendation”, follow the steps described in this field.

  • If the FailedCheck field of the error message is “Normalisation-RDN”, use the
    db check -rob -repair operation to fix the inconsistency.

  • After performing any recovery step(s) indicated in the error message, check the database consistency again with either the dbamverify -D command or the dirxadm db check -rob command. If the error was resolved by the specific recovery step, the database is consistent, and you can proceed to the next step in the general recovery process.

  • If the problem cannot be resolved by the previous steps, use one of the general recovery methods described in the next chapter to recover the database.