Using the DirX Audit Tools

The DirX Audit tools have the following prerequisites:

The DirX Audit tools are located in sub-folders of the folder:

install_path/tools/tool_identifier

where

tool_identifier corresponds to the tool as follows:
db_maintenance indicates the DB maintenance tool.
db_fact_population indicates the fact population tool.

The sub-folder bin contains the binary of the tool.

The names of the binaries are:

The general usage of the tools is:

tool_name command [options]

where

tool_name
is either dxtdbtool or dxthistdbtool for the DB maintenance tool or dxtPopulateFacts for the fact population tool.

command
is one of the following keywords:

options
is a list of common and command specific options. (See the following sections for details.)

If a value contains a SPACE character enclose the value in double quotes (").

All DirX Audit tools recognize the following options:

-debug
Creates a debug log.

-debugMemory
Displays more detailed memory allocation information.

-silent
Suppresses the user confirmation dialog for performing a given action. By default, you must confirm (by entering yes) that you really want to perform an action. This option allows you to bypass the confirmation dialog.

-simulate
Shows the results of an action without actually performing the action.

-tenantid tenantID
Specifies the tenant whose configuration file is accessed to acquire database credentials. The tenantID specifies the identifier of a configured tenant.

-transize tranSize
Specifies the transaction size used during an operation. A larger size will increase performance, but will require larger transaction log files.

-types type[,type]…
Specifies a list of comma-separated history entry types to which the operation is restricted. All types are used, if none is specified.

The tenant ID is used to obtain the database credentials of the specific tenant. The credentials are acquired from the tenant configuration file, properly configured by the Configuration Wizard. If only one tenant is configured, the -tenantid option can be omitted from the command line and the configuration file of this tenant is used automatically. If more than one tenant is configured, the -tenantid option needs to be specified on the command line.

Some DirX Audit tools require the following LDAP connection parameters:

-ldapbase ldapBase
LDAP node used as a search base.

-ldapconfig ldapConfigFile
LDAP connection properties file. It contains all mentioned required LDAP parameters.
You can find the ldap.properties file example in the install_path/tools/db_maintenance/doc/samples.

-ldapdomain ldapDomain
LDAP domain DN. It will be prepended before the ldapBase node.

-ldapfilter ldapFilter
Search filter applied to the LDAP search.

-ldaphost ldapHost
LDAP host used to establish an LDAP connection.

-ldappassword ldapPassword
LDAP connection password.

-ldapport ldapPort
Port on which the LDAP connection will be established.

-ldapscope ldapScope
Scope of the LDAP search.

-ldapuser ldapUser
LDAP user used to establish an LDAP connection.

-ldapusessl ldapUseSSL
Specifies whether to use SSL for the LDAP connection.

-ldaptruststorepath ldapTruststorePath
Path to the truststore file used for the LDAP connection.

-ldaptruststorepassword ldapTruststorePassword
Truststore password used for the LDAP connection.

To export audit messages from the DirX Audit Data Database to XML files, execute the following command:

dxtdbtool export common_options
-dstdir folder_name
[-delete]
[-from date_time | function]
[-recsperfile number]
[-recsperquery number]
[-to date_time | function]

where

common_options
Specifies the common options. See the section “Common Options” for details.

-dstdir folder_name
Specifies the destination folder for the XML files containing the exported audit messages.

-delete
Specifies that the audit messages are to be deleted from the DirX Audit Data Database after export.

-from date_time | function
Specifies that only audit messages created after the specified date and time (inclusive) or by the specified function are exported.

-recsperfile number
Specifies the maximum number of audit messages to be written to the XML files.
The default value is 500.

-recsperquery number
Specifies the maximum number of audit messages per database query. The value can affect command processing on the database-server side. A low value can increase the total processing time because of frequent network communication and more actions to be processed. On the other hand, a high value can require more database server resources and command processing can fail when these resources are not sufficient.
The default value is 500. Decrease this number if the export fails.

-to date_time | function
Specifies that only audit messages created up to the specified date and time (exclusive) or by the specified function are exported.

The format for the value of date_time is yyyy.MM.dd_HH.mm.ss_z . See the Java documentation for the SimpleDateFormat class for details and examples.

You can use the following functions in -from and -to options:

The parameter argument is a required integer value, where:

Note that audit messages usually use UTC zone in the audit message Identification - When field.

If the -delete option has been specified and an error occurs during export or the tool is interrupted, it can resume work: simply re-run the tool with exactly the same arguments.

To import audit messages from XML files to the DirX Audit Data Database, execute the following command:

dxtdbtool import common_options
-src path
[-delete | -dstdir folder_name]
[-recursive]

where

common_options
Specifies the common options. See the section “Common Options” for details.

-src path
Specifies the source file or folder. You can also use a folder containing data previously exported with dxtdbtool export saved as dxtdata_xxx.xml.zip.

-delete
Specifies that the XML files are deleted after import.

-dstdir folder_name
Specifies the destination folder to which the XML files are moved after import.

-recursive
Specifies that the XML files in all sub-folders are imported.

The options -delete and -dstdir are mutually exclusive. You can specify only one of them. After a successful import, the source file is either deleted from the file system (if -delete was specified) or moved from the source folder into destination folder (if -dstdir was specified).

The option -src can be either a path to a file or a folder. If it is a folder, only the files in this folder are processed unless the -recursive option was specified (all sub-folders and files are processed in this case).

The import operation expects the files to be in the same format and syntax as generated by the export operation; that is, in the form of archive files.

If an error occurs during the import process or the tool is interrupted, it can resume the import: simply re-run the tool with exactly the same arguments.

You can also use the DirX Audit file collector component of the DirX Audit Server to import the messages from the XML files into the DirX Audit Data Database. However, you must unzip all audit message files before you copy them into the collector’s input folder. Do not copy the information files there. Only the audit message files should be copied to this folder. See the section “Server File Collector for DirX Audit Format” in the DirX Audit Installation Guide for the configuration details.

To extend audit messages in the DirX Audit Data Database with audit events and dimensions, execute the following command:

dxtdbtool extend common_options
[-from date_time]
[-recsperquery number]
[-to date_time]

where

common_options
Specifies the common options. See the section “Common Options” for details.

-from date_time
Specifies that only audit messages created after the specified date and time (inclusive) are extended.

-recsperquery number
Specifies the maximum number of audit messages per database query. The value can affect command processing on the database-server side. A low value can increase the total processing time because of frequent network communication and more actions to be processed. On the other hand, a high value can require more database server resources and command processing can fail when these resources are not sufficient. The default value is 500. Decrease this number if the command fails.

-to date_time
Specifies that only audit messages created up to the specified date and time (exclusive) are extended.

The format for the value of date_time is yyyy.MM.dd_HH.mm.ss_z. See the Java documentation for the SimpleDateFormat class for details and examples.

Note that audit messages usually use UTC zone in the audit message Identification - When field.

Extending audit messages is not usually necessary, especially if you use the import tool. It is only necessary in specific cases and is then indicated in the appropriate places; in the DirX Audit Release Notes, for example.

Original messages can be optionally stored together with audit messages. In DirX Audit 7.0 SP1 and earlier versions, they were saved as text. As of DirX Audit 7.1, they can be saved in a compressed form, which can significantly reduce the database size.

The original message in the text form is saved in the ORIGINALMESSAGE column of the DAT_ORIGINALMESSAGES table. The original message in the compressed form is saved in the ORIGINALMESSAGE_COMPRESS column of the same table.

The command compresses the original message text data and stores it in the compressed form. It removes the text data from the table. When the command is completed, the table’s clustered index is rebuilt. Data space should be significantly reduced. The table size should be reduced to about 20 % of its previous size.

To compress or decompress original messages in the DirX Audit Data Database, execute the following command:

dxtdbtool compress|decompress common_options
[-from date_time]
[-recsperquery number]
[-to date_time]

where

common_options
Specifies the common options. See the section “Common Options” for details.

-from date_time
Specifies that only audit messages created after the specified date and time (inclusive) are compressed.

-recsperquery number
Specifies the maximum number of audit messages per database query. The value can affect command processing on the database-server side. A low value can increase the total processing time because of frequent network communication and more actions to be processed. On the other hand, a high value can require more database server resources and command processing can fail when these resources are not sufficient. The default value is 500. Decrease this number if the command fails.

-to date_time
Specifies that only audit messages created up to the specified date and time (exclusive) are compressed.

The format for the value of date_time is yyyy.MM.dd_HH.mm.ss_z. See the Java documentation for the SimpleDateFormat class for details and examples.

Note that audit messages usually use UTC zone in the audit message Identification - When field.

To purge audit messages data from the DirX Audit Data Database, execute the following command:

dxtdbtool purge common_options
-scope {DAT_AUDITMESSAGES | DAT_AUDITMESSAGES_ADDITIONS |
DAT_ORIGINALMESSAGES}
-filter file_path
[-paramsfile file_path]
[-params key[.DATETIME]=value[,key[.DATETIME]=value]…​]

where

common_options
Specifies the common options. See the section “Common Options” for details.

-scope {DAT_AUDITMESSAGES | DAT_AUDITMESSAGE_ADDITIONS |
DAT_ORIGINALMESSAGES}
Specifies the scope of the audit message data to be purged with a table name:

-paramsfile file_path
Specifies a path to the file containing a list of SQL select statement parameters and their values in the format key=value for common types and key.DATETIME=value for the date and time type.
The -paramsfile parameter can be omitted when no SQL select statement parameter is used or when only inline parameters are provided. When a parameter key is used both in the file and inline, the inline value takes precedence.

-params key[.DATETIME]=value[,key[.DATETIME]=value]…​
Represents a set of SQL select statement parameter key – value pairs. Optionally, the parameter can be of date and time data type.

The format for the value of date_time is yyyy.MM.dd_HH.mm.ss_z. See the Java documentation for the SimpleDateFormat class for details and examples.

You can use the same functions as described in the section “Export Audit Trail” to specify values of date and time parameters (dateparam).

Note that audit messages usually use UTC zone in the audit message Identification - When field.

These options allows computing missing audit events context data independently from the similar server job.

To generate missing audit events context in the DirX Audit Data Database, execute the following command:

dxtdbtool computeContext common_options
[-from date_time]
[-to date_time]
[-childrenlinkingbatch number]
[-maxresult number]
[-ordering string]
[-orphanprocessingbatch number]
[-orphanto date_time]

where

common_options
Specifies the common options. See the section “Common Options” for details.

-from date_time
Specifies that context is computed for audit messages created after the specified date and time (inclusive).

-to date_time
Specifies that context is computed only for audit messages created up to the specified date and time (exclusive).

-childrenlinkingbatch number
Batch size for linking children to parents. All children are always processed within a context processing iteration. This parameter only influences how many children are processed at once in a single batch. Default is 20000 for MSSQL and 1000 for ORACLE.

-maxresult number
Batch size per iteration. If an iteration lasts too long, lower the number.

-ordering string
Specifies the ordering of the context calculation. Possible values are ASC|DESC. DESC means, that contexts will be calculated from the newest messages to the oldest. The default value is ASC.

-orphanprocessingbatch number
Specifies the batch size for processing orphaned messages. This parameter determines how many orphaned messages are processed in a single context processing iteration. The default value is 5000.

-orphanto date_time
Specifies that orphaned messages older than the specified date and time (exclusive) are processed. It allow to handle old messages, that could not be associated with a context because they are older than the specified date and time.

The DB maintenance tool can delete history entries data from the DirX Audit History Database that have already ended. Specifically, it can remove rows of the following tables according to their VALID_TO column value:

To purge history entries data from the DirX Audit History Database, execute the following command:

dxthistdbtool purge common_options
-endedbefore date_time | function
[-endedafter date_time | function]

where

common_options
Specifies the common options. See the section “Common Options” for details.

-endedbefore date_time | function
Specifies that only history entries data ended before the specified date and time (exclusive) or by the specified function are deleted.

-endedafter date_time | function
Specifies that only history entries data ended after the specified date and time (inclusive) or by the specified function are deleted.

The format for the value of date_time is yyyy.MM.dd_HH.mm.ss_z. See the Java documentation for the SimpleDateFormat class for details and examples.

You can use any of the functions described in the section “Export Audit Trail” in the -from and -to options.

Purging history entries data using the purge tool can leave so-called “orphaned” entries. Orphaned entries are entries that have no corresponding attributes. Purging those orphans can be performed by executing the following command:

dxthistdbtool purgeorphans common_options
[-forcedelete]

where

common_options
Specifies the common options. See the section “Common Options” for details.

-forcedelete
In case of data inconsistencies in the database, some orphaned entries can have lingering attributes which fail the purge. This flag forces the deletion of orphaned entries together with any would-be lingering attributes.

Another approach to purging history entries is by purging ended entries. Ended entries are identified as entries, that have no valid attributes. As opposed to the purge tool, which only purged the ended attributes of entries, this approach purges the entire entries along with their attributes. Purging ended entries can be performed by executing the following command:

dxthistdbtool purgeended common_options
-endedbefore date_time | function

where

common_options
Specifies the common options. See the section “Common Options” for details.

-endedbefore date_time | function
Purge ended history entries whose validity ended before this date and time (exclusive).

To export history entries data from the DirX Audit History Database to JSON files, execute the following command:

dxthistdbtool export common_options
-dstdir folder_name
[-delete]
[-includestarted]
[-endedafter date_time | function]
[-endedbefore date_time | function]
[-entsperfile number]
[-nouidlist]
[-noorphans]

where

common_options
Specifies the common options. See the section “Common Options” for details.

-dstdir folder_name
Specifies the destination folder for the JSON files containing the exported history entries data.

-delete
Specifies that the history entries data that were ended in given date-time range are to be purged from the DirX Audit History Database after export.
A date and time range must be defined at least by the -endedbefore option. The default is not to delete history entries data. Started entries and attributes that have not ended are not affected.

-includestarted
Includes history entries data that were started in the given date and time range into the export process. The default exports only ended history entries data in the given date and time range.

-endedafter date_time | function
Exports only those history entries data ended after the specified date and time (inclusive) or by the specified function.

-endedbefore date_time | function
Exports only those history entries data ended up to the specified date and time (exclusive) or by the specified function.

-entsperfile number
Specifies the maximum number of history entries represented by JSON files to be written to the ZIP files. The default value is 500.

-nouidlist
Specifies that lists of entries identifiers will not be created. The default is to create such a list for every type folder.

-noorphans
Specifies that assignments without references to link attributes will not be exported. The default is to export such assignments into a separate folder.

The format for the value of date_time is yyyy.MM.dd_HH.mm.ss_z. See the Java documentation for the SimpleDateFormat class for details and examples.

To remove duplicate history entries data from the DirX Audit History Database, execute the following command:

dxthistdbtool remdup common_options
-searchby {ENTRY | DIRX_ENTRY_UUID | DN}

where

common_options
Specifies the common options. See the section “Common Options” for details.

-searchby {ENTRY | DIRX_ENTRY_UUID | DN}
Specifies whether the duplicate entries should be identified by entry, dirxEntryUUID (default) or DN:

To remove entries with duplicate dirxEntryUUID attributes from LDAP, and optionally synchronize the changes made in LDAP to the DirX Audit History Database, execute the following command:

dxthistdbtool ldapremdup common_options ldap_parameters
[-filluuid]
[-showonly]
[-synctohistdb]

where

common_options
Specifies the common options. See the section “Common Options” for details.

ldap_parameters
Specifies LDAP connection parameters. See the section “LDAP Connection Parameters” for details.

-filluuid
If entries with missing dirxEntryUUID attribute are found, they will have new values generated. By default, they are ignored.

-showonly
Only shows the empty or duplicated values for dirxEntryUUID attribute and DN values of corresponding entries.

-synctohistdb
Synchronize the changes made in LDAP to DirX Audit History Database. Therefore, if a duplicate dirxEntryUUID is changed in LDAP, the same action occurs in the DirX Audit History Database if the corresponding entry exists. Entries in LDAP and DirX Audit History Database are determined to be corresponding if they match either on dirxEntryUUID/DN or dirxEntryUUID/dxrUID value pairs.

To generate new unique values for missing dirxEntryUUID attributes of history entries in DirX Audit History Database, execute the following command. No modifications are made in LDAP.

dxthistdbtool filluuid common_options ldap_parameters
[-uuidfiltersize size]

where

common_options
Specifies the common options. See the section “Common Options” for details.

ldap_parameters
Specifies LDAP connection parameters. See the section “LDAP Connection Parameters” for details. If an entry with a missing dirxEntryUUID value is found in the DirX Audit History Database, the tool searches LDAP for a matching entry and if possible, uses this dirxEntryUUID value in DirX Audit History Database.

- uuidfiltersize size
Specifies the size of searchFilter used, when retrieving entries from LDAP by their dirxEntryUUID values. A value of 100 means, that the searchFilter will contain 100 dirxEntryUUID values, and up to 100 entries might be returned in a single batch. Increasing the value improves performance, however, if there are errors on the LDAP server, try selecting a lower value. Default value is 100.

To ensure unique history entries in DirX Audit History Database (with unique and non-missing dirxEntryUUID values), execute the following command:

dxthistdbtool makeunique common_options ldap_parameters
[-purge]
[-csvfile csvFile]
[-uuidfiltersize size]

where

common_options
Specifies the common options. See the section “Common Options” for details.

ldap_parameters
Specifies LDAP connection parameters. See the section “LDAP Connection Parameters” for details.

-purge
Resolve exact duplicates (same on all attributes) by purging all extra entries. For example in case of three exact duplicates, two will be purged from the DirX Audit History Database, and only one will remain (the one with the smallest ENTRY_ID will usually remain).

-csvfile csvFile
Output digests (summaries) of duplicate entries found in the DirX Audit History Database. This parameter can be used in combination with the -simulate option to check what entries would be modified without actually making changes in the DirX Audit History Database.

- uuidfiltersize size
Specifies the size of searchFilter used, when retrieving entries from LDAP by their dirxEntryUUID values. A value of 100 means, that the searchFilter will contain 100 dirxEntryUUID values, and up to 100 entries might be returned in a single batch. Increasing the value improves performance, however, if there are errors on the LDAP server, try selecting a lower value. Default value is 100.

To import LDAP entries from an LDIF file into the DirX Audit History Database, execute the following command:

dxthistdbtool importldif common_options files

where

common_options
Specifies the common options. See the section “Common Options” for details.

files
List of LDIF files to be imported into the DirX Audit History Database. Separate the filenames with a SPACE character. You can also specify directory names and use wildcard characters like *. Each LDIF file must contain a single entry type. The entry type is extracted from the filename. For example: account.ldif imports entries of entry type Account. A prefix delimited by an underscore (_) can also be used. For example: test_account.ldif also imports entries of entry type Account.

If you want to generate facts for all the audit messages in the tenant database (using Java based population; omit the start and end date as shown in the following command. Please make sure that you use the “simple plain hyphen” - in the command line.

dxtPopulateFacts -tenantid 4f753e1d-d0de-4aef-bb22-caace7342e99

This example assumes that DirX Audit is installed and configured for multi-tenancy and you want to start fact population for a specific tenant. The connection data for the DirX Audit Data Database, DirX Audit History Database and localization of the configuration files for the fact and dimension tables are stored in the tenant configuration.

The next example calculates all the facts until the end of year 2011 (for Java-based population). The connection data and path to the fact configuration files are taken from the configuration file of the only configured tenant (stored configuration) and it starts without prompting the user to confirm the DirX Audit Data and History Databases to be updated (because the -silent option is used; see the section “Common Options” for details). If multiple tenants are configured you must specify the tenant ID.

dxtPopulateFacts -endDate 2011.12.31 -silent

The next example calculates the facts from January 1st until the end of March 2015 (for Java-based population). The connection data for the databases and localization of the configuration files are taken from the tenant configuration. Only the DirX Audit History Database is populated and the user is prompted to confirm the database to be updated (because the -silent option is not used; see the section “Common Options” for details):

dxtPopulateFacts -tenantid 4f753e1d-d0de-4aef-bb22-caace7342e99 -startDate 2015.01.01 -endDate 2015.03.31 -disableData

The next example calculates the facts for all the audit messages only in the DirX Audit Data Database. The connection data for the databases and localization of the configuration files are taken from the tenant configuration.

dxtPopulateFacts -tenantid 4f753e1d-d0de-4aef-bb22-caace7342e99 -disableHistory

The next example calculates the facts in the DirX Audit Data Database for specified fact tables (FCT_EVENTS, FCT_USERS) and their required dimensions. The DirX Audit History Database is not populated. The connection data for the databases and localization of the configuration files are taken from the tenant configuration. Note that if calculating The DirX Audit History Database is not disabled, all its fact will be calculated.

dxtPopulateFacts -tenantid 4f753e1d-d0de-4aef-bb22-caace7342e99 -factTablesData FCT_EVENTS,FCT_USERS -disableHistory

The next example calculates the facts in the DirX Audit History Database for specified fact tables (FCT_HST_ENTRIES, FCT_HST_USERS) and their required dimensions. The DirX Audit Data Database is not populated. The connection data for the databases and localization of the configuration files are taken from the tenant configuration. Note that if calculating The DirX Audit Data Database is not disabled, all its fact will be calculated.

dxtPopulateFacts -tenantid 4f753e1d-d0de-4aef-bb22-caace7342e99 -factTablesHistory FCT_HST_ENTRIES,FCT_HST_USERS -disableData