Implementing a RESTful Connector

This chapter describes guidelines for implementing a connector for RESTful Web Services. Please consider these guidelines, especially the ones on workflow configuration and mapping: most often they can be applied with only slight adaptations.

Documentation

To understand this topic, we recommend that you read Chapter 9, "Implementing a Custom Connector (General)" in this guide and the additional documentation listed in its "Documentation" section.

The inline Java documentation of the REST-related interfaces and implementing classes is part of the whole connector framework in the folder Documentation/DirXIdentity/ConnFrameWork on the product media. See the section REST framework for the REST-related interfaces and classes.

Workflow Configuration and Connector Attribute Handling

This section provides guidelines for configuring the workflows, especially the mapping. Please follow these guidelines unless there is an important reason not to do so.

Configuring All Entry Types

This section provides guidelines for configuration that apply to all entry types.

Handling the Entry Identifier

Most REST services generate a unique identifier for each entry (user, group or role) that they create. They use it to identify the entry in service requests or in (user) attributes as a reference to a linked entry.

The REST service returns this identifier in the response to a create request. The connector should return it in the identifier of the Add Response with an identifier type "GenericString". The workflow should map it to the dxrPrimaryKey Identity domain attribute. In the mapping to the connected system, the dxrPrimaryKey is used as the identifier for all requests except the Add. As the identifier is not initially known in Identity, the identifier in an Add Request is left empty.

Distinguishing Users and Groups

Use operational attributes to allow the connector to distinguish operations between users and groups. For the channel to the connected system, set the operational attribute "objType" to the right value; use either "user" or "group". For additional types, take a value of your own; for example, "role". Here is the snippet for the accounts channel:

<operationalAttrMapping mappingType="constant" name="objType">
    <value>user</value>
</operationalAttrMapping>

Searching All Entries of a Type

For searching all entries of a type (for example, all users), leave the search base empty. This means no string "all" or similar.

Searching an Entry with ID

For finding the joined entry in the connected system, use its Identifier (dxrPrimaryKey) and pass it as the search base in the search request.

Handling Deleted Entries

If the REST service supports a delete operation, the mapping is straightforward as for other types of systems:

If the dxrState of the Identity entry (account or group) is DELETED, the mapping needs to create a delete request. It performs this task by setting the request type to DELETE as in this snippet:

targetMapResult.setRequestType(Request.Type.DELETE)

In the opposite direction from the connected system to Identity, set dxrTsState to DELETED if the REST entry doesn’t exist or some state attribute indicates the deleted state.

Some systems do not support deleting entries. If they provide an attribute for indicating the deleted state, the mapping is straightforward: configure a Java mapping that provides the appropriate values for both directions. If the service does not provide this kind of an attribute out of the box, consider either appropriating an existing attribute that is not needed in your environment or extending the schema of the connected system to introduce this attribute.

Configuring Accounts

This section provides guidelines for configuring accounts.

Handling the Logon Name

The dxrName attribute of an Identity account should contain the logon name of the user in the connected system. The object description for the accounts must generate a unique value. A good choice is to take the user’s email address. Check the existing template descriptions for samples.

Handling Enable / Disable

In principle, we distinguish between the following cases:

The REST user contains a state attribute; for example, isSuspended.

Use a direct mapping between the Identity attributes dxrState / dxrTsState and the REST attribute. You will typically need to write a Java mapping because the REST service will almost never support the same values as Identity (ENABLED, DISABLED, TOBEDELETED, etc.).

The REST service supports suspend / enable operations.

The connector should support a virtual attribute for the state, let’s say dxrState. The mapping can be as in the case above and map the Identity states to ENABLED / DISABLED.

The connector must evaluate this attribute in SPML requests passed by the join engine and issue the appropriate suspend / enable operations to the REST service. The connector must also populate this state attribute with ENABLED / DISABLED when returning a searchResultEntry. Often the REST responses contain appropriate attributes that are read-only in these cases.

Configuring Account – Group Memberships

In the Identity domain, always store group memberships at the account. The referenced attribute in the group is dxrPrimaryKey.

At the target system entry, you configure this in the Advanced tab by enabling the flag "Reference Group from Account" and setting "Referenced property" to "dxrPrimaryKey. This setting assumes that the identifier of the REST entry is stored in dxrPrimaryKey (as described in previous sections) and is used in the REST service for linking users and groups.

In the workflow configuration in the Connectivity database, you need to link the members channel from the accounts channel (General tab). In the mapping to the REST service, use the virtual attribute "members" for the member values (that is, the values taken from dxrPrimaryKey). The REST connector must evaluate them appropriately, as described in the following sections.

Configuring the Connector

Use the existing standard names for the connector’s configuration parameters as much as possible. Use one of the existing connectors as a template; for example, Salesforce or Office365.

The <connection> attributes user, password, ssl, server and path contain the user’s logon name, its password, whether to use “http” or “https”, the address of the REST service and the relative path to the application respectively.

Authentication is often done via OAuth. In these cases, the connector must send authentication requests to an OAuth service. The attribute authEndpoint should contain the relative or absolute path to the OAuth service depending on whether or not it is located on the same server as the "productive" REST service. The connector must identify itself as an application and therefore provide its identifier (in OAuth, it is called the client or application ID) and its secret (kind of password). The client ID and client secret should be configured in the attributes clientID and clientSecret respectively. Use the LDAP attributes dxmClientID and dxmClientSecret to store them in the connected directory. Usually, the connector must also provide the name of a user and a password. They should be configured in the attributes user and password and taken from the bind profile.

If the REST service can only be accessed through a proxy, store its server and port in an additional connected directory, reference it from the connected directory via the attribute dxmProxyServer-DN and pass the values as proxyHost and proxyPort properties to the connector.

Here is a sample snippet of a <connection> template with proxy and OAuth properties:

<connection
password="${DN4ID(THIS)@dxmBindProfile-DN@dxmPassword}"
server="${DN4ID(THIS)@dxrConnectionLink@dxmService-DN@dxmAddress}"
ssl="${DN4ID(THIS)@dxmSSL}”
user="${DN4ID(THIS)@dxmBindProfile-DN@dxmUser}">
	<property name="clientId" value="${DN4ID(THIS)@dxrConnectionLink@dxmClientId}"/>
	<property name="clientSecret" value="${DN4ID(THIS)@dxrConnectionLink@dxmClientSecret}"/>
	<property name="securityToken" value="${DN4ID(THIS)@dxrConnectionLink@dxmBindProfile-DN@dxmSpecificAttributes(securitytoken)}"/>
	<property name="authEndpoint" value${DN4ID(THIS)@dxrConnectionLink@dxmSpecificAttributes(oauth_path)@dxmAddress}"/>
	<property name="path" value="${DN4ID(THIS)@dxrConnectionLink@dxmSpecificAttributes(servicepath)}"/>
	<property name="proxyHost" value="${DN4ID(THIS)@dxrConnectionLink@dxmProxyServer-DN@dxmService-DN@dxmAddress}"/>
	<property name="proxyPort" value="${DN4ID(THIS)@dxrConnectionLink@dxmProxyServer-DN@dxmService-DN@dxmDataPort}"/>
</connection>

Only the parameters user, password, ssl, server and port are XML attributes. The others must be configured as <property> elements within the <connection> element.

Connector Structure

A connector typically performs the following steps when processing a provisioning operation (add, modify, delete, search):

Before the first operation: authenticate, mostly via OAuth or basic authentication.
Determine the type of an entry: user, group. Sometimes also other types need to be supported.
Transform to the specific REST operation(s): the HTTP operation POST, PATCH (practically never: PUT), DELETE or GET, sometimes with a payload that is most often JSON.
Often not only one, but a series of operations must be sent. As a result, entry attributes need to be split. A typical example: user – group memberships are managed and queried in additional operations.
Send the HTTP operation and evaluate the response code.
For create, extract the identifier of the new entry. For search, evaluate the JSON payload and transform to SPML search result entries.

The connector framework supports this process by providing a connector template that follows this sequence and uses a couple of interfaces and sample implementations together with other common utilities.

The REST template connector is implemented in the AbstractRestSampleConnector class. Its source is provided in the folder Additions/RESTfulConnector of the product media. The following sections describe how it processes selected requests and the interfaces it requires.

A specific connector can extend this template and can override some of its methods when necessary. Normally, it must provide its own implementations for the important interfaces: the request transformer, the command producer and the response evaluator.

REST-Specific Interfaces

The following figure illustrates the REST-specific interfaces that the AbstractRestSampleConnector uses:

Figure 1. REST Interfaces used by the Abstract REST Connector

The AbstractRestConnector class provides the configuration parameters in the ConfigurationOptions convenience class.

The interfaces and basic implementations used include:

ISpmlRequestTransformer – transforms the given SPML requests to attribute sets or – for a search request – to filters and requested attributes. A sample implementation is SimpleSpmlRequestTransformer.

ICmdProducer – produces the REST operations (GET, POST, etc.) that correspond to the SPML requests (search, add, etc.). A sample implementation is SimpleCommandProducer.

IRESTSender – sends the REST operations to the remote REST service. If available, it uses an authentication control for authentication. The recommended implementation is CXFRSSender, based on Apache CXF.

IAuthenticationControl – performs authentication and adds the corresponding HTTP header to the REST operation. An implementation for OAuth2 is OauthPasswordAuthenticator.

IResponseEvaluator - evaluates the REST responses and supports transforming them to SPML responses. A sample implementation is BasicRspEvaluator.

All these interfaces support an open method which is used for passing the configuration parameters. For details, see the next sections.

Add User

This section shows how these interfaces work together to create a user.

Each connector implements an add method that receives a SPML AddRequest and returns a SPML AddResponse. The template connector extracts the entry type (here we assume User) from the operational attribute (see the section "Configuring All Entry Types") and then calls the specific method (here: addUser).

The following sequence diagram illustrates the first set of actions in addUser:

Figure 2. addUser Actions - Part 1

The REST connector prepares an AddResponse and then calls the request transformer’s userAdd method to separate the attributes from the AddRequest into sets as a preparation for the subsequent commands. For adding a user, we expect one command for creating the user and a list of commands to add the new user to groups. Consequently, the transformer will produce one set with all the attributes required for the user creation and another set with the group memberships. The attribute set for the create operation is a map with the attribute names as keys and the values as either String or list of Strings.

The connector then forwards these attribute sets to the command producer in the method addUser. The command producer produces a REST request, which consists of the HTTP method (in this case: POST), an HTTP path, the attribute set as payload and query parameters.

The connector forwards the command to the REST sender. The sender uses Apache CXF to issue REST operations. The sender handles authentication. If an authenticationControl was set, the sender asks it to extend the HTTP headers. The authenticationControl also might change the endpoint of the REST service. If no authenticationControl is available, the sender assumes basic authentication and produces the appropriate header itself. For more details on authentication, see the "Authentication" section.

The sender transforms the payload to JSON and then sends the REST operation using CXF. It accepts only JSON as the response format and populates the result (payload, HTTP result code along with the REST request) into a REST response and then returns it to the connector.

The connector passes this response to the response evaluator. The evaluator creates and returns an evaluated response. For an addRequest, the response should contain the identifier of the new entry.

The connector puts this identifier into the prepared SPML AddResponse and into the attribute sets so that it can be used in the subsequent commands.

The next actions for handling group memberships and additional attributes are shown in the next diagram:

Figure 3. addUser Actions – Part 2

The connector asks the command producer for the commands to create user-group memberships, passes each of them to the REST sender and then asks the evaluator to check the responses. On failures, the evaluator throws an exception and thus stops processing, causing the connector to return an error response.

For some applications, additional commands should be sent to fully provision a new user. This function is handled in the connector method addUserAdditional. The connector asks the command producer for the list of specific REST commands, passes them to the sender and then lets the evaluator check them.

The handling of modify and delete requests is very similar.

Search

This section shows how the interfaces work together to search users. It includes searching in iterations, where the REST service provides the result in chunks, each of which must be explicitly retrieved.

First Search

In the search method, the connector receives an SPML SearchRequest and returns a SearchResponse. As in the other methods, the template connector extracts the entry type (here we assume User) from the operational attribute (see the section "Configuring All Entry Types") and then calls the specific method (here: searchUser).

The following sequence diagram gives an overview of the actions in searchUser. Searches for other entry types are very much the same.

Figure 4. searchUser Actions

First, the connector prepares a SearchResponse that is returned to the join engine at the end. To support paging, it creates a special PagingSearchResponse. The PagingSearchResponse extends the normal SPML Search Response and provides additional features for retrieving the subsequent page of result entries when the current page has been processed. This implementation ensures that always only one page is in memory so that very large result sets can be processed.

By calling the method userSearch, the connector asks the request transformer to produce a search context. A search context contains the information needed for issuing a query: filter, search base, requested attributes, and operational attributes as well as the original SPML request and the current SPML response.

With this search context, the command producer creates a search command context. A search command context contains a list of REST requests: one basic (that is, the first or main) and an optional list of additional ones. The basic search command is supposed to obtain a list of entries matching the filter with a set of attributes. The other commands serve to obtain additional attributes per entry.

The rest of the search is handled in an extra method, searchUsers, which expects the search response and the search context as parameters. This method is also used when retrieving subsequent result pages so that the common actions can be handled by only one method.

The connector passes the basic search command to the REST sender and passes the returned REST response to the response evaluator along with the search command and the search context. The evaluator transforms the result to a list of SPML SearchResultEntries and returns them in an evaluated response.

For each result entry, the connector asks the command producer to create further commands for retrieving additional attributes of the entry (here: user). The connector passes them to the REST sender and asks the response evaluator to extract them from the REST response to the found SPML result entry. After the last command, the connector adds the result entry to the SPML search response.

Retrieving Subsequent Pages

For convenience, the join engine of the provisioning job (aka job controller) obtains the list of result entries one by one by calling the method enumerateSearchResultEntry of the search response. This gives the search response class control and allows it to re-issue subsequent searches after the first chunk has been processed. As a disadvantage, it might not know the total number of result entries at the beginning.

The PagingSearchResponse delegates enumeration of entries to the underlying SPML Search Response. As long as it has more elements, it just returns them to the join engine.

If the first chunk is exhausted, the PagingSearchResponse requests the next chunk from the connector by calling its method getNextPageUser (for users) and passes the current search context.

The connector asks the command producer for the command to retrieve the next page (here: of users). If there is one, it calls the same method searchUsers as in processing the original search request. When it returns, the entries of the current page are in the search response.

Now the PagingSearchResponse can again enumerate over the result entries and pass them to the join engine.

Read an Entry

As mentioned above, reading a single entry should be implemented by a search request with a search base. The search base represents the identifier of the entry.

The connector performs the same actions as outlined above. Just setting the response should be a bit different. If the entry cannot be found, the response should have an error code NOSUCHIDENTIFIER and a custom error code NO_SUCH_OBJECT. This is important for the provisioning join engine; in that case it continues trying to find the joined entry with the next join expression.

Sample Connector

We provide the sources for a sample connector BasicScimConnector as a sample. The sample extends the AbstractRestSampleConnector template and just sets the OAuth Authentication control and a simple response evaluator.

Note that despite its name, the connector does not (yet) implement the SCIM specification. It takes some elements of SCIM to provide a simple demonstration rather than invent a new REST protocol.

Authentication

Authentication is handled within the REST sender.

The (derived) connector can set an authentication control in the REST sender. An authentication control must implement the interface IAuthenticationControl.

IAuthenticationControl methods include:

open – used for passing the configuration parameters.

getAuthenticationHeaders – allows the authentication control to return the authentication- and authorization-relevant HTTP headers that must be added to a REST request.

getDynamicEndpoint – allows the authentication control to return an application endpoint generated dynamically from the authentication service, thereby instructing the REST sender to use it for the next REST request.

evaluateErrorResponse – allows the authentication control to check the result code and error message of a JAX-RS response to determine whether or not to refresh an access token for the next request.

If no authentication control is passed, the sender uses the user name and password from the configuration for basic authentication.

If an authentication control is available, the connector asks the control for additional (authentication and authorization) headers and then adds them to the REST command. The control may also provide an endpoint to which requests are subsequently to be sent. This parameter supports scenarios where the address of the REST service is determined by the authentication service.

The framework provides an authentication control for OAuth2, the OauthPasswordAuthenticator. It obtains a new token when required and a refresh token once the previous is expired and adds it in a HTTP header "Authorization Bearer".

For accessing the OAuth2 service, the control evaluates the configuration options authorizationServerUrl (address), clientId and clientSecret (see OAuth2): identification or user name of the connector as a client application and its password.

ISpmlRequestTransformer Interface

An implementation of ISpmlRequestTransformer must evaluate SPML requests for users, groups and other entry types and then return appropriate attribute sets:

For add, modify and delete requests - an object that implements ISeparatedAttributes. A sample implementation is BasicSeparatedAttributes. This kind of object provides the attributes in various sets:

Basic attributes for the first or main operation to create or modify an entry.

Attributes for either adding a user to a group or removing it.

The identifier of the (new) entry.

The list of additional attribute sets, when more operations are needed to create, update or delete an entry.

For search requests – an object that implements ISearchContext. A search context provides the search filter, requested attributes, a search base and operational attributes. A sample implementation is BasicSearchContext.

In the open method, the connector passes the configuration options for the entire connector that might also be used by the request transformer.

A sample implementation for a request transformer is SimpleSpmlRequestTransformer. It assumes that the attribute "memberOf" is the one that contains the groups of which a user is member and provides them as groupMemberToAdd/Delete. All the other attributes go into the basic attribute set. It also respects a list of multi-valued attributes. The values for these attributes are provided as lists. The names of multi-valued attributes can be set from the connector.

From a search request, it extracts the search base and the requested attributes, but not (yet) the filter.

ICmdProducer Interface

An implementation of the ICmdProducer interface produces the REST commands for creating, updating, deleting and searching for users, groups and other entries. The main inputs are the attribute sets and search contexts produced by the request transformer.

In the open method, the connector passes the configuration options for the entire connector that may also be used by the command producer.

For each entry type (user, group or other) there are the following methods:

*Add / modify / delete*Type – for creating / updating or deleting the basic attributes of an entry. The method is expected to produce a REST request with either a POST, PUT, PATCH or DELETE operation, the relative path that needs to be added to the application endpoint, the query parameters, the headers and the (preferably JSON) payload.

ProcessUserToGroups – for returning the REST requests for adding a user to and removing it from groups. The returned list typically has a request per user-group membership.

Add / modify / delete*Type*Additional – for performing additional requests for managing special attributes or privileges such as licenses, profiles, etc. The command producer returns a (potentially empty) list of REST requests that are processed one after the other.

*Search*Type – for finding all entries (of the specific type user, group or some proprietary one) matching the filter. Typically, the searches will try to find all entries of a given type (for example, all users) or only one entry associated with the given source entry (called the joined entry). In this case, the filter will just reference a unique identifier (the primary key) or a few basic attributes. These methods expect a search context as input and return a search command context.

Search*Type*Additional – for finding the additional attributes of a given entry. The method expects the search context, the search command context produced by the previous SearchType method and the evaluated REST response associated with the previous search as input parameters. It returns a search command context with explicitly the additional typed search requests populated. The type allows for distinguishing several attribute sets (for example, groups, licenses). Unfortunately, the identifier of the individual entry is not known to the producer at the time this method is called. As a result, the connector must currently insert it somehow into the REST command.

*GetNextPage*Type – for retrieving the next page (chunk) of result entries if there are some left. The information needed to decide that should be put to the search context by the response evaluator.

The SimpleCommandProducer class provides a straightforward implementation of a command producer inspired by SCIM. It can be used as a copy/paste starting point for specific command producers. The framework also provides implementations for search command context and REST requests that are basically property holders.

IRESTSender Interface

An implementation of the IRESTSender interface sends REST requests according to REST command objects passed to it and returns corresponding REST responses. The sender handles authentication. The connector may pass an authentication controller, which the sender then uses to obtain authentication headers and/or query parameters.

Interface Methods

In the open method, the connector passes the configuration options for the entire connector that can also be used by the sender.

The setAuthenticationControl method expects an authentication controller implementing the interface IAuthenticationControl. For details on authentication, see the "Authentication" section.

In the send method, the sender sends a REST request according the given REST command (interface IRESTRequest; see the section "ICmdProducer Interface". The sender returns a REST response object that holds the HTTP result code, the full original response, the payload, the headers and potentially an error message.

CXFRSSender Implementation

The framework provides the CXFRSSender class as an implementation of the IRESTSender interface. CXFRSSender uses Apache CXF for sending REST requests and evaluating the responses.

The sender accepts only JSON REST responses. It also assumes JSON for the payload of requests, which can be overridden by the request media type. The Jackson JSON provider helps transform the message payload to and from Maps. Consequently, the connector and the other components don’t need to manage JSON handling; they simply produce and consume Java Maps.

When the connector doesn’t set an authentication control, CXFRSSender assumes basic authentication and uses the configured user and password for producing the authentication header.

When a proxy host is configured, the sender instructs the CXF Web client to send the requests to the proxy host and port from the configuration. Currently, no authentication for the proxy is supported.

IResponseEvaluator Interface

A response evaluator implements the IResponseEvaluator interface and needs to check the REST responses, extract identifiers and attributes from the payload, transform them to SPML and then feed them into the evaluated response.

The interface methods are organized according to entry types (user, group, other) and operations (add, modify, delete, search). In the open method, they receive the configuration parameters for the connector.

All the methods must check the response for errors. If the HTTP result code indicates failure, the REST sender sets the result of the internal REST response to failure. All the methods get the REST request and the corresponding REST response with the payload.

Some of the methods (modify, delete) normally do not contain a payload or the payload is not evaluated by the join engine. So the evaluator can simply ignore them.

When an entry is created, the response payload most often contains the identifier of the new entry. In the methods evaluateTypeAdd, the evaluator must extract it and put it into the evaluated response.

The payload of GET operations contains the matching entries as a list or a single entry. The evaluator receives the payload in two groups of operations:

evaluateTypeSearch – the payload is expected to contain the basic attributes of the matching entries. The evaluator must transform each of them into a SPML result entry and then put them into the search result entries list of the evaluated response. Each result entry needs to have an identifier and can have a list of single- or multi-valued attributes.

For paging support, the evaluator must also read response attributes that tell whether the result is complete or how to retrieve the next chunk. This information should be stored in the search context.

addTypeAttributes – the payload contains additional attributes of an entry. The evaluator must add them to the existing SPML entry. The entry with the attributes already obtained is passed as an input parameter. Another parameter passes a type that if present indicates the attribute set that is expected in the payload. A typical type will indicate the groups of which a user is a member.

The BasicResponseEvaluator class provides a simple implementation of the interface and returns the simple implementation BasicEvaluatedResponse. This evaluator checks the REST response and throws an exception on failure. A specific implementation can extend it and then must implement the evaluate*Search methods at a minimum, and frequently the add*Attributes methods, too. The ScimRspEvaluator implementation shows how to extract simple String attributes from the payload and create SPML search result entries and the SPML identifier. It also reads the number of total results, start index and items per page that can be used to decide whether a request to retrieve a subsequent chunk of entries makes sense.

REST Utilities

The RESTUtils class provides some utility methods that are helpful for sending and evaluating REST requests and responses.

Some methods implement transformation between JSON strings and maps and lists or to base64 strings.

Other methods check the HTTP status code with respect to the HTTP operations.

The remaining methods support logging requests and responses.

SPML and Framework Utilities

The connector framework provides some utilities that can help to manage SPML requests and responses. You can find them in the Java package com.siemens.dxm.spml.util.

The ResponseCreator provides methods for creating a success or failure response that accept the SPML request and an error message or an exception.

The Serializer serializes a SPML request and response to a string, which can be useful for logging.

The SpmlUtils class provides methods for:

Creating attributes from a name and value or list of values.
Obtaining attributes as a map, get values as map or list.
Obtaining the first value from an attribute map or list.
Obtaining the identifier string or produce an identifier from a string value.
Getting or setting some specific operational attributes, such as the scope or the object type.

Examples

The sources for all the REST-related interfaces, implementing classes and utilities are provided in the folder RestFramework on the product media.