Chapter 6 OpenLDAP slapd.conf database bdb and hdb

This section deals with bdb (and hdb) database specific directives. Generic database directives are here. The Berkeley DataBase (BDB or HDB) (now owned by Oracle) was the preferred and recommended back-end for OpenLDAP (now superceded by mdb) but requires very careful configuration if the application demands high performance. The parameters defined here form only part of the performance definition - others may be contained in the DB_CONFIG file (see also Performance).

Warning: The BDB transaction log file (log.xxxxxxxxxxxx in the LDAP directory) can become very big, very quickly and requires regular maintenance - see BDB Reference and Chapter 25 for maintenance utilities and Chapter 12 for log file removal.

A number of the parameters in this section of the slapd.conf file are passed through to BDB. These parameters may be removed from cn=config (slapd.conf file) by using a DB_CONFIG file (defined in olcDbConfig for cn=config) with the corresponding directive. It appears that increasingly the OpenLDAP advice is to use DB_CONFIG in preference to othe configuration attributes or slapd.conf directives. Where this is possible, it is noted in each directive.

Contents

1. cachesize
2. checkpoint
3. dbnosync
4. directory
5. dirtyread
6. index
7. lockdetect
8. mode
9. searchstack
10. DB_CONFIG sample file
11. Sample database bdb fragments

cachesize

Format:

cachesize integer

The cachesize directive defines the number of entries that the LDAP backend will maintain in memory. Do not confuse this directive with the BDB set_cachesize directive - they control different behaviours.

For maximum performance this figure should be as high as practical or as close as practical to the number of records maintained in the directory. The default is 1000. Examples:

cachesize 10000
# LDAP maintains 10,000 entries in memory 

See also Performance chapter.

checkpoint

Format:

checkpoint kbyte min

The checkpoint directive defines the time between checkpoint operations in BDB (the database can only be recovered from the last checkpoint).

The frequency of checkpointing determines the time during which data may be unrecoverable by BDB in the event of a system failure. If NOT using the dbnosync this time could be set to a reasonably long period, say, 10 mins or more, if the dbnosync directive is being used 5 - 15 mins or less if practical. kbytes is the number of kilobytes written to the directory and min is the time in minutes. Whichever occurs first determines the period between checkpoints.

OpenLDAP default is NO CHECKPOINTING - you should always supply a checkpoint directive. See the BDB Chapter 12 Section 15 documentation for more information. This directive may be replaced by using DB_CONFIG file with the txn_checkpoint directive. Examples:

checkpoint 128 15
# check point whenever 128k data bytes written or 
# 15 minutes has elapsed whichever occurs first

dbnosync

dbnosync

dbnosync
# updates are allowed to remain pending 

directory

Format:

directory /path/to/directory

The directory directive defines the location of the BDB database files. By convention these files are placed /var/db/openldap-data (BSD) or /var/lib/ldap (FC/Linux) but they can be placed anywhere convenient. The directory MUST exist before OpenLDAP is started for the first time.

If you do not define this entry then OpenLDAP will default to a directory name of /var/db/openldap-data (BSD) or /var/lib/ldap (FC). If you specify a relative directory name (without a leading '/') BDB will prepend its DB_HOME environmental variable - which could be anywhere in the known universe.

While defaults are available the OpenLDAP documentation suggests (but does not enforce) that a directory directive is MANDATORY. If for no other reason than to save time trying to figure out where files are located you should always include this directive.

If you are going to maintain or have plans to create a multi-DIT (needs multiple database sections) directory it is sensible to create a sensible subdirectory naming scheme for these database files. Examples:

directory /var/ldap/example-com
# the directory example-com must exist 
# directory permissions for read and write for ldap user

directory example-com
# the value of DB_HOME (for BDB) will be prepended
# to create /user/local/share/openldap/example-com (BSD)

dirtyread

Format:

dirtyread

The dirtyread directive allows OpenLDAP to return in-memory data that may not yet have been written to disc. If the subsequent write operation fails this data will be discarded. The user may have received a result that incorrectly reflects the final state of the directory.

Specifying this option may speed up directory performance at the risk of some inconsistency. The default (parameter not present) is not to return in-memory data that has not yet been written to disc. Examples:

dirtyread
# user MAY receive a result that is inconsistent
# with the final state of the directory

olcDbIndex (index)

The index directive of slapd.conf is only effective on initial load of the directory (using ldapadd). If indexes are subsequently changed the directory needs to be re-indexed using slapindex (caution: depending of the version must stop slapd first).

In OLC (cn=config) the attribute used is olcDbIndex which can only appear in an olcDatabase={0}config entry. Additionally re-indexing is done in real time so any change to an attribute immediately triggers an indexing change (slapindex is not required). However, it is not clear using OLC (cn=config) when re-indexing is completed - there is no visible sign to indicate completion.

Format:

# OLC (cn=config) form
olcDbIndex: attrlist | default indices

# slapd.conf form
index attrlist | default indices

# indices = [pres [,approx] [,eq] [,sub] [,special]]

The olcDbIndex/index attribute/directive defines what indexes will be maintained by OpenLDAP. Any number of index parameters may be included. If an attribute does not appear in an index directive it can still be used in a search filter - but if it occurs frequently it will hurt performance - once in a lifetime is not too bad!

attrlist may be either a single attribute or a comma separated list.

The optional parameter default stores the supplied indices and uses them on any subsequent index parameter that does does not have an indices entry. The default value must be defined before any olcDbIndex/index which does not have a indices value. A subsequent default will be used for index parameters following the new default.

pres should be used if use searches of the form 'objectclass=person' or 'attribute=mail' will be used.

approx MUST be used if use searches of the form 'sn~=person' (a 'sounds-like' search) will be used.

eq should be used if searches of the form 'sn=smith' will be used i.e no wildcards are included (uses the EQUALITY rule only).

sub should be used if use searches of the form 'sn=sm*' i.e wildcards are included (uses the SUBSTR rule). This rule may be enhanced by a using subinitial (optimised for 'sn=*s'), subany (optimised for 'sn=*n*') or subfinal (optimised for 'sn=th*'). One or more sub parameters may be included.

special may be either nolang or nosubtypes which are related to subtypes.

Careful attention to what indexes are maintained based on the application requirements will significantly affect directory read performance - conversely there is no point in indexing an attribute if no searches ever access it. If all the searches use EQUALITY rules only then there is no point in indexing for sub. Indexes (indeces) consume memory (more indexes = more memory) and write or modify operations will take longer due to index updates.

Examples:

# simple use of the default value
# OLC (cn=config) form
olcDbIndex: default pres,eq
olcDbIndex: cn,sn,uid

# slapd.conf form
index default pres,eq
index cn,sn,uid

# defines presence and equality indexes for 
# attributes cn, sn and uid
# exactly the same as the three index directives below
index cn pres,eq
index sn pres,eq
index uid pres,eq

index cn eq,sub,subinitial
# creates indexes for attribute cn (commonname)
# EQUALITY, SUBSTR searches and further optimises
# for sc=a* type searches

index sn eq,approx,sub
# creates indexes for sn (surname) on
# EQUALITY and SUBSTR searches
# NOTE: The approx index is a waste of time because
#       there is no ORDERING rule for sn approx is present
#       only to illustrate the parameter exists

index mail pres,eq,sub
# creates indexes for attribute mail on
# presence, EQUALITY and SUBSTR

index objectclass eq
# optimises searches of form objectclass=person

See also Performance chapter for a further review of performance.

lockdetect

Deprecated in favour of the DB_CONFIG file set_lk_detect parameter.

mode

Format:

mode mask

The mode directive defines file permissions used when the BDB database files are initially created. The default is 0600 (ldap user only - read and write permission - no others). Examples:

mode 0660
# ldap user - read and write
# ldap group - read and write
# others - no access 

searchstack

Format:

searchstack depth

The searchstack directive defines the depth of the stack used for search filter evaluation. Search filters are evaluated on a stack to accommodate nested AND / OR clauses. An individual stack is assigned to each server thread. The depth of the stack determines how complex a filter can be evaluated without requiring additional memory allocation.

Filters that are nested deeper than the search stack depth will cause a separate stack to be allocated for that particular search operation. If the number of nested conditions exceeds the value defined (or the default) significant performance penalties will be incurred.

Each search stack uses 512K bytes per nesting level. The default stack depth is 16, thus 8MB per thread is used. Examples:

searchstack 5
# allows up to 5 nested search statements

DB_CONFIG Sample File

The decision as to whether or not to use a DB_CONFIG file is simple - if you care about performance use it. If you don't care - forget it. BUT if you elect not to use a DB_CONFIG you MUST use a checkpoint directive. Increasingly the OpenLDAP advice seems to be to use DB_CONFIG rather that pass parameters through via slapd.conf.

The DB_CONFIG file is located as defined in the directory directive. The directives used in this file are fully, if not clearly, explained in the BDB Manual. You can replace the checkpoint, lockdetect, dbnosync directives with the equivalent directives in the DB_CONFIG file as well as add additional tuning directives as required.

Example DB_CONFIG file:

# replaces lockdetect directive
set_lk_detect DB_LOCK_EXPIRE
# OR set lk_detect DB_LOCK_DEFAULT

# uncomment if dbnosync required
# set_flags DB_TXN_NOSYNC
# multiple set_flags directives allowed

# sets max log size = 5M (BDB default=10M)
set_lg_max 5242880

set_cachesize 0 5242880 1
# sets a database cache of 5M and 
# allows fragmentation
# does NOT replace slapd.conf cachesize 
# this is a database parameter

txn_checkpoint 128 15 0
# replaces checkpoint in slap.conf
# writes checkpoint if 128K written or every 15 mins
# 0 = no writes - no update 

Note:

1. If you have a multiple DITs (multiple database sections in slapd.conf) a DB_CONFIG file is required for each directory containing a database.
2. To check some performance stats of the database use:

   cd /to/bdb/directory
   db_stats -m
   
   # OR on BSD
   
   db4_stat -m
   

3. It is not clear from the documentation what happens if you have conflicting parameters in slapd.conf and DB_CONFIG. In general - DON'T! i.e if you have a checkpoint directive in slapd.conf and a txn_checkpoint directive in DB_CONFIG - remove one or the other.

Sample database bdb fragments

The following database specific fragments are shown to illustrate configuration of a DBD database. The performance is not optimised.

Example 1 - no DB_CONFIG file:

database bdb
...
# database specific fragment only
# order not important
cachesize 10000
checkpoint 128 15
directory /var/ldap/first-dit
dbnosync
dirtyread
index mail pres,eq
index objectclass pres
index default eq,sub
index sn eq,sub,subinitial
index telephonenumber
index cn
searchstack 5

The same configuration with a DB_CONFIG file:

slapd.conf fragment:

database bdb
...
# database specific fragment only
# order not important
cachesize 10000
directory /var/ldap/first-dit
dirtyread
index mail pres,eq
index objectclass pres
index default eq,sub
index sn eq,sub,subinitial
index telephonenumber
index cn
searchstack 5

DB_CONFIG file:

The DB_CONFIG file is located as defined in the directory directive. The directives for this file are fully, if not clearly, explained in the BDB Manual. You can replace the checkpoint, lockdetect, dbnosync directives with the equivalent directives in the DB_CONFIG file as well as add additional tuning directives.

set lk_detect DB_LOCK_DEFAULT
set_flags DB_TXN_NOSYNC
set_lg_max 5242880
set_cachesize 0 5242880 1
txn_checkpoint 128 15 0

Problems, comments, suggestions, corrections (including broken links) or something to add? Please take the time from a busy life to 'mail us' (at top of screen), the webmaster (below) or info-support at zytrax. You will have a warm inner glow for the rest of the day.