So, today we’ve had us a wee bit of an issue with our OpenLDAP servers – to LDAP clients, the server, on attempts to do MOD operations, would say unhelpful things like “error code 80 – other”. (Coincidentally, the full error should be “Error code: 80 – other (implementation-specific)” After some digging, the logs show:

Feb 14 12:30:02 head02 slapd[19062]: conn=3229729 op=22190 MOD attr=mail
Feb 14 12:30:02 head02 slapd[19062]: bdb(dc=XXXXXXX,dc=com): Lock table is out of available lock entries
Feb 14 12:30:02 head02 slapd[19062]: => bdb_idl_insert_key: c_get failed: Cannot allocate memory (12)

A slightly better way to ask “WTF, Over?” is thusly:

[root@head02 log]# sudo -u ldap /usr/sbin/slapd_db_stat -h /var/lib/ldap/ -c
40 Last allocated locker ID
0x7fffffff Current maximum unused locker ID
9 Number of lock modes
1000 Maximum number of locks possible
1000 Maximum number of lockers possible
1000 Maximum number of lock objects possible
80 Number of lock object partitions
9 Number of current locks
1063 Maximum number of locks at any one time
4 Maximum number of locks in any one bucket
0 Maximum number of locks stolen by for an empty partition
0 Maximum number of locks stolen for any one partition
45 Number of current lockers
47 Maximum number of lockers at any one time
9 Number of current lock objects
511 Maximum number of lock objects at any one time
3 Maximum number of lock objects in any one bucket
0 Maximum number of objects stolen by for an empty partition
0 Maximum number of objects stolen for any one partition
742188 Total number of locks requested
742179 Total number of locks released
0 Total number of locks upgraded
11 Total number of locks downgraded
0 Lock requests not available due to conflicts, for which we waited
0 Lock requests not available due to conflicts, for which we did not wait
0 Number of deadlocks
0 Lock timeout value
0 Number of locks that have timed out
0 Transaction timeout value
0 Number of transactions that have timed out
872KB The size of the lock region
4 The number of partition locks that required waiting (0%)
2 The maximum number of times any partition lock was waited for (0%)
0 The number of object queue operations that required waiting (0%)
0 The number of locker allocations that required waiting (0%)
0 The number of region locks that required waiting (0%)
3 Maximum hash bucket length

Specifically:

1000 Maximum number of locks possible
[...]
1063 Maximum number of locks at any one time

One of the peculiarities inherent to BerkeleyDB is the notion of the “environment” (a directory which contains the BerkeleyDB files) which defines run-time parameters, and once the databases (in this case – the actual LDAP data as well as associated indexes) are initialized, the values are actually baked in. There is a file called DB_CONFIG, but it, too, only applies at initialization time. Or recovery time. Yep. In order to apply different run-time parameters, the database has to be “recovered”. (There is a third option – writing C code to push a new run-time configuration struct into the environment, but That Way Madness Lies a.k.a. much higher potential for getting it Wrong)

So, had to stop slapd (do a recovery without doing that, and all sorts of FUN will ensue), modify DB_CONFIG, and then run sudo -u ldap /usr/sbin/slapd_db_recover -h /var/lib/ldap -v (Making backups of the BDB environment is a Good Idea, just in case)

Once that completes (pretty quick), restart slapd and verify that the new run-time parameters are in effect. Happy happy, joy joy?

The actual DB_CONFIG change:

# pmay, 14February2012
# We’re running out of locks.
# Errors:
# bdb(dc=XXXXXX,dc=com): Lock table is out of available lock entries
# => bdb_idl_insert_key: c_get failed: Cannot allocate memory (12)
# sudo -u ldap /usr/sbin/slapd_db_stat -h /var/lib/ldap -c
# -> shows that the max 1000 locks is being exceeded by the Max # of locks at any one time.
# Trying to increase it via setting of
# set_lk_max_locks to 1500
# set_lk_max_locks 1500