OpenLDAP: A comparison of back-mdb and back-hdb performance

One of the biggest changes to OpenLDAP in years has made its way into the latest OpenLDAP 2.4 releases, and that is a brand new backend named “back-mdb”.  This new backend leverages the Lightning Memory-Mapped Database from Symas.  To see why this new backend was introduced, it is useful to look at the differences in performance and resource utilization between old BDB based back-hdb and the new LMDB based back-mdb.

Hardware details

  • Dell PowerEdge R710
  • 36GB of RAM
  • ESXi 5.1 hypervisor
  • 2 CPU, 4 cores per CPU, with hyperthreading (16 vCPUs)
  • 1.2 TB RAID array from 4x SEAGATE ST9300603SS 300GB 10kRPM drives
  • Ubuntu12 64-bit OS (3.5.0-28-generic #48~precise1-Ubuntu SMP kernel)
  • LDAP data is stored on its own /ldap partition, using ext2 as the filesystem type
  • ext2 options: noatime,defaults

Software details

  • OpenLDAP 2.4 Engineering from 5/10/2013
  • Berkeley DB 5.2.36 for the back-hdb backend
  • For read tests, slamd 2.0.1 was used to generate results

Data description

For this test, I examined the databases of several of Zimbra’s clients to get an idea of what a typical user entry looked like.  I took the larger average of the common user entries I found between the customer databases to create a template from which to generate purely random data.  Using this template, I generated a 5,000,000 random user database.  The resulting LDIF file was 10GB in size.  By comparison, a large client with 6.3 million entries has a 4.6 GB LDIF, so my user entries are larger than average.

OpenLDAP tuning

Frontend (slapd) specific tuning

  • 16 server threads
  • 2 tool threads (mdb)
  • 16 tool threads (hdb)

MDB tuning

  • writemap
  • nometasync

HDB tuning

  • 31 GB Shared Memory BDB cache for loading the db (slapadd)
  • 22 GB Shared Memory BDB cache for searching (slapd)
  • DN cachesize: unlimited
  • Entry cachesize: 50,000
  • IDL cachesize: 50,000
  • Cachefree: 500

Database Indexing

  • objectClass eq
  • entryUUID eq
  • entryCSN eq
  • cn pres,eq,sub
  • uid pres,eq
  • zimbraForeignPrincipal eq
  • zimbraYahooId eq
  • zimbraId eq
  • zimbraVirtualHostname eq
  • zimbraVirtualIPAddress eq
  • zimbraMailDeliveryAddress eq,sub
  • zimbraAuthKerberos5Realm eq
  • zimbraMailForwardingAddress eq
  • zimbraMailCatchAllAddress eq,sub
  • zimbraMailHost eq
  • zimbraShareInfo sub
  • zimbraMailTransport eq
  • zimbraMailAlias eq,sub
  • zimbraACE sub
  • zimbraDomainName eq,sub
  • mail pres,eq,sub
  • zimbraCalResSite eq,sub
  • givenName pres,eq,sub
  • displayName pres,eq,sub
  • sn pres,eq,sub
  • zimbraCalResRoom eq,sub
  • zimbraCalResCapacity eq
  • zimbraCalResBuilding eq,sub
  • zimbraCalResFloor eq,sub

Notes

You will note that the entry cache and idl cache for HDB does not cover all entries in the database.  To do so would have required more RAM than the server has available.  My best calculation is it would take at least 64GB of total RAM to hold the entry cache and idl cache in full.  Part of the point of this benchmark is to detail the differences in resource requirements between the two backends, and this is where the first immediate difference becomes apparent.  This does not affect load time via slapadd, but may affect read rate.

Database resource usage and load times

MDB load time

  • Total time to load the database using slapadd -q: 59m3.377s
  • Database size: 26GB

HDB load time

  • Total time to load the database using slapadd -q: 108m57.655s
  • Database size: 30GB in .bdb files, plus shared memory key (55GB or 61GB depending on “run” vs “load” configuration)

MDB is able to load the DB 1.847 times faster than HDB.  MDB consumes 2.11 times fewer resources than HDB.

Read rates

SLAMD was used for generating load against the different backends.  For the purposes of this benchmark, there were 16 slamd clients configured to generate load against the LDAP server.  The slamd clients would start with a single thread generating load for 20 minutes, logging the total read rate.  The number of threads per client would then be incremented by 1, and the same 20 minute interval measured again.  After 3 intervals of no increase in read rate, the test run would stop.  Prior to running an optimizing search job, the full database was iterated through in a slamd job to ensure all data was loaded into memory and off disk so that comparisons were fair between the two backends.  The following two search filters were used in an even split for searches.  I chose these search filters as they are what our postfix service uses to find users for mail delivery, so reflect real-world usage.

  • (&(zimbraMailDeliveryAddress=user.[1-5000000]@zre-ldap001.eng.vmware.com)(zimbraMailStatus=enabled))
  • (&(zimbraMailAlias=user.[1-5000000]@zre-ldap001.eng.vmware.com)(zimbraMailStatus=enabled))

MDB read rates

  • Optimal search rate reached with 16 slamd clients, 23 threads per client, for a total of 368 querying clients
  • Read rate is 61,433 searches/second
  • Slamd report

HDB read rates

  • Optimal search rate was reached with 16 slamd clients, 4 threads per client, for a total of 64 querying clients
  • Read rate is 20,818.336 searches/second
  • Slamd report

MDB is 295% faster than HDB.  In addition, MDB scales with high read rates for a greater amount of querying clients than HDB.

Conclusions

MDB outperforms HDB in all areas tested.  MDB consumes significantly fewer resources than HDB, allowing for better utilization of server resources.  MDB scales better than HDB, allowing it to server more clients per server.

About these ads

8 Responses to “OpenLDAP: A comparison of back-mdb and back-hdb performance”

  1. hspencer77 Says:

    Reblogged this on More Mind Spew-age from Harold Spencer Jr. and commented:
    Great insight on how much performance improvement you get with OpenLDAP when you use back-mdb instead of back-hdb.

  2. Christian Hollstein Says:

    How does the DIT look like? Could you submit (or post) a small but representative fragment of your LDIF you used to populate the data base?

    • mishikal Says:

      Hi Christian,

      Certainly. This is based off of Zimbra’s schema and design, so we root our database at “”. Underneath that there is a cn=zimbra tree for specific zimbra configuration bits. After that, the number of 1-level nodes depends on the domain(s) being hosted. If it was a single .com domain, for example, there would be just dc=com. If there were multiple domains, there could be dc=org, dc=edu, dc=com, etc.

      In my test LDIF, there is only dc=com in addition to cn=zimbra.

      This is broken down into:

      # com
      dn: dc=com
      objectClass: organization
      objectClass: dcObject
      o: com domain
      dc: com

      # vmware.com
      dn: dc=vmware,dc=com
      objectClass: organization
      objectClass: dcObject
      o: vmware domain
      dc: vmware

      # eng.vmware.com
      dn: dc=eng,dc=vmware,dc=com
      objectClass: organization
      objectClass: dcObject
      o: eng domain
      dc: eng

      # zre-ldap001.eng.vmware.com
      dn: dc=zre-ldap001,dc=eng,dc=vmware,dc=com
      zimbraDomainType: local
      zimbraDomainStatus: active
      objectClass: dcObject
      objectClass: organization
      objectClass: zimbraDomain
      objectClass: amavisAccount
      zimbraId: cb9a3147-0884-4e5a-9a89-bedbf147a171
      zimbraCreateTimestamp: 20130419225322Z
      zimbraDomainName: zre-ldap001.eng.vmware.com
      zimbraMailStatus: enabled
      o: zre-ldap001.eng.vmware.com domain
      dc: zre-ldap001
      zimbraACE: c9c110d7-5631-46ff-9ea2-9988b57b7f92 grp +domainAdminConsoleRights

      # groups, zre-ldap001.eng.vmware.com
      dn: cn=groups,dc=zre-ldap001,dc=eng,dc=vmware,dc=com
      objectClass: organizationalRole
      cn: groups
      description: dynamic groups base

      # people, zre-ldap001.eng.vmware.com
      dn: ou=people,dc=zre-ldap001,dc=eng,dc=vmware,dc=com
      objectClass: organizationalRole
      ou: people
      cn: people

      People is where the heart of everything is for this test. There are 5,000,000 people entries that have been created, using a base template with some data randomized where it made sense. Here’s an example user:

      # user.1, people, zre-ldap001.eng.vmware.com
      dn: uid=user.1,ou=people,dc=zre-ldap001,dc=eng,dc=vmware,dc=com
      objectClass: top
      objectClass: person
      objectClass: organizationalPerson
      objectClass: inetOrgPerson
      objectClass: amavisAccount
      objectClass: zimbraAccount
      givenName: Quintilla
      sn: Marleau
      cn: Quintilla Marleau
      uid: user.1
      mail: user.1@zre-ldap001.eng.vmware.com
      userPassword:: bXZlQ0hEemMyeVhO
      zimbraMailDeliveryAddress: user.1@zre-ldap001.eng.vmware.com
      zimbraId: FC3779AA-AEA1-11E2-B62F-C1F0D2107332
      displayName: Quintilla Marleau
      zimbraMailAlias: user.1@zre-ldap001.eng.vmware.com
      zimbraPasswordModifiedTime: 20110624003207Z
      zimbraPrefReplyToAddress: user.1@zre-ldap001.eng.vmware.com
      zimbraPrefReadingPaneEnabled: TRUE
      zimbraAllowFromAddress: user.1@vmware.com
      zimbraZimletUserProperties: com_zimbra_social:social_pref_trendsPopularIsOn:tr
      ue
      zimbraZimletUserProperties: com_zimbra_social:social_pref_tweetmemePopularIsOn
      :true
      zimbraZimletUserProperties: com_zimbra_social:social_pref_cardWidthList:400px
      zimbraZimletUserProperties: com_zimbra_social:social_pref_showTweetAlertsOn:tr
      ue
      zimbraZimletUserProperties: com_zimbra_social:social_pref_toolbarButtonOn:true
      zimbraZimletUserProperties: com_zimbra_social:social_pref_numberofTweetsToRetu
      rn:50
      zimbraZimletUserProperties: com_zimbra_social:social_pref_numberofTweetsSearch
      esToReturn:50
      zimbraZimletUserProperties: com_zimbra_social:social_alertUpdateTime:131613450
      6914
      zimbraZimletUserProperties: com_zimbra_social:social_pref_SocialMailUpdateOn:t
      rue
      zimbraZimletUserProperties: com_zimbra_social:social_emailLastUpdateDate:2011/
      9/15
      zimbraZimletUserProperties: com_zimbra_social:social_pref_dontShowWelcomeScree
      nOn:false
      zimbraZimletUserProperties: com_zimbra_social:social_pref_diggPopularIsOn:true
      zimbraZimletUserProperties: com_zimbra_social:social_pref_autoShortenURLOn:tru
      e
      zimbraMailHost: mbs08-zcs.vmware.com
      zimbraMailStatus: enabled
      zimbraMailTransport: lmtp:mbs08-zcs.vmware.com:7025
      zimbraAccountStatus: active
      zimbraPrefSkin: carbon
      zimbraPrefSortOrder: BDLV:,CAL:,CLV:,CNS:,CNSRC:,CNTGT:,CV:,TKL:,TV:,TV-main:d
      ateDesc
      zimbraLastLogonTimestamp: 19840218004400Z

      All of the user data was generated via a simple perl script, which for some of the randomization (like names) pulls in the names database that ships with slamd:

      #!/usr/bin/perl
      use strict;
      use Data::UUID;
      use Data::Random;

      my $bdn = “ou=people,dc=zre-ldap001,dc=eng,dc=vmware,dc=com”;
      my $maildomain = “zre-ldap001.eng.vmware.com”;
      my $numusers=10000000;

      my @first_names = do {
      my $filename=”MakeLDIF/first.names”;
      open my $fh, “<", $filename
      or die "could not open $filename: $!";
      ;
      };

      my @last_names = do {
      my $filename=”MakeLDIF/last.names”;
      open my $fh, “<", $filename
      or die "could not open $filename: $!";
      ;
      };

      my $i=1;

      while ($i new;
      my $id = $ug->create_str();
      my $dt = &genDate();

      print “dn: uid=$uid,$bdn\n”;
      print “objectClass: top\n”;
      print “objectClass: person\n”;
      print “objectClass: organizationalPerson\n”;
      print “objectClass: inetOrgPerson\n”;
      print “objectClass: amavisAccount\n”;
      print “objectClass: zimbraAccount\n”;
      print “givenName: $gn\n”;
      print “sn: $sn\n”;
      print “cn: $cn\n”;
      print “uid: $uid\n”;
      print “mail: “.$uid.”\@”.$maildomain.”\n”;
      $tmp=join”, map +(0..9,’a’..’z’,’A’..’Z’)[rand(10+26*2)], 1..12;
      print “userPassword: $tmp\n”;
      print “zimbraMailDeliveryAddress: “.$uid.”\@”.$maildomain.”\n”;
      print “zimbraId: $id\n”;
      print “displayName: $cn\n”;
      print “zimbraMailAlias: “.$uid.”\@”.$maildomain.”\n”;
      print “zimbraPasswordModifiedTime: “.$dt.”Z\n”;
      print “zimbraPrefReplyToAddress: “.$uid.”\@”.$maildomain.”\n”;
      print “zimbraPrefReadingPaneEnabled: TRUE\n”;
      print “zimbraAllowFromAddress: “.$uid.”\@vmware.com\n”;
      print “zimbraZimletUserProperties: com_zimbra_social:social_pref_trendsPopularIsOn:tr\n”;
      print ” ue\n”;
      print “zimbraZimletUserProperties: com_zimbra_social:social_pref_tweetmemePopularIsOn\n”;
      print ” :true\n”;
      print “zimbraZimletUserProperties: com_zimbra_social:social_pref_cardWidthList:400px\n”;
      print “zimbraZimletUserProperties: com_zimbra_social:social_pref_showTweetAlertsOn:tr\n”;
      print ” ue\n”;
      print “zimbraZimletUserProperties: com_zimbra_social:social_pref_toolbarButtonOn:true\n”;
      print “zimbraZimletUserProperties: com_zimbra_social:social_pref_numberofTweetsToRetu\n”;
      print ” rn:50\n”;
      print “zimbraZimletUserProperties: com_zimbra_social:social_pref_numberofTweetsSearch\n”;
      print ” esToReturn:50\n”;
      print “zimbraZimletUserProperties: com_zimbra_social:social_alertUpdateTime:131613450\n”;
      print ” 6914\n”;
      print “zimbraZimletUserProperties: com_zimbra_social:social_pref_SocialMailUpdateOn:t\n”;
      print ” rue\n”;
      print “zimbraZimletUserProperties: com_zimbra_social:social_emailLastUpdateDate:2011/\n”;
      print ” 9/15\n”;
      print “zimbraZimletUserProperties: com_zimbra_social:social_pref_dontShowWelcomeScree\n”;
      print ” nOn:false\n”;
      print “zimbraZimletUserProperties: com_zimbra_social:social_pref_diggPopularIsOn:true\n”;
      print “zimbraZimletUserProperties: com_zimbra_social:social_pref_autoShortenURLOn:tru\n”;
      print ” e\n”;
      $tmp=int(rand(10));
      my $mailhost=”mbs0″.$tmp.”-zcs.vmware.com”;
      print “zimbraMailHost: $mailhost\n”;
      print “zimbraMailStatus: enabled\n”;
      print “zimbraMailTransport: lmtp:$mailhost:7025\n”;
      print “zimbraAccountStatus: active\n”;
      print “zimbraPrefSkin: carbon\n”;
      print “zimbraPrefSortOrder: BDLV:,CAL:,CLV:,CNS:,CNSRC:,CNTGT:,CV:,TKL:,TV:,TV-main:d\n”;
      print ” ateDesc\n”;
      $dt=&genDate();
      print “zimbraLastLogonTimestamp: “.$dt.”Z\n”;
      print “\n”;
      $i++;
      }

      sub genDate() {
      my $date=Data::Random::rand_datetime(min=>’1978-9-21 4:0:0′, max=>’now’);
      $date=~ s/://g;
      $date=~ s/-//g;
      $date=~ s/\s+//g;
      return $date;
      }

  3. Christian Hollstein Says:

    Regarding top performance please take a look at DVTDS (Distributed Virtual Transaction Directory Server). It reaches more than one million searches/s (asynchronous) and more than 441000 searches/s (synchronous). Modifies reaches 761000/s (asynchronous) and almost 300000/s synchronous. The server is in advanced prototype status (not yet available). For details go to dvtds.com or google for: ldap world record

  4. mishikal Says:

    When you can provide benchmarks on comparable machines between different directory servers, that’d be of interest. It has already been found after your LDAPCon presentation that cycle for cycle, back-mdb is still faster than your server.

  5. Christian Hollstein Says:

    Faster in which request types?

    In benchmarks performed on the same machine I found OpenLDAP serializing (means: no scaling at all) parallel modify requests even when I used up to six mdb – backends with different suffix DNs with data stored on six different hard disks. This behavior points to a serialization mechanism in the OpenLDAP frontend.

    Search and Compare requests scaled good, but were well below the DVTDS performance. Anyway, best thing is another try. I will repeat my OpenLDAP tests and send you the complete configuration, data model, machine, OS. Then please do me a favor: Tell me if I missed any important knobs to squeeze the maximum out of it and I will repeat the tests again. For DVTDS I know all the tuning parameters and its just fair to try every possible means for OpenLDAP as well.

    Sorry for still failing to deliver a demo version of DVTDS.

  6. Christian Hollstein Says:

    As promised, here are my benchmark results for OpenLDAP and DVTDS on the same machine and configuration. First of all I have to apologize for my statement about modify serialization in OpenLDAP. This seems to be an effect of the OS syncing database content to disk, rather than being a property of the OpenLDAP frontend. It scaled well as long as the test is through before the Linux Kernel starts syncing.

    However, DVTDS is between 1.4 and 4 times faster for add, search, compare, bind and modify. Delete and moddn was not tested. Further DVTDS needed just a third of the disk space and memory compared to OpenLDAP.

    Here are the results:

    OpenLDAP DVTDS
    Performance Performance
    [Operations/s] / [Operations/s] /
    # of Client Threads # of Client Threads

    12000000 entries
    LDAP ADD 39860 / 6 160946 / 6
    Sequential UID

    Random UID
    LDAP SEARCH 235282 / 290 445651 / 270
    LDAP COMPARE 311331 / 270 447374 / 290
    LDAP BIND 315519 / 290 463190 / 300
    LDAP MODIFY 118782 / 48 420158 / 96

    Random UID
    LDAP SEARCH 92811 / 10 158284 / 10
    LDAP COMPARE 125187 / 10 168038 / 10
    LDAP BIND 128221 / 10 184163 / 10
    LDAP MODIFY 9590 / 1 19290 / 1

    OpenLDAP DVTDS
    Resource Usage Resource Usage

    Hard Disk Space 17425 MByte 5029 MByte
    Memory 18 GByte 6 Gbyte

    Here is the configuration:

    Server:
    Intel core i7 4960x (6 core, running @ 4.6 GHz water cooled)
    32 GB RAM @ 2133 MHz
    6 x SATA 750 GB @ 7200 RPM
    2 x 10 GBit/s network adapter
    OpenLDAP 2.4.39, Compiled with CFLAGS=-O4, LDFLAGS=-O4

    Client:
    Intel core i7 3770k (4 core, running @ 4.3 GHz water cooled)
    32 GB RAM @ 2133 MHz
    6 x SATA 750 GB @ 7200 RPM
    2 x 10 GBit/s network adapter
    ELDC 1.002 (Can be downloaded from http://www.teracortex.com)

    OS: OpenSuSE 13.1 64 Bit
    The database files are on six different filesystems (EXT2). The
    file systems are located in six different logical volumes of 64 GByte each
    The logical volumes are on six different hard disks at the second position.
    On each hard disk the first position is occupied by a 64 GByte raw volume.

    slapd configuration:

    include /download/openldap-2.4.39/run/etc/openldap/schema/core.schema
    include /download/openldap-2.4.39/run/etc/openldap/schema/cosine.schema
    include /download/openldap-2.4.39/run/etc/openldap/schema/inetorgperson.schema

    pidfile /download/openldap-2.4.39/run/var/run/slapd.pid
    argsfile /download/openldap-2.4.39/run/var/run/slapd.args
    conn_max_pending 1000
    conn_max_pending_auth 1000
    listener-threads 16
    threads 16
    loglevel 0

    database mdb
    directory /vgdata0/openldap
    suffix “dc=dm0000000,dc=com”
    rootdn “cn=Manager,dc=dm0000000,dc=com”
    rootpw secret
    dbnosync
    maxreaders 400
    maxsize 4398046511103
    index uid eq

    database mdb
    directory /vgdata1/openldap
    suffix “dc=dm0000001,dc=com”
    rootdn “cn=Manager,dc=dm0000001,dc=com”
    rootpw secret
    dbnosync
    maxreaders 400
    maxsize 4398046511103
    index uid eq

    database mdb
    directory /vgdata2/openldap
    suffix “dc=dm0000002,dc=com”
    rootdn “cn=Manager,dc=dm0000002,dc=com”
    rootpw secret
    dbnosync
    maxreaders 400
    maxsize 4398046511103
    index uid eq

    database mdb
    directory /vgdata3/openldap
    suffix “dc=dm0000003,dc=com”
    rootdn “cn=Manager,dc=dm0000003,dc=com”
    rootpw secret
    dbnosync
    maxreaders 400
    maxsize 4398046511103
    index uid eq

    database mdb
    directory /vgdata4/openldap
    suffix “dc=dm0000004,dc=com”
    rootdn “cn=Manager,dc=dm0000004,dc=com”
    rootpw secret
    dbnosync
    maxreaders 400
    maxsize 4398046511103
    index uid eq

    database mdb
    directory /vgdata5/openldap
    suffix “dc=dm0000005,dc=com”
    rootdn “cn=Manager,dc=dm0000005,dc=com”
    rootpw secret
    dbnosync
    maxreaders 400
    maxsize 4398046511103
    index uid eq

    LDIF template:
    UID is a 10 digit number
    Each client thread targets one of the six databases by means of
    its thread id % 6
    So the variable “x” is calculated: x = threadid % 6, which generates
    the suffixes dc=dm0000000,dc=com

    dc=dm0000005,dc=com

    dn: uid=%010d_v%,dc=dm%07d_x%,dc=com
    changetype: add
    sn: who_else
    cn: who_else
    businessCategory: telco
    carLicense: 12345678
    departmentNumber: 000
    displayName: Samsung
    employeeNumber: nocare
    employeeType: CEO
    givenName: Chris
    homePhone: 12345678
    homePostalAddress: itshere
    initials: CH
    mail: mymail
    manager: cn=itsme,dc=mydomain,dc=com
    mobile: 76543210
    o: dvtds
    pager: 76543210
    preferredLanguage: en
    roomNumber: 000_21
    secretary: cn=jane,dc=mydomain,dc=com
    userPassword: qwertz

  7. Téssio Fechine Says:

    Any news about OpenLDAP 2.5?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Follow

Get every new post delivered to your Inbox.