Paging support for LDAP FDW

[JulianVolodia]

Hi, I met problem with access Active Directory using LDAP protocol with multicorn.ldapfdw.LdapFdw

It needs "paging-style" access (async). I have written fork of that plugin and want to contribute but first want to know how you accept it:
a) better to keep one wrapper with simple access and paging support for LDAP protocol,
b) another wrapper with different name?
c) any other ideas?

I also think about add SSL conn. support to LDAP FDW so (a) case is the best in my opinion. On the other hand it could become more complicated (so could be not well tested on all cases different than mine) - then (b) could be good choice probably.

What you think?

Best regards.

[helix84]

@JulianVolodia I'm not affiliated with the project but I'd like to test your fork. Do you have it available somewhere?

[JulianVolodia]

@helix84 sorry for waiting...

mod of ldapfdw.py:
https://gist.github.com/JulianVolodia/516c130b13dedf514ae4e5cf5d919444

Tested with Active Directory.

[JulianVolodia]

@helix84 have you tried that? Could I help you in any way?

I make some fast&furious research and attach links to ref.

Why not to change magic 1k number: http://jeftek.com/219/avoid-changing-the-maxpagesize-ldap-query-policy/

Some over-thinking... https://stackoverflow.com/a/24512928
refering to: http://support.microsoft.com/kb/2009267

From 2008 R2 version of MS srv OS - the limit is set.

http://www.fots.nl/set-the-maxpagesize/

LDAP Server Settings: By default in Microsoft Active Directory the query limit is set to 1000 objects per request.

https://communities.ca.com/thread/241759621-active-directory-1000-page-limit-and-ldapsearch-e-option

Example of ldapsearch with the -E option to allow paging over 1000 objects.
ldapsearch.exe -h imps001 -p 20389 -D "eTGlobalUserName=etaadmin,eTGlobalUserContainerName=Global Users,eTNamespaceName=CommonObjects,dc=im,dc=eta" -w Password01 -b "eTADSDirectoryName=Example_01_with_Full_Domain_Admin_Access,eTNamespaceName=ActiveDirectory,dc=im,dc=eta" -s base "(objectClass=eTADSDirectory)" -E pr=1000/noprompt eTADSexchangeStores eTExploreUpdateEtrust
Note: The -E option should follow after the base / objectClass filter has been defined to avoid warning message from ldapsearch of unknown switch.

...against src of ldapsearch compatible with LDAPv2 ; see https://github.com/openldap/openldap/blob/master/clients/tools/ldapsearch.c#L348

Windows Server 2008 and newer domain controller returns only 5000 values in a LDAP response
[...]
Note: On both domain controllers the setting MaxPageSize is set to 50000 (default 1000) and MaxValRange to 25000 (default 1500).

Adding the comment about maximize it. Maybe there is easy way to pre-fetch that setting ;)
http://it-toolkit.blogspot.com/2015/11/how-to-set-maxpagesize-in-active.html
https://technet.microsoft.com/en-us/library/aa997076(v=exchg.80).aspx
https://msdn.microsoft.com/en-us/library/aa746459(v=vs.85).aspx

https://www.petri.com/forums/forum/microsoft-networking-services/active-directory/21207-maxpagesize-change-in-ad

The maxpagesize value is a default query policy which is applied to the forest, the LDAP policy is located at:
CN=Default Query Policy,CN=Query-Policies,CN=Directory Service,CN=Windows NT,CN=Services,DC=.....

...about 260k ?? https://books.google.pl/books?id=Ga1nLk_xqcUC&lpg=PA163&ots=ML75yfCQGo&dq=AD%20max%20page%20size%20default&hl=pl&pg=PA163#v=onepage&q=AD%20max%20page%20size%20default&f=false

So @helix84 , if you have any find outs I will be glad to see that :) Best!

[alexnur]

I install master (1.4.0) and replace ldapfdw.py from https://github.com/Segfault-Inc/Multicorn/pull/240/files
After any query:
ERROR: Error in python: KeyError
DETAIL: 'attributes'

Before replace ldapfdw.py if > 3000 records also
ERROR: Error in python: KeyError
DETAIL: 'attributes'