Quantcast

Lots of LDAP Queries

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Lots of LDAP Queries

Jan Martens
Hi,

we use ownCloud 9.1.1 with the ldap_user APP and have a LDAP server with
about 11,000 user entries. We investigated that our ownCloud test server
(only two syncclients connected) made about 60,000 to 350,000 queries
per hour to the LDAP server. The productive Server makes even more for
example because for every sync client there is one bind operation every
five minutes.

For example from 2016-10-13 15:00:00 to 2016-10-13 16:00:00 the only the
ownCloudUser 17706 binds and 60572 queries.

It looks like ownCloud makes every ten minutes for every user multiple
queries to the LDAP even if the user never used the ownCloud. The
queries look like the following:

base="uid=usernameXY,ou=people,dc=company,dc=de" scope=0 deref=0
filter="(ownCloudQuota=*)"
base="uid=usernameXY,ou=people,dc=company,dc=de" scope=0 deref=0
filter="(objectClass=*)"
base="ou=People,dc=company,dc=de" scope=2 deref=0
filter="(&(ownCloudQuota=*)(uid=usernameXY))"
base="uid=usernameXY,ou=people,dc=company,dc=de" scope=0 deref=0
filter="(objectClass=*)"
base="uid=usernameXY,ou=people,dc=company,dc=de" scope=0 deref=0
filter="(objectClass=*)"
base="uid=usernameXY,ou=people,dc=company,dc=de" scope=0 deref=0
filter="(objectClass=*)"
base="ou=ownCloud Groups,dc=company,dc=de" scope=2 deref=0
filter="(&(cn=*)(memberUid=usernameXY))"

Of cause we are using APCu as memcache. By increasing the TTL from 600
seconds to 3600 we have alredy reduced the number of queries by about 10
percent.

My question is, if this is normal and nessesary behavior of owncloud. Or
can we adjust the intervall? It would be okay if ownCloud would
synchronize the users three times per day. The rest of the time it would
be enougth to handle the user authentification.

Or could a misconfiguration cause the amount of queries?

Best regards
Jan


_______________________________________________
User mailing list
[hidden email]
http://mailman.owncloud.org/mailman/listinfo/user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Lots of LDAP Queries

Vincent Petry
Possibly related ? https://github.com/owncloud/core/issues/26065


On 20.10.2016 12:14, Jan Martens wrote:

> Hi,
>
> we use ownCloud 9.1.1 with the ldap_user APP and have a LDAP server
> with about 11,000 user entries. We investigated that our ownCloud test
> server (only two syncclients connected) made about 60,000 to 350,000
> queries per hour to the LDAP server. The productive Server makes even
> more for example because for every sync client there is one bind
> operation every five minutes.
>
> For example from 2016-10-13 15:00:00 to 2016-10-13 16:00:00 the only
> the ownCloudUser 17706 binds and 60572 queries.
>
> It looks like ownCloud makes every ten minutes for every user multiple
> queries to the LDAP even if the user never used the ownCloud. The
> queries look like the following:
>
> base="uid=usernameXY,ou=people,dc=company,dc=de" scope=0 deref=0
> filter="(ownCloudQuota=*)"
> base="uid=usernameXY,ou=people,dc=company,dc=de" scope=0 deref=0
> filter="(objectClass=*)"
> base="ou=People,dc=company,dc=de" scope=2 deref=0
> filter="(&(ownCloudQuota=*)(uid=usernameXY))"
> base="uid=usernameXY,ou=people,dc=company,dc=de" scope=0 deref=0
> filter="(objectClass=*)"
> base="uid=usernameXY,ou=people,dc=company,dc=de" scope=0 deref=0
> filter="(objectClass=*)"
> base="uid=usernameXY,ou=people,dc=company,dc=de" scope=0 deref=0
> filter="(objectClass=*)"
> base="ou=ownCloud Groups,dc=company,dc=de" scope=2 deref=0
> filter="(&(cn=*)(memberUid=usernameXY))"
>
> Of cause we are using APCu as memcache. By increasing the TTL from 600
> seconds to 3600 we have alredy reduced the number of queries by about
> 10 percent.
>
> My question is, if this is normal and nessesary behavior of owncloud.
> Or can we adjust the intervall? It would be okay if ownCloud would
> synchronize the users three times per day. The rest of the time it
> would be enougth to handle the user authentification.
>
> Or could a misconfiguration cause the amount of queries?
>
> Best regards
> Jan
>
>
> _______________________________________________
> User mailing list
> [hidden email]
> http://mailman.owncloud.org/mailman/listinfo/user


_______________________________________________
User mailing list
[hidden email]
http://mailman.owncloud.org/mailman/listinfo/user

signature.asc (169 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Lots of LDAP Queries

Gerald Vogt
In reply to this post by Jan Martens
I just noticed the same thing this morning. The owncloud client
application for Windows checks for updates every 30 seconds. For that it
does a PROPFIND on /remote.php/webdav/ using HTTP basic authentication.

In the course of that it checks the password with a BIND and then
requests several attributes from the user entry.

If you have a lot of users with the owncloud client running at all times
that generates a lot of LDAP queries.

As far as I can see there is no caching of the LDAP attributes thus it
requests them each time.

Due to this this the owncloud server quickly became our main load on
your ldap servers...

-Gerald


On 20/10/2016 12:14, Jan Martens wrote:

> Hi,
>
> we use ownCloud 9.1.1 with the ldap_user APP and have a LDAP server with
> about 11,000 user entries. We investigated that our ownCloud test server
> (only two syncclients connected) made about 60,000 to 350,000 queries
> per hour to the LDAP server. The productive Server makes even more for
> example because for every sync client there is one bind operation every
> five minutes.
>
> For example from 2016-10-13 15:00:00 to 2016-10-13 16:00:00 the only the
> ownCloudUser 17706 binds and 60572 queries.
>
> It looks like ownCloud makes every ten minutes for every user multiple
> queries to the LDAP even if the user never used the ownCloud. The
> queries look like the following:
>
> base="uid=usernameXY,ou=people,dc=company,dc=de" scope=0 deref=0
> filter="(ownCloudQuota=*)"
> base="uid=usernameXY,ou=people,dc=company,dc=de" scope=0 deref=0
> filter="(objectClass=*)"
> base="ou=People,dc=company,dc=de" scope=2 deref=0
> filter="(&(ownCloudQuota=*)(uid=usernameXY))"
> base="uid=usernameXY,ou=people,dc=company,dc=de" scope=0 deref=0
> filter="(objectClass=*)"
> base="uid=usernameXY,ou=people,dc=company,dc=de" scope=0 deref=0
> filter="(objectClass=*)"
> base="uid=usernameXY,ou=people,dc=company,dc=de" scope=0 deref=0
> filter="(objectClass=*)"
> base="ou=ownCloud Groups,dc=company,dc=de" scope=2 deref=0
> filter="(&(cn=*)(memberUid=usernameXY))"
>
> Of cause we are using APCu as memcache. By increasing the TTL from 600
> seconds to 3600 we have alredy reduced the number of queries by about 10
> percent.
>
> My question is, if this is normal and nessesary behavior of owncloud. Or
> can we adjust the intervall? It would be okay if ownCloud would
> synchronize the users three times per day. The rest of the time it would
> be enougth to handle the user authentification.
>
> Or could a misconfiguration cause the amount of queries?
>
> Best regards
> Jan
>
>
> _______________________________________________
> User mailing list
> [hidden email]
> http://mailman.owncloud.org/mailman/listinfo/user
_______________________________________________
User mailing list
[hidden email]
http://mailman.owncloud.org/mailman/listinfo/user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Lots of LDAP Queries

Jan Martens
In reply to this post by Vincent Petry
Thank you, I subscribed to this issue. It looks like the same problem.



Am 20.10.2016 um 12:23 schrieb Vincent Petry:

> Possibly related ? https://github.com/owncloud/core/issues/26065
>
>
> On 20.10.2016 12:14, Jan Martens wrote:
>> Hi,
>>
>> we use ownCloud 9.1.1 with the ldap_user APP and have a LDAP server
>> with about 11,000 user entries. We investigated that our ownCloud test
>> server (only two syncclients connected) made about 60,000 to 350,000
>> queries per hour to the LDAP server. The productive Server makes even
>> more for example because for every sync client there is one bind
>> operation every five minutes.
>>
>> For example from 2016-10-13 15:00:00 to 2016-10-13 16:00:00 the only
>> the ownCloudUser 17706 binds and 60572 queries.
>>
>> It looks like ownCloud makes every ten minutes for every user multiple
>> queries to the LDAP even if the user never used the ownCloud. The
>> queries look like the following:
>>
>> base="uid=usernameXY,ou=people,dc=company,dc=de" scope=0 deref=0
>> filter="(ownCloudQuota=*)"
>> base="uid=usernameXY,ou=people,dc=company,dc=de" scope=0 deref=0
>> filter="(objectClass=*)"
>> base="ou=People,dc=company,dc=de" scope=2 deref=0
>> filter="(&(ownCloudQuota=*)(uid=usernameXY))"
>> base="uid=usernameXY,ou=people,dc=company,dc=de" scope=0 deref=0
>> filter="(objectClass=*)"
>> base="uid=usernameXY,ou=people,dc=company,dc=de" scope=0 deref=0
>> filter="(objectClass=*)"
>> base="uid=usernameXY,ou=people,dc=company,dc=de" scope=0 deref=0
>> filter="(objectClass=*)"
>> base="ou=ownCloud Groups,dc=company,dc=de" scope=2 deref=0
>> filter="(&(cn=*)(memberUid=usernameXY))"
>>
>> Of cause we are using APCu as memcache. By increasing the TTL from 600
>> seconds to 3600 we have alredy reduced the number of queries by about
>> 10 percent.
>>
>> My question is, if this is normal and nessesary behavior of owncloud.
>> Or can we adjust the intervall? It would be okay if ownCloud would
>> synchronize the users three times per day. The rest of the time it
>> would be enougth to handle the user authentification.
>>
>> Or could a misconfiguration cause the amount of queries?
>>
>> Best regards
>> Jan
>>
>>
>> _______________________________________________
>> User mailing list
>> [hidden email]
>> http://mailman.owncloud.org/mailman/listinfo/user
>
>
>
> _______________________________________________
> User mailing list
> [hidden email]
> http://mailman.owncloud.org/mailman/listinfo/user

_______________________________________________
User mailing list
[hidden email]
http://mailman.owncloud.org/mailman/listinfo/user

j_martens.vcf (239 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Lots of LDAP Queries

Gerald Vogt
In reply to this post by Vincent Petry
On 20/10/2016 12:23, Vincent Petry wrote:
> Possibly related ? https://github.com/owncloud/core/issues/26065

At least for me that's not related.

I have analyzes the ldap server logs and the bind is only happening
every 5 minutes.

However, every 30 seconds the owncloud client on windows does a
PROPFIND /remote.php/webdav/ and it seems each time the owncloud server
does the same 9 LDAP searches:

op=1 base="uid=UID,ou=people,..." scope=0 filter="(USERFILTER)" attrs=" "

op=2 base="ou=people,..." scope=2 filter="(uid=UID)" attrs="dn uid
samaccountname memberof  mail gecos  jpegphoto thumbnailphoto "

op=3 base="uid=UID,ou=people,..." scope=0 filter="(USERFILTER)" attrs=" "

op=4 base="ou=people,..." scope=2 filter="(uid=UID)" attrs="dn uid
samaccountname memberof  mail gecos  jpegphoto thumbnailphoto "

op=5 base="uid=UID,ou=people,..." scope=0 filter="(USERFILTER)" attrs=" "

op=6 base="uid=UID,ou=people,..." scope=0 filter="(objectClass=*)"
attrs="primaryGroupID "

op=7 base="uid=UID,ou=people,..." scope=0 filter="(objectClass=*)"
attrs="uid "

op=8 base="ou=group,..." scope=2
filter="(&(GROUPFILTER)(memberUid=UID))" attrs="cn dn "

op=9 base="cn=GRP,ou=group,..." scope=0 filter="(objectClass=*)" attrs=" "

That happens each time a PROPFIND comes in. op=9 is for the single group
to which the user belongs.

op=1, op=3, op=5 are identical. As are op=2 and op=4.

Worse even, op=1 to op=7 could actually be done with a single search.
You could check for existence of the entry, match of user filter an get
all attributes required in a single search.

What's also bad is that it requests the memberof attribute, which is a
dynamic attribute, i.e. it is determined at run time from the group
memberships. Requesting memberof when it is not used is a huge waste of
time and resources...

Can anyone confirm this?

Thanks,

Gerald


>
>
> On 20.10.2016 12:14, Jan Martens wrote:
>> Hi,
>>
>> we use ownCloud 9.1.1 with the ldap_user APP and have a LDAP server
>> with about 11,000 user entries. We investigated that our ownCloud test
>> server (only two syncclients connected) made about 60,000 to 350,000
>> queries per hour to the LDAP server. The productive Server makes even
>> more for example because for every sync client there is one bind
>> operation every five minutes.
>>
>> For example from 2016-10-13 15:00:00 to 2016-10-13 16:00:00 the only
>> the ownCloudUser 17706 binds and 60572 queries.
>>
>> It looks like ownCloud makes every ten minutes for every user multiple
>> queries to the LDAP even if the user never used the ownCloud. The
>> queries look like the following:
>>
>> base="uid=usernameXY,ou=people,dc=company,dc=de" scope=0 deref=0
>> filter="(ownCloudQuota=*)"
>> base="uid=usernameXY,ou=people,dc=company,dc=de" scope=0 deref=0
>> filter="(objectClass=*)"
>> base="ou=People,dc=company,dc=de" scope=2 deref=0
>> filter="(&(ownCloudQuota=*)(uid=usernameXY))"
>> base="uid=usernameXY,ou=people,dc=company,dc=de" scope=0 deref=0
>> filter="(objectClass=*)"
>> base="uid=usernameXY,ou=people,dc=company,dc=de" scope=0 deref=0
>> filter="(objectClass=*)"
>> base="uid=usernameXY,ou=people,dc=company,dc=de" scope=0 deref=0
>> filter="(objectClass=*)"
>> base="ou=ownCloud Groups,dc=company,dc=de" scope=2 deref=0
>> filter="(&(cn=*)(memberUid=usernameXY))"
>>
>> Of cause we are using APCu as memcache. By increasing the TTL from 600
>> seconds to 3600 we have alredy reduced the number of queries by about
>> 10 percent.
>>
>> My question is, if this is normal and nessesary behavior of owncloud.
>> Or can we adjust the intervall? It would be okay if ownCloud would
>> synchronize the users three times per day. The rest of the time it
>> would be enougth to handle the user authentification.
>>
>> Or could a misconfiguration cause the amount of queries?
>>
>> Best regards
>> Jan
>>
>>
>> _______________________________________________
>> User mailing list
>> [hidden email]
>> http://mailman.owncloud.org/mailman/listinfo/user
>
>
>
>
> _______________________________________________
> User mailing list
> [hidden email]
> http://mailman.owncloud.org/mailman/listinfo/user
>
_______________________________________________
User mailing list
[hidden email]
http://mailman.owncloud.org/mailman/listinfo/user
Loading...