Support Center

Name is required.
Email address is required.
Invalid email address
Answer is required.
Exceeding max length of 5KB

PerfTest issues with jest library version 0.0.3

Saikat Kanjilal Apr 09, 2013 01:13AM EEST

Hello,
I posted this question on the elasticsearch mailing list but I think this is a better palce. I am using the jest library version 0.0.3 and am running into some issues, we have a tomcat rest service that abstracts away our elasticsearch infrastructure, we are using spring and autowiring in the JestClientFactory, when we go to perform a search or an insert we basically call getObject to get a JestClient object. The code works fine under normal circumstances, however under a perf test it falls over with the following tomcat error:

org.apache.tomcat.util.net.JloEndpoint - Socket accept failed java.net.SocketException: Too many open files


Code for insert:

if(isEmpty(documents))
{
return null;
}
Bulk bulk = new Bulk();
for(Document document : documents)
{
bulk.addIndex(new Index.Builder(document).index(getIndex().toLowerCase()).type(getType()).id((String)document.get("id")).build());
}
JestClient jestClient=(JestClient)getJestClientFactory().getObject();
JestResult result=jestClient.execute(bulk);
if (!result.isSucceeded()) {
logger.error("IndexDocumentCommand::the bulk operation to index documents failed with message"+result.getErrorMessage());
} else {
logger.error("IndexDocumentCommand::the bulk operation to index documents succeeded with message"+result.getJsonString());
}



I dont believe I am doing anything out of the ordinary here, any thoughts on this? Also I have checked the ulimit on the vm's in our dev environment and increased that to a large number. Also please let me know if I should use another forum to talk to the Jest folks, my apologies if this is not the right place. Finally is there a timeout I can set associated with the JestClient that closes the socket connection quickly?


Thanks for your help on this.
Regards

Up 24 rated Down
Ferhat Apr 09, 2013 08:45AM EEST Searchly Agent
Hi Saikat,

Jest client is designed to be singleton, so please first check your application only constructed "one" JestClient.

Another point is the configuration of JestClient. For instance configuring Jest as "IS_MULTI_THREADED" means to use connection pool, which is very advised.

// Configuration
ClientConfig clientConfig = new ClientConfig();
LinkedHashSet<String> servers = new LinkedHashSet<String>();
servers.add("http://localhost:9200");
clientConfig.getProperties().put(ClientConstants.SERVER_LIST, servers);
clientConfig.getProperties().put(ClientConstants.IS_MULTI_THREADED, true);

// Construct a new Jest client according to configuration via factory
JestClientFactory factory = new JestClientFactory();
factory.setClientConfig(clientConfig);
JestClient client = factory.getObject();


Let me know if it is not clear,

Regards,
Ferhat


Up 21 rated Down
Saikat Kanjilal Apr 09, 2013 06:45PM EEST
Thanks Ferhat, this is very helpful, we certainly dont have the JestClient wired up to be a singleton, I will change this asap. On the other ClientConstants issue we are setting the is_multi_threaded flag.

I will try the change above and let you know results.
Regards
Up 27 rated Down
Ferhat Apr 09, 2013 08:38PM EEST Searchly Agent
Hi Saikat,

Any update on this ?
Up 28 rated Down
Saikat Kanjilal Apr 09, 2013 09:04PM EEST
Ferhat,
I went ahead and created a singleton object for JestClient, I basically have a static method inside my class that only tracks/keeps one instance of the JestClient object, however now I am running into an issue with reusing this connection, I am seeing the following message:

Invalid use of BasicClientConnManager: connection still allocated.
Make sure to release the connection before allocating another one

Before I go down the path of introducing routes etc based on your wiki shown here () https://github.com/searchbox-io/Jest/blob/master/src/test/java/io/searchbox/client/JestClientFactoryTest.java) am I missing anything else here that might be obvious, code is shown below, note that I also added the shutdownClient method after each indexing call:

Bulk bulk = new Bulk();
for(Document document : documents)
{
bulk.addIndex(new Index.Builder(document).index(getIndex().toLowerCase()).type(getType()).id((String)document.get("id")).build());
}
JestResult result=getJestHttpClient().execute(bulk);
if (!result.isSucceeded()) {
logger.error("IndexDocumentCommand::the bulk operation to index documents failed with message"+result.getErrorMessage());
} else {
logger.error("IndexDocumentCommand::the bulk operation to index documents succeeded with message"+result.getJsonString());
}
getJestHttpClient().shutdownClient();


Any ideas on what else needs to be tweaked?

Up 16 rated Down
Ferhat Apr 09, 2013 09:42PM EEST Searchly Agent
Hi Saikat,

Error coming from BasicClientConnectionManager so it means jest client is not using connection pool, configured for single threaded usage.
I have prepared a gist to work on https://gist.github.com/ferhatsb/5348186
Also you should not shutdown the client, in multi threaded more, it will create a connection pool and manage it (uses apache http client under the hood).
Up 7 rated Down
Saikat Kanjilal Apr 09, 2013 10:55PM EEST
Ok I've taken out the shutdown statement, am still seeing the error message, first let me show some more details on my setup:

Spring Config:
<bean id="elasticSearchClientFactory" class="io.searchbox.client.JestClientFactory">
<property name="clientConfig">
<ref local="searchClientConfig"/>
</property>
</bean>


<util:set id="serverList">
<value>${elasticSearchClusterHost1}:${elasticSearchClusterPort}</value>
<value>${elasticSearchClusterHost2}:${elasticSearchClusterPort}</value>
<value>${elasticSearchClusterHost3}:${elasticSearchClusterPort}</value>
</util:set>


<bean id="searchClientConfig" class="io.searchbox.client.config.ClientConfig">
<property name="properties">
<map>
<entry key="serverList" value-ref="serverList"/>
<entry key="io.searchbox.client.config.ClientConstants.IS_MULTI_THREADED">
<value type="java.lang.Boolean">true</value>
</entry>
<entry key="io.searchbox.client.config.ClientConstants.DISCOVERY_ENABLED">
<value type="java.lang.Boolean">true</value>
</entry>
</map>
</property>
</bean>


Code for singleton:
private static JestHttpClient jestHttpClient;
public JestHttpClient getJestHttpClient() {
if (jestHttpClient==null) {
jestHttpClient=(JestHttpClient)getJestClientFactory().getObject();
}
return jestHttpClient;
}


As far as I can tell my setup is exactly the same as you identified in your gist.


Now am seeing this message still:
Invalid use of BasicClientConnManager: connection still allocated.
Make sure to release the connection before allocating another one.


It seems that the BasicClientConnManager is still being used, any ideas on what I may be missing in my spring configs or elsewhere?

Much appreciate your help
Up 10 rated Down
Ferhat Apr 10, 2013 12:16AM EEST Searchly Agent
Your spring configuration seems to be correct,
That is the source https://github.com/searchbox-io/Jest/blob/master/src/main/java/io/searchbox/client/JestClientFactory.java
of factory. Can you please configure your logger as debug to see "Multi Threaded http client created" (If not we will understand that client is not started as multi threaded)

Also one another point which is not very likely "auto discovery". Perhaps you should try without it.

Last can be to try same setup programatically instead creating via spring configuration, then we can eliminate if this is somehow related to spring configuration.
Up 12 rated Down
Saikat Kanjilal Apr 10, 2013 12:49AM EEST
Here's my latest findings, if I use the following code everything works and I dont see any errors (of course I only ran the load test for 20 minutes so far but it looks good):

public JestHttpClient getJestHttpClient() {
if (jestHttpClient == null) {
ClientConfig clientConfig = new ClientConfig();
LinkedHashSet<String> servers = new LinkedHashSet<String>();
servers.add("http://host1");
servers.add("http://host2");
servers.add("http://host3");
clientConfig.getProperties().put(ClientConstants.SERVER_LIST, servers);
clientConfig.getProperties().put(ClientConstants.IS_MULTI_THREADED, true);
jestClientFactory.setClientConfig(clientConfig);
jestHttpClient = (JestHttpClient) jestClientFactory.getObject();
return jestHttpClient;
} else {
return jestHttpClient;
}
}


However if I autowire the spring config I seem to run into the BasicConnectionManager message I identified above. I even ran the code under the debugger and can see that the multithreaded flag is being set int the autowired clientConfig object. Any ideas? I'm completely baffled by this.

Regards
Up 4 rated Down
Ferhat Apr 10, 2013 09:04AM EEST Searchly Agent
Can you also add discovery enabled with programmatic configuration to make both same ?
Up 4 rated Down
Saikat Kanjilal Apr 10, 2013 08:29PM EEST
Ferhat,
So it looks like I spoke too soon, this morning after making the desired changes that you mentioned, I am running into the following exception:

Service exception:
com.google.common.util.concurrent.UncheckedExecutionException: java.lang.NullPointerException

in the following line of the code from the sample below:
jestHttpClient = (JestHttpClient) jestClientFactory.getObject();





public JestHttpClient getJestHttpClient() {

if (jestHttpClient == null) {
ClientConfig clientConfig = new ClientConfig();
LinkedHashSet<String> servers = new LinkedHashSet<String>();
servers.add("http://host1");
servers.add("http://host2");
servers.add("http://host3");
clientConfig.getProperties().put(ClientConstants.SERVER_LIST, servers);
clientConfig.getProperties().put(ClientConstants.DISCOVERY_ENABLED, true);
clientConfig.getProperties().put(ClientConstants.IS_MULTI_THREADED, true);

jestClientFactory.setClientConfig(clientConfig);
jestHttpClient = (JestHttpClient) jestClientFactory.getObject();
return jestHttpClient;

} else {
return jestHttpClient;
}
}


Note that in order to get something working and at least have a baseline I am doing everything programmatically as opposed to spring configs. Any ideas on this?

I feel like I am really close and it seems like I am doing everything you mentioned in the gist.


Regards

Up 7 rated Down
Saikat Kanjilal Apr 10, 2013 08:39PM EEST
Detailed stack trace:

Caused by: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.NullPointerException
at com.google.common.util.concurrent.Futures.wrapAndThrowUnchecked(Futures.java:1015)
at com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1001)
at com.google.common.util.concurrent.AbstractService.startAndWait(AbstractService.java:220)
at com.google.common.util.concurrent.AbstractScheduledService.startAndWait(AbstractScheduledService.java:284)
at io.searchbox.client.JestClientFactory.getObject(JestClientFactory.java:86)
Up 2 rated Down
Saikat Kanjilal Apr 10, 2013 09:17PM EEST
Here's an update, I fixed my spring configuration issue, I was just not representing booleans correctly, so I've traced the issue to the line of code below:

jestHttpClient = (JestHttpClient) jestClientFactory.getObject();


It seems like there's a race condition somewhere hidden in that code that's causing the following exception:

500 com.google.common.util.concurrent.UncheckedExecutionException: java.lang.NullPointerException

Caused by: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.NullPointerException
at com.google.common.util.concurrent.Futures.wrapAndThrowUnchecked(Futures.java:1015)
at com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1001)
at com.google.common.util.concurrent.AbstractService.startAndWait(AbstractService.java:220)
at com.google.common.util.concurrent.AbstractScheduledService.startAndWait(AbstractScheduledService.java:284)
at io.searchbox.client.JestClientFactory.getObject(JestClientFactory.java:86)

Caused by: java.lang.NullPointerException


Would appreciate any input.
Thanks again.
Up -13 rated Down
Saikat Kanjilal Apr 10, 2013 11:53PM EEST
Problem Fixed:
Ferhat, I looked at some of your code inside the Scheduler and it was expecting the following parameters to be defined when running the client in multithreaded mode:

<entry key="discoveryFrequency">
<value type="java.lang.Long">3600000</value>
</entry>
<entry key="isDiscoveryEnabled">
<value type="java.lang.Boolean">true</value>
</entry>
<entry key="discoveryFrequencyTimeUnit">
<value type="java.util.concurrent.TimeUnit">MILLISECONDS</value>
</entry>


As soon as I set these parms in my spring config everything worked, it was throwing an NPE because in multithreaded mode for the client it looks up the client configs and expects these parms to be defined. You should really document this somewhere on your wiki. Also fyi I'd like to contribute to your codebase so ping me if I can write some code to help you guys out sometime. My perf tests are now running successfully for long periods now. Thanks again for your input and help.

Regards
Up 28 rated Down
Ferhat Apr 11, 2013 08:46AM EEST Searchly Agent
Hi Saikat,

Great to hear that you have figured out and Jest is working fine for you.

Actually discovery feature and multi-threaded mode is two different things.
Discovery gives Jest to discover other nodes via ElasticSearch's node api, so if one server goes down in your cluster, Jest removes it from its server list. It is vica versa (if a new server comes in, it is added)

Feel free to fork, add new features and ask for a pull request https://github.com/searchbox-io/Jest. All unit & integration test are very welcomed.

Ferhat
Up 19 rated Down
Saikat Kanjilal Apr 11, 2013 06:04PM EEST
Hey Ferhat,
Yes I already understood that discovery and multithreaded mode are two different things, I actually pasted the wrong bits, I noticed that if I put the following two properties in everything worked:

<entry key="discoveryFrequency">
<value type="java.lang.Long">3600000</value>
</entry>
<entry key="discoveryFrequencyTimeUnit">
<value type="java.util.concurrent.TimeUnit">MILLISECONDS</value>
</entry>


As I mentioned in your code that runs the scheduler you were looking for these properties to be set when running the client in multithreaded mode. Maybe it was because I set discovery to be enabled that these other parms were reuqired. Anyways as far as contributing are there features that you still need or is it only unit and integration tests that need beefing up?

Thanks again
Up 21 rated Down
Ferhat Apr 12, 2013 08:46AM EEST Searchly Agent
Hi Saikat,

Please check out source code you will see many empty implementations. I was mentioning about tests with these implementations.
Up 9 rated Down
Spoolvevy May 31, 2013 05:37PM EEST
Dans Certains Cas, Le Principe De La Solution La Moins Contraignante Faire Correspondre Les Besoins Des Personnes Avec Ce Qu'il Y A De Plus Courant Est Le Plus Pratique: <a href="http://tadalafil-elifr2013.cd.st/#5691">acheter du tadalafil par cheque</a> prix du tadalafil en allemagne.
J'aime L'impala Glucotrol Travaillant Alors Je Retourne L'approche Principale Jusqu' Ce Que Vous Sachiez Comment GLUCOTROL Capable Prt Disposй Doit Manger Jusqu' Ce Que GLUCOTROL Ne Soit Allй Recevoir Cette Idйe Que De Hauts Niveaux De Sardaigne Incendient Vraiment Le Boise Mais Mme Sur Mon Chque D'accu Mtre Est Restй Dans Le Monde, Les Discussions De Beaucoup Plus Que J'ai Eu Besoin: <a href="http://tamoxifenmgvia.cd.st/#1891">tamoxifen original achat</a> acheter du tamoxifen en toute sйcuritй.

This question has received the maximum number of answers.

support@searchly.com
http://assets2.desk.com/
false
searchboxio
Loading
seconds ago
a minute ago
minutes ago
an hour ago
hours ago
a day ago
days ago
about
false
Invalid characters found
/customer/en/portal/articles/autocomplete