Elasticsearch Java Integration

Elasticsearch functionalities can be easily integrated into any Java application via a REST API or native APIs. In Java, it is easy to call a REST HTTP interface with one of the many libraries available, such as the Apache HttpComponents client. From Elasticsearch 6.x onward, Elastic has provided a battle low-/high-level HTTP for clients to use. In version 8.x, Elastic released a modern/functional/strongly typed client. . The Elasticsearch community recommends using the REST APIs when integrating them, as they are more stable between releases and are well documented.

If you want learn more about monitoring using Elasticsearch please read this post

Creating a standard Java HTTP client

An HTTP client is one of the easiest clients to create, as it allows for the calling of internal methods and third-party calls implemented in plugins that can only be called via HTTP.

To enable the compilation in your Maven pom.xml project, just add the following code:

<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.13</version>
</dependency>

2. Instantiate a client and fetch a document with a get method, the code will look like the following:

import org.apache.http.HttpEntity;
import org.apache.http.HttpHost;
import org.apache.http.HttpStatus;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.AuthCache;
import org.apache.http.client.CredentialsProvider;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.protocol.HttpClientContext;
import org.apache.http.conn.ssl.NoopHostnameVerifier;
import org.apache.http.ssl.SSLContextBuilder;
import org.apache.http.conn.ssl.TrustAllStrategy;
  import org.apache.http.impl.auth.BasicScheme;
 import org.apache.http.impl.client.BasicAuthCache;
 import org.apache.http.impl.client.
BasicCredentialsProvider;
 import org.apache.http.impl.client.CloseableHttpClient;
 import org.apache.http.impl.client.HttpClients;
 import org.apache.http.util.EntityUtils;
 import java.io.IOException;
 import java.security.KeyManagementException;
 import java.security.KeyStoreException;
 import java.security.NoSuchAlgorithmException;
  public class AppWithSecurity {
     private static String wsUrl =
"https://127.0.0.1:9200";
     public static void main(String[] args) throws
KeyManagementException, NoSuchAlgorithmException,
KeyStoreException {
 CloseableHttpClient client = HttpClients.custom()
                 .setSSLContext(new SSLContextBuilder().
loadTrustMaterial(null, TrustAllStrategy.INSTANCE).
build())
              .setSSLHostnameVerifier(NoopHostnameVerifier.
INSTANCE).setRetryHandler(new MyRequestRetryHandler())
 .build();
         HttpGet method = new HttpGet(wsUrl + "/mybooks/_
doc/1");
 // Execute the method.
          HttpHost targetHost = new HttpHost("localhost",
9200, "https");
         CredentialsProvider credsProvider = new
BasicCredentialsProvider();
 credsProvider.setCredentials(AuthScope.ANY,
                 new UsernamePasswordCredentials(System.
getenv("ES_USER"), System.getenv("ES_PASSWORD")));
 // Create AuthCache instance
 AuthCache authCache = new BasicAuthCache();

Creating a standard Java HTTP client 479
 
// Generate BASIC scheme object and add it to
local auth cache
BasicScheme basicAuth = new BasicScheme();
authCache.put(targetHost, basicAuth);
// Add AuthCache to the execution context
HttpClientContext context = HttpClientContext.
create();
context.setCredentialsProvider(credsProvider);
method.addHeader("Accept-Encoding", "gzip");
try {
CloseableHttpResponse response = client.
execute(method, context);
if (response.getStatusLine().getStatusCode()
!= HttpStatus.SC_OK) {
System.err.println("Method failed: " +
response.getStatusLine());
} else {
HttpEntity entity = response.getEntity();
String responseBody = EntityUtils.
toString(entity);
System.out.println(responseBody);
}
} catch (IOException e) {
System.err.println("Fatal transport error: "
+ e.getMessage());
e.printStackTrace();
} finally {
// Release the connection.
method.releaseConnection();
}} }

If you run the code, the result output will be as follows:

{"_index":"mybooks","_type":"_doc","_id":"1","_
version":1,"_seq_no":0,"_primary_term":1,"found":true,"_
source":{"uuid":"11111","position":1,"title":"Joe
Tester","description":"Joe Testere nice guy","date":"
2021-10-22","price":4.3,"quantity":50}}

Creating a low-level Elasticsearch client

The two official Elasticsearch clients are the low-level one and the new typed one available from Elasticsearch 8.x. The low-level one is used to communicate with Elasticsearch, and its main features include minimal dependencies, load balancing across all available nodes, failover in the case of node failures, failed connection penalization, persistent connections, trace logging of requests and responses, and optional automatic discovery of cluster nodes. To execute the following commands, an index must be populated with the ch04/populate_kibana.txt commands, and a Maven tool or an IDE that natively supports it for Java programming must be installed.

To create RestClient, we will perform the following steps:

• We need to add the Elasticsearch HTTP client library that’s used to execute HTTP calls. This library is available in the main Maven repository at search.maven. org. To enable compilation in your Maven pom.xml project, just add the following code:

<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>elasticsearch-rest-client</artifactId>
<version>8.0.0</version>
</dependency>

• If we want to instantiate a client and fetch a document with a get method, the code will look like the following:

import org.apache.http.HttpEntity;
import org.apache.http.HttpHost;
import org.apache.http.HttpStatus;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.CredentialsProvider;
import org.apache.http.conn.ssl.NoopHostnameVerifier;
import org.apache.http.conn.ssl.TrustAllStrategy;
import org.apache.http.impl.client.
BasicCredentialsProvider;
import org.apache.http.impl.nio.client.
HttpAsyncClientBuilder;
import org.apache.http.ssl.SSLContextBuilder;
import org.apache.http.util.EntityUtils;
import org.elasticsearch.client.*;
import java.io.IOException;
import java.security.*;
import javax.net.ssl.SSLContext;
public class App {
public static void main(String[] args) throws
KeyManagementException, NoSuchAlgorithmException,
KeyStoreException {
RestClientBuilder clientBuilder = RestClient.
builder(new HttpHost("localhost", 9200, "https"))
.setCompressionEnabled(true);
final CredentialsProvider credentialsProvider =
new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY,
new UsernamePasswordCredentials(System.
getenv("ES_USER"), System.getenv("ES_PASSWORD")));
final SSLContext sslContext = new
SSLContextBuilder().loadTrustMaterial(null,
TrustAllStrategy.INSTANCE).build();
clientBuilder.setHttpClientConfigCallback(new
RestClientBuilder.HttpClientConfigCallback() {
Creating a low-level Elasticsearch client 485
 
public HttpAsyncClientBuilder
customizeHttpClient(HttpAsyncClientBuilder
httpClientBuilder) {
return httpClientBuilder
.setSSLContext(sslContext)
.setSSLHostnameVerifier(NoopHostnameVerifier.INSTANCE)
.setDefaultCredentialsProvider(credentialsProvider);
}
});
RestClient client = clientBuilder.build();
try {
Request request = new Request("GET", "/_doc/1");
Response response = client.
performRequest(request);
if (response.getStatusLine().getStatusCode()
!= HttpStatus.SC_OK) {
System.err.println("Method failed: " +
response.getStatusLine());
} else {
HttpEntity entity = response.getEntity();
String responseBody = EntityUtils.
toString(entity);
System.out.println(responseBody);
}
} catch (IOException e) {
System.err.println("Fatal transport error: "
+ e.getMessage());
e.printStackTrace();
} finally {
// Release the connection.
try {
client.close();
} catch (IOException e) {
e.printStackTrace();
}}} }

After compiling and executing the code, the result will be as follows:

{"_index":"mybooks","_type":"_doc","_id":"1","_
version":1,"_seq_no":0,"_primary_term":1,"found":true,"_
source":{"uuid":"11111","position":1,"title":"Joe
Tester","description":"Joe Testere nice guy","date":"
2021-10-22", "price":4.3, "quantity":50 }}

Using the Elasticsearch official Java client

• The official Java client is built on top of a low-level one and provides strong typed communication with Elasticsearch.
• Initially released with Elasticsearch in the latest versions of 7.15 or above, this client is the official Java client for Elasticsearch 8.x.
• It provides many extra functionalities, such as the following: • Integration from application classes to JSON instances via an object mapper such as Jackson • Request/response marshaling/unmarshaling that provides stronger typed programming • Support for both synchronous and asynchronous calls • Use of fluent builders and functional patterns to allow writing concise yet readable code when creating complex nested structures • Built on top of previous low-level client Note RestClient is a low-level client; it has no helpers on build queries or actions.
• For now, using it consists of building the JSON string of the request and then parsing the JSON response string.
• A Maven tool or an IDE that natively supports it for Java programming, such as Visual Studio Code, Eclipse, or IntelliJ IDEA, must be installed.
• Because Elasticsearch 8.x or above is secured by default, to run all the examples of this chapter, put the credentials in ES_USER and ES_PASSWORD environment variables when executing the code.

The following screenshot shows how to put them (Environment variables) in IntelliJ IDEA:

Managing indices

In this section we will learn how to manage indices via client calls.

An Elasticsearch client maps all index operations under the indices object of the client, such as create, delete, exists, open, close, and optimize. The following steps retrieve a client and execute the main operations on the indices:

• First, we import the required classes, as shown in the following code:

import co.elastic.clients.elasticsearch.
ElasticsearchClient;
import java.io.IOException;
import java.security.KeyManagementException;
import java.security.KeyStoreException;
import java.security.NoSuchAlgorithmException;

• Then, we define an IndicesOperations class that manages the index operations, as shown in the following code:

public class IndicesOperations {
private final ElasticsearchClient client;
public IndicesOperations(ElasticsearchClient client)
{ this.client = client; }

• Next, we define a function that is used to check whether the index is there, as shown in the following code:

public boolean checkIndexExists(String name) throws
IOException {return client.indices().exists(c -> c.index(name)).
value(); }

• Then, we define a function that can be used to create an index, as shown in the following code:

public void createIndex(String name) throws IOException {
client.indices().create(c -> c.index(name));
}

• We then define a function that can be used to delete an index, as follows:

public void deleteIndex(String name) throws IOException {
client.indices().delete(c -> c.index(name)); }

• Then, we define a function that can be used to close an index, as follows:

public void closeIndex(String name) throws IOException {
client.indices().close(c -> c.index(name)); }

• Next, we define a function that can be used to open an index, as follows:

public void openIndex(String name) throws IOException {
client.indices().open(c -> c.index(name)); }

• Then, we test all the previously defined functions, as follows:

public static void main(String[] args)
throws InterruptedException, IOException,
NoSuchAlgorithmException, KeyStoreException,
KeyManagementException {
    ClientHelper nativeClient = new ClientHelper();
    ElasticsearchClient client = nativeClient.
    getClient();
    IndicesOperations io = new IndicesOperations(client);
    String myIndex = "test";
    if (io.checkIndexExists(myIndex)) io.deleteIndex(myIndex);
    io.createIndex(myIndex);
    Thread.sleep(1000);
    io.closeIndex(myIndex);
    io.openIndex(myIndex);
    io.deleteIndex(myIndex);
}

Managing mappings

After creating an index, the next step is to add some mappings to it.

In the following steps, we add a mapping to a myindex index via the native client:

1. Import the required classes using the following code:

import co.elastic.clients.elasticsearch.
ElasticsearchClient;
import java.io.IOException;
import java.security.KeyManagementException;
import java.security.KeyStoreException;
import java.security.NoSuchAlgorithmException;

2. Define a class to contain our code and to initialize client and index, as follows:

public static void main(String[] args) throws
NoSuchAlgorithmException, KeyStoreException,
KeyManagementException {
    String index = "mytest";
    ElasticsearchClient client = ClientHelper.
    createClient();
    IndicesOperations io = new IndicesOperations(client);
    try {
        if (io.checkIndexExists(index))
            io.deleteIndex(index);
        io.createIndex(index);

3. Prepare the mapping request to put in the index, as follows:

var response = client.indices()
    .putMapping(p - > p.index(index).properties("nested1", m - > m.nested(f - > f)));

4. We can now check the response, as follows:

if (!response.acknowledged()) {
System.out.println("Something strange happens"); }

5. We’ll remove index to clean up, as follows:

io.deleteIndex(index);

Managing documents

The native APIs for managing documents (index, delete, and update) are the most important after the search APIs. In this section, we will learn how to use them. In the next section, we will proceed to bulk actions to improve performance.

1. We’ll need to import the required classes to execute all the document CRUD operations via the high-level client, as follows:

public static void main(String[] args) throws
NoSuchAlgorithmException, KeyStoreException,
KeyManagementException {
    String index = "mytest";
    ElasticsearchClient client = ClientHelper.
    createClient();
    IndicesOperations io = new IndicesOperations(client);
    try {
        if (io.checkIndexExists(index))
            io.deleteIndex(index);

2. In this example, we will map our indexed data in the following data class:

public static class Record {
private String text;
public String getText() { return text; }
public void setText(String text) { this.text = text;
}
}

3. The following code will create the client and remove the index that contains our data, if it exists:

public static void main(String[] args) throws
NoSuchAlgorithmException, KeyStoreException,
KeyManagementException {
    String index = "mytest";
    ElasticsearchClient client = ClientHelper.
    createClient();
    IndicesOperations io = new IndicesOperations(client);
    try {
        if (io.checkIndexExists(index))
            io.deleteIndex(index);

4. We will call the create index by providing the required mapping, as follows:

try {
    client.indices()
        .create(c - > c.index(index).mappings(m -
            > m.properties("text", t - > t.text(fn - >
                fn.store(true)))));
} catch (IOException e) {
    System.out.println("Unable to create mapping");
}

5. Now, we can create a typed document record and store a document in Elasticsearch via the index call, as follows:

final Record document = new Record();
document.setText("unicorn");
IndexResponse ir = client.index(c - > c.index(index).id("2").document(document));
System.out.println("Version: " + ir.version());

6. Let’s retrieve the stored document via the get call, as follows:

GetResponse<DocumentOperations.Record> gr = client.get(c
-> c.index(index).id("2"), Record.class);
System.out.println("Version: " + gr.version());

7. We can update the stored document via the update call using a painless script, as follows:

UpdateResponse < DocumentOperations.Record > ur = client.
update(u - >
    u.index(index).id("2")
    .scriptedUpsert(true)
    .script(Script.of(s - > s.inline(code - >
        code.source("ctx._source.text = 'v2'")))),
    Record.class
);
System.out.println("Version: " + ur.version());

8. We can delete the stored document via the delete call, as follows:

client.delete(d -> d.index(index).id("2"));

9. We can now free up the resources that were used, as follows:

io.deleteIndex(index);
} catch (IOException e) {
e.printStackTrace();}

10. The console output result will be as follows:

Version: 1
Version: 1
Version: 2

Managing bulk actions

Executing automatic operations on items via a single call will often be the cause of a bottleneck if you need to index or delete thousands/millions of records. The best practice, in this case, is to execute a bulk action.

To manage a bulk action, we will perform these steps:

1. We’ll need to import the required classes to execute bulk actions via the high-level client, as follows:

import co.elastic.clients.elasticsearch.
ElasticsearchClient;
import co.elastic.clients.elasticsearch.core.bulk.
BulkOperation;
import java.io.IOException;
import java.security.KeyManagementException;
import java.security.KeyStoreException;
import java.security.NoSuchAlgorithmException;
import java.util.ArrayList;
import java.util.List;

2. Next, we’ll create the client, remove the old index if it exists, and create a new one, as follows:

public static void main(String[] args) throws
NoSuchAlgorithmException, KeyStoreException,
KeyManagementException {
    String index = "mytest";
    ElasticsearchClient client = ClientHelper.
    createClient();
    IndicesOperations io = new IndicesOperations(client);
    try {
        if (io.checkIndexExists(index))
            io.deleteIndex(index);
        try {
            client.indices()
            mappings(m - > m
                .create(c - > c.index(index).
                    .properties("position", p - > p.integer(i - >
                        i.store(true)))
                )
            );
        } catch (IOException e) {
            mapping ");
            System.out.println("Unable to create
            }

3. To execute a bulk operation, we need to collect all the documents in BulkOperation to prepare the list to execute against the client.bulk method, as follows:

List < BulkOperation > bulkOperations = new ArrayList < > ();
for (int i = 1; i <= 1000; i++) {
    
final AppData obj = new AppData();
    
obj.setPosition(i);
    
bulkOperations.add(BulkOperation.of(o - > o.index(idx -
> idx.index(index).id(Integer.toString(obj.position)).document(obj))));
}
System.out.println("Number of actions for index: " +
bulkOperations.size());
client.bulk(c -> c.operations(bulkOperations));

4. We can bulk update the previously-created 1,000 documents via a script, adding the bulk update action to bulker, as follows:

bulkOperations.clear();
for (int i = 1; i <= 1000; i++) {
    
final int id = i;
    
bulkOperations.add(BulkOperation.of(o - > o.update(u -
> u.index(index).id(Integer.toString(id)).action(a ->
                        a.script(s - > s.inline(code -> code.source("ctx._source.
                                    position += 2 ")))))));
}
                                
System.out.println("Number of actions for update: " +
                                    bulkOperations.size()); client.bulk(c - > c.operations(bulkOperations));

5. We can bulk delete 1,000 documents, adding the bulk delete actions to bulker, as follows:

bulkOperations.clear();
for (int i = 1; i <= 1000; i++) {
    
final int id = i;
    
bulkOperations.add(BulkOperation.of(o -> o.delete(u -
> u.index(index).id(Integer.toString(id)))));
}

System.out.println("Number of actions for delete: " +
    bulkOperations.size());
client.bulk(c - > c.operations(bulkOperations));

6. We can now free up the resources that were used, as follows:

o.deleteIndex(index);

}
catch (IOException e) {
 
   e.printStackTrace();
}

The result will be as follows:

Number of actions for index: 1000
Number of actions for update: 1000
Number of actions for delete: 1000

Building a query

Before a search, a query must be built. Elasticsearch provides several ways to build these queries.

To create a query, we will perform the following steps:

1. We need to import SearchRequest using the following code:

import co.elastic.clients.elasticsearch.core.SearchRequest;

2. Next, we’ll create a query using SearchRequest, as follows:

SearchRequest searchRequest = new SearchRequest.Builder()
.index(index)
.query(q – >
q.bool(b – > b
.must(must – >
must.range(r – >
r.field(“number1”).gte(JsonData.of(500)))
).filter(f – > f.term(t – >
t.field(“number2”).value(FieldValue.of(1))))
)
).build();

3. Now, we can execute a search, as follows (searching via a native API will be discussed in the following recipes):

SearchResponse response = client.
search(searchRequest, Record.class);
assert response.hits().total() != null;
System.out.println(“Matched records of elements: ” + response.hits().total().value());

Executing a standard search

We will now execute a query to retrieve some documents.

To execute a standard query, we will perform the following steps:

1. We need to import SearchRequest.QueryBuilder to create the query, as follows:

import co.elastic.clients.elasticsearch.core.SearchRequest;

2. We can create an index and populate it with some data, as follows:

String index = "mytest";
QueryHelper qh = new QueryHelper();

qh.populateData(index);
ElasticsearchClient client = qh.getClient();

3. Now, we will build a query with the number1 field greater than or equal to 500, we filter it for number2 equal to 1, and we highlight the name field as follows:

SearchRequest searchRequest = new SearchRequest.
Builder().index(index)
    .query(q - >
        q.bool(b - > b
            .must(must - > must.range(r - >
                r.field("number1").gte(JsonData.of(500)))).filter(f - > f.term(t - >
                t.field("number2").value(FieldValue.of(1))))))
    .highlight(h - > h.fields("name", h1 - >
        h1.field("name"))).build();

4. After creating a query, it is enough to execute it using the following code:

SearchResponse < QueryHelper.AppData > response = client.
search(searchRequest, QueryHelper.AppData.class);
System.out.println("Matched number of documents: " +
    response.hits().total().value());
System.out.println("Maximum score: " + response.hits().maxScore());;

5. When we have SearchResponse, we need to check its status and iterate it on hit(s), as follows:

for (Hit < QueryHelper.AppData > hit: response.hits().hits()) {
    System.out.println("hit: " + hit.index() + ":" + hit.id());
}

The result should be similar to the following:

Matched number of documents: 251 hits
Maximum score: 1.0
hit: mytest:_doc:499
hit: mytest:_doc:501
hit: mytest:_doc:503
hit: mytest:_doc:505
hit: mytest:_doc:507
hit: mytest:_doc:509
hit: mytest:_doc:511
hit: mytest:_doc:513
hit: mytest:_doc:515
hit: mytest:_doc:517

Conclusion

In conclusion, integrating Elasticsearch with Java can greatly enhance the search functionality and performance of your Java applications. Elasticsearch provides powerful and flexible search capabilities, and its integration with Java makes it easy to implement in your application. Overall, the integration of Elasticsearch with Java allows for a robust and efficient search functionality in Java applications, and can greatly improve the user experience and overall performance of your application. By leveraging the power of Elasticsearch and Java together, developers can provide a powerful and efficient search experience to their users.

Tags: elasticsearch index using javaelasticsearch javaelasticsearch java integraionelasticsearch using javamanage elasticsearch documents java