22 December 2021

Blog Moved to Medium


 

 

Hi All I have moved my blog to Medium. Please follow me there

Posted by at No comments:

16 May 2020

Spring Boot Reactive Testing Part 1



I was worked in spring 4 in the last couple of years. Then I thought to learn reactive spring using spring 5. Then I thought to write an article that I have discovered new in reactive programming testing.

Here I will not explain how to write reactive rest endpoints. Here It will explain how to test different kinds of scenarios. In this thread I will explain how to test data repository & reactive API endpoint. Next tutorial I will explain how to test reactive client

It is like same in synchronous mongo client. First write repository & then using that repository we can write a simple JUnit test. Here I have used StepVerifier to validate the response.

import dev.innova.mockito.mockitoserver.domain.UserData;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.autoconfigure.data.mongo.DataMongoTest;
import org.springframework.test.context.junit4.SpringRunner;
import reactor.core.publisher.Flux;
import reactor.test.StepVerifier;

@DataMongoTest
@RunWith(SpringRunner.class)
public class UserDataRepositoryTest {

    @Autowired
    private UserRepository userRepository;

    @Test
    public void testUserQuery() {

        Flux<UserData> userDataFlux = this.userRepository.deleteAll()
                .thenMany(Flux.just("sajith", "sajith", "sajith2", "sajuth3")
                .flatMap(item -> this.userRepository.save(new UserData(null, item, item)))
                .thenMany(this.userRepository.findByName("sajith")));

        StepVerifier.create(userDataFlux)
                .expectNextCount(2)
                .verifyComplete();
    }
}

As the first step I have created a User Resource configuration. It will return the user list.

import dev.innova.mockito.mockitoserver.domain.UserData;
import dev.innova.mockito.mockitoserver.repository.UserRepository;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.http.MediaType;
import org.springframework.web.reactive.function.server.RouterFunction;
import org.springframework.web.reactive.function.server.RouterFunctions;
import org.springframework.web.reactive.function.server.ServerResponse;

import static org.springframework.web.reactive.function.server.RequestPredicates.GET;

@Configuration
public class UserResourceConfiguration {

    @Bean
    RouterFunction<ServerResponse> routes(UserRepository userRepository) {
        return RouterFunctions.route(GET("/allUsers"), request -> ServerResponse.ok()
                .contentType(MediaType.APPLICATION_JSON).body(userRepository.findAll(), UserData.class));
    }
}


Then using mockito we can mock the object & methods. Then using WebTestClient test case like below.


import dev.innova.mockito.mockitoserver.config.UserResourceConfiguration;
import dev.innova.mockito.mockitoserver.domain.UserData;
import dev.innova.mockito.mockitoserver.facade.ClassifiedService;
import dev.innova.mockito.mockitoserver.repository.UserRepository;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.Mockito;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.autoconfigure.web.reactive.WebFluxTest;
import org.springframework.boot.test.mock.mockito.MockBean;
import org.springframework.context.annotation.Import;
import org.springframework.test.context.junit4.SpringRunner;
import org.springframework.test.web.reactive.server.WebTestClient;
import reactor.core.publisher.Flux;

@WebFluxTest
@Import(UserResourceConfiguration.class)
@RunWith(SpringRunner.class)
public class AddsRestApiTest {

    @MockBean
    private UserRepository userRepository;

    @Autowired
    private WebTestClient webTestClient;

    @MockBean
    private ClassifiedService classifiedService;

    @Test
    public void getAllAddsTest() throws Exception {

        UserData add1 = new UserData();
        add1.setId("001");
        add1.setName("Sample1");

        UserData add2 = new UserData();
        add2.setId("002");
        add2.setName("Sample2");

        UserData add3 = new UserData();
        add3.setId("003");
        add3.setName("Sample3");

        Mockito.when(this.userRepository.findAll())
                .thenReturn(Flux.just(add1, add2, add3));

        this.webTestClient.get()
                .uri("http://localhost:8084/allUsers")
                .exchange()
                .expectStatus().isOk()
                .expectBody().jsonPath("@.[0].id").isEqualTo("001");

    }
}

Code base available in here.If you have any suggestions or questions put comments here.

Posted by at 1 comment:
Labels: , , , , ,

12 October 2018

Analyze Custom Logs Using ELK

Last Month I was searching for a topic to write a good article. Then I thought to write an article regarding ELK stack. This article will describe the practical usage of the ELK stack. Let’s assume there is a merchant registration system in the eCommerce platform. Then at the end of the month we have to show the number of merchant registrations and merchant details ( location, country, etc ). For the above scenario there are lots of options to solve the problem. The common approach is to write data to the database and using any reporting library to show the report. But Is this approach is extensible ??. That is the problem. If the new requirement comes there are a lot of changes. Here what will do is write data to log files and analyze the data using ELK.

For this example I have generated log as json format. I have used this to generate json data set. This is the sample data set entry. 


1
{“id”:1,”first_name”:”Freeman”,”last_name”:”Jowers”,”email”:”fjowers0@mashable.com”,”gender”:”Male”,”ip_address”:”15.128.77.162",”latitude”:9.9004655,”longitude”:13.0544185,”date”:”2017–10–29T17:47:59Z”,”country”:”Nigeria”}


Logstash Configuration

First I will show the logstash configuration I have used for read custom json logs & describe section by section.


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
input {
 file{
 path => [“/data/sample-data.json”]
 type => “json”
 start_position => “beginning”
 }
}
filter {
 grok {
 match => [‘message’,’(?<body>\”id\”:.*\”country\”:\”[^”]+\”)’]
 add_field => [“json_body”,”{%{body}}”]
 }
 json {
 source => “json_body”
 remove_field => [“message”,”body”,”json_body”]
 }
 mutate{
 add_field => [“[geoip][location]”,”%{[latitude]}”]
 add_field => [“[geoip][location]”,”%{[longitude]}”]
 }
 mutate{
 convert => [“[geoip][location]”,”float”]
 }
}
output {
 stdout {
 codec => rubydebug
 }
 elasticsearch {
 hosts => [“localhost:9200”]
 }
}

In logstash configuration three config sections
  1. Input
  2. Filter
  3. OutPut
Logstash configurations build on filter based . There are currently 200 plugins available . In above configuration Input section describe how the event source can connect to logstash. In here event coming from file.
Filter section describe how the event are manipulated. In my case I have used two plugins to manipulate data. grok plugin convert separate json data entry. And mutate plugin convert longitude and latitude data to geo_point data.
Output section describes where the data need to output. It can be stream processing system [kafka] or database or may be to another logstash input. In above scenario set output as elastic search engine.

Read Logs Using Logstash



1
./logstash -f config-file.conf


Read Full article
Posted by at No comments:
Labels: , , , ,

27 May 2018

Spring JPA/Data with Postgres Array Types


Last couple of weeks I have to work with spring boot pet project. This solution for one of the difficulty faced with postgres string array with spring JPA. The problem vs It Always failed to map the domain class array list with postgress text array. This Solution is common for any type of Arrays in postgres database.

 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
package com.example.mytastyserver.util;

import org.hibernate.HibernateException;
import org.hibernate.engine.spi.SharedSessionContractImplementor;
import org.hibernate.usertype.UserType;

import java.io.Serializable;
import java.sql.*;

public class GenericArrayUserType<T extends Serializable> implements UserType {
    protected static final int[] SQL_TYPES = {Types.ARRAY};
    private Class<T> typeParameterClass;

    @Override
    public Object assemble(Serializable cached, Object owner) throws HibernateException {
        return this.deepCopy(cached);
    }

    @Override
    public Object deepCopy(Object value) throws HibernateException {
        return value;
    }

    @SuppressWarnings("unchecked")
    @Override
    public Serializable disassemble(Object value) throws HibernateException {
        return (T) this.deepCopy(value);
    }

    @Override
    public boolean equals(Object x, Object y) throws HibernateException {

        if (x == null) {
            return y == null;
        }
        return x.equals(y);
    }

    @Override
    public int hashCode(Object x) throws HibernateException {
        return x.hashCode();
    }

    @Override
    public Object nullSafeGet(ResultSet resultSet, String[] names, SharedSessionContractImplementor sharedSessionContractImplementor, Object o) throws HibernateException, SQLException {
        if (resultSet.wasNull()) {
            return null;
        }
        if (resultSet.getArray(names[0]) == null) {
            return new Integer[0];
        }

        Array array = resultSet.getArray(names[0]);
        @SuppressWarnings("unchecked")
        T javaArray = (T) array.getArray();
        return javaArray;
    }

    @Override
    public void nullSafeSet(PreparedStatement statement, Object value, int index, SharedSessionContractImplementor sharedSessionContractImplementor) throws HibernateException, SQLException {
        Connection connection = statement.getConnection();
        if (value == null) {
            statement.setNull(index, SQL_TYPES[0]);
        } else {
            @SuppressWarnings("unchecked")
            T castObject = (T) value;
            Array array = connection.createArrayOf("integer", (Object[]) castObject);
            statement.setArray(index, array);
        }
    }

    @Override
    public boolean isMutable() {
        return true;
    }

    @Override
    public Object replace(Object original, Object target, Object owner) throws HibernateException {
        return original;
    }

    @Override
    public Class<T> returnedClass() {
        return typeParameterClass;
    }

    @Override
    public int[] sqlTypes() {
        return new int[]{Types.ARRAY};
    }
}

Then you can add the type of the entity class.

 
1
2
@Type(type = "com.example.mytastyserver.util.GenericArrayUserType")
private String[] tags;

Hope this will helps to resolve your issue. Thanks
Posted by at No comments:
Labels: , , ,

10 February 2018

Pure Restful web services with Spring Data Rest

Last couple of weeks I had a chance to work with spring data rest project. When I read the documentation I thought I pretty awesome.Because It has lot of features for REST api development.Here I will describe main features contains in spring data rest. At the end of this thread I will create simple rest api for product management.

What is Pure Restful Webservice ?

First we talk about what is restful web service.  Restful web service is architectural style for request and response mapping in web service. There are different kind of HTTP request coming to API. But when it comes to REST api every url have different meaning for different HTTP request methiods.

Ex: /api/v1/student/  --- > GET
      /api/v1/student/ ----> POST
     /api/v1/student/ ---> PUT

And there are different response Status codes for different purposes.
Method Method
200 OK 201 Created
202 Accepted 203 Non-Authoritative Info
204 No content 205 Reset content
206 Partial content
300 Multiple choice 301 Moved permanently
302 Found 303 See other
304 Not modified 306 (unused)
307 Temporary redirect
400 Bad request 401 Unauthorized
402 Payment required 403 Forbidden
404 Not found 405 Method not allowed
406 Not acceptable 407 Proxy auth required
408 Timeout 409 Conflict
410 Gone 411 Length required
412 Preconditions failed 413 Request entity too large
414 Requested URI too long 415 Unsupported media
416 Bad request range 417 Expectation failed
500 Server error 501 Not implemented
502 Bad gateway 503 Service unavailable
504 Gateway timeout 505 Bad HTTP version

As Example When we POST student obejct to rest API it will need to Send 204 no content status code rather than 200 status code.likewise there are standard ways to send response.If all the standard are fulfill  it will become pure restful web service.

These are the core areas we are discus in this tutorial.
    

  1. Projection
    2. 

  2. Search
    3. 

  3. Advance Search
    4. 

  4. Pagination & Sorting 
    5. 

  5. Spring Security
    6. 




To Discuss spring data rest we used simple sales management demo. These are the entity classes. We have Product,Sale,SalesAgent



package dev.firelimez.io.domain;


import org.springframework.data.annotation.Id;


public class Product {
    @Id
    private String productId;
    private String name;
    private String price;

    public String getProductId() {
        return productId;
    }

    public void setProductId(String productId) {
        this.productId = productId;
    }

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public String getPrice() {
        return price;
    }

    public void setPrice(String price) {
        this.price = price;
    }
}









package dev.firelimez.io.domain;

import org.springframework.data.annotation.Id;
import org.springframework.data.mongodb.core.mapping.DBRef;

import java.util.List;

public class Sale {
    @Id
    private String salesId;
    private String date;

    @DBRef
    private SalesAgent salesAgent;

    @DBRef
    private List<Product> products;

    public String getSalesId() {
        return salesId;
    }

    public void setSalesId(String salesId) {
        this.salesId = salesId;
    }

    public String getDate() {
        return date;
    }

    public void setDate(String date) {
        this.date = date;
    }

    public List<Product> getProducts() {
        return products;
    }

    public void setProducts(List<Product> products) {
        this.products = products;
    }

    public SalesAgent getSalesAgent() {
        return salesAgent;
    }

    public void setSalesAgent(SalesAgent salesAgent) {
        this.salesAgent = salesAgent;
    }
}









package dev.firelimez.io.domain;


import org.springframework.data.annotation.Id;

import java.io.Serializable;

public class SalesAgent implements Serializable {

    @Id
    private String agentId;
    private String name;
    private String lastName;
    private String age;

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public String getLastName() {
        return lastName;
    }

    public void setLastName(String lastName) {
        this.lastName = lastName;
    }

    public String getAge() {
        return age;
    }

    public void setAge(String age) {
        this.age = age;
    }

    public String getAgentId() {
        return agentId;
    }

    public void setAgentId(String agentId) {
        this.agentId = agentId;
    }
}






So this tutorial I will use mongo database as database reference. First you have to add these dependencies to your maven project.








    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>1.5.9.RELEASE</version>
    </parent>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-rest</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-mongodb</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-security</artifactId>
        </dependency>
        <dependency>
            <groupId>com.google.code.gson</groupId>
            <artifactId>gson</artifactId>
            <version>2.8.2</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>






Spring data rest all are written using interfaces. So first thing we have to do is create repository. All the entity classes should have repository classes. As example this is the example Repository class.








package dev.firelimez.io.repo;

import dev.firelimez.io.domain.Product;
import org.springframework.data.mongodb.repository.MongoRepository;
import org.springframework.data.repository.query.Param;
import org.springframework.data.rest.core.annotation.RepositoryRestResource;
import org.springframework.data.rest.core.annotation.RestResource;

import java.util.List;

@RepositoryRestResource(collectionResourceRel = "products", path = "products")
public interface ProductRepository extends MongoRepository<Product, String> {

    @RestResource(path = "names", rel = "demo1")
    List<Product> findByName(@Param("name") String name);


    @RestResource(path = "similar", rel = "demo")
    List<Product> findByNameLike(@Param("name") String name);

    @Override
    @RestResource(exported = true)
    void delete(String productId);
}






And Final step is start program as spring boot application.






package dev.firelimez.io;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class StarterApplication {

    public static void main(String[] args) {
        SpringApplication.run(StarterApplication.class, args);
    }
}






And next step is add application.yml and configuration should be like this.






server:
    port: 9008

spring:
 data:
    mongodb:
      host: 127.0.0.1
      port: 27017
      database: sales_management






And that is your REST API  is ready.











Projection





Projection is concept expose specific fields to rest api end points. As example in sales rest API  how to get the sales agent details who are done the  sales. In this case we can use projection. For that we have to add what are the properties we expect when we using projection. 







Ex:  sales expand projection return sales agent details with the sales objects.










package dev.firelimez.io.domain.projection;

import dev.firelimez.io.domain.Sale;
import org.springframework.data.rest.core.config.Projection;

import java.util.List;

@Projection(name = "expand", types = Sale.class)
interface SalesProjection {

    String getSalesId();

    String getDate();

    SalesAgentProjection getSalesAgent();

    List<ProductProjection> getProducts();
}















package dev.firelimez.io.domain.projection;

import org.springframework.beans.factory.annotation.Value;

interface SalesAgentProjection {

    int getAge();

    String getAgentId();

    @Value("#{target.name} #{target.lastName}")
    String getFullName();
}








And then you can access projection data using this way.








http://localhost:9008/sales?projection=expand










Search & Advance Search





When we creating rest api biggest issue is dealing with database and based on user REST end point what we did was write query and get result from database and return to user. But spring data rest support queryless way to get result from database. 







Ex:  if we want to get product by product name 










@RestResource(path = "names", rel = "name")
    List<Product> findByName(@Param("name") String name);










These are the inbuild query methods in spring jpa repository.



Advanced Search



For advance search we have to use ExampleMatcher class and it will support advance search in spring data rest.








    @Autowired
    SalesAgentRepository salesAgentRepository;

    @RequestMapping(method = RequestMethod.POST, path = "/api/v2/agent/search/advance", consumes = MediaType.APPLICATION_JSON_VALUE)
    @ResponseBody
    public List<SalesAgent> claimSupportTicket(@RequestBody SalesAgent salesAgent) {
        ExampleMatcher exampleMatcher = ExampleMatcher.matching().withIgnoreNullValues().withIgnoreCase();
        List<SalesAgent> advanceSearch = salesAgentRepository.findAll(Example.of(salesAgent, exampleMatcher));
        return advanceSearch;
    }
























Posted by



at





No comments:
  

















Labels:
,
,
,
















        

      









Subscribe to:
Posts (Atom)