Spring Cloud Netflix - Hystrix

Spring Cloud Netflix - Hystrix

1. Introduction

Microservice architecture is a distributed system where multiple services collaborate with one another. They typically communicate with each other using REST API.

However, all the services are prone to failure. It might happen due to various reasons like network latency, instance running out of memory. Thus in a typical distributed system, failure of any given microservice is inevitable.

If a Service a fails, then other services which were directly consuming the APIs of Service A will also fail. This in turn will cause failure of the APIs of other services which were dependent on them. This is called cascading effect caused due to a failure of a single service. This might render a huge part of distributed system unaccessible or in worst case bring the entire application down too.

Also debugging of the root cause in the distributed system is more difficult than simple monolithic applications.

Netflix Hystrix is a circuit breaker implementation library which provides fault tolerance.It improves the overall resiliency of a distributed system and stops the cascading effect.

In this article, we’ll explore more about Hystrix.

2. Hystrix

Hystrix is a fault-tolerance library which implements circuit breaker pattern to improve the resiliency of distributed system, Before jumping into Hystrix, let’s first find out what exactly is circuit breaker design pattern.

2.1 Circuit Breaker Design Pattern:

The basic idea behind the circuit breaker is very simple. Wrap a protected/remote function call in a circuit breaker object, which monitors for failures.

Closed State: The circuit breaker continues delegating the calls to the underlying function and monitors the failures. Once the failures reach a certain threshold, the circuit breaker trips open.

Open State: In the open, state the circuit breaker won’t make a protected call at all. Instead, it will route the call to fallback logic (if fallback is configured). The circuit remains in the open state for a certain sleep interval.

After a certain sleep-interval where the circuit is open, it will again attempt to close the circuit to check if the underlying protected call can still be made. If it again fails, the circuit will again trip open for the sleep interval duration. This state is sometimes referred to as Half Open state.

More information about the Circuit Breaker pattern is provided in this beautiful post.

2.2 Hystrix Features:

  1. Hystrix provides the implementation of Circuit Breaker Pattern.
  2. Github link - http://github.com/Netflix/Hystrix
  3. Documentation - http://github.com/Netflix/Hystrix/wiki
  4. It isolates the calls to other services, by wrapping them inside a circuit breaker object. This circuit breaker object acts as a proxy and routes the calls to underlying service, only when it’s closed.
  5. If the circuit is open, then calls won’t be redirected to underlying service. Instead, a fallback logic can be configured and all the calls will to route to it when the circuit is open.
  6. Spring Cloud integration available to integrate it with Spring Boot projects.

3. Project

In our project, we’ll be creating 3 microservices

  1. Eureka Server - service registry
  2. Cricket Service - a microservice providing APIs for a cricket score.
  3. Notify Service - A client microservice which consumes APIs of Cricket service. We’ll be implementing the circuit breaker pattern here to examine how Hystrix handles failures in underlying cricket service.

3.1 Eureka Server

Eureka Server acts as Service Discovery which maintains the list of all the microservices as KEY-VALUE pair. Key is serviceId and value is the instance information (host, port).

All microservices must register themselves on Eureka.

To implement the Eureka server, all we need to do is:

  1. Add spring-cloud-starter-eureka-server dependency
  2. Enable Eureka Server by adding annotation @EnableEurekaServer on our main Spring boot application class.

The project structure is as follows:

Spring Cloud Netflix - Hystrix

3.1.1 Maven Dependencies

<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.hemant</groupId> <artifactId>eureka-server</artifactId> <version>0.0.1-SNAPSHOT</version> <packaging>jar</packaging> <name>eureka-server</name> <description>Demo project for Spring Boot</description> <parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId> <version>1.5.1.RELEASE</version> <relativePath /> </parent> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding> <java.version>1.8</java.version> </properties> <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <!-- https://mvnrepository.com/artifact/org.springframework.cloud/spring-cloud-starter-eureka-server --> <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-eureka-server</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> </dependencies> <dependencyManagement> <dependencies> <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-dependencies</artifactId> <version>Camden.SR5</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement> <build> <plugins> <plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> </plugin> </plugins> </build> </project>

3.1.2 Spring Boot Main Class

package com.hemant.eurekaserver; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.SpringBootApplication; import org.springframework.cloud.netflix.eureka.server.EnableEurekaServer; @SpringBootApplication @EnableEurekaServer publicclassEurekaServerApplication { publicstaticvoidmain(String[] args) { SpringApplication.run(EurekaServerApplication.class, args); } }

3.1.3 Application Properties

# Server port server: port: 8761 eureka: client: # Dont register itself with eureka registerWithEureka: false fetchRegistry: false
  1. Eureka server will run on port 8761
  2. All the clients register themselves on Eureka. Since this is Eureka itself, it doesn’t need to register itself.
  3. FetchRegistry is needed for clients to get information about instances and cache it, hence disabled here.

3.1.4 Deploying and Running Eureka Server:

  1. To start the Eureka Server application, go to the project’s root directory and execute
  2. >>> mvn clean install
  3. If the build is success, we can launch the application by:
  4. >>> java -jar target/eureka-server-0.0.1-SNAPSHOT.jar
  5. The following logs indicate that application is up and running
  6. e.s.EurekaServerInitializerConfiguration : Started Eureka Server s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 8761 (http) c.n.e.EurekaDiscoveryClientConfiguration : Updating port to 8761 c.h.e.EurekaServerApplication : Started EurekaServerApplication in 19.632 seconds (JVM running for21.345)
  7. You can access, the Eureka Server console at http://localhost:8761/
  8. Spring Cloud Netflix - Hystrix
  9. Currently no client is started, hence it’s displaying ‘No Instances Available’.

3.2 Cricket Service Application:

This is a Eureka Registered microservice which exposes some APIs (currently about dummy cricket scores).

Its structure is as follows:

Spring Cloud Netflix - Hystrix

3.2.1 Maven Dependencies

<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.hemant</groupId> <artifactId>cric-service</artifactId> <version>0.0.1-SNAPSHOT</version> <packaging>jar</packaging> <name>cric-service</name> <description>Demo project for Spring Boot</description> <parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId> <version>1.5.1.RELEASE</version> <relativePath /><!-- lookup parent from repository --> </parent> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding> <java.version>1.8</java.version> </properties> <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-eureka</artifactId> <version>1.2.6.RELEASE</version> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> </plugin> </plugins> </build> </project>

3.2.2 Spring Boot Main Class

package com.hemant.cricservice; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.SpringBootApplication; import org.springframework.cloud.netflix.eureka.EnableEurekaClient; @SpringBootApplication @EnableEurekaClient publicclassCricServiceApplication { publicstaticvoidmain(String[] args) { SpringApplication.run(CricServiceApplication.class, args); } }

3.2.3 Controller

The controller exposes API for the mock cricket match data.

package com.hemant.cricservice.controller; import java.util.HashMap; import java.util.Map; import java.util.Properties; import javax.annotation.PostConstruct; import org.springframework.web.bind.annotation.PathVariable; import org.springframework.web.bind.annotation.RequestMapping; import org.springframework.web.bind.annotation.RequestMethod; import org.springframework.web.bind.annotation.RestController; @RestController publicclassCricketController { private Map<Integer, Properties> currentMatchData; /** * Invoked once when the bean is created and all the dependencies are injected. * This contains initialization logic */ @PostConstruct publicvoidcreateMockData() { currentMatchData = new HashMap<>(); Properties p1 = new Properties(); p1.setProperty("MatchName", "IND VS PAK"); p1.setProperty("Team1", "IND"); p1.setProperty("Team2", "PAK"); p1.setProperty("Team1 Score", "189 For 7 IN 20 Overs"); p1.setProperty("Team2 Score", "120 For 7 IN 15 Overs"); currentMatchData.put(1, p1); Properties p2 = new Properties(); p2.setProperty("MatchName", "AUS VS ENG"); p2.setProperty("Team1", "AUS"); p2.setProperty("Team2", "ENG"); p2.setProperty("Team1 Score", "177 For 8 IN 20 Overs"); p2.setProperty("Team2 Score", "157 For 9 IN 20 Overs"); currentMatchData.put(2, p2); } @RequestMapping(value = "/{id}", method = RequestMethod.GET) public Properties getMatchById(@PathVariable(value = "id") int matchId) { Properties matchData = currentMatchData.get(matchId); if(null == matchData) { thrownew IllegalArgumentException("No match exists for id = " + matchId); } return matchData; } }

3.2.4 Application Properties

server : port : 8080 spring : application : name : cricket eureka: client: serviceUrl: defaultZone: http://localhost:8761/eureka

Here

  1. Cricket service’s serviceId = cricket
  2. It will run on port 8080
  3. Also its registered on our Eureka server running on Port 8761

3.2.5 Running The Application

  1. 1. Go to the root directory and execute
  2. >>> mvn clean install
  3. If the build is success, the JAR can be executed by
  4. >>> java -jar target/cric-service-0.0.1-SNAPSHOT.jar
  5. The following logs suggest that the application is started on 8080 And also registered on Eureka.
  6. s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 8080 (http) c.n.e.EurekaDiscoveryClientConfiguration : Updating port to 8080 c.h.cricservice.CricServiceApplication : Started CricServiceApplication in 9.915 seconds (JVM running for10.784) com.netflix.discovery.DiscoveryClient : DiscoveryClient_CRICKET/hembook:cricket:8080 - registration status: 204
  7. Also we can confirm that the service instance is registered on Eureka dashboard at http://localhost:8761/
  8. Spring Cloud Netflix - Hystrix
  9. Confirm whether its API is returning mock match data.
  10. GET http://localhost:8080/1 { "Team2": "PAK", "Team1 Score": "189 For 7 IN 20 Overs", "Team1": "IND", "Team2 Score": "120 For 7 IN 15 Overs", "MatchName": "IND VS PAK" }

3.3 Notify Service (Uses Hystrix)

Notify service is the 3rd microservice.

It is Eureka registered service and further it consumes the APIs of the cricket service. However for API invocation of underlying cricket service, Hystrix is enabled.

The project structure is as follows:

Spring Cloud Netflix - Hystrix

3.3.1 Maven dependencies

<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.hemant</groupId> <artifactId>notify-service</artifactId> <version>0.0.1-SNAPSHOT</version> <packaging>jar</packaging> <name>notify-service</name> <description>Demo project for Spring Boot</description> <parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId> <version>1.5.1.RELEASE</version> <relativePath /><!-- lookup parent from repository --> </parent> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding> <java.version>1.8</java.version> <spring.cloud.version>1.2.6.RELEASE</spring.cloud.version> </properties> <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>com.h2database</groupId> <artifactId>h2</artifactId> <scope>runtime</scope> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-eureka</artifactId> <version>${spring.cloud.version}</version> </dependency> <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-hystrix</artifactId> <version>${spring.cloud.version}</version> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> </plugin> </plugins> </build> </project>

Spring-cloud-starter-hystrix - This dependency provides Hystrix support.

Spring-cloud-starter-eureka - This dependency provides basic Eureka Client support.

3.3.2 Spring Boot Main Application Class

package com.hemant.notification; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.SpringBootApplication; import org.springframework.cloud.client.circuitbreaker.EnableCircuitBreaker; import org.springframework.cloud.netflix.eureka.EnableEurekaClient; import org.springframework.context.annotation.Bean; import org.springframework.web.client.RestTemplate; /** * @EnableCircuitBreaker - enables the underlying circuit breaker config * Hystrix looks for any method annotated with the @HystrixCommand annotation, * and wraps it inside proxy so that Hystrix can monitor it. * * * @EnableEurekaClient - to enable this as eureka client for registry * * @author hemant * */ @SpringBootApplication @EnableEurekaClient @EnableCircuitBreaker publicclassNotifyServiceApplication { publicstaticvoidmain(String[] args) { SpringApplication.run(NotifyServiceApplication.class, args); } @Bean public RestTemplate restTemplate() { returnnew RestTemplate(); } }

3.3.3 Controller

The controller exposes endpoints which in turn consume APIs from Cricket service.

package com.hemant.notification.controller; import java.util.Date; import java.util.HashMap; import java.util.Map; import java.util.UUID; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.web.bind.annotation.PathVariable; import org.springframework.web.bind.annotation.RequestMapping; import org.springframework.web.bind.annotation.RequestMethod; import org.springframework.web.bind.annotation.RestController; import com.hemant.notification.service.NotificationService; @RestController publicclassNotificationController { @Autowired NotificationService notificationService; @RequestMapping(value = "/{matchId}", method = RequestMethod.GET) public Map<String, Object>getMatchNotifications(@PathVariable int matchId) { Map<String, Object> map = new HashMap<>(); map.put("currentTime", new Date()); map.put("latestScore", notificationService.getMatchDetailsById(matchId)); map.put("nextUpdateAvailableIn", "60 sec"); map.put("userId", UUID.randomUUID().toString()); return map; } }

3.3.4 Notification Service and Its Implementation

The Hystrix configuration and the fallback logic is implemented here.

NotificationService interface

package com.hemant.notification.service; import java.util.Properties; publicinterfaceNotificationService { Properties getMatchDetailsById(int matchId); }

Implementation Class

package com.hemant.notification.service; import java.util.List; import java.util.Properties; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.cloud.client.ServiceInstance; import org.springframework.cloud.client.discovery.DiscoveryClient; import org.springframework.http.HttpMethod; import org.springframework.http.ResponseEntity; import org.springframework.stereotype.Service; import org.springframework.web.client.RestTemplate; import com.netflix.hystrix.contrib.javanica.annotation.HystrixCommand; @Service publicclassNotificationServiceImplimplementsNotificationService { privatestaticfinal Logger LOG = LoggerFactory.getLogger(NotificationServiceImpl.class); @Autowired private RestTemplate restTemplate; @Autowired private DiscoveryClient discoveryClient; privatestaticfinal String CRICKET_SERVICE_ID = "cricket"; /** * @HystrixCommand - This annotation works on method or spring component * * As per application properties * -------------------------------------- * <li> circuitBreaker.requestVolumeThreshold = 10 * This property sets the minimum number of requests in a rolling window * that will trip the circuit. * * <li> circuitBreaker.errorThresholdPercentage = 50 * If 50% of the requests fail in rolling window, then circuit will trip open * * <li> circuitBreaker.sleepWindowInMilliseconds = 10000 * This property sets the amount of time after opening circuit, it will reject requests * before reattempting to try if underlying service has recovered or not * -------------------------------------- * * Thus as per configured properties, * Hystrix will observe the requests in rolling window of size 10. * If 50% of these requests fail, then it will open the circuit. * * When circuit is open, Hystrix won't attempt to call underlying service * and directly call the fallback logic. * * The circuit open duration or sleep duration is 10s or 10000 millis * When this duration lapses, circuit breaker will close the circuit and check if underlying * service has recovered or not. * * If YES, then circuit will be closed * If Still NO, then it will open the circuit again for 10s and route all calls to fallback * before retrying again. * * * * More on HYSTRIX PROPERTIES IN * https://github.com/Netflix/Hystrix/wiki/Configuration#intro * */ @Override @HystrixCommand(fallbackMethod = "getMatchDetailsByIdFallback") public Properties getMatchDetailsById(int matchId) { LOG.info("Calling the underlying service"); List<ServiceInstance> cricketServiceInstances = discoveryClient.getInstances(CRICKET_SERVICE_ID); if (cricketServiceInstances.isEmpty()) { LOG.error("No cricket service found for ID : {}", CRICKET_SERVICE_ID); thrownew RuntimeException("No cricket service found"); } ServiceInstance cricketServiceInstance = cricketServiceInstances.get(0); String url = cricketServiceInstance.getUri().toString() + "/" + matchId; ResponseEntity<Properties> response = restTemplate.exchange(url, HttpMethod.GET, null, Properties.class); Properties matchDetails = response.getBody(); matchDetails.setProperty("source", url); LOG.info("Found match:{} for matchId:{} from underlying cricket service", matchDetails, matchId); return matchDetails; } /** * A fallback method should be defined in the same class where is HystrixCommand * The signature of the fallback method must exactly match the Hystrix enabled method. * @param matchId * @return */ public Properties getMatchDetailsByIdFallback(int matchId) { LOG.warn("HYSTRIX FALLBACK METHOD IS INVOKED"); Properties properties = new Properties(); properties.setProperty("source", "getMatchDetailsByIdFallback"); properties.setProperty("matchDetails", "NOT FOUND"); return properties; } }
  1. Here the method getMatchDetailsById which invokes underlying cricket service is annotated by Hystrix command.
  2. This method will fail to invoke when underlying service is down or unreachable due to network latency. In this case, normally the API of this service will fail too.
  3. But since Hystrix is in place, failures are handled gracefully. Upon any failure in execution of the method getMatchDetailsById, fallback method is invoked.
  4. The fallback method should be defined in same class and must have the same signature as the original method.
  5. The @HystrixCommand, gets the remaining values from the application properties file.

3.3.5 Application Properties

server: port : 8081 spring: application: name : notification eureka: client: serviceUrl: defaultZone: http://localhost:8761/eureka hystrix: command: default: circuitBreaker: requestVolumeThreshold : 10 sleepWindowInMilliseconds : 10000 errorThresholdPercentage : 50
  1. The service has serviceId as notification and will run on port 8081.
  2. circuitBreaker.requestVolumeThreshold = 10 ->This property sets the minimum number of requests in a rolling window that will trip the circuit.
  3. circuitBreaker.errorThresholdPercentage = 50 -> If 50% of the requests fail in rolling window, then circuit will trip open.
  4. circuitBreaker.sleepWindowInMilliseconds = 10000 ->This property sets the amount of time after opening circuit, it will reject requests before reattempting to try if underlying service has recovered or not.

Thus the Hystrix configuration is as follows:

  1. Hystrix will observe the requests in rolling window of size 10. If 50% of these requests fail, then it will open the circuit.
  2. When circuit is open, Hystrix won't attempt to call underlying service and directly call the fallback logic.
  3. The circuit open duration or sleep duration is 10s or 10000 Millis

When this duration lapses, circuit breaker will close the circuit and check if underlying service has recovered or not.

If YES, then circuit will be closed

If Still NO, then it will open the circuit again for 10s and route all calls to fallback before retrying again.

3.3.6 Running the application:

  1. Go to the root directory and execute
  2. >>> mvn clean install
  3. If the build is success, the JAR can be executed by
  4. >>> java -jar target/notify-service-0.0.1-SNAPSHOT.jar
  5. The following logs suggest that the application is started on 8081 And also registered on Eureka.
  6. 12:20:14 INFO c.n.d.DiscoveryClient - Saw local status change event StatusChangeEvent [timestamp=1535352614798, current=UP, previous=STARTING] 12:20:14 INFO c.n.d.DiscoveryClient - DiscoveryClient_NOTIFICATION/hembook:notification:8081: registering service... 12:20:14 INFO c.n.d.DiscoveryClient - DiscoveryClient_NOTIFICATION/hembook:notification:8081 - registration status: 204 12:20:14 INFO o.s.b.c.e.t.TomcatEmbeddedServletContainer - Tomcat started on port(s): 8081 (http) 12:20:14 INFO o.s.c.n.e.EurekaDiscoveryClientConfiguration - Updating port to 8081 12:20:15 INFO c.h.n.NotifyServiceApplication - Started NotifyServiceApplication in 12.91 seconds (JVM running for14.149)
  7. Upon checking Eureka dashboard, we should see the instance available.
  8. Spring Cloud Netflix - Hystrix

4. Demonstration

  1. Since all applications are up, on invoking the API of notify service, the underlying cricket service should be accessible.
  2. GET http://localhost:8081/1 { "currentTime": 1535352802470, "nextUpdateAvailableIn": "60 sec", "latestScore": { "Team2": "PAK", "Team1": "IND", "Team1 Score": "189 For 7 IN 20 Overs", "Team2 Score": "120 For 7 IN 15 Overs", "source": "http://localhost:8080/1", "MatchName": "IND VS PAK" }, "userId": "36861b4b-6ea0-4fe9-b8ac-d3eff3ee4364" }

    This implies the circuit is closed as all seems well.

  3. Now shutdown the instance of Cricket Service and try accessing the API. From the logs it’s evident that call to main method failed and Hystrix invoked the fallback method in response.
  4. 12:26:19 INFO c.h.n.s.NotificationServiceImpl - Calling the underlying service 12:26:19 ERROR c.h.n.s.NotificationServiceImpl - No cricket service found for ID : cricket 12:26:19 DEBUG c.n.h.AbstractCommand - Error executing HystrixCommand.run(). Proceeding to fallback logic ... java.lang.RuntimeException: No cricket service found at com.hemant.notification.service.NotificationServiceImpl.getMatchDetailsById(NotificationServiceImpl.java:76) ~[classes/:na]

    Since fallback logic was invoked the API returned with 200 OK (with data from fallback logic). This avoided the cascading effect of failure of a service.

    GET http://localhost:8081/1 { "currentTime": 1535352979843, "nextUpdateAvailableIn": "60 sec", "latestScore": { "matchDetails": "NOT FOUND", "source": "getMatchDetailsByIdFallback" }, "userId": "23e58c4a-7b4c-445d-9480-5b10fb2e6988" }
  5. Make quick attempts to hit the API In this case, once the circuit is open, then it will keep it open for specified sleepDuration (in application its 10 sec). In this duration, it will simply redirect to fallback logic and not try to invoke underlying API.
  6. 12:28:45 WARN c.h.n.s.NotificationServiceImpl - HYSTRIX FALLBACK METHOD IS INVOKED 12:28:48 WARN c.h.n.s.NotificationServiceImpl - HYSTRIX FALLBACK METHOD IS INVOKED

    The response remains the same:

    GET http://localhost:8081/1 { "currentTime": 1535352979843, "nextUpdateAvailableIn": "60 sec", "latestScore": { "matchDetails": "NOT FOUND", "source": "getMatchDetailsByIdFallback" }, "userId": "23e58c4a-7b4c-445d-9480-5b10fb2e6988" }
  7. Try accessing the URL after 10 seconds When we try accessing the API after 10 sec, since sleepDuration has expired, Hystrix will again attempt to connect to underlying service. However since cricket service is still down, it will fail and invoke the fallback logic and again trip open the circuit for next 10 sec.
  8. 12:31:26 DEBUG c.n.h.c.j.c.GenericCommand - execute command: getMatchDetailsById 12:31:26 INFO c.h.n.s.NotificationServiceImpl - Calling the underlying service 12:31:26 ERROR c.h.n.s.NotificationServiceImpl - No cricket service found for ID : cricket 12:31:26 DEBUG c.n.h.AbstractCommand - Error executing HystrixCommand.run(). Proceeding to fallback logic ... java.lang.RuntimeException: No cricket service found at com.hemant.notification.service.NotificationServiceImpl.getMatchDetailsById (NotificationServiceImpl.java:76) ~[classes/:na]

    The response still remains same as in Step 3.

  9. Now restart the cricket service and try accessing notify service API. Once the service is started and the sleep duration of open circuit expires, then Hystrix will again attempt to invoke underlying service and will succeed this time. Thus returning proper service response.
  10. GET http://localhost:8081/1 { "currentTime": 1535353459825, "nextUpdateAvailableIn": "60 sec", "latestScore": { "Team2": "PAK", "Team1": "IND", "Team1 Score": "189 For 7 IN 20 Overs", "Team2 Score": "120 For 7 IN 15 Overs", "source": "http://localhost:8080/1", "MatchName": "IND VS PAK" }, "userId": "e65d0222-2ff2-4f96-8ab1-1874951b9e0e" }

5. Conclusion:

Thus we have successfully added resiliency in our distributed system by implementing Hystrix circuit breaker.

If though underlying service is down/not accessible, Hystrix allows APIs to continue to perform by invoking the fallback logic. This ensures graceful degradation of microservices in case of failure. Also this avoids the dreaded cascading effect of failure.

Note that the code shared by Expert Java Application Development team is for reference reason only. If you have any doubt or need to ask anything related to the issue, make remarks below.

For further information, mail us at info@aegissofttech.com