Case Study of Batch Processing With Spring Cloud Data Flow Server in Cloud Foundry - Bruce Thelen

Problem:
CoreLogic RiskMeter is a platform for insurance underwriters to assess natural hazard risk to insured properties. RiskMeter is going through a replatforming as part of CoreLogic’s cloud native transformation with help from Pivotal Labs. The replatformed Riskmeter is implemented in Spring Boot and deployed on Cloud Foundry as a cloud native 12 factor app. While executing this replatforming, the team discovered the need for recreating the old application’s batch processing features. In this new cloud native version of the application, there was a strong desire to be able to blue green deploy the application without disrupting running batches.

Solution:
In order to accomplish this, we chose to use Spring Cloud Data Flow Server with Spring Cloud Task to execute a Spring Batch Job. The batch job updates its status via a StepExecutionListener sending messages to a RabbitMQ message queue which is read via a websocket connection from the Angular frontend thus providing a real time progress bar.

Specific Items To Emphasize As Lessons Learned:

Decomposing the system to extract the short lived microservice that composes the batch run
Injecting configuration into Spring Cloud Data Flow Server
Injecting configuration into Spring Tasks
Techniques for blue/green deployment of Spring Tasks
Realtime batch status progress bar via websockets to Angular front end
Conclusion:
Previous CoreLogic projects which implemented homegrown batch processing relied heavily on complex infrastructure, deployment processes (i.e. allocating dedicated VM instances, configuring app servers), and long development cycles. In one case, it took approximately a year to implement a similar system. Using Cloud Foundry, Spring Cloud Data Flow Server, Spring Task, and Spring Batch, we were able to decompose our app and deploy the batch solution in about two months without having to write custom and complex batch management tooling.

Bruce Thelen, Principal Software Architect, CoreLogic
Filmed at SpringOne Platform 2017