Large-Scale, Real-time Complex Event Processing with Akka.NET
Our last customer case study “Akka.NET Goes to Wall Street” remains as one of the most popular articles on Petabridge, and today I’m pleased to share with you a new case study written by Kim-Lisa Gad and the DigiOutsource team from sunny Cape Town, South Africa.
The DigiOutsource team, lead by Jean-Pierré Vermeulen, developed an extremely high-speed complex event processing system real-time customer analysis system on top of Akka.NET from conception to production within 5 months, and that system has gone on to increase overall revenue by 35%.
What follows is their story!
DigiOutsource, a leading online marketing company situated in the heart of sunny Cape Town, South Africa, required a new way to analyse their customer data real time in order to better understand their customer’s experience on each website, and to further provide their customers with offers and benefits tailored specifically to the individual in real time as well. With this goal in mind, lead Enterprise Architect Jean-Pierré Vermeulen was tasked with not only designing a real time complex event processing system but also finding the best technologies to utilize to achieve this.
Due to the extremely high number of events processed, averaging 10,000 events per second from multiple websites across multiple platforms (web and mobile), a solution able to cater for high volume processing was required that could also be scalable as business grew, and adapted to meet customer requirements as they changed with the times too. The system designed needed to allow for a high throughput of data and be self-healing in order to guarantee an almost 100% up time. The business would be fully reliant on the functionality contained within the solution to continuously deliver accurate information to the customer and, in so doing, ensure customer retention and that they returned to the websites and increased their use thereof.
With many years’ experience in software development, Jean-Pierré was already familiar with the actor model system and how it could be been used to build powerful, resilient, concurrent message driven applications which is exactly what he was looking for in a solution. The business, however, had been driven on object oriented programming so his first decision hinged on deciding which route to go in order to design the solution. The actor model proved to be the better solution for communicating the concepts he had in mind as the actors in the model are so life like, which allowed him to explain his thought processing to both developers and business units alike. The system also provided an easy way to gather requirements across the business and the understanding and explanation of these requirements was simplified through the drawing of UML Use Case diagrams.
Jean-Pierré Vermeulen, Lead Enterprise Architect, DigiOutsource
“The Akka.Net actor model system provides an easy and understandable way to communicate ideas and concepts to developers and business units alike because the actors in the model are so life like.” Jean-Pierré Vermeulen
Having made the decision to use an actor model based system in order to bed down and define what the business requirements were, and also simplify the design and development of a solution, the next problem Jean-Pierré encountered was to find the right actor model technology to utilize. Having attended a Scala course previously, he was already familiar with Akka as a framework and how the Java based system had been successfully utilized in the development of high volume message oriented products such as Twitter. DigiOutsource, however, is a predominantly .Net environment and to change direction in the type of technology used for a solution that would impact a large portion of the business seemed an impossible and expensive task.
With further investigation, Jean-Pierré found resolution and peace of mind in the Akka.Net technology available from Petabridge.
Akka.Net gave Jean-Pierré the confidence that he was providing his development team with a technology that was understandable, easy for them to adapt to, and would allow them the opportunity to focus on building a solution rather than worrying about complexities such as parallel processing, concurrent execution, supervision strategies, clustering and sharding as these were already catered for within the Akka.Net product. Aaron and the team at Petabridge were also extremely positive about how well Akka.Net suited the environment at DigiOutsource and how it could cater for all the businesses needs too. Jean-Pierré found engaging with Petabridge to be a refreshing and rewarding experience as they were always working at ways to strengthen the relationship.
With requirements now clearly defined, a viable development technology on the table, and valuable input from the team at Petabridge, Jean-Pierré and his team began to build their complex event processing engine. Other technologies were incorporated into the solution such as RabbitMQ to transport events as messages from the multiple servers processing all the customer activity, and MongoDB to store the results from the complex calculations being done on the data processed behind the scenes. Kibana was further used to monitor the events received and the time taken to process them, and Managed Engine Op Manager was implemented to ensure all resources and applications stayed up and running all the time and provide alerts should any resource issues or downtime be encountered. The final stage of the project involved using socket servers to deliver push notifications to customer devices in real time.
Within a record short time of only 5 months from the inception of the project, a new super solution was rolled out across all DigiOutsources client sites worldwide. This has seen a 35% increase in revenue from our customers since then and guaranteed continued return of both active and formerly lapsed customers to our websites due to the improved customer experience. The system has also remained up and running consistently with no downtime encountered, even with the increased customer activity and higher volumes of data being processed.
With such phenomenal results recorded, it has made sense to the technology teams at DigiOutsource to further skill up their developers in Akka.Net and to look at utilizing this technology across other areas of the business as well. Currently, a project is in place to use Akka.Net in conjunction with Service Fabric as the hosting platform for other actor based systems and our next Akka.Net project is set to be deployed and go live with these two technologies in November of 2016.
You can learn more about DigiOutsource at their website: http://digioutsource.com/share it with your followers or follow us on Twitter!