One of the questions that has been coming up a lot lately as many people are building with Akka.Remote is this:
How big of a message can I send over the network?
I’ve been asked about this four or five times this week alone, so it’s time to put out a blog post and stop re-writing the answer. This is a great question to cover, as it starts to reveal more about what is going on under the hood with Akka.Remote and the networking layer.
Up until I worked on Akka.NET, I honestly hadn’t thought much about the networking layer and so this was a fun question for me to dig into and research.
When Does This Come up?
This comes up all the time when people have large chunks of data that they need to transmit and process. I’ve been asked about this lately in the context of doing big ETL jobs, running calculations on lab data, web scraping, for video processing, and more. People asking about sending files that range anywhere from 20MB to 5GB.
All of these contexts involve large amounts of data that need to be processed, but no clear way to link that up with the distributed processing capabilities that Akka.NET provides.
So what’s a dev to do?
Here’s the first answer people try: “network-shmetwork! JUST SEND IT!”
Why This Is A Bad Idea
This is basically what that does to the network:
That is a python trying to eat an entire pig. Ewwwww. Gross.
If it could feel, that’s how the network would feel about our large messages.
What else do people try to do? Here are some of the common approaches I’ve seen:
- crank up...