Vertech is an industrial automation and information solutions company based out of the United States that develops solutions for a broad range of industries.

This case study was authored by John Wolnisty, a Solutions Architect at Vertech who, in the context of this case study, used Akka.NET to help test, simulate, and build solutions for airport baggage handling systems.

What does Vertech do?

Based in Phoenix, Arizona, Vertech designs and implements industrial automation solutions including traditional automation systems, SCADA systems, MES solutions, and OT networks. One of our industry focuses is airport baggage handling systems. A large airport can have hundreds of conveyors that need to be integrated and controlled by the system. Airport security requirements add layers of complexity requiring control systems that account for every bag on a conveyor and track suspicious activities, such as a bag suddenly disappearing or a new bag appearing from the middle of a conveyor. At Vertech we provide comprehensive solutions that handle all these details by personally writing everything from automation code in PLCs to the high-level systems that interface with airline data feeds.

What problem were you trying to solve?

A PLC (Programmable Logic Controller) is an industrial computer repetitively running logic that reads input sensors, implements operational behaviors and controls outputs. Inputs are control panel switches, push buttons, photo-eyes and the like. Outputs are motors, lights, warning horns and other devices. The PLC programmers have a problem, especially with a large system, of being unable to develop and test their code without the physical hardware being present. We had a need for a system that could simulate the conveyor hardware, provide relatively seamless connections with the PLCs, display the output conditions, and log what was going on without having to set up hundreds of physical conveyors. It would also be useful to simulate random situations that the PLC code must handle, such as jams, removing a bag midstream, etc.

The Devices

The specific devices we needed to emulate:

1. Conveyor - Respond to start/stop commands, provide run indication, report belt speed, provide tracking pulse
2. Photo-eyes - Indicate PE is blocked or clear
3. Control Station - Control Station Respond to pushbuttons, illuminate indicator lights
4. Bag - Generate random pieces of baggage of varying sizes and orientation with respect to the belt
5. Scanner - Emulate a tunnel bar-code scanner (12-heads) that can read a bag tag from any angle unless under the bag
6. EDS - Emulate an Explosive Detection Scanner and generate random suspicious bags

A conveyor operates in its own micro-world. It knows about the conveyor immediately before it and after it. It knows about the photo-eyes it has. And it knows about the control stations that can command it. The conveyor has a few properties: the length of the belt, its speed, what type of photo-eyes it has.

That’s it. The challenge is, at any given time, multiple (or all) conveyors are running with 1 or more bags on them. When trying to emulate this, we ideally needed a way to have each emulated conveyor run in its own thread of code. So we had to come up with a generic rule set, a way to manage the bags on the conveyor, and a way to pass bags off to the next conveyor.

Developing code that involves potentially hundreds of running threads is not for the faint-hearted. Trying to debug that code can be trying at best.

Why did you choose Akka.NET?

While attending Microsoft Build in 2019, I picked up Reactive Applications with Akka.Net honestly not knowing a thing about it other than it was a way to make scalable asynchronous software. After diving into the book and spending time at the free Petabridge.com bootcamp website, a few prototype programs later, I was convinced Akka.Net would be the perfect solution for writing a simulation system.

• Scalable? Check.
• Multi-threaded, but you don’t have to be concerned about thread management? Check.
• Able to pass information easily between those threads? Check.

Anytime I use something new, I need to understand it well enough to know it will be the correct tool for the job. I spent time with the previously mentioned resources until convinced it would work for this application.

How does Vertech use Akka.Net?

The premise behind Akka.Net is the Actor. An Actor runs in a thread context, receives data via messages, can respond to the messages or forward them to another actor, and is scalable by running more actors. The conveyor simulation system runs off of models of portions of the conveyor system stored in a database. This database is populated by another tool we use for developing the PLC code, so there’s no knowledge duplication. Once smaller portions of a conveyor system have been tested, the models can be merged into larger models until eventually the entire airport is running as a simulation.

So, what are the actors? See below:

1. Conveyor Actor - Emulates a single conveyor. Handles external inputs / outputs. Emulates photo-eyes. Queues virtual “bags” on the conveyor at the belt speed.
2. BagStatUpdateActor - A global, single actor that runs on the UI thread that receives status messages from other actors and updates relevant data on the display window.
3. ShaftActor - Conveyors that use precision tracking require pulses from a shaft encoder to determine how far a conveyor belt has moved in a given amount of time. This actor is only created for those belts that require it.
4. DiverterActor - Similar to a conveyor, but coordinates sending a bag down one conveyor or another at a junction.
5. SecurityDoorActor - Emulates a security / fire door over a conveyor
6. ATRScannerActor - Emulates a 12-head tag reading bar-code scanner array. Returns the tag contents to the PLC and which heads successfully read the tag.
7. EDSScannerActor - Emulates explosive detection equipment.
8. PLCActor - Actor that writes data to the PLC. Uses Dependency Injection to isolate specific PLC brand protocols. App.config hocon section is used to create multiple actors as load requires.
9. PLCCommandActor - Actor that reads commands from the PLC.

The Conveyor Actor

Using the Conveyor Actor as an example, the bulk of the work of the simulator is done by the Conveyor Actor. This Actor has two states: Stopped and Running. In each state there are multiple message types that can be handled.

Before the Conveyor can start accepting messages, the PreStart routine sets up the runtime:

protected override void PreStart() {
  base.PreStart();
  BagsOnBelt = new Queue<BeltBag>();
  conLogger.Tell(new ConsoleMessage(System.Drawing.Color.Gray,
   $"Conveyor address is {Self.Path} and length is {conv.Configuration.Length}." ));
  // Set Photoeye initial state (Clear - no bags)
  if (conv.HES != null) 
    plc.Tell(new PLCOp(PLCOPERATION.WRITE, conv.HES.PLCTag, true));
  if (conv.TES != null)
    plc.Tell(new PLCOp(PLCOPERATION.WRITE, conv.TES.PLCTag, true));
  if (conv.OHS != null)
    plc.Tell(new PLCOp(PLCOPERATION.WRITE, conv.OHS.PLCTag, true));
  Context.System.Scheduler.ScheduleTellRepeatedly(TimeSpan.FromSeconds(2),
     tickInterval, Self, ConveyorCommand.TICK_CMD, ActorRefs.NoSender);
}

In the Stopped state, the following messages are handled:

private void Stopped() {
    Receive<ConveyorCommand>(cmd => cmd.Command == CONVEYORCOMMAND.STATEREQ,
                  cmd => Sender.Tell(GetStatus()));
    Receive<Scanner>(cmd => CreateScanner(cmd));
    Receive<ConveyorCommand>(cmd => cmd.Command == CONVEYORCOMMAND.START,
                  cmd => StartConveyor());
    Receive<ConveyorCommand>(cmd => cmd.Command == CONVEYORCOMMAND.UNJAM,
                  cmd => UnJamAll());
    Receive<ConveyorCommand>(cmd => cmd.Command == CONVEYORCOMMAND.CLEARBAGS,
                  cmd => ClearBags());
    Receive<Luggage>(bag => bag.Laps == -1, bag => AddStaticBag(bag)); // only accepts initial inductions
    Receive<PhotoEyeCommand>(cmd => ForcePhotoEye(cmd));
}

While in the stopped state, a Conveyor can:

1. Return its current status.
2. Start running.
3. Clear a jam, etc.

The StartConveyor Command updates the UI, starts child actors and finally changes state:

private void StartConveyor() {
    state = CONVEYORSTATE.RUNNING;
    conLogger.Tell(new ConsoleMessage(System.Drawing.Color.Green,
             $"Conveyor {conv.Name} has started."));
    conLogger.Tell(new BeltGUIUpdate(conv.Name, true, false, false));
    AckCommand(CONVEYORCOMMAND.START);
    if (encoderActor != null)
        encoderActor.Tell(ConveyorCommand.START_CMD);
    Become(Running);
}

In the running state, the accepted messages are:

private void Running() {
    Receive<ConveyorCommand>(cmd => cmd.Command == CONVEYORCOMMAND.STOP,
                  cmd => StopConveyor());
    Receive<ConveyorCommand>(cmd => cmd.Command == CONVEYORCOMMAND.TICK, 
                  cmd => Tock());
    Receive<ConveyorCommand>(cmd => cmd.Command == CONVEYORCOMMAND.CAUSEJAM, 
                  cmd => JamABag());
    Receive<ConveyorCommand>(cmd => cmd.Command == CONVEYORCOMMAND.JAMATHES, 
                  cmd => JamABag(true));
    Receive<ConveyorCommand>(cmd => cmd.Command == CONVEYORCOMMAND.STATEREQ, 
                  cmd => Sender.Tell(GetStatus()));
    Receive<Luggage>(bag => AddBag(bag));
    Receive<BagCommand>(cmd => cmd.Command == BAGCOMMAND.REMOVEBYINDEX, 
               cmd => RemoveFromQueue(cmd.Identity));
    Receive<BagCommand>(cmd => cmd.Command == BAGCOMMAND.REMOVEBYIATA, 
                   cmd => RemoveFromQueue(cmd.IATA));
    Receive<BagCommand>(cmd => cmd.Command == BAGCOMMAND.CONVEYORACKBYINDEX, 
                 cmd => AckBag(cmd.Identity));
    Receive<BagCommand>(cmd => cmd.Command == BAGCOMMAND.GROWBAG, 
                cmd => GrowBag(cmd.IATA));
    Receive<BagCommand>(cmd => cmd.Command == BAGCOMMAND.HANGBAG, 
                cmd => HangBag(cmd.IATA, true));
    Receive<BagCommand>(cmd => cmd.Command == BAGCOMMAND.UNHANGBAG, 
                cmd => HangBag(cmd.IATA, false));
    Receive<PhotoEyeCommand>(cmd => ForcePhotoEye(cmd));
}

While in the Running state a conveyor can:

1. Stop Running.
2. Process an internal clock TICK
3. Cause a Jam, etc.

If a conveyor is running, TICK messages are processed. Each tick interval, each BeltBag object in the FIFO queue, has the following operations done:

1. The bag is advanced along the belt a distance that is calculated from the tick interval and the belt speed.
2. If the bag is blocking any photo-eye, a logic 0 is written to the appropriate PLC tag.
3. If there is a JAM or a HANG, the bag can’t advance but is “jostled” in-place by forcing the bag orientation angle to turn more towards perpendicular depending on how many bags are “hitting” it from behind.
4. If the bag is at the belt’s end, the next conveyor actor is asked if it can accept a new bag. If it is running and not jammed, the bag is passed on to the new conveyor and removed from the old conveyor’s queue.
5. When a conveyor accepts a bag and creates a new BagBelt object, a new orientation angle is created and the bag “turns”.

If a bag passes through a scanner, the Conveyor actor tells its scanner child actor, passing the Luggage object.

if (scanners != null) {
    foreach(ScannerHandler sh in scanners) {
        // if centerpoint of bag cuts through plane of scanner this move
        if (oldPosition < sh.offset && newPosition >= sh.offset)                            
            sh.actor.Tell(bb.Bag);
    }
}

SCADA Programmers

Industrial applications often make use of SCADA packages such as Ignition from Inductive Automation, to implement a UI for operators. The simulator has a side benefit of allowing the changing process values written to the PLC be read by our SCADA package. This gives our SCADA developers realistic data to verify it was being read and displayed correctly long before going on-site.

The User Interface

The user interface is designed to allow the PLC programmer flexibility in running the simulator to provide various scenarios in testing their code.

Starting at the upper left of the display, let’s look at the features of the UI (below).

A simulation model is selected from a drop-down list. This list is driven from the database and entries are created by another tool used for PLC code creation. Once a system is selected, it can’t be changed without restarting the program. The Tick interval sets the Conveyor Actor’s TICK command interval. The Speed Factor effectively multiplies the travel speed of the conveyor. This is especially useful when very long conveyors are involved.

The grid on the left side lists all the bags on all conveyors (just one in this example). You can see the bag tag, internal ID, time inducted, the current conveyor it is on, and the position of the leading edge of the bag relative to the conveyor. The right-hand list box logs all the messages from actors. The lines are color coded so types of messages can be quickly located while running. The far-right side lists all the conveyors and their states. Black = not running, grey = running. Green / red lines show the photo-eye state (red = blocked).

The bottom section controls bags on the conveyors. In a normal simulation, the starting conveyor is selected from the drop-down (IB8_01 in this case). The “Generate” button is clicked, which fills in the IATA and ID text boxes with random values. Clicking “Induct” sends a new bag message to the IB8_01 actor and, if it’s running, will start moving the bag down the belt. Other buttons and checkboxes are used to cause jams and other problems on a belt.

Selecting a Control Station from the drop-down and clicking “Show” brings up a representation of the control panel associated with a conveyor. The graphic is created on the fly based on the types of buttons and indicators that control station has. The example that follows controls the conveyors behind a ticket counter along with the security door:

Using Akka.Net has been a game-changer for writing this type of code. I could focus on the functionality of the actors implementing my simulation and not spend any time on the various plumbing, multi-threading, message passing, or other infrastructure that really is secondary to the problem I was trying to solve. To me a real proof of its value was when changes and enhancements needed to be made. It was very easy to add functionality to the actors without fear of breaking current code.

Another good example is the Shaft Encoder. This needs to simulate a rotating shaft turning at conveyor speeds. The entire code (minus the constructor) is as follows:

private void Stopped() {
    Receive<ConveyorCommand>(cmd => cmd.Command == CONVEYORCOMMAND.START,
        cmd => StartEncoderPulse());
}

private void Running() {
    Receive<ConveyorCommand>(cmd => cmd.Command == CONVEYORCOMMAND.STOP,
        cmd => StopEncoderPulse());
    Receive<ConveyorCommand>(cmd => cmd.Command == CONVEYORCOMMAND.TICK,
        cmd => Tock());
}

private void Tock() {
    pulse = !pulse;
    plc.Pulse(pulse);
}

private void StartEncoderPulse() {
    cancelKey = new Cancelable(Context.System.Scheduler);
    Context.System.Scheduler.ScheduleTellRepeatedly(TimeSpan.FromSeconds(2),
            tickInterval, Self, ConveyorCommand.TICK_CMD,
            ActorRefs.NoSender, cancelKey);
    Become(Running);
}

private void StopEncoderPulse() {
    cancelKey.Cancel();
    Become(Stopped);
}

As you can see, the code is simple, clean and is concerned with the generating of the pulses, not the infrastructure. I was also impressed with the performance. Even with a pulse rate of every 50 milliseconds, performance was good. Plotting the tag using PLC coding tools showed that the pulse was being written at that interval.

Akka.Net is an excellent match with industrial applications. I have already implemented and deployed system that uses Akka.Net at its core collecting real-time data values from a set of PLCs and storing them in a database. It is a great addition to any industrial programmer’s utility belt.

Observe and Monitor Your Akka.NET Applications with Phobos

Did you know that Phobos can automatically instrument your Akka.NET applications with OpenTelemetry?

Click here to learn more.