Large systems require integration. Why? Because building large systems requires a huge team. A huge team operates efficiently by breaking a problem down into pieces that can be purchased or developed. At some point, those pieces must be brought back together.In project planning, I have seen projects show a roadmap of all the pieces coming together for an integration event. In the back of my mind, I hear “train wreck.” Have you ever seen a massive railyard that eventually converges to a single track? Can you image all the trains simultaneously fighting for time on that final track? Failure seems not only likely, but the only option for a complex, large system.
At Albers Aerospace, we take a different approach to integration. These techniques allow the “train” (of subsystems and components) progressively come together, piece by piece, verifying and validating interfaces so that you eventually have a complete (large, complex) system that hangs together. Our approach allows one subsystem to make progress, without having to wait for other subsystems to catch-up. Our approach moves beyond just message verification and continues to the protocols and handshakes necessary for operation of any meaningful capability.Experienced integrators will not see any of these approaches as novel. These are techniques taught to us by those who have done it before, or they were “discovered” through experience and necessity (necessity being the mother of invention, per Plato). These concepts do not stand-alone, but rather they leverage and extend one another. None are free - the size and breadth of your system will dictate which are appropriate.
Imagine you are a software engineer, and you are handed a 500-page interface specification. Where do you begin? How do you know if you’re doing the simplest or most common thing correctly, let alone designing in a way that your compiled result will comply with the subtleties of one of these huge specifications?As software minded systems engineers, we build interface simulators (composed of stimulations (exciters) and emulations) for our industry partners. These serve has reference implementations and “goal-lines” for our software teams to work against. They provide the feedback you need so you can know development and integration teams are making the correct progress.Message parsing is an important first step – when I get a message, can I read and interpret the bits? But interfaces are more than message sets. What handshakes exist? What rhythm of messages are expected? Who should initiate? Interface emulations allow integration to begin without the hardware or other software in the loop.Imagine your organization produces multi-million-dollar gimbals. They are incredibly capable, with lots of options, and thus a significant and daunting interface definition accompanies them. You want your gimbal on every military platform, but that requires lots of integration with many different platforms – no small task. You need to make integration with your product easy and inexpensive. Are you going to loan your system out to every software developer? Are those software shops going to sign your multi-million-dollar bailment agreement? Did I mention that it’s classified when powered on – I hope the software developers have a secure lab to work in.What if you had an emulation of your gimbal that was distributed as a setup.exe or RPM file? What if you had a cheap emulation of all the hardware elements with analog and discrete inputs and outputs, serial, RF, video outputs? Imagine how much shorter integration would be if you had this feature set in-hand during development. How much risk reduction could you claim with software working against the vendor provided emulation prior to the event?As an integrator, we know the sense of relief that comes when a supplier provides an emulation of their hardware: “Oh good, we don’t have to wait for the hardware to know if it’ll work.”At Albers Aerospace, we build interface emulation capabilities for our partners. We provide them to those who integrate with us. For customers with products, we build them to enable growth and sales. This capability, to imitate and stimulate an interface, is the basis of other capabilities, like stub/sub integration and model-based testing that we’ll discuss below.
Sub in this context means “substitution”, but that doesn’t come off the tongue like “stub / sub integration.” Stubs are throw-away capabilities that mimic aspects of a subsystem.When one of your subsystems is ready early (maybe because you could buy it off the shelf), but other systems are not, we encourage setting up a “Stub / Sub” environment in your System Integration Lab (SIL). together with “stubs” initially, eventually substituting in functional subsystems. These stubs respond to stimuli and produce stimuli, emulating the interface of the not-yet-ready-for-integration subsystems. When a subsystem is further along, it can be “substituted” where the stub was and the “as-is” system becomes more real.
At Albers Aerospace, our development team enjoys this kind of rapid development. We recognize that these “stub” work products are throw-away. Our team is accustomed to working in this “gear.” We know and love engineers with one “gear” – high quality, all the time, every time. We have that “gear.” But we also understand the value of having a “go fast, make it usually work, good enough” when in a rapid development mode, such as stubs and rapid prototypes. Our stub / sub approach allows a large development project to demonstrate a system capability quickly. This approach enables the system to mature as subsystems come online, even when they are partially capable. A partially complete subsystem might be “wrapped” in a stub, where the stub responds to certain messages and the subsystem responds to others.
Often, our teams are geographically dispersed. We don’t have the luxury of constantly working together in a lab. This physical separation, compounded by the security requirements of our industry partner spaces, creates significant challenges to overcome when creating subsystems that integrate cleanly and work together as a system.In our industry, security is both paramount and under resourced – which sacrifices availability. Security is considered overhead: a cost which must be reduced. Our most competent staff are not on overhead – they are designing the next operational capability for the warfighter. This makes for effective firewalls, both figuratively and from a networking standpoint.At Albers Aerospace, we value the full definition of information assurance: security and availability of our information. The latter enables systems to communicate, the former prevents it.In a recent project, we purchased some commercial hardware that our customer needed to integrate. It had a well-documented API, but there is nothing like the real thing. It was useful in a military system but was itself unsecured. So – we poked a hole in our firewall and exposed it on the internet by the next morning. If you are unscrewing yourself from the ceiling right now – you likely work for a large organization with an under-funded infrastructure team. Our customer, working from home (as their corporate firewall only allows TCP on limited ports), was able to integrate and fix normal issues when you bring two systems together for the first time. When they showed up to the test event, “it just works.” Integration was done early and remotely.“But you cheated!!” No, we applied systemic thought to security and availability. When more security is necessary, we apply it. When it isn’t…At Albers Aerospace, we setup secure (meeting all the rules of Department of Defense), collaboration environments for software source control, issue tracking, and document management, and we can usually set it up in the time it takes you provide your external IP address and verify your identity. Secure CI/CD pipelines can be established faster than your travel time to your integration facility. Secure Virtual Private Clouds deployed into AWS GovCloud, securely managed deployments of servers and virtual machines for everyone that needs it.
Imagine, you had the integration event – some stuff worked, others didn’t. When you’re back in your lab at home you think, “it’d be really nice to be able to receive that message again and see if it worked.”At Albers Aerospace, we record our integration events. What do we record? Everything that might be useful. Many of our applications have their own recording and playback capability – we just have to make sure we mash the red button to start it. But for others, we must perform the recording: video streams, entire screen captures, network level packet detail, etc.Wireshark and the PCAP family of products is one of our go-to applications for network recording. It has native recording capabilities. Using the recorded files, it can utilize its byte-level inspection tools and playback functionality. One issue you run into in playback is that it is time specific. Bit by bit playback will often result in time being out of sync with a system which is running now (in real time) versus two weeks ago when you recorded it. Recordings also don’t support a request/response situation well, nor do they have the ability to modify/update message sequence numbers.Albers Aerospace has experience in building “smart” playback capability. We often use the PCAP provided recording capability. In playback, we leverage the open-source PCAP library to read the recording, make a value-added improvement (like updating a timestamp, sequence number, etc.), and then rebroadcast. Recordings also make a great starting place for request/response caching for your “stub” or interface emulation to respond like the physical system in the lab.These recordings can serve as excellent initial source data for a test environment. We’ll grab a single message frame, wrap it in a test case, and ensure our message parsing works as expected.
Integration isn’t easy. “All-at-once” integration has been proved through experience as ineffective. By experience, we have learned that integration events can be successful when meaningful, intentional pre-conditions achieved. The techniques outlined in this article are not academic – they are battle-tested by our own experience in industry. If this experience would be valuable to your project or program, please don’t hesitate to contact us: engineering.solutions@albers.aero
©2024 Albers Aerospace, All Rights Reserved.