As a controls engineer who has implemented and assisted with dozens of robotic manufacturing cells, the problem isn't the motion planning or compute, it's legacy product support and ease of onboarding new hires.
Any given cell may be in production for 10 years - some only for 5, but some for 20 or more - and you need <24 hour response times to any mechanical issues that may arise over that duration. Also, one can't expect that the original programmer will be available to diagnose and debug issues that arise in the field after a year or two, so you need a large pool of technicians and maintenance personnel that can understand the software and be productive in the first hour after being introduced to the machine. None of this is valued by academics, instead, the fundamentals of the process of a single lab trying to prove merit as PhD students and, once that credential is acquired, move on to greener pastures is anathema to both of these ideas.
The problem is that academia is working in ROS (now ROS 2), writing Python and VHDL, while the real work in the industry is being done in proprietary vendor tooling that's backwards compatible with training from the 80s. They're proudly advertising touchscreens and full-color 5.7" displays on their teach pendants, as if their customers were still 20 years in the past.
Yes, academia, I do want sub-millimeter repeatability, and flawlessly smooth motion planning. I want collision detection that accommodates both the inertia of the arm itself and any end-of-arm tooling, including cable harnesses with distributed loads. I want automatic singularity avoidance and trivial point and frame manipulation. I want 24-bit encoders for resolution, and I want angular velocities to the moon, and to make those work together I need servo loop rates and high-speed skip responses at 1, 2, or 10 kHz. I want 4D stop position prediction to avoid intersection of the robot arm with complex assemblies and safety zones imported from CAD. I want all these things cheaper and faster, with more storage.
But I need these to be accessible by Billy Joe, whose only credentials that got him the job in the maintenance department is that he helped out his daddy working with a welder and an old clapped-out Bridgeport on the farm growing up. Billy is probably my best tech on 3rd shift, and he's never learned to type on a full-size keyboard, just his smartphone... and on a robot teach pendant.
Well said. Robotics is indeed the art of system integration and such remains (and will) one the biggest tech hurdles.
We wrote about this, and raised-funding, and worked for years on it. Hell, we even created an extension of ROS for it called the "Hardware Robot Operating System" (H-ROS) that aimed to address many of these issues you describe using ROS as the common language and Programmable Logic (FPGAs) to deal with physical interfaces. More on this at https://ieeexplore.ieee.org/document/8046383.
Problem was that the market didn't really accept it. I still believe on the tech problem landscape in here. I'm just not so sure anymore if there's any real market/business for such a solution. After all, vendor lock-ins are great business, and generate business.
After reading this, I'm persuaded this space would benefit greatly from having a few dominant open-source software frameworks and a few dominant open-source hardware standards, making product support and new-hire onboarding much easier for vendors of all sizes. Alas, I recognize that no established vendor in the space would ever want to see the emergence of industry standards. :-P
It definitely would. RoboDK is one product that is moving in the right direction.
In the PLC space, Beckhoff controllers using commodity x86/64 hardware and standardized IEC 61131 programming languages are gobbling up Rockwell and Siemens' market share.
Given they're perpetually about 20 years behind the times, perhaps this year someone at Fanuc, ABB, Kuka, and Yaskawa will catch up to the year 2002 and read Spolsky's admonition [1] that says:
> Smart companies try to commoditize their products’ complements.
Stop trying to turn every division of your company into its own product center. I loathe paying $800/year to Rockwell for the privilege of reading "update your firmware" (and subsequently bricking my controller as an unwilling beta tester) on TechConnect. They're a PLC vendor, they should focus on selling PLC hardware, not making their customers hate them by nickel-and-diming them for software licenses and support contracts. They were down for FOUR DAYS this weekend for planned maintenance. FOUR DAYS!
Likewise, Fanuc should focus on selling servos and castings - robotic manipulator hardware - and stop trying to get the training division to turn a profit. If, instead of a 40 hour course costing $2,000, it was free, there would be more people able to use their $40,000 robot arms! If Roboguide software was free instead of $2,650, clients could more efficiently build more robot cells, and sell more yellow paint! Gah!
A very brief search turned up assorted overviews[1][2][3], and a github topic[4]. The PLCopen standard[5] is paywalled - only $400 single user, with low low per-seat multipliers! :/
What this field really needs is a massive push. The reason it always fail to deliver is that the incentive system is screwed. We need something like a Manhattan project capital allocation to a few very talented people to make progress. Building endless frameworks and trying to get thousands of people to cooperate on high level decision making dooms the whole endeavor. The risk for product development is too high for private industry to take and academia can't deliver products.
What I describe is what ROS supporters want it to be. As I wrote, no established vendor in the space wants to see the emergence of commodity industry standards. It seems that for the foreseeable future, we're stuck with the status quo.
I may be misunderstanding your comment, but as I understood it you are talking about industry failures academia can't do anything about. If industry is massively behind on outdated, closed environments and academia work's with different, open tools then that's not academias fault. Like:
> it's legacy product support and ease of onboarding new hires.
I don't see how academia can help with this, this so very much an industry problem.
> But I need these to be accessible by Billy Joe, whose only credentials that got him the job in the maintenance department is that he helped out his daddy working with a welder and an old clapped-out Bridgeport on the farm growing up.
While accessible interfaces and forms of human-computer interactions are researched by academia, they are not building products. Research has to be relevant, but it's still research. Those problems, they might be better addressed by a startup than academia, if that's even realistic. Academia can't solve a problem if it's not a research-problem.
Exactly. Advancements and capabilities in university press releases like this don't result in real-world improvements because the two fields are divergent.
you are leaving out a whole host of use cases bro. maybe in the past robotics was just for industrial use cases. but there are orders of magnitude other use cases that a ROS can help solve. simple things around the house , in the neighborhood, and other mechatronic integrations that would not properly called a robot. I would say the ones you are addressing with billy bob on the 3rd shift are like 1 percent of the cases. There are plenty of things you can actuate in the world that dont require sub mm repeatability. like what about tending my hydroponic vertical garden? theres nothing about the ubiquitous robotic future of humankind that needs any of the things you mentioned. thats more of an improvement on current use cases.
What are your thoughts on OPCUA? That's the direction they're trying to get to having different vendors be able to talk to each other as it's all pretty much federated.
Did you know it takes almost a hundred ms for a signal to go from your fingers to your brain? Did you know that the brain is constantly predicting the future to make up for its slow reaction time? Did you know even small computers are really really fast? If you did then you'd probably guess that initiatives like this are a waste of time and money. The difficulty is in the software, not the hardware, or the compute layer. With good algorithms our current crop of robots are already fully capable of producing trillions more in economic value than they do today.
We do not need hardware acceleration in robotics. Can we use it if it exists? Sure, but do we care about "democratizing" it? No.
There is a place for hardware acceleration in robotics that is making a huge difference and that's in simulation. GPUs, the once and future king, make training robust neural networks possible and fast. If you're an investor reading this who needs a reminder, bet on software.
Did you know that for predicting what you'll see 100ms into the future you need a level of real-time 3D recognition and world knowledge that we haven't ever solved acceptably, not even offline and parts of it in isolation?
While I agree with you that in 50 years, advances in AI will make fast hardware obsolete, it may well be that in 5 years, fast hardware will be able to avoid problems that we cannot yet solve in software. That would mean this company has 45 years to cash in before they get replaced by software.
The human brain does what it does with roughly 20 watts, if we take the popular figure of the brain taking 1/5th the power of your whole body and the average human body burning 100 watts.
How much neural net can you run on silicon with 20 watts?
Sure it's not fair, but there's no handicap in this game. Silicon isn't given any leg up for being millions of years late. Raw results are what really matters, not results weighted by whatever criteria would render the comparison subjectively fair.
I agree that one of the major aspects to improve in robotics is software. There's still a lot do there and many are working towards it. Most leading initiatives around ROS 2. I also believe that we need faster robotics simulation, and hardware acceleration will be valuable there, not only with GPUs, but also other accelerators (e.g. FPGAs outperform GPUs in many benchmarks both performance and energy consumption-wise).
I strongly disagree with the rest and your comments indicate a clear lack of (hardware) understanding. I often encounter this in the AI world. Where folks building ANNs just "forget" that this computational abstraction only "grew in popularity" (there were previous accelerators though) when CNNs got implemented as an accelerator in a GPU empowering newer results. The same's likely to happen in robotics.
Robots are real-time systems. Meeting time deadlines in their computations is the most important feature. (robot) Behaviors often take the form of computational graphs, with data flowing between compute abstractions (Nodes), across physical networks (communication buses) and while mapping to underlying sensors, compute technologies and actuators. ROS enables you to build these computational graphs and create robot behaviors by providing libraries, a communication infrastructure, drivers and tools to put it all together.
From a more technical compute architecture perspective, ROS 2 presents an event-driven programming interface. The resulting computational graphs built with it are "event-driven software architectures". Mapping these event-driven software architectures to hardware using CPUs leveraging classic control flow architectures (von Neumann architectures) leads to various issues. A key challenge in applying classic event-driven programming is that CPUs hardly provide real-time and safety guarantees. The de facto strategy in industry to meet the timing deadlines is a laborious, empirical, and case-by-case tuning of the system. This “whack-a-mole” approach is being realized by some, but unsustainable and hard to scale due to the lack of a hardware-supported timing-safe even driven programming interface. This is where accelerators come in. As already adopted in other industries including aerospace, automotive and healthcare, through the creation of custom compute building blocks (accelerators), a hardware/software co-design strategy provides clean behavioral specifications, describing clearly its guarantees in terms of timing (in other words, avoiding memory-centric von Neumann architectures).
Hardware is important. Note the current scarcity of semiconductors, which is one of the drivers of this research. Note also that creating custom accelerators for robotics application not only can lead to performance improvements (i.e. software that runs faster!) but also to more deterministic responses (which affects the downstream pipeline of software!). Rephrasing Alan Kay's quote, if you're serious about robotics, you should care about hardware ;).
Yeah, started my own robotics company, early engineer at another that sold for >$200M. You seem to have misinterpreted my statement of "hardware is already good enough" to mean "hardware is unimportant."
While hardware accelerators in robotics are very cool, and needed.
The fact that the entire workflow is ROS2 centric is a bummer.
FPGA and Accelerator Programming is already a huge PITA, and despite it's success ROS is really one of the worst frameworks I've ever used in terms of DX.
It makes an already hard problem, needlessly harder.
I wouldn't be surprised if they developed their own FPGA programming environment based on C++ and CMake, and all kinds of other custom language and tool shenanigans.
You have to remember that in a professional setting, the alternatives to ROS are atrocious pieces of crap like LabVIEW.
From there, ROS is a step up! Both are programming environments designed by mechanical engineers, who care more about making easy problems really easy than making hard cases any better (after all, to a mechanical engineer, hard programming problems are really hard on their own).
Regarding who's going to create the accelerators, enablers like this allow any ROSser to jump into creating their own accelerator from C++ by purely adding some metadata into their CMakeLists.txt files (which every ROS package has already).
On top of that, organizations are creating professional-grade kernels meant for production. NVIDIA and AMD are investing in that direction and so are we at Acceleration Robotics. Here're a few of the first ones announced:
- Accelerated perception stack (ROS 2 API-compatible) https://accelerationrobotics.com/robotcore-perception.php
- Accelerated coordinate system transformation (ROS 2 API-compatible, tf2) https://accelerationrobotics.com/robotcore-transform.php
That's exactly what this work does while remaining ROS 2-centric (with C++, CMake extensions and other friends). Goal in this research was to provide a way for the path of least friction for roboticists.
There's a huge gap between robotics development (more sw-centric today) and the average hardware engineer in big semiconductor companies. Totally different perspectives and viewpoints. We felt bridging the gap was necessary.
Indeed, ROS set out to create an ecosystem of reusable robot software infrastructure, but a monoculture has grown instead.
I started building a ROS alternative called Mech (see my profile) after becoming frustrated with ROS. It’s getting there but not ready for prime time yet.
To answer the sibling comment, my biggest problem with ROS is how inscrutable it is for undergrads. I’ve witnessed students struggle to even install it while following directions step by step. It is a huge mess of incidental complexity, layering the complexity of C++ build systems on top of an IPC system on top of an expansive middleware.
The learning curve is so high that when I spend a semester teaching robotics, students still struggle with ROS basics at midterms. The course turns into more about learning ROS rather than learning actual robotics.
I’m hoping one day Mech will work along side and in conjunction with ROS to provide an easier way of doing things for those who want to get right to the heart of robotics.
Why not provide a pre-set up machine?
I would also not have them install eg Matlab or LabView themselves at the start of classes.
But yes, I recognize the struggle. At my RoboCup team (TechUnited Eindhoven), the learning curve is also steep: from non-CS students that use Matlab on Windows, to using Linux with C++ & Python, ROS, git, not to mention our own stuff on top.
They all have their own laptops and prefer to use them. We also don’t have enough money to provide a computer for everyone. But I mean the setup is just one part of the issue, as you note. What I’ve found is I’ve had way more success teaching robotics to younger students via matlab/windows than Linux/ros/c++.
I'm a huge fan of your work.
Is there a reason why you didn't pursue the triple model of EVE and went with tables instead?
It feels like the former would be such a great fit for the knowledge representation side of robotics.
Hey, thanks! That's one of the main and crucial differences between Mech and Eve, actually. The main reason I went with tables in Mech is because of the robotics domain. Robotics has a lot of linear algebra and much of the math can be parallelized naturally. This means in Mech we can store columns as contiguous arrays, and take advantage of SIMD optimizations. The triple store in Eve worked well with the whole relational algebra nature of that language, so it was a natural fit there. Not so much for my use case.
Maybe there's room for an Eve-Mech bridge where Eve can be used on the knowledge representation side, and Mech for the lower-level motion planning stuff. That's a fun thought!
Really interesting point.
I always figured I'd go with an ECS system like specs for the high perf case, but you're absolutely right, columnar storage is gonna be orders of magnitude faster.
(Fun fact, the hibitset of specs looks _a lot_ like a worst case optimal join.)
A GPU/FPGA accelerated Mech would be really really cool :D.
A hybrid system is a really interesting idea. I actually wrote my bachelors thesis on a system heavily inspired by EVE to be used for knowledge representation in robotics and have been experimenting a lot with implementations of the idea over the years.
You should really open a Mech discord channel, I'd love to hang around and help build something like that :D.
Do you have a link to your thesis? I'd really like to read it!
GPU/FPGA Mech has been my dream since like... forever. I started taking a crack at it here: https://gitlab.com/mech-lang/core/-/blob/gpu/src/bin/gpu.rs, but mostly I was just figuring out wgpu's capabilities in Rust. But that was months ago and I've since decided to push it off until after I launch what I have done already.
I have a private slack that I've been using to work with my students, I'd love to send you an invite. Send me an email at corey@mech-lang.org and I'll send it to you. I've been putting off doing a discord channel because I wanted to code up a chat app in Mech instead :P I think I'm almost there but I've been thinking that for years >_<. If I don't have one by the time I launch in October I'll probably bite the bullet and do discord then.
Bruh, take a step back for a second. ROS is a messaging system. Its like Microservices for Robotic components. If students are having trouble with ROS you might take a different angle on the purpose. When you teach ROS you are learning middleware patterns of integration first. Secondly you have to teach fundamental geometric concepts like TF. So thats why your students are having problems you are giving them too broad or too narrow or some convoluted context to reason about what they are doing. Try flipping the script.
If only ROS were that. ROS wasn't a particularly good Middleware, and ROS2 isn't a middleware at all (DDS is a middleware, ROS is a framework using DDS).
The issue with ROS isn't the general idea of communicating nodes over a blackboard and the accompanying standard libraries, but the huge pile of low quality tools, sparse documentation, ad-hoc architecture, bad code, and mediocre introspection tooling.
A lot of that can be traced back to C++ and CMake, but ROS manages to take already complex tools and make them even more complex in an attempt to make them easy with automagic.
If ROS2 had just opted for ProtoBuffers/FlatBuffers/CaptnProto over MQTT/NATS and build TF et.al as a library over that, we could just point students to the excellent documentation of the Serialisation Format, of the Middleware and to the (with all that freed developer time) much better documentation of the Robotics libraries, and call it a day.
Instead they spend months with the idiosyncrasies of the build system, only to continue struggling with the documentation, and the sub-par tooling when something goes wrong. Cough only two seconds of time travel in ROSBags cough
Use ROS 2. The misconceptions about ROS have been mostly addressed with ROS 2.
Check out industry adoption of ROS 2 in various industries including automotive, healthcare and warehouse automation. ROS 2 is (and will remain as) the de facto standard for robot application development.
The issue with ROS are mostly cultural, and heavily ingrained into the community. ROS2 isn't going to fix that and has been shown to repeat the same mistakes of ROS.
The atrocious build system has stayed the same. People continue to add even more brittle automagic to the pile. (.e.g. automagic compilation of C++ to FPGAs)
The grown semi ad-hoc messaging semantics for services has stayed the same. Even if DDS was an acceptable foundation, adding a ROS message translation layer and ROS semantics on top of a perfectly valid schema language is bringing the same pitfalls that Object Relational Mappers have, in that you now have to understand three systems, the source, the compiler, and the target.
And given that ROS1 had a 'wontfix' issue on the message schema checksums being computed wrongly (ignoring the cardinality of fields), which could result in your robot literally ripping your head of when it received messages from two different library versions, I don't trust anybody from the community to get it right this time.
The whole DDS vendor based ecosystem is antithetical to open source. The DDS standard is way too big, and a bunch of vendors meeting every couple years to demonstrate that their systems can send messages about shapes between one another simply doesn't cut it to prove correctness.
ROS(2) is an attempt to make C++ more easy by layering automagic and complexity on top. Most robot shops I know waste most of their time fighting that magic and complexity. The other ones don't use ROS(2).
ROS should have been:
* A bunch of libraries (not frameworks) to solve robotics, vision, and knowledge representation related tasks.
* A bunch of best practices to work with common middlewares, none of which are developed as part of ROS.
* A repository of competing data schemata for the above libraries and middlewares to facilitate interoperability but allow for experimentation.
* Dataviz libraries for common notebooks, e.g. jupyter, observablehq, that work with the common data schemata.
Instead we got one of the hairiest C++ projects and a bunch of shell scripts to open your default text editor more conveniently. Yay.
Looks impressive to a layman like myself. I'm guessing it addresses the global shortage of single-board computers like Raspberry Pis right? Curious to know how long it'll take to do that as well.
Well, yes but not only. There's only so much you can do with Pis and other SBCs that offer a CPU-centric compute solution. Robots are real-time systems and CPUs don't excel at that.
When optimizing dataflows for lower latencies and higher throughputs, you typically seek specialized compute architectures and that's wherein GPUs and FPGAs come in. That's what this work really enables.
comparison using Jetson TX2 or Xavier could have provided better insight instead of Nano 2G which struggles even in basic task but otherwise looks quite promising
Any given cell may be in production for 10 years - some only for 5, but some for 20 or more - and you need <24 hour response times to any mechanical issues that may arise over that duration. Also, one can't expect that the original programmer will be available to diagnose and debug issues that arise in the field after a year or two, so you need a large pool of technicians and maintenance personnel that can understand the software and be productive in the first hour after being introduced to the machine. None of this is valued by academics, instead, the fundamentals of the process of a single lab trying to prove merit as PhD students and, once that credential is acquired, move on to greener pastures is anathema to both of these ideas.
The problem is that academia is working in ROS (now ROS 2), writing Python and VHDL, while the real work in the industry is being done in proprietary vendor tooling that's backwards compatible with training from the 80s. They're proudly advertising touchscreens and full-color 5.7" displays on their teach pendants, as if their customers were still 20 years in the past.
Yes, academia, I do want sub-millimeter repeatability, and flawlessly smooth motion planning. I want collision detection that accommodates both the inertia of the arm itself and any end-of-arm tooling, including cable harnesses with distributed loads. I want automatic singularity avoidance and trivial point and frame manipulation. I want 24-bit encoders for resolution, and I want angular velocities to the moon, and to make those work together I need servo loop rates and high-speed skip responses at 1, 2, or 10 kHz. I want 4D stop position prediction to avoid intersection of the robot arm with complex assemblies and safety zones imported from CAD. I want all these things cheaper and faster, with more storage.
But I need these to be accessible by Billy Joe, whose only credentials that got him the job in the maintenance department is that he helped out his daddy working with a welder and an old clapped-out Bridgeport on the farm growing up. Billy is probably my best tech on 3rd shift, and he's never learned to type on a full-size keyboard, just his smartphone... and on a robot teach pendant.