Welcome to robotics research and development
Microsoft researchers controlled this robotic arm using ChatGPT. | Credit: Microsoft
By now, you’ve likely heard of ChatGPT, OpenAI’s language model that can generate somewhat coherent responses to a variety of prompts and questions. It’s primarily being used to generate text, translate information, make calculations and explain topics you’re looking to learn about.
Researchers at Microsoft, which has invested billions into OpenAI and recently integrated ChatGPT into its Bing search engine, extended the capabilities of ChatGPT to control a robotic arm and aerial drone. Earlier this week, Microsoft released a technical paper that describes a series of design principles that can be used to guide language models toward solving robotics tasks.
“It turns out that ChatGPT can do a lot by itself, but it still needs some help,” Microsoft wrote about its ability to program robots.
Prompting LLMs for robotics control poses several challenges, Microsoft said, such as providing a complete and accurate description of the problem, identifying the right set of allowable function calls and APIs, and biasing the answer structure with special arguments. To make effective use of ChatGPT for robotics applications, the researchers constructed a pipeline composed of the following steps:
In one example, Microsoft researchers used ChatGPT in a manipulation scenario with a robot arm. It used conversational feedback to teach the model how to compose the originally provided APIs into more complex high-level functions that ChatGPT coded by itself. Using a curriculum-based strategy, the model was able to chain these learned skills together logically to perform operations such as stacking blocks.
The model was also able to build the Microsoft logo out of wooden blocks. It was able to recall the Microsoft logo from its internal knowledge base, “draw” the logo as SVG code, and then use the skills learned above to figure out which existing robot actions can compose its physical form.
Researchers also tried to control an aerial drone using ChatGPT. First, they fed ChatGPT a rather long prompt laying out the computer commands it could write to control the drone. After that, the researchers could make requests to instruct ChatGPT to control the robot in various ways. This included asking ChatGPT to use the drone’s camera to identify a drink, such as coconut water and a can of Coca-Cola. It was also able to write code structures for drone navigation based solely on the prompt’s base APIs, according to the researchers.
“ChatGPT asked clarification questions when the user’s instructions were ambiguous and wrote complex code structures for the drone such as a zig-zag pattern to visually inspect shelves,” the team said.
Microsoft said it also applied this approach to a simulated domain, using the Microsoft AirSim simulator. “We explored the idea of a potentially non-technical user directing the model to control a drone and execute an industrial inspection scenario. We observe from the following excerpt that ChatGPT is able to effectively parse intent and geometrical cues from user input and control the drone accurately.”
The researchers did admit this approach has a major limitation: ChatGPT can only write the code for the robot based on the initial prompt the human gives it. A human engineer has to thoroughly explain to ChatGPT how the application programming interface for a robot works, otherwise, it will struggle to generate applicable code.
“We emphasize that these tools should not be given full control of the robotics pipeline, especially for safety-critical applications. Given the propensity of LLMs to eventually generate incorrect responses, it is fairly important to ensure solution quality and safety of the code with human supervision before executing it on the robot. We expect several research works to follow with the proper methodologies to properly design, build and create testing, validation and verification pipelines for LLM operating in the robotics space.
“Most of the examples we presented in this work demonstrated open perception-action loops where ChatGPT generated code to solve a task, with no feedback provided to the model afterwards. Given the importance of closed-loop controls in perception-action loops, we expect much of the future research in this space to explore how to properly use ChatGPT’s abilities to receive task feedback in the form of textual or special-purpose modalities.”
Microsoft said its goal with this research is to see if ChatGPT can think beyond text and reason about the physical world to help with robotics tasks.
“We want to help people interact with robots more easily, without needing to learn complex programming languages or details about robotic systems. The key challenge here is teaching ChatGPT how to solve problems considering the laws of physics, the context of the operating environment, and how the robot’s physical actions can change the state of the world.”
The post How ChatGPT can control robots appeared first on The Robot Report.
Luxonis offers high-resolution cameras with depth vision and on-chip machine learning. | Source: Luxonis
Luxonis announced the release of its newest DepthAI ROS driver for its stereo depth OAK cameras. The driver aims to make the development of ROS-based software easier.
When using the DepthAI ROS driver, almost everything is parameterized with ROS2 parameters/dynamic reconfigure, which aims to provide more flexibility to help users customize OAK to their unique use cases.
The DepthAI ROS driver is being developed on ROS2 Humble and ROS1 Noetic. This allows users to take advantage of ROS Composition/Nodelet mechanisms. The driver supports both 2D and spatial detection and semantic segmentation networkers.
The driver offers several different modes that users can run their camera in depending on their use case. For example, users can use the camera to publish Spatial NN detections and publish RGBD pointcloud. Alternatively, with the DepthAI ROS driver users can stream data straight from sensors for host processing, calibration and modular camera setup.
With the driver, users can set parameters for things like exposure and focus for individual cameras at runtime and IR LED power for better depth accuracy and night vision. This allows users to experiment with onboard depth filter parameters.
The driver enables encoding to get more bandwidth with compressed images and provides an easy way to integrate a multi-camera setup. It also provides docker support for easy integration, users can build one themselves or use one from Luxonis’ DockerHub repository.
Users can also reconfigure their cameras quickly and easily using ‘stop’ and ‘start’ services. The driver also allows users to use low-quality streams and switch to higher quality when they need or switch between different neural networks to get their robot the data it needs.
Earlier this month, Luxonis announced a partnership with ams OSRAM. As part of the partnership, Luxonis will use OSRAM’s Belago 1.1 Dot Projector in its 3D vision solutions for automatic guided vehicles (AGVs), robots, drones and more.
The post Luxonis releases DepthAI ROS driver appeared first on The Robot Report.
A vehicle’s check engine light turning on is many car owners’ worst nightmare. Without specific knowledge about the inner workings of the vehicle, it’s unlikely you’d know what is actually wrong with it based solely on the check engine light. It could be a quick and easy fix, or a significant component failure that could end up costing thousands of dollars.
But what if your car could tell you exactly what the issue is instead of relying on something so vague? Thankfully the internet of things has made this, among other exciting developments, a reality. Read on to learn how IoT-enabled “connected cars” are revolutionizing the automotive industry.
The origin of connected cars
While the term connected car is often used nowadays to describe vehicles utilizing IoT technology, that’s far from where the term first gained traction. In 1996, GM and Ford introduced their OnStar and RESCU services respectively. These services primarily served as an emergency response tool, putting customers in contact with a call center in the case of a crash, breakdown, theft, or other similar emergency. By utilizing GPS tracking built into the vehicle, call center operators could quickly and easily send help to the drivers location even if the driver didn’t know where they were.
These were the first connected cars, and as the years went by both GM and Ford expanded the features offered by their services. Remote diagnostics, navigation assistance, Wi-Fi hotspots and more were all eventually incorporated in connected cars. These advancements became a launchpad for full IoT integration, connecting vehicles to nearly every IoT-enabled device imaginable. Smartphones are one of the most well-known examples of this, being able to connect to a car to control music, remotely lock, unlock, and start the car, and more.
Predictive maintenance for connected cars
Now, let’s circle back to the hypothetical check engine light scenario. Without IoT connectivity for your vehicle, knowing the exact cause of the problem likely isn’t an easy task. However, the IoT enables predictive maintenance across the entire vehicle, informing the driver of specific components that are deteriorating before they fail. Guesswork is taken out of vehicle maintenance entirely, making consumers’ lives easier and keeping them on the road longer. No more frustrating trips to a mechanic, unsure if you’re being given the runaround or ripped off.
Even beyond engine problems, connected cars can alert drivers to more routine maintenance needs. Air filter replacement, transmission fluid service, tire rotation and brake servicing are all examples of maintenance that drivers aren’t necessarily reminded of without a connected car. Knowing when exactly to conduct maintenance helps keep the vehicle running longer, and is a cheaper alternative to replacing components after they’ve failed due to lack of maintenance.
Managing fleets of connected cars
With the IoT, managing large fleets of transport vehicles is easier, safer, and more efficient. Predictive maintenance can be used in the same exact way as previously discussed, informing operators of potential component deterioration before it fails, which in the case of these large transport vehicles can cause catastrophic damage. Beyond that, cargo and driver safety can be improved during operation through integrated vision systems. Blind spots can be monitored to reduce accident risk, and AI can even be integrated to detect other vehicles outside of just blind spots and alert the driver if an accident is imminent. The video used for this detection system can also be saved and used later on, whether for training purposes or in cases of an accident or litigation (Source: Intel).
Communication between drivers of the fleet can also be improved with the IoT. 5G or Wi-Fi modules can be installed in the vehicles, giving drivers the means to quickly communicate with each other with minimal latency (Source: Intel). Fleet managers can also keep track of where each vehicle in the fleet is, the status of both the vehicle and cargo, and provide instructions to drivers if rerouting is necessary.
Connected cars made autonomous
There’s one final application of IoT for connected cars to discuss, and it’s arguably the most exciting for the majority of consumers. Semi-autonomous and fully-autonomous vehicles have been a high-profile topic for years, with Tesla being one of the highest-profile proponents of the technology. Tesla “Full Self-Driving” (FSD) vehicles utilize eight cameras installed around the front, back and sides of the vehicle to achieve a full 360° view of the surrounding environment. These cameras, along with other sensors and a neural network built into the vehicle, form the basis of the FSD technology. The neural network is the most important aspect here though. Elon Musk, Tesla CEO, has stated that the neural network in Tesla vehicles “will learn over time”, primarily using data collected from other Tesla cars/drivers. This process is made possible with the IoT, connecting sensors to the cloud so all of the data gathered can be compiled and used to improve the self-driving features of the vehicle.
Conclusion
There’s no shortage of applications that connected cars make possible. Not only can drivers’ lives be made easier thanks to predictive maintenance, larger corporations can better manage fleets of vehicles, or develop solutions to complex problems like autonomous driving.
For questions about IoT-enabled plain bearings, linear bearings and slides for automotive manufacturing, contact a drytech® smart plastics expert. For IoT-enabled cables and cable carriers, contact an e-chain® smart plastics expert.
Sponsored content by igus
The post Connected Cars: Driving the automotive industry forward appeared first on The Robot Report.
From seeing Boston Dynamics’ Stretch robot at work in a DHL facility to Alphabet closing down one of its robotics subsidiaries, there was no shortage of robotics news in February 2023.
Here are the top 10 most popular stories on The Robot Report in February 2023. Subscribe to The Robot Report Newsletter or listen to The Robot Report Podcast to stay updated on the robotics stories you need to know about.
Some 30,000 people in the U.S. are affected by amyotrophic lateral sclerosis (ALS), also known as Lou Gehrig’s disease. Now, a team of researchers from the Harvard John A. Paulson School of Engineering and Applied Sciences (SEAS) and Massachusetts General Hospital (MGH) has developed a soft robotic wearable capable of significantly assisting upper arm and shoulder movement in people with ALS. Read More
9. What slowdown? – December 2022 robotics investments reach $1.14BDecember 2022’s 55 robotics investments totaled US $1.14B, a 7.7% increase over December 2021’s funding total. Investment into companies providing robotics solutions for autonomous ground transportation such as sensors, autonomy ‘stacks’ and ‘drivers’, and whole cloth systems, was particularly strong. Read More
8. Alphabet closes Everyday Robots among layoffsAlphabet, Google’s parent company, is shutting down its subsidiary Everyday Robots, according to reporting from Wired. On January 20, 2023, Alphabet announced it would be laying off around 12,000 workers, 6% of its workforce, and Everyday Robots is one of the few projects disbanded as part of the budget cuts. Read More
Rapid Robotics announced a brand-new integration with Yaskawa America, Inc., Motoman Robotics Div. (Yaskawa Motoman) that will bring industrial capabilities to Rapid Robotics’ Rapid Machine Operator (RMO). The company is now integrated with the entire Yaskawa robotics portfolio. Read More
Rapid Robotics and Universal Robots (UR) just announced a new partnership. UR, a Danish company that makes collaborative robots (cobots), will supply Rapid Robotics with collaborative robot arms so that it can set up cobot work cells all over North America. This means that Rapid Robotics will be able to serve more customers and keep the quick deployment times that customers have come to expect, even as Rapid Robotics continues to grow across the country. Read More
5. 5 top robotics trends to watch in 2023Robot installations hit an all-time high in 2021, with the International Federation of Robotics’ (IFR) data showing over 500,000 new industrial robots were installed that year. In North America, robot sales hit an all-time high for the second year in a row in 2022, according to the Associate for Advancing Automation (A3), bringing in $2.38 billion. Read More
Researchers at Microsoft, which has invested billions into OpenAI and recently integrated ChatGPT into its Bing search engine, extended the capabilities of ChatGPT to control a robotic arm and aerial drone. Microsoft released a technical paper that describes a series of design principles that can be used to guide language models toward solving robotics tasks. Read More
China’s Ministry of Industry and Information Technology, along with 17 other agencies, has created a new action plan called the “Robot + Application Action Plan.” This plan lays out 10 industries the country wants to focus on developing robotic systems for and overarching goals for the country’s robotics industry to hit by 2025. Read More
Since its founding eight years ago, Zoox, a company developing autonomous vehicles (AVs) and now a subsidiary of Amazon, has been working towards its goal of creating a purpose-built autonomous vehicle, with no steering wheel. Zoox has an extensive, recently upgraded testing fleet that is verifying the effectiveness of its autonomous driving technology. Read More
Boston Dynamics is officially putting its Stretch robot into the hand of its customers. Its first commercial application is with DHL Supply Chain, a company that Boston Dynamics has been collaborating with since 2018, when it began developing Stretch. Read More
The post Top 10 robotic stories of February 2023 appeared first on The Robot Report.