Augmented Reality (AR)
Assembly Line
Hitachi's metaverse tech helps keep Japanese craftsmanship alive
Hitachi has developed a metaverse-based system that lets skilled workers run remote demonstrations using real-world factory equipment, with an eye toward industries where know-how is concentrated among aging veterans. This reporter experienced it firsthand with demonstration equipment. Through a virtual reality headset, an image of an instructor’s hands was superimposed on a drum-shaped machine.
Warping in the real world: Remote factory meetings in 360°
In a research project with SB&D, ETH students Michael Stegmaier and Cesare Primultini led our field test of interactive mixed reality live streaming technology in manufacturing. Michael and Cesare conceptualized two ways to transmit live-streaming 360° video from the shop floor of SB&D: A first-person view where a camera was mounted to a hard hat and a fixed-position view where the camera could be mounted inside a machine during a changeover. Live-streamed 360° video would be transmitted to a cloud, where participants would meet virtually from anywhere (see figure below). Back in 2021, we identified Avatour, a startup from California, as our project’s only viable software provider.
Where can this technology find good use? High-potential use cases are safety audits, training programs, and tasks that require the presence of an expert or coach. Other use cases are factory tours, external or internal audits, technology transfer projects, inspections, and on-the-job coaching of individuals.
How Prominent Video Game Technology is Changing Industrial Training and Manufacturing
Panigrahi explains that GridRaster’s XR equipment is currently being used in the manufacturing of airplane wings, spacecraft parts and automobile components, adding that AR and VR have emerged as a crucial part of repair maintenance for these sectors. “Having a headset that can give you instructions, workflows overlayed on top of the asset you are fixing, and working through each and every instruction has been not only a time-saver but a life-saver as well because you’re ensuring those human errors can be completely eliminated,” Panigrahi says.
Eliminating Defects with AR Technology
L3Harris Technologies implemented LightGuide AR software to ensure standardization across a variety of complex processes. Following the implementation of LightGuide on a line with 17 variants of one product, visually guided workflows helped consolidate parts, which eliminated changeover per variant and resulted in zero assembly-related defects. Since these results, L3Harris has implemented LightGuide on complex manual lines at multiple locations.
According to one Engineering Manager at the L3Harris, the system uses infrared and 3D sensing to know where an operator’s hand is within an inch in any direction. The cues highlighting what pieces go where allow operators to focus on the task at hand, not where they are in the process; further, the system will notify the operator via visual cues and messaging and stop instruction if they skip a step or reach for the wrong component.
AR in Manufacturing Industry
The manufacturing sector’s usage of augmented reality (AR) is the main topic of this blog. By superimposing digital information over the real world, augmented reality (AR) technology gives workers access to real-time information and direction. Design and prototyping, assembly and production, maintenance and repair, as well as training and teaching, are just a few of the manufacturing-related uses for augmented reality technology. AR may improve productivity, improve safety, and reduce costs in manufacturing operations, making it an attractive technology for organizations in the industry. The initial expenditure in hardware and software, as well as ongoing maintenance and updates, might make integrating AR challenging.
How Smart Glasses Helped a Manufacturer Through the Pandemic
How does an expert in one facility guide an engineer halfway across the country through implementing a new technology? Food and beverage packaging manufacturer Crown Holdings found an innovative answer during the pandemic, when its employees couldn’t travel freely: it equipped them with smart glasses.
“When the pandemic started, we were trying to coordinate commissioning activities through email and WhatsApp messaging,” said Crown Holdings Project Industrial Engineer Leon Azzi. “Tasks that normally took two to three days were taking weeks.” But with the glasses, “[The workers] could share with each other the PLC (programmable logic controllers) electrical diagrams and pictures using the glasses viewer, and the remote experts could point them to areas to focus on in real time,” said Crown Holdings Digital Optimization Team Head Alberto Rodriguez.
Reality Show: X-ray Vision Can See Through Metal
A typical aircraft maintenance inspection involves maintenance technicians and engineers walking around an aircraft recording new defects and damage with a pencil in a notebook. Locations are often described in language like ‘3 inches from the left side of the window.’ The inspection can often take hours or days. But what if you could hold a digital device and see locations of all previous damage and repairs highlighted in 3D?
Young Guns shoot for AI automated shipping container
“There is a significant drive from leading retailers internationally to innovate within their supply chain transformation strategies – predictability, visibility, and productivity will be a key component in these transformations, and we believe our shipping container innovation will create opportunities for this to occur at the container face that are not currently availability in the industry.”
The team mocked up their distribution centre, containers, and the Polaris product in a 3D CAD (computer-aided design) model, and transitioned the data into an augmented reality environment in HoloLens. The simulation used in the development of Polaris tracks the actions and interaction of each aspect of container de-stuffing. Each unpacker, forklift, container, and pallet are represented as an agent within a distribution centre. “We showed what the product would look like at various stages of utilisation in the distribution centre,” Kelly said. “We had YGCC’s frontline team come in and they could put the headsets on, and they could walk around in the virtual future distribution centre environment with two people at a time. “They could discuss what it would be like to use the device in different stages and get a feel for it and what it’s trying to do. That was hugely beneficial for them in terms of being able to engage with what the device is, because they didn’t just have to imagine being with it, they could actually be with it.”
“Waste time in the form of movement and waiting can be reduced from 25 down to 5 per cent,” Mr Kelly said. “Allocating the correct number of workers per stock type is important in achieving these improvements.”
HoloLens 2 helps Novo Nordisk employees see work in new ways
The pharmaceutical giant, a leader in the global production of diabetes medicine, has been pioneering the use of emerging digital technologies for everything from simplifying floor operations in their manufacturing facilities across the globe to conducting virtual site visits from thousands of miles away. Some of those solutions include Microsoft HoloLens 2, Microsoft Dynamics 365 Remote Assist, Microsoft Dynamics 365 Guides and Microsoft Power Platform. This includes introducing what is known as augmented instructions, via HoloLens 2, to shop floor employees, providing them with new possibilities to support both process performance and getting onboarded for new procedures.
Kawasaki Heavy Industries (Satya Nadella 2022 Build Keynote)
Flexible robotic arm put to work with AR
According to Imperial, the flexible arm can twist and turn in all directions, making it customisable for applications in manufacturing, spacecraft maintenance, and injury rehabilitation. In use, people working with the robot would manually bend the arm into the precise shape needed for each task, a level of flexibility made possible by layers of mylar sheets inside, which slide over one another and can lock into place. So far, configuring the robot into specific shapes without guidance has presented challenges.
To enhance the robot’s user-friendliness, researchers at Imperial’s REDS (Robotic manipulation: Engineering, Design, and Science) Lab designed a system for users to see in AR how to configure their robot. Wearing mixed reality smartglasses and through motion tracking cameras, users see templates and designs in front of them superimposed onto their real-world environment. They then adjust the robotic arm until it matches the template, which turns green on successful configuration so that the robot can be locked into place.
Semi-virtual site visits deliver enhanced customer value
AR technology offers opportunities to schedule visits that have been difficult to arrange in the past due to sites being in remote locations, offshore environments or other restricted areas. Headset technology allows field engineers to reach someone where they are and streamline the visit process. As a result, someone can receive insights in real time because remote engineers can see what is happening onsite versus trying to identify issues through messages or images (see Figure 4). Problems can be solved more quickly, which saves time and money in the process.
Gemini - Collaboration using Augmented Reality
Our New Reality: The Rise of AR and VR
“The ability to use digital twins and simulation in AR [and] VR is gaining steam,” says Kelly Malone of Taqtile. “New technologies like 5G and edge computing enable new use cases where access to data-intensive operations is feasible in-situ using HMDs and mobiles devices. So rather than minutes or hours to see the impact of a design change, these results can be viewed in-situ within minutes or seconds to support the evaluation of more design options and iterations. Sensor information (industrial internet of things – IIoT) can also be integrated and overlaid upon the real-world or digital twin, providing an added layer of data to inform design change impacts and decision making.”
This Factory Is Using AR To Help With A Hiring Crunch
One of the challenges associated with AR has been in trying to turn a complex physical process, such as wiring a component or working a machine, into code that could run on a headset. Taqtile CEO Dirck Schou said the company’s software makes programming for AR glasses simple, and based on my conversation with Tim Lecrone and Beau Wileman of PBC, the software Taqtile developed is easy to use. Once PBC has created a module for training it pays for itself after 1.44 employees train with it according to Wileman.
The cobots help handle processes that are repetitive and free up people to take on different tasks. Given how tough it is to hire people to work in the factory, using them helps reduce the overall staffing load. But the biggest gains so far have been in training and getting employees quickly up to speed. Now PBC can hire a person and get them working on a machine in a few days as opposed to that taking up to six weeks. It also helps reduce the cost of training a cobot and staff. Wileman told me that an intern, which costs $17 an hour, can train a cobot or map out a process in less than four hours, while it might cost around $30,000 for an outside expert to manually train a cobot.
CEOs Are Dooming Business Travel — Maybe for Good
Take Akzo Nobel NV, Europe’s biggest paint maker, for instance. At its Amsterdam headquarters, Chief Executive Officer Thierry Vanlancker has spent the past year watching his manufacturing head, David Prinselaar, flap his arms, madly gesticulate and seemingly talk to himself while “visiting” 124 plants by directing staff with high-definition augmented-reality headgear on factory floors. A task that meant crisscrossing the globe in a plane before is now done in a fraction of the time — and with no jet lag. For Vanlancker, there’s no going back.
Royal Dutch Shell Plc has created online control rooms with interactive 3D simulations of oil platforms and plants, giving engineers virtual access from home. In Troy, Michigan, Kevin Clark, the CEO of Aptiv Plc, a former car parts unit of General Motors Co., is using drones and Oculus augmented-reality headsets to show customers the performance and manufacturing run rates of plants in Mexico, Hungary, or China.
Industry 4.0 and the Automotive Industry
“It takes about 30 hours to manufacture a vehicle. During that time, each car generates massive amounts of data,” points out Robert Engelhorn, director of the Munich plant. “With the help of artificial intelligence and smart data analytics, we can use this data to manage and analyze our production intelligently. AI is helping us to streamline our manufacturing even further and ensure premium quality for every customer. It also saves our employees from having to do monotonous, repetitive tasks.”
One part of the plant that is already seeing benefits from AI is the press shop, which turns more than 30,000 sheet metal blanks a day into body parts for vehicles. Each blank is given a laser code at the start of production so the body part can be clearly identified throughout the manufacturing process. This code is picked up by BMW’s iQ Press system, which records material and process parameters, such as the thickness of the metal and oil layer, and the temperature and speed of the presses. These parameters are related to the quality of the parts produced.
How Augmented Reality Strengthens Biotech Manufacturing
Probably, the biggest advantage of AR is it enables seeing the production process virtually, without the need to be there. “It’s a game-changer for the industry. Individuals can have eyes and ears on site at a moment’s notice to address an emerging issue, or to host routine remote collaboration sessions,” Stracquatanio highlights.
AR can also increase control over the manufacturing process. Pharma and biotech companies cannot afford mistakes during the production phase. A little oversight might lead to serious consequences such as having to start from scratch, which can be very expensive and time-consuming. A recent example is that of Johnson & Johnson’s manufacturing partner Emergent BioSolutions, whose workers erroneously mixed ingredients from two different Covid-19 vaccines; this led to wasting around 15 million vaccine doses.
During a working day, we could see an operator who loads 3D models of biotech instruments, looking at specific pieces and relevant information appearing in the smart glasses or tablet. Meanwhile, another engineer walks up to a machine with a QR code, and instructions pop up in the glasses, facilitating access to the adjustments. A few steps from there, another colleague is looking at batch records, saving values into the system just via voice.
Augmented reality becomes actual reality
When applied to electrical power distribution across a wide range of businesses and industries, AR has the potential to greatly increase power availability, electrical safety, and efficiency. Here’s why:
- Availability: AR helps organizations optimize operations and maximize continuity for better productivity and profitability
- Safety: AR helps to reduce the risk of occupational injuries and fatalities
- Efficiency: AR help reduces the total cost of ownership by offering more accessible and effective training
How Augmented Reality Became a Serious Tool for Manufacturing
Making monsters appear in games like Pokémon Go is not the only application for augmented reality these days. Industry is using the technology too, harnessing CAD data for training workers, standardizing workflows, and enabling collaboration.
Augmented Reality Gets Pandemic Boost
Augmented reality, which superimposes digital content onto a user’s view of the real world, became more valuable for some companies such as Mercedes-Benz USA and L’Oréal SA last year amid social distancing requirements and lockdowns. The companies are using the technology to provide assistance for employees and consumers in real-time, without needing to be physically present.
Advanced Technologies Adoption and Use by U.S. Firms: Evidence from the Annual Business Survey
While robots are usually singled out as a key technology in studies of automation, the overall diffusion of robotics use and testing is very low across firms in the U.S. The use rate is only 1.3% and the testing rate is 0.3%. These levels correspond relatively closely with patterns found in the robotics expenditure question in the 2018 ASM. Robots are primarily concentrated in large, manufacturing firms. The distribution of robots among firms is highly skewed, and the skewness in favor of larger firms can have a disproportionate effect on the economy that is otherwise not obvious from the relatively low overall diffusion rate of robots. The least-used technologies are RFID (1.1%), Augmented Reality (0.8%), and Automated Vehicles (0.8%). Looking at the pairwise adoption of these technologies in Table 14, we find that use of Machine Learning and Machine Vision are most coincident. We find that use of Automated Guided Vehicles is closely associated with use of Augmented Reality, RFID, and Machine Vision.
Boeing Tests Augmented Reality in the Factory
Installing electrical wiring on an aircraft is a complex task that leaves zero room for error. That’s why Boeing is testing augmented reality as a possible solution to give technicians real-time, hands-free, interactive 3D wiring diagrams - right before their eyes.
“Our theory studies have shown a 90 percent improvement in first-time quality when compared to using two-dimensional information on the airplane, along with a 30 percent reduction in time spent doing a job.”
Google Glass Didn't Disappear. You Can Find It On The Factory Floor
With Google Glass, she scans the serial number on the part she’s working on. This brings up manuals, photos or videos she may need. She can tap the side of headset or say “OK Glass” and use voice commands to leave notes for the next shift worker.
Peggy Gullick, business process improvement director with AGCO, says the addition of Google Glass has been “a total game changer.” Quality checks are now 20 percent faster, she says, and it’s also helpful for on-the-job training of new employees. Before this, workers used tablets.
Augmented Reality Is Already Improving Worker Performance
The video below, for example, shows a side-by-side time-lapse comparison of a GE technician wiring a wind turbine’s control box using the company’s current process, and then doing the same task while guided by line-of-sight instructions overlaid on the job by an AR headset. The device improved the worker’s performance by 34% on first use.
There’s been concern about machines replacing human workers, and certainly this is happening for some jobs. But the experience at General Electric and other industrial firms shows that, for many jobs, combinations of humans and machines outperform either working alone. Wearable augmented reality devices are especially powerful, as they deliver the right information at the right moment and in the ideal format, directly in workers’ line of sight, while leaving workers’ hands free so they can work without interruption. This dramatically reduces the time needed to complete a job because workers needn’t stop what they’re doing to flip through a paper manual or engage with a device or workstation. It also reduces errors because the AR display provides explicit guidance overlaid on the work being done, delivered on demand. Workers need only follow the detailed instructions directly in front of them in order to move through a sequence of steps to completion. If they encounter problems, they can launch training videos or connect by video with remote experts to share what they see through their smart glasses and get real-time assistance.