Intelligent Assistant Helps with Picking

A gigantic warehouse with endless rows of shelving. And just where are the screws for the picking order from production? Untrained employees who help out in the warehouse during periods of peak activity don’t know their way around, but they still have to handle picking orders quickly. Under this time pressure, they often deliver incorrect items, so returns are always part of a day’s work. Even experienced employees, who can handle several orders simultaneously, are not immune from mistakes. The risk of errors is especially high when entering follow-up data into the warehouse management software.

Warehouse employee with MICA transport cart
Warehouse employee with MICA transport cart

MICA helps in this challenging situation. The adaptive and multimodal system supports warehouse employees according to their level of knowledge. With sensors and positioning technologies like radio frequency identification (RFID), the intelligent assistant is able to realize the process context of individual employees. It can also offer help if needed – audio suggestions over a headset or graphic suggestions on a tablet PC on the transport cart. Audio and optical sensors in the headset and on the cart capture employee movements, as well as environmental data, such as noise level and brightness. If the noise is too loud, the system issues notifications on screen. It also switches to an audio notification if the employee is away from the transport cart or when poor lighting makes it difficult to read the monitor.

The end of orientation problems

Thanks to a real-time connection to the SAP Extended Warehouse Management application, which is planned as the Mica pilot scheme’s next step on a customer’s operation, users don’t need to enter and maintain data manually. MICA receives the orders directly from the warehouse management software and informs employees accordingly. MICA also explains individual work steps to inexperienced employees. The monitor displays, if necessary, an image of the items to be picked and even highlights important details, such as an ID number and the location of the number.

MICA shows the way
MICA shows the way

MICA also handles route planning; an integrated routing system leads employees through the warehouse and schedules picking orders so that they can be fulfilled with the shortest route. Heavy and oversize items are loaded into the transport cart first, so they’re located at the bottom of a crate. If an inexperienced employee deviates from the planned route, MICA triggers an audio tone to alert the employee of the mistake and indicate the correct direction. The system tolerates deviations by experienced employees – as long as the new route leads to another item in the order. If the new route doesn’t lead to a relevant item, the warning signal is triggered and the screen offers assistance via the navigation area and the highlighted yellow button.

Talking to the boss: audio conferencing

MICA offers proactive assistance when warehouse employees stay in one location and search for an item or move up and down the warehouse aisles. Employees can decide for themselves if they want to accept the help. If they continue to work, the highlighting disappears. With MICA, employees don’t have to interrupt their work to ask a manager where they can find a particular item. But if they do need a manager’s help – if the item isn’t in the proper storage location or if it’s damaged – they can start an audio conference with MICA. The manager can also use a camera in an employee’s helmet to view the actual situation.

RFID grid on the transport cart
RFID grid on the transport cart

RFID sensors at the bottom of the transport carts register the loading and unloading of items. MICA notes when an incorrect item ends up in the cart and instructs the warehouse employee to remove it from the crate.

Four system layers for special tasks

MICA interface architecture
MICA interface architecture

To enable the adaptive and multimodal design of MICA, FIT developed a new, modular system architecture with special layers for sensors, actuating elements, modeling, and dialog management. The modules for sensors and actuating elements are housed on clients; modeling and dialog management modules are housed on servers. All modules are stored flexibly; they can be added or deleted, and they can be moved from clients to servers and vice versa.
The sensor layer captures user interactions, tracking data, and environmental parameters. Users can interact with MICA using microphones, pens, or keyboards. Tracking systems and sensors for movement, brightness, and volume provide user-relevant parameters and context information. RFID tags capture order-specific data.
The raw data is processed in the modeling layer with recognition and integration software that interprets language, movement patterns, and gestures and then derives meaning from that information. The recognition and integration software also brings tracking data into the context of a stored spatial model to locate employees in the warehouse and to facilitate the orientation.
Dialog management plans and refines the behavior of the system. Users can switch off the tracking mode to respect an individual’s privacy. The layer also reacts to learning goals by blocking inexperienced employees from specific functions. As an employee gains experience, the functions are released little by little. Dialog management capabilities also coordinate content output to users according to their preferences and the appropriate modalities.
The output layer guarantees that warehouse clerks receive appropriate information. When it outputs data, it considers the technical characteristics of the headset and the tablet PC and adjusts the volume or brightness accordingly.

Developers as guests in the warehouse

SAP Research has supported FIT’s MICA project since the end of 2004. MICA is part of the research focus “Arbeitsplatz der Zukunft” (“Workplace of the Future”) of the SAP research site in Darmstadt, Germany. The intention is to explore technologies which will exert influence on the organization of our future work environment. The goal for MICA was to realize the system with a prototype for a warehouse. To meet the requirements of an adaptive, multimodal assistant, the developers interviewed warehouse employees, foremen, and managers at two large warehouses.
They also spent a full day in each warehouse to obtain a real-world look at the problems that can develop. The developers found that returns caused by the delivery of incorrect items could actually run higher than 5 percent. Returns not only create extra costs, they also aggravate customers. Even minimal configuration of MICA solves the problem with RFID technology. A prototype has been available since the end of 2006 and is ready to undergo testing in pilot operations at a client site.

Dr.  Markus Eisenhauer
Dr. Markus Eisenhauer