This Is Probably What a Store Will Be Like in the Future

Anyone finding this post and video 10-15 years from now will think, “Huh? How else would you shop? What’s checking out? Lines, really?” Amazon has announced a new brick-and-mortar “Amazon Go” store in Seattle they plan to open early in 2017. It fundamentally changes the way we shop. And it’s an idea that makes so much sense, if it does work, that other sellers will inevitably follow suit over time. The store is currently — this is a little weird — being beta-tested by Amazon employees.

For some time now, you’ve been able to go into an Apple Store and purchase items yourself from your phone without requiring a sales person. The Amazon Go store and phone app go way beyond that. They let you keep your phone in your pocket, place the items you want in your own bags, and then just leave when you’re done. Here’s the company’s introductory video.

The system works through an Amazon Go app you install on your phone. As a registered Amazon customer, you arrive at the store, swipe yourself in using the app, put your phone away, and hit the shelves, adding items to your “virtual shopping cart” It’s like a shopping cart on a website, not the kind with a bent wheel that you try and push and drag through a physical store.

While Amazon’s not being overly transparent about how they’re doing this, their video does say, “So how does it work? We used computer vision, deep learning algorithms and sensor fusion, much like you’d find in self-driving cars.”

Last year, tech site Recode uncovered a patent filing from Amazon that likely offers some clues about the tech being used.

Normal-looking shelves. (AMAZON GO)

The first challenge a smart store like this would face is knowing with certainty what products are on their shelves and which products are being removed. Amazon’s patent suggests a system of multiple sensors working in tandem to identify the goods:

In some implementations, data from other input devices may be used to assist in determining the identity of items picked and/or placed in inventory locations. For example, if it is determined that an item is placed into an inventory location, in addition to image analysis, a weight of the item may be determined based on data received from a scale, pressure sensor, load cell, etc., located at the inventory location. The image analysis may be able to reduce the list of potentially matching items down to a small list. The weight of the placed item may be compared to a stored weight for each of the potentially matching items to identify the item that was actually placed in the inventory location. By combining multiple inputs, a higher confidence score can be generated increasing the probability that the identified item matches the item actually picked from the inventory location and/or placed at the inventory location.

The deep learning algorithms apparently have to do with using the buyer’s Amazon shopping history as a backup-check to help ensure the system is correctly identifying each item being purchased.

For example, if the inventory management system cannot determine if the picked item is a bottle of ketchup or a bottle of mustard, the inventory management system may consider past purchase history and/or what items the user has already picked from other inventory locations. For example, if the user historically has only picked/purchased ketchup, that information may be used to confirm that the user has likely picked ketchup from the inventory location.

Making a purchase. (AMAZON GO)

Since the system watches everything so closely, should you change your mind and put a product back, it’s automatically removed from your virtual cart.

Finally, you leave the store, shopping done. A receipt appears on the phone for what’s just been charged to your Amazon account:

“[W]hen the customer passes through the exit (transition area) of the retail location, the items picked by the user may be automatically transitioned from the materials handling facility to the user and the user may be charged a fee for the items. … For example, if the user is purchasing items from a retail location, rather than the user having to stop and ‘check out’ with a cashier, teller or automated check station, because the picked items are already known and identified on an item identifier list associated with the user, the user may simply exit the retail location with the items. The exit of the user will be detected and, as the user passes through the exit (transition area), the user, without having to stop or otherwise be delayed, will automatically be charged a fee for the items (the items are transitioned to the user).”

Receipt (AMAZON GO)

Of course, all of this happens invisibly to the customer — except for the receipt and charge — and we can assume Amazon has been refining these systems since the patent was filed in June of 2013. Still, this is probably pretty close to what’s going on, though some suspect RFID tags are also involved now.

There are tantalizing questions for which we don’t yet have the answers. Will stores like this have delis where servings to order can be obtained? Will we have to bring our own bags? (We’d guess not.) Will there be actual carts? (Presumably.) Where will they display tabloid rags and impulse-buy treats? And maybe most important, without checkouts and baggers where will teenagers work over the summer?