Everything, everywhere, will soon be continuously recorded and uploaded to the internet. This will start with dense, urban areas, but over time every single square meter of every part of the globe will be recorded. Advances in computer vision & AI mean this data will be usable at scale, which will revolutionise advertising, law enforcement, and bring us back to a pre-privacy world.
(and yes, you guessed it, the movie to pair this post with is Minority Report)
Photo Courtesy of Tesla
Video Everywhere
Here is where we are today:
- Satellites & Drones record videos of anywhere on earth. This is what enables for instance Terra Bella (a Google subsidiary) to sell HD video feeds of any place on earth for commercial applications. In most cases the resolution is still quite coarse — for instance you can monitor whether a truck is on a building site from space, but not see its license plate.
- CCTVs & Home Camera systems have proliferated as sensor costs have dropped and surveillance requirements increase both at the national security level and the individual level (home security, babycams, etc). There are more than 6 Million CCTVs in the UK (source).
- Consumer Video recordings have increased ridiculously. For an order of magnitude: more than 4 Million Hours of video are uploaded to Youtube each day (source). This isn’t slowing down as the 50% of the planet who do not yet have smartphones get access to them, and products such as Facebook Live, Twitter’s Periscope, and Snapchat drive current smartphone users to take more (now ‘live’) video.
But the real step change is around the corner:
- Augmented Reality “googles” will become mass market, and will need to record a constant video feed of what the user is seeing to be able to augment the view with software.. Today Microsoft Hololens and Snapchat Spectacles are leading, and have very small penetration. But many smart, well funded people are working on making AR work, spearheaded by $5bn in funding and 700+ employee stealth startup Magic Leap.
Photo courtesy of Snap Inc
- Autonomous Vehicles (AVs) will become mass market and will capture continuous, high definition, high frame rate, 360° video. This is huge when you consider the scale of this AV revolution as a replacement for all cars today, but even more mind-blowing when you consider the role AVs in an even bigger market — logistics. As the market demands increasingly “on-demand” goods (Amazon now offer 1 hour deliveries) AVs (including drone) will be the cost effective path for companies to satiate this need. If driving the vehicle is free and powering it becomes practically free (with the advances in electric engines efficiency and Solar Power) — why not get 10 shirts shipped to your door for you to try on and return the 9 you don’t like? (For more on this: my friend Alex Flamant gave a prescient talk on this topic at the London Futurists meetup.)
Source: Google Self-Driving Car Project Monthly Report
As these technological evolutions march on, we are getting closer to continuous HD video recording of every square meter of every part of the globe.
Now. As awesome as some videos are (here’s a funny one), no one is going to sit through and watch all this stuff…
From Video to Insight
Fuelled by this explosion of video data, there have been incredible advances in Computer Vision over the last 20 years.
Let’s start with something you may not be aware of:
- Videos as Microphones: Very high definition silent video feeds can now be processed by algorithms to reconstruct the sound as it was in the area filmed. This works by being able to recognise tiny vibration motions caused by sound on surrounding objects (some of the same technology that enables us to get someone’s heart-rate from a video feed). In a striking example in this video, researchers are able to recover a conversation by filming a packet of crisps from behind sound-proof glass. The recovery is good enough to understand what people were saying, and there was no sound recorded at all! While there is no reason for AVs, CCTVs, etc not to record sound, these techniques may help get better sound definition, especially at long distances and in noisy places…
Hearing through a pack of crisps. The Visual Microphone: Passive Recovery of Sound from Video
- Natural Language Processing: Your clumsy Google Now or Siri may betray this today, but in the research lab we’re there in terms of reliably going from sound to text. As with Facial Recognition, the explosion of data on which to train NLP algorithms (from conversational interfaces in smartphones, bots, and Amazon Alexa/Google Home speakers) has spurred this innovation. In 2016 Microsoft beat human benchmarks for speech-recognition. Shortly after, Google showed off new ways to do live translation for 8 language pairs, showing that the innovation is perfectly comfortable whizzing past what (most) humans can do!
Finding a Voice — The Economist
- Facial Recognition can be considered solved. In June 2015, Yann LeCun, head of artificial intelligence at Facebook, announced that on a set of 40 000 pictures from Flickr, Facebook’s algorithms were able to correctly identify faces in 83% of cases. Interestingly people’s faces didn’t have to always be in the picture — with enough data about a user the algorithm could use other cues such as clothing to identify them (“Mark seems to wear a lot of grey t-shirts…”) Photo-based recognition is continuously improved with the explosion of photos and the continuous feedback Facebook and Google get as users tag their friends in pictures. With video, and a history of each user’s habits, clothing style, and locations, the algorithms will easily surpass people in recognising who is who. (remember the last time you struggled to recognise someone you hadn’t seen in a few years?). They might also take visual cues better than you:
Extracting the needle from the haystack
AI systems are capable of inferring higher level concepts (‘features’) from raw data. For example, faces are a feature that is inferred from raw pixels forming the video feed. This effectively makes all video, and all sound, searchable. Clarify.io as an example of a company providing a service to do this:
Clarify.io (seems to work — the top result was an Elon Musk video!)
But AI goes much further than enabling humans to search. It can effectively perform all possible searches for you and return higher level ‘features’: interesting patterns, anomalous events.
It also does so across as many data sources as are available, joining the data sets on common identifiers across them. So it’s not “there is a face in this video”, it’s also the name behind the face, the internet ‘breadcrumb’ of this person (including all other locations where this face was seen ever, all browsing activity, all messaging, all social network opinions, etc) and any other data set that might be available.
These AI systems will continue to improve, and each improved version will reprocess the raw data. Say the UK police runs an algorithm through all public CCTV recordings from the last week, looking for a criminal’s face. Today’s algorithm may not find it, but a future version will.
So, what are the implications of these advances?
Commercial Implications
Advertisers, insurers, retailers, banks, will feast on this new data like they have d0ne with all other internet data so far. The aim remains to build a detailed profile of you to sell you more stuff and quantify the risk/liability you represent.
Ubiquitous video combined with ever improving AI systems provide:
- Your socioeconomic status: where you live, where you work and how much, whether you go drop off kids at school, who you hang out with, where, how frequently
- Your shopping habits: where you shop, what you wear, what your friends wear, what you have to wear at work
- Your health: how active you are, whether you smoke/drink, which kinds of restaurants you visit or order from, how many doctor visits you have been to
truly 121 advertising with a side of leather jackets and guns (Minority report)
It also brings up some interesting forecasting opportunities at a more macro level:
What happens to the fashion industry when half a dozen static $100 cameras can tell you everything that anyone in Shoreditch wore this year — when you can trace a trend through social and street photography from start to the mass-market, and then look for the next emerging patterns?
— Benedict Evans, Cameras, ecommerce and machine learning
It seems inevitable that these video-sourced insights will be resold by the AV companies, in the same way that Google and Facebook sell targeting based on browsing history and likes. The commercial opportunity is huge.
Whether this happens officially will depend on the nature of the regulation surrounding the use of this video data. Unofficially, given febrility of security on the internet, this data will get used for commercial means.
Privacy Implications
“The Right To Privacy” ~ December 15, 1890, Harvard Law Review
As Gregory Ferenstein brilliantly lays out in The Birth And Death Of Privacy, full transparency is humanity’s natural state. Privacy as a concept is only about 150 years old.
For a long time, it was not materially possible to have any form of privacy as we’d think about it today. Tribal communities all lived together in caves, Romans all lived in one room houses, medieval families and servants all slept in the same bed — there was no practical concept of private space or individual intimacy.
As private life became possible — physically with separate houses, rooms, beds, and through communication with new technologies like writing and the telephone — the desire for privacy emerged. The first privacy-orientated law in the US was the 1710 Post Office Act, which banned sorting through the mail by postal employees.
Nonetheless, privacy has always remained a secondary concern to convenience and cost. This explains the consistent, broad adoption of new technologies which encroach on our privacy but are deemed worth it.
Today, Autonomous Vehicles is one of these new technologies to evaluate. Isn’t it crazy that today we let people drive a ton of steel at 100s of miles per hour, just with a few signs telling them not to press on a pedal too much? It is — and it causes 30 000 deaths per year in the US (that’s one every 17 minutes). With autonomous vehicles, this number will significantly decrease, which by itself makes this technology very attractive. But what are the privacy implications?
This time, you can’t opt out
The smartphone & internet revolution has also been a profound change, but you could choose not to have a smartphone or not to use Facebook, Google, or any other services that make a living collecting and selling your data.
In the Video Everywhere world, tracking moves fully offline. The moment you step outside, all your movements are streamed to a database, timestamped, geolocated and added to your digital profile. You can’t opt out.
(To delay this day as much as possible, you could (1) make sure your face is not on the internet, (2) live in a place that will have AVs last, e.g North Korea, (3) live in a cloudy place such that drones and satellites struggle to see you from far. On the plus side, I bet your rent will be cheaper there)
100% Enforceability
As we all know, the USA-pioneered surveillance-without-warrant state is in vogue at the moment, for example with the recent UK ‘Snooper’s charter’ bill becoming law, or the emergency state being preserved in France.
Video Everywhere means potentially one more source data from which to find anomalous citizen behaviour. This will be used by intelligence agencies for defence purposes, but it directly affect you and me in the hands of your average prosecutor.
Law becomes 100% enforceable. If the prosecutor can codify all infractions in a video processing algorithm — say jaywalking (to take a particularly ridiculous law) — and run it on all video ever taken of public life in the US, it could then immediately charge all jaywalkers.
I think the most important idea is to remember that there have been times throughout American history where what is right is not the same as what is legal
— Edward Snowden
Another factor to consider is the increased breadth and effectiveness of cybercrime that results from more data collection, particularly video data. As tracking moves to the offline, physical world, so does the targeting — you could for instance imagine a drone being hijacked to go and hurt someone who was identified by hacking into an AV’s video feed.
😬. What now?
Today, the public debate about Autonomous Vehicles and AI focuses on the most pressing issue: the economic and societal impact of automating millions of jobs (such as truck driver which is the most popular job in America.) This is what the electorate is most worried about.
However the privacy implications of Autonomous Vehicles also need a place in this public debate.
Autonomous Vehicles should be a deliberate societal choice made after the pros and cons have been carefully weighed, including the privacy impact.
The output of this debate should be clear regulation that addresses:
- Which data can be collected from AV video cameras, in which format, what should be anonymised and how?
- Where will this data be stored, shared, and secured, for how long, and what control will the public have on it?
- Who will control the algorithms for processing video information and how will we assess their effectiveness and regulatory compliance? (will judges need to learn to code? Will law become code…?)
As the finishing touches are applied to the technology (Elon Musk expects the first fully autonomous Telsa by 2018) it is time for the policy debate to seriously consider the impact of privacy.
So far in my research, all I have seen in the US on this is one bill introduced by two Democrat senators called “Security and Privacy in Your Car Act of 2015”
Introduced in July 2015, crickets since. (Bill Tracker at Congress.gov)
This bill aims to:
protect against unauthorized access to: (1) electronic controls or driving data, including information about the vehicle’s location, speed, owner, driver, or passengers; or (2) driving data collected by electronic systems built into a vehicle while that data is stored onboard the vehicle, in transit from the vehicle to another location, or subsequently stored or used off-board the vehicle.
As reported here, this bill was introduced in July 2015, and executives from Google, General Motors, Delphi, and Lyft appeared before a US Senate committee on March 15 2015 to discuss it. The executives were asked whether there should be minimum standards for privacy and cybersecurity, and all but one participant declined to say yes or no. Since then, the bill has not been discussed further.
Autonomous Vehicles may bring about a step change in public tracking and surveillance — everything, everywhere continuously recorded, this time with no way to opt out.
I’d love to be proved wrong, but it seems that privacy is a complete afterthought in the push towards our driverless future.
Thanks for reading — feel free to reach out with any thoughts by replying here or on my twitter 👋
Thanks to Christian Moroy, Alex Flamant, and Gabriel Oppetit for great conversations on all this mind blowing stuff (and patiently critiquing this post.)