Intellectual Property in the Real World
According to an analysis conducted in 2014 of publicly available data provided by the U.S.P.T.O. (Greenbaum, 2014a), virtual reality patents began to appear in 1992. Since many virtual reality (VR) patents are more than 10 years old, some may be unenforceable due to unpaid maintenance fees; however, there are also manyvalid patents that are waiting to be infringed upon.
VR patents are held by various companies, with IBM, Canon, Sony, Phillips, Samsung, and Microsoft accounting for approximately 65% of issued patents. Nintendo, Sony, Microsoft, Konami, and Namco hold the most VR gaming patents. Patent transactions are also occurring, with Geomagic, Avaya, and Imersion Corp. as the top buyers of VR patents.
In Virtual Reality: Patent Landscape Analysis, LexInnova (2015) identifies Sony, Samsung, and Microsoft as leaders, with a large patent portfolio and frequent patent filing activity. Most patent filings have occurred in the United States, followed by South Korea and China.
What is being patented? TechRepublic published a list comprising “10 Cool Virtual Reality Patents from the Past 25 Years,” showing the breadth of these patents:
- Optical system for virtual reality helmet
- Data-suit for real-time computer animation and virtual reality applications
- Visual field perimetry using virtual reality glasses
- Virtual reality generator for use with financial information
- Virtual reality system for treating patients with anxiety disorders
- Virtual reality headset (granted to Oculus Rift)
- Virtual reality GTAW and pipe welding simulator and setup (a VR welding simulator, with the welding mask acting as the headset)
- Virtual reality glove system with fabric conductors
- Virtual reality interactivity system and method
- Scalable virtual world chat client-server system
What is being litigated? Some patent issues arising in virtual and augmented reality will be familiar to anyone who follows technology law. For example, Oculus has been sued by game publisher ZeniMax Mediaalleges that Oculus poached ZeniMax employees to misappropriate trade secrets. As of this writing, the case is likely to go to trial.
User interface patents will continue to be a focal point, similar to what has occurred in the smartphonepatent wars. Apple v. Samsung provides a good model of what to expect and the ramifications and repercussions involved. As I discussed in a recent article, Apple viewed Samsung’s perceived infringement of their iOS user interface as a serious issue, because they view their user experience as key to their success in the growing mobile market. Similarly, if a dominant user interface emerges in virtual or augmented reality, patent holders will fight to protect it, with the stakes getting higher as acceptance of virtual and augmented reality grows. Precedents established for current software interfaces are likely to be applied.
InGlasses.com v. Ditto, a case in which I was involved as an expert witness, the issue was infringement over anaugmented reality technology for trying on glasses at home. The technology provided a virtual fitting room, where people could see how they would look wearing many different glasses. In an interesting arrangement, Ditto partnered with patent consultant IPNav who, in return for covering Ditto’s legal costs, will receive equity in Ditto if they win the case (Wohlsen, 2013). I was involved with a similar virtual try on case for the makeup company Sephora.
“Patent trolling” will continue to be a factor as well. In a series of well-known cases, non-practicing entity Lennon Image Technologies filed against thirteen retailers for patent infringement (including Ditto, in a separate case), regarding the retailers’ use of augmented reality to allow customers to virtually try on clothes.
I also was an expert witness working for Nintendo on theARCzar v. Nintendo case, where the non-practicing entity ARCzar sued Nintendo for a function Nintendo used to provide an augmented reality experience. This case settled.ARCzar’s ‘936 patent had one claim that was so broad that it apparently claimed that they own the entire field of augmented reality, covering among other things:
A method of displaying a graphical user interface in a computer vision system comprising the steps: addressing a scene with a computer vision system; forming an image of the scene; generating a graphical user interface; combining the image of the scene with the graphical user interface to form a composite image; and displaying the composite image in a display field.
In another case involving a seemingly generic patent, Recognicorp, LLC, filed a complaint for patent infringement against CCP hf of Reykjavik, Iceland, for infringement for “character creation and customization that allow the selection of a facial feature, incorporation of the facial feature into a composite image, and instructions for modifying the selected facial feature image that has been incorporated into the composite image”(Heard, 2012).
Similarly, Impulse Technology, Ltd., has sued Microsoft and eight video game makers over motion tracking technology used in virtual reality that allegedly infringes on seven Impulse Technology patents. The patents are described as covering a “wide variety of games where the movement of a player is tracked in three dimensions … and certain exercise games where the motion of the player is tracked to effect movement of a virtual avatar, and the exertion of the user is monitored, including where the tracking of the player is done by use of a camera”(Coe, 2011).
In Abarta, LLC, v. Sensics, Inc., Alberta claims:
Defendant infringes ‘326 patent by providing customers a reality system designed to provide a 360 degree viewing experience, including tracking movement in the X and Y directions, to display portions of corresponding images. The directional sensors are embedded in the eye displays included with the Defendant’s system, to track the user’s movements that correspond with imagery used in the virtual environment and viewed on the display. The SmartGoggles System detects movement through multiple trackers and displays a portion of the environment as a function of the X and Y directions. (Abarta, LLC, v. Sensics, Inc., 2012)
I have not conducted detailed analyses of these cases (aside from the above cases that I was involved in) and can’t comment on them in detail, but they are a strong indicator that many fundamental virtual and augmented reality technologies are already considered protected intellectual property and worth fighting for in the courtroom. These cases as well as otherpatent disputes relating to virtual and augmented reality, will becomeasprevalent as they already are in smartphones and mobile technology.
Intellectual Property in Virtual and Augmented Worlds
Intellectual property law will increasingly be applied to activities occurring in virtual worlds, raising questions of how real-world copyrights and trademarks will apply. Here’s an overview of some of the issues that may arise and have arisen (many of the examples here were taken from Wassom (2014)).
Some questions are fundamental. For instance, if I build and sell iPhones in Second Life, have I created a counterfeit product?If so, have I harmed the real-world manufacturer? Or have I provided free advertising for the real-world manufacturer? Companies will need to evaluate these issues, and decide how to best respond – ignore the “infringement,” take action against the infringing party, or even get into the game themselves and sell a virtual version of the product themselves, as many manufacturers do. Going forward, they also need to determine their own patent strategy to best protect themselves.
Questions of public display may also arise. If a physical location triggers an image in an augmented reality application, does that constitute a public display of that image? Enforcement may also prove challenging, as virtual worlds are constantly changing and are unique to what one individual views. In this environment, it may not be clear who saw what, from what angle, and when they saw it.
Copyright laws restrict rights of reproduction and alteration. Augmented reality will test these restrictions by providing new ways of creating derivative works and will also push the fair use doctrine. On the other hand, at some point in the future, augmented reality eyewear could conceivably enforce copyright by alerting the viewer of improper display of copyrighted content or even obscuring the content.
Trademark infringement will also become more complex, particularly when an augmented reality application overlays trademarked content on the real world. Trademarking will need to be re-evaluated; for example, whether a logo is used “in commerce” when appearing in a virtual or augmented application. The virtual information may be copyrighted or derived from copyrighted material, and may be viewed in situations where there is no precedent in the physical world. For example, consider an augmented reality application that recognizes a poster and overlays content, changing the way the poster appears to the virtual viewer. Has the poster been altered? There is an iPhone app called The Leak in Your Home Town that does exactly this. The app was inspired by the BP Deepwater Horizon oil spill. When the app recognizes a BP logo, it overlays the logo with a generated image of a pipe spewing oil.
Augmented reality may also change how trademarking is used. Corporate logos could easily trigger augmented content (similar to how QR codes currently trigger content). This could lead to an augmented world saturated with advertising – one satirical video shows a vision of a highly branded world, with logos and advertisements overlaid on all surfaces. However, what if the logo triggers content from a competitor or a third party, like the BP app I mentioned above. To what extent will trademark holders want to secure their branding in the virtual world?
Product liability issues are also likely to arise. Virtual reality systems that track full-body motion can require a lot of space. Wall Street Journal technology reviewer Geoffrey Fowler (2016) reports that the HTC Vive requires “an unobstructed 15-by-15 foot patch” and describes “crawling around on the floor with a Vive headset on, jabbing at the air with an 8-inch wand. Inside my headset, I was floating in the sea, probing an anemone.”Virtual reality users, while immersed in their virtual worlds, are at least somewhat oblivious to the real world around them. The Vive does include safeguards to alert users if they are nearing wall or other obstruction, but I believe we will soon see YouTube “fail” videos of headset-wearing gamers, tripping, falling, stubbing toes, and so on, because they cannot see their surroundings. Liability issues are certain to follow.
Virtual reality itself might present hazards, as it affects us cognitively, and can lead to motion sickness and even dizziness, seizures, or blackouts. Stress reactions are also possible, and become more likely the more realistic and immersive the virtual experience becomes. It is not hard to envision a lifelike simulation triggering already engrained phobias such as claustrophobia or fear of heights. It is also possible to envision players of a violent virtual game experiencing a post-traumatic stress reaction.
Virtual reality may also encourage bad, and possibly criminal, behavior among users. We know that bullying, intimidation, and hateful speech are sadly prevalent in gaming and other virtual forums today.There is no reason to believe such behavior will not continue in virtual reality worlds, where it may become even more serious in the more realistically presented environment. Similarly, the realism of virtual reality games may give new credence to theories that violent virtual experiences (e.g., video games) lead to violence in the real world.These effects may not be evident immediately, so we may see lawsuits years in the future arising from repercussions of today’s and tomorrow’s virtual reality.
In these cases, the hardware manufacturer, the creator of the content, and provider of the content may all be considered liable.
Privacy will obviously be a concern as well. One issue is that virtual and augmented reality will contribute further to growing corporate data collection. Oculus has already been hit by allegations that it is collecting too much data (Hern, 2016):
Oculus’ relationship with Facebook may make them particularly susceptible to such concerns if information from Rift further adds to Facebook’s vast collection of data about their users.
Other concerns are more Orwellian. Facial recognition is already quite advanced. Our faces, or even our entire bodies, could trigger automatic searches of information about us – information from social media, our address, our credit history, criminal records, etc. A world where an augmented reality eyewear recognizes people on the street and overlays their address, credit history, or criminal record would be frightening indeed.
As I mentioned at the end of my articleon the upcoming virtual reality patent wars, it is truly a brave new world.
Dr. Craig Rosenberg is an entrepreneur, human factors engineer, computer scientist, and expert witness. You can learn more about Dr. Rosenberg and his expert witness consulting business atwww.ui.expert
- Abarta, LLC, v. Sensics, Inc., Case 2:12-cv-00681 (United States District Court for the Eastern District of Texas, Marshall Division October 23, 2012).
- Coe, E. (2011, July 5). Microsoft, Others Sued Over Motion Tracking IP. Retrieved April 9, 2016, from Law360: http://www.law360.com/articles/255776/microsoft-others-sued-over-motion-tracking-ip
- Fowler, G. (2016, April 5). HTC Vive Review: Oculus Rift’s Rival Makes VR Bigger. Retrieved April 7, 2016, from The Wall Street Journal: http://www.wsj.com/articles/htc-vive-review-oculus-rifts-rival-makes-vr-bigger-1459865158
- Greenbaum, E. (2014a, January 23). A Virtual reality Patent Landscape Analysis. Retrieved March 23, 2016, from Greenbaum Patent: https://greenbaumpatent.wordpress.com/2014/01/23/a-virtual-reality-patent-landscape-analysis/
- Greenbaum, E. (2014b, February 19). Oculu vs. Oculus: Complaint. Retrieved March 23, 2016, from Greenbaum Patent Blog: https://greenbaumpatent.wordpress.com/2014/02/19/oculu-vs-oculus-complaint/
- Heard, P. (2012, July 16). Recognicorp Files New Patent Infringement Action Against CCP Over Popular Video Game Eve Online. Retrieved April 9, 2016, from Womble Carlyle Blogs: http://georgiaiplit.blogspot.com/2012/07/recognicorp-files-new-patent.html
- Hern, A. (2016, April 7). Facebook’s Oculus hit by fears it will spy on users and pass on data. Retrieved April 7, 2016, from The Guardian: https://www.theguardian.com/technology/2016/apr/07/facebook-oculus-virtual-reality-fears-spy-on-users-pass-on-data
- Le, T. P. (2013, May). More Than Just a Game. Retrieved April 1, 2016, from Washington Lawyer: https://www.dcbar.org/bar-resources/publications/washington-lawyer/articles/may-2013-virtual-game.cfm
- LexInnova. (2015). Virtual Reality: Patent Landscape Analysis. San Jose: LexInnova Technologies.
- Wassom, B. D. (2014, January/February 23). IP in an Augmented Reality. Retrieved 2016 March, from Landslide: A Publication of the ABA Section of Intellectual Property Law: http://www.americanbar.org/publications/landslide/2013-14/january-february/ip_an_augmented_reality.html
- Wohlsen, M. (2013, September 10). Patent Trolls Are Killing Startups — Except When They’re Saving Them. Retrieved April 7, 2016, from Wired: http://www.wired.com/2013/09/patent-trolls-versus-startups/