Patent Application
20220319269
A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent disclosure, as it appears in the Patent and Trademark Office patent files or records, but otherwise reserves all copyright rights whatsoever. Copyright 2019, SG Gaming, Inc.
The present invention relates generally to gaming systems, apparatus, and methods and, more particularly, to authentication and authorization for restricted actions at a gaming device.
At least some gaming devices of the gaming industry are used to provide products or services to players and users without requiring an attendant to be present and fully engaged with the players. In some cases, the gaming devices facilitate providing products or services without any attendants (i.e., unattended devices). Examples of such gaming devices may include, but are not limited to, free-standing electronic gaming machines, lottery terminals, sports wager terminals, and the like.
Although these gaming devices may operate with little to no attendant overview, the gaming devices may provide products or services that may be restricted to one or more potential users. For example, wager-based games and lottery games may be age-restricted in certain jurisdictions. In another example, the gaming devices may enable a user to link a user account and/or digital wallet to a gaming session at the gaming devices, and these features may be limited to the specific user associated with the user account. As a result, security measures may be implemented by the gaming devices to limit or otherwise prevent unauthorized users from accessing such restricted activities.
However, as security measures are implemented, countermeasures may be employed by unauthorized users to access these restricted activities. The unattended (or near unattended) nature of these gaming devices may leave the gaming devices susceptible to unauthorized attempts to access restricted activities. Accordingly, further improvements for securing such gaming devices are required.
According to another aspect of the disclosure, a gaming terminal includes an input device that receives physical user input from a user, an image sensor that captures image data of a user area associated with the gaming terminal and is at a predetermined location relative to the user area, and logic circuitry communicatively coupled to the input device and the image sensor. The logic circuitry detects user input received at the input device and that is associated with a restricted action, receives, via the image sensor, image data that corresponds to the user input, applies at least one neural network model to the received image data to classify pixels of the received image data as representing human characteristics including at least one face and at least one pose model, compares, based at least partially on (i) pixel coordinates of the human characteristics within the received image data and (ii) pixel coordinates of an user input zone within the image data and associated with the detected user input, each of the pose models to the user input zone and the faces, and permits, in response to one of the at least one pose model matching (i) a face of the at least one face and (ii) the user input zone, the restricted action.
According to yet another aspect of the disclosure, a method for authentication a user at a gaming terminal of a gaming system is provided. The gaming system includes at least one image sensor and logic circuitry in communication with the gaming terminal and the image sensor. The method includes receiving, by an input device of the gaming terminal, physical user input from the user and that is associated with a restricted action, receiving, by the logic circuitry via the image sensor, image data that corresponds to the physical user input, applying, by the logic circuitry, at least one neural network model to the received image data to classify pixels of the received image data as representing human characteristics including at least one face and at least one pose model, comparing, by the logic circuitry and based at least partially on (i) pixel coordinates of the human characteristics within the received image data and (ii) pixel coordinates of an user input zone within the image data and associated with the detected user input, each of the at least one pose model to the user input zone and the at least one face, and permitting, by the logic circuitry and in response to one of the at least one pose model matching (i) a face of the at least one face and (ii) the user input zone, the restricted action.
According to one yet another aspect of the present disclosure, a gaming system comprises a gaming terminal including an input device that receives physical user input from a user, an image sensor that captures image data of a user area associated with the gaming terminal and is at a predetermined location relative to the user area, and logic circuitry communicatively coupled to the input device and the image sensor. The logic circuitry detects user input received at the input device and that is associated with a restricted action, receives, via the image sensor, image data that corresponds to the user input, applies at least one neural network model to the received image data to classify pixels of the received image data as representing human characteristics including at least one face and at least one pose model, compares, based at least partially on (i) pixel coordinates of the human characteristics within the received image data and (ii) pixel coordinates of an user input zone within the image data and associated with the detected user input, each of the pose models to the user input zone and the faces, and permits, in response to one of the at least one pose model matching (i) a face of the at least one face and (ii) the user input zone, the restricted action. The gaming system may be incorporated into a single, freestanding gaming machine.
Additional aspects of the invention will be apparent to those of ordinary skill in the art in view of the detailed description of various embodiments, which is made with reference to the drawings, a brief description of which is provided below.
While the invention is susceptible to various modifications and alternative forms, specific embodiments have been shown by way of example in the drawings and will be described in detail herein. It should be understood, however, that the invention is not intended to be limited to the particular forms disclosed. Rather, the invention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the invention as defined by the appended claims.
While this invention is susceptible of embodiment in many different forms, there is shown in the drawings and will herein be described in detail preferred embodiments of the invention with the understanding that the present disclosure is to be considered as an exemplification of the principles of the invention and is not intended to limit the broad aspect of the invention to the embodiments illustrated. For purposes of the present detailed description, the singular includes the plural and vice versa (unless specifically disclaimed); the words “and” and “or” shall be both conjunctive and disjunctive; the word “all” means “any and all”; the word “any” means “any and all”; and the word “including” means “including without limitation.”
For purposes of the present detailed description, the terms “wagering game,” “casino wagering game,” “gambling,” “slot game,” “casino game,” and the like include games in which a player places at risk a sum of money or other representation of value, whether or not redeemable for cash, on an event with an uncertain outcome, including without limitation those having some element of skill. In some embodiments, the wagering game involves wagers of real money, as found with typical land-based or online casino games. In other embodiments, the wagering game additionally, or alternatively, involves wagers of non-cash values, such as virtual currency, and therefore may be considered a social or casual game, such as would be typically available on a social networking web site, other web sites, across computer networks, or applications on mobile devices (e.g., phones, tablets, etc.). When provided in a social or casual game format, the wagering game may closely resemble a traditional casino game, or it may take another form that more closely resembles other types of social/casual games.
The systems and methods described herein facilitate authorization and authentication of users at gaming devices (particularly unattended gaming devices) for restricted actions (e.g., placing a wager, accessing a player account, purchasing a lottery ticket, etc.). More specifically, the systems and methods described herein detect user input associated with a restricted action of the gaming machine and capture image data of a user area associated with a gaming device and perform image analysis to determine whether or not an authorized user is attempting to access the restricted action. The image analysis may include, but is not limited to, applying one or more neural networks to the image data for detecting and classifying one or more potential users, generating a depth map of the user, and the like. If one of the potential users matches the user input, the gaming device may perform the restricted action for the matching user or proceed with other security procedure. If none of the detected potential users match the detected user input, the systems and methods described herein prevent the restricted action from being performed and may escalate authorization for subsequent action, such as issuing an authentication challenge to the user and/or notifying an attendant. The systems and methods described herein facilitate an automatic and dynamic authentication layer to unattended gaming devices that provide additional security against unauthorized access to restricted actions.
Referring to
The gaming machine 10 illustrated in
The input devices, output devices, and input/output devices are disposed on, and securely coupled to, the cabinet 12. By way of example, the output devices include a primary display 18, a secondary display 20, and one or more audio speakers 22. The primary display 18 or the secondary display 20 may be a mechanical-reel display device, a video display device, or a combination thereof in which a transmissive video display is disposed in front of the mechanical-reel display to portray a video image superimposed upon the mechanical-reel display. The displays variously display information associated with wagering games, non-wagering games, community games, progressives, advertisements, services, premium entertainment, text messaging, emails, alerts, announcements, broadcast information, subscription information, etc. appropriate to the particular mode(s) of operation of the gaming machine 10. The gaming machine 10 includes a touch screen(s) 24 mounted over the primary or secondary displays, buttons 26 on a button panel, a bill/ticket acceptor 28, a card reader/writer 30, a ticket dispenser 32, and player-accessible ports (e.g., audio output jack for headphones, video headset jack, USB port, wireless transmitter/receiver, etc.). It should be understood that numerous other peripheral devices and other elements exist and are readily utilizable in any number of combinations to create various forms of a gaming machine in accord with the present concepts.
In the example embodiment, the gaming machine 10 includes a camera 34 that, via the one or more image sensors within the camera 34, captures image data at least of a user area in front of the gaming machine 10. As used herein, the “user area” refers at least to an area in which players are expected to be or intended to be located to operate the gaming machine 10 or other gaming device. The image data may include single images or video data, and the camera 34 may be a depth camera or other form of camera that collects additional sensor data in combination with the image data. In certain embodiments, the gaming machine 10 may include additional cameras 34 and/or cameras 34 positioned in a different configuration around the gaming machine 10. In other embodiments, the gaming machine 10 may not include a camera 34, but rather a separate camera associated with the gaming machine 10 is oriented to capture the user area.
The player input devices, such as the touch screen 24, buttons 26, a mouse, a joystick, a gesture-sensing device, a voice-recognition device, and a virtual-input device, accept player inputs and transform the player inputs to electronic data signals indicative of the player inputs, which correspond to an enabled feature for such inputs at a time of activation (e.g., pressing a “Max Bet” button or soft key to indicate a player's desire to place a maximum wager to play the wagering game). The inputs, once transformed into electronic data signals, are output to game-logic circuitry for processing. The electronic data signals are selected from a group consisting essentially of an electrical current, an electrical voltage, an electrical charge, an optical signal, an optical element, a magnetic signal, and a magnetic element.
The gaming machine 10 includes one or more value input/payment devices and value output/payout devices. In order to deposit cash or credits onto the gaming machine 10, the value input devices are configured to detect a physical item associated with a monetary value that establishes a credit balance on a credit meter such as the “credits” meter 84 (see
Turning now to
The game-logic circuitry 40 is also connected to an input/output (I/O) bus 48, which can include any suitable bus technologies, such as an AGTL+frontside bus and a PCI backside bus. The I/O bus 48 is connected to various input devices 50, output devices 52, and input/output devices 54 such as those discussed above in connection with
The external system 60 includes, in various aspects, a gaming network, other gaming machines or terminals, a gaming server, a remote controller, communications hardware, or a variety of other interfaced systems or components, in any combination. In the example embodiment, the external system 60 may include an attendant device that manages one or more gaming machines 10 and/or other gaming devices. The attendant device may be associated with, directly or indirectly, with a party that deploys and/or manages the gaming machine 10. For example, in a casino environment, the attendant device may be associated with the casino operator. In another example, for a gaming device deployed in a commercial environment (e.g., a lottery terminal deployed at a gas station), the attendant device may be associated with the operator of the commercial environment. In yet other aspects, the external system 60 comprises a player's portable electronic device (e.g., cellular phone, electronic wallet, etc.) and the external-system interface 58 is configured to facilitate wireless communication and data transfer between the portable electronic device and the gaming machine 10, such as by a near-field communication path operating via magnetic-field induction or a frequency-hopping spread spectrum RF signals (e.g., Bluetooth, etc.).
The gaming machine 10 optionally communicates with the external system 60 such that the gaming machine 10 operates as a thin, thick, or intermediate client. The game-logic circuitry 40—whether located within (“thick client”), external to (“thin client”), or distributed both within and external to (“intermediate client”) the gaming machine 10—is utilized to provide a wagering game on the gaming machine 10. In general, the main memory 44 stores programming for a random number generator (RNG), game-outcome logic, and game assets (e.g., art, sound, etc.)—all of which obtained regulatory approval from a gaming control board or commission and are verified by a trusted authentication program in the main memory 44 prior to game execution. The authentication program generates a live authentication code (e.g., digital signature or hash) from the memory contents and compare it to a trusted code stored in the main memory 44. If the codes match, authentication is deemed a success and the game is permitted to execute. If, however, the codes do not match, authentication is deemed a failure that must be corrected prior to game execution. Without this predictable and repeatable authentication, the gaming machine 10, external system 60, or both are not allowed to perform or execute the RNG programming or game-outcome logic in a regulatory-approved manner and are therefore unacceptable for commercial use. In other words, through the use of the authentication program, the game-logic circuitry facilitates operation of the game in a way that a person making calculations or computations could not.
When a wagering-game instance is executed, the CPU 42 (comprising one or more processors or controllers) executes the RNG programming to generate one or more pseudo-random numbers. The pseudo-random numbers are divided into different ranges, and each range is associated with a respective game outcome. Accordingly, the pseudo-random numbers are utilized by the CPU 42 when executing the game-outcome logic to determine a resultant outcome for that instance of the wagering game. The resultant outcome is then presented to a player of the gaming machine 10 by accessing the associated game assets, required for the resultant outcome, from the main memory 44. The CPU 42 causes the game assets to be presented to the player as outputs from the gaming machine 10 (e.g., audio and video presentations). Instead of a pseudo-RNG, the game outcome may be derived from random numbers generated by a physical RNG that measures some physical phenomenon that is expected to be random and then compensates for possible biases in the measurement process. Whether the RNG is a pseudo-RNG or physical RNG, the RNG uses a seeding process that relies upon an unpredictable factor (e.g., human interaction of turning a key) and cycles continuously in the background between games and during game play at a speed that cannot be timed by the player, for example, at a minimum of 100 Hz (100 calls per second) as set forth in Nevada's New Gaming Device Submission Package. Accordingly, the RNG cannot be carried out manually by a human and is integral to operating the game.
The gaming machine 10 may be used to play central determination games, such as electronic pull-tab and bingo games. In an electronic pull-tab game, the RNG is used to randomize the distribution of outcomes in a pool and/or to select which outcome is drawn from the pool of outcomes when the player requests to play the game. In an electronic bingo game, the RNG is used to randomly draw numbers that players match against numbers printed on their electronic bingo card.
In the example embodiment, the gaming machine 10 further includes one or more image sensors 62 that are configured to capture image data, which may be (at least temporarily) stored by the memory unit 44 and/or the storage unit 56. In certain embodiments, the external system 60 may include one or more image sensors that transmit image data to the logic circuitry 40. The image data includes at least one user area of the gaming machine 10 such that image analysis performed of the image data may result in detection of one or more potential users of the gaming machine 10.
The gaming machine 10 may include additional peripheral devices or more than one of each component shown in
It is to be understood that other suitable gaming devices may be configured similar to the gaming machine described with respect to
The camera 304 is configured to capture a user area associated with the terminal 300. More specifically, the user area is defined to include an area in front of the touchscreen 306 where users are expected to be positioned when operating the terminal 300. In certain embodiments, the camera 304 may also be configured to capture at least a portion of the touchscreen 306 to facilitate mapping the location of physical user input at the touchscreen 306 to a location within the image data captured by the camera 304 as described herein.
In some embodiments, the camera 304 may include more sensors than just an image sensor for detecting objects and/or people within the user area. In one example, the camera 304 is a depth or three-dimensional (3D) camera including additional image sensors and/or depth sensors (e.g., infrared sensors) for generating a depth map that provides depth information for each pixel in the image data captured by the camera 304, which may be used to distinguish people and objects from each other and a static background within the image data. In another example, the camera 304 may include a time-of-flight sensor to detect whether or not a potential user is located in a position relative to the terminal 300 that indicates the potential user is operating the terminal 300. That is, the time-of-flight sensor may facilitate distinguishing between people walking by the terminal at a distance from potential users standing next to the terminal 300. In other embodiments, the terminal 300 may include or be in communication with other sensors separate from the camera 304 that assist in the object detection, object classification, and/or authorization performed using sensor data from the camera 304 as described herein.
The touchscreen 306 is configured to present information to users and receive physical user input from the users to interact with the information presented on the touchscreen 306. That is, a user touches (directly or indirectly, such as via a stylus) the touchscreen 306, and the logic circuitry of the terminal 300 (or another processing component associated with the touchscreen 306) detects the coordinates of the touch on the touchscreen via a suitable touchscreen technology (e.g., resistive or capacitive). The coordinates of the touch may be referred to herein as “touch coordinates”, and the touch coordinates may be matched to a portion of the displayed information to determine what, if any, action should be taken in response to the touch. For example, a user at the terminal 300 may order one or more lottery tickets by pressing the touchscreen 306 at a location corresponding to a graphical “ORDER” button.
In the systems and methods described herein, gaming devices such as the gaming machine 10 (shown in
The systems and methods described herein capture image data of a user area for a gaming device, perform image analysis of the captured image data to detect potential users within the user area, and determine whether or not an authorized user is attempting to access a restricted action of the gaming device. If it is determined the authorized user is in fact attempting to access the restricted action, the gaming device performs the restricted action (or proceeds to additional security measures implemented by the gaming device). If not, then the gaming device may escalate to additional authentication challenges and/or notifying an attendant of suspicious activity.
In some embodiments, the logic circuitry that performs at least a portion of the functionality described herein with respect to the gaming devices may be separate from the gaming device. For example, the logic circuitry may be within a server-computing device in communication with the gaming device to receive image data from the gaming device (or a separate image sensor associated with the gaming device) and transmit a message to the gaming device in response to determining the authorization status of the user. The server-computing device may be in communication with a plurality of gaming devices to perform the functionality described herein.
The input device 450 is in communication with the logic circuitry 440 and is configured to receive physical user input from a user. The physical user input may vary according to the form and functionality of the input device 450. For example, a touchscreen may receive touch input, while a button might be pressed, or a joystick may be moved. The input device 450 enables a user to interact with the gaming device 410. In the example embodiment, at least one restricted action of the gaming device 410 may be selectable using the input device 450. That is, the user may provide user input via the input device 450 to prompt the gaming device 410 to perform one or more restricted actions, such as, but not limited to, placing wagers and/or purchasing lottery tickets. The logic circuitry 440 may be configured to detect the physical user input and the selection of a restricted action, which may cause the logic circuitry 440 to initiate an authorization process as described herein.
The image sensor 460 is configured to capture image data of a user area associated with the gaming device 410 and transmit the image data to the logic circuitry 440. The image data may be continuously captured at a predetermined framerate or periodically. In one example, if user input associated with a restricted action is detected, the logic circuitry 440 causes the image sensor 460 to capture the image data. In some embodiments, the image sensor 460 is configured to transmit the image data with limited image processing or analysis such that the logic circuitry 440 and/or another device receiving the image data performs the image processing and analysis. In other embodiments, the image sensor 460 may perform at least some preliminary image processing and/or analysis prior to transmitting the image data. In such embodiments, the image sensor 460 may be considered an extension of the logic circuitry 440, and as such, functionality described herein related to image processing and analysis that is performed by the logic circuitry 440 may be performed by the image sensor 460 (or a dedicated computing device of the image sensor 460).
The logic circuitry 440 is configured to establish data structures relating to each potential user detected in the image data from the image sensor 460. In particular, in the example embodiment, the logic circuitry 440 applies one or more image neural network models during image analysis that are trained to detect aspects of humans. Neural network models are analysis tools that classify “raw” or unclassified input data without requiring user input. That is, in the case of the raw image data captured by the image sensor 460, the neural network models may be used to translate patterns within the image data to data object representations of, for example, faces, hands, torsos etc., thereby facilitating data storage and analysis of objects detected in the image data as described herein. The neural network models may be implemented via software modules executed by the logic circuitry 440 and/or implemented via hardware of the logic circuitry 440 dedicated to at least some functionality of the neural network models.
At a simplified level, neural network models are a set of node functions that have a respective weight applied to each function. The node functions and the respective weights are configured to receive some form of raw input data (e.g., image data), establish patterns within the raw input data, and generate outputs based on the established patterns. The weights are applied to the node functions to facilitate refinement of the model to recognize certain patterns (i.e., increased weight is given to node functions resulting in correct outputs), and/or to adapt to new patterns. For example, a neural network model may be configured to receive input data, detect patterns in the image data representing human faces, and generate an output that classifies one or more portions of the image data as representative of human faces (e.g., a box having coordinates relative to the image data that encapsulates a face and classifies the encapsulated area as a “face” or “human”).
To train a neural network to identify the most relevant guesses for identifying a human face, for example, a predetermined dataset of raw image data including human faces and with known outputs is provided to the neural network. As each node function is applied to the raw input of a known output, an error correction analysis is performed such that node functions that result in outputs near or matching the known output may be given an increased weight while node functions having a significant error may be given a decreased weight. In the example of identifying a human face, node functions that consistently recognize image patterns of facial features (e.g., nose, eyes, mouth, etc.) may be given additional weight. The outputs of the node functions (including the respective weights) are then evaluated in combination to provide an output such as a data structure representing a human face. Training may be repeated to further refine the pattern-recognition of the model, and the model may still be refined during deployment (i.e., raw input without a known data output).
At least some of the neural network models applied by the logic circuitry 440 may be deep neural network (DNN) models. DNN models include at least three layers of node functions linked together to break the complexity of image analysis into a series of steps of increasing abstraction from the original image data. For example, for a DNN model trained to detect human faces from an image, a first layer may be trained to identify groups of pixels that may represent the boundary of facial features, a second layer may be trained to identify the facial features as a whole based on the identified boundaries, and a third layer may be trained to determine whether or not the identified facial features form a face and distinguish the face from other faces. The multi-layered nature of the DNN models may facilitate more targeted weights, a reduced number of node functions, and/or pipeline processing of the image data (e.g., for a three-layered DNN model, each stage of the model may process three frames of image data in parallel).
In at least some embodiments, each model applied by the logic circuitry 440 may be configured to identify a particular aspect of the image data and provide different outputs such that the logic circuitry 440 may aggregate the outputs of the neural network models together to distinguish between potential users as described herein. For example, one model may be trained to identify human faces, while another model may be trained to identify the bodies of players. In such an example, the logic circuitry 440 may link together a face of a player to a body of the player by analyzing the outputs of the two models. In other embodiments, a single DNN model may be applied to perform the functionality of several models.
The inputs of the neural network models, the outputs of the neural network models, and/or the neural network models themselves may be stored in one or more data structures that may be retrieved for subsequent use. In certain embodiments, the logic circuitry 440 may store the inputs and/or outputs in data structures associated with particular potential users. That is, data structures may be retrieved and/or generated for a particular user such the user may be known during subsequent image analysis (via unique human characteristics detected by the neural network models) to the system 400. It is to be understood that the underlying data storage of the user data may vary in accordance with the computing environment of the memory device or devices that store the data. That is, factors such as programming language and file system structures (e.g., FAT, exFAT, ext4, NTFS, etc.) may vary the where and/or how the data is stored (e.g., via a single block allocation of data storage, via distributed storage with pointers linking the data together, etc.). In addition, some data may be stored across several different memory devices or databases.
Although the output of the image neural network models may vary depending upon the specific functionality of each model, the outputs generally include one or more data elements that represent a physical feature or characteristic of a person or object in the image data in a format that can be recognized and processed by logic circuitry 440 and/or other computing devices. For example, one example neural network model may be used to detect the faces of players in the image data and output a map of data elements representing “key” physical features of the detected faces, such as the corners of mouths, eyes, nose, ears, etc. The map may indicate a relative position of each facial feature within the space defined by the image data (in the case of a singular, two-dimensional image, the space may be a corresponding two-dimensional plane) and cluster several facial features together to distinguish between detected faces. The output map is a data abstraction of the underlying raw image data that has a known structure and format, which may be advantageous for use in other devices and/or software modules.
In the example embodiment, applying the image neural network models to the image data causes the logic circuitry 440 to generate one or more key user data elements. The key player user elements are the outputs of the image processing (including the neural network models). Other suitable image processing techniques and tools may be implemented by the logic circuitry 440 in place of or in combination with the neural network models. As described above, the key user data elements represent one or more physical characteristics of the potential users (e.g., a face, a head, a limb, an extremity, or a torso) detected in the image data. The key user data elements may include any suitable amount and/or type of data based at least partially on the corresponding neural network model. At least some of the key user data elements include position data indicating a relative position of the represented physical characteristics within a space at least partially defined by the scope of the image data. The position data may be represented as pixel coordinates within the image data.
The key user data elements may include, but are not limited to, boundary boxes, key feature points, vectors, wireframes, outlines, pose models, and the like. Boundary boxes are visual boundaries that encapsulate an object in the image and classify the encapsulated object according to a plurality of predefined classes (e.g., classes may include “human”, “tokens”, etc.). A boundary box may be associated with a single class or several classes (e.g., a player may be classified as both a “human” and a “male”). The key feature points, similar to the boundary boxes, classify features of objects in the image data, but instead assign a singular position (i.e., pixel coordinates) to the classified features. In certain embodiments, the logic circuitry 440 may include neural network models trained to detect objects other than the players. For example, the logic circuitry 440 may include a neural network model trained to detect display devices (e.g., a smartphone or tablet) that are displaying faces within the image data. In such an example, the logic circuitry 440 may automatically determine that a potential user is attempting to trick the system 400 into providing unauthorized access to the restricted action. The logic circuitry 440 may prevent the restricted action from being performed and/or notify an attendant of the potential fraud.
Although the key user data elements are described above as outputs of the neural network models, at least some key user data elements may be generated using other object detection and/or classification techniques and tools. For example, a 3D camera of the sensor system 106 (shown in
In addition to detecting potential users, the logic circuitry 440 is configured to establish one or more user input zones within the image data. A user input zone is a portion of the image data representing an area including one of the user input devices 450 and/or an area proximal to the user input device 450 from which a user would operate the user input device 450. For example, for a touchscreen, the user input zone may include pixels representing an area near the touchscreen that is likely to be occupied by a user's hand to operate the touchscreen. The user input zone may also capture the touchscreen itself to enable the logic circuitry to identify human characteristics in the image data (e.g., a finger or a hand) that correspond to the detected user input on the touchscreen. The user input zone may be static and predefined, or the user input zone may be at least partially a function of the detected user input. For example, with a touchscreen, user input is detected with touch coordinates indicating an area or point on the touchscreen that the user has selected. The logic circuitry 440 may be configured to map the touch coordinates to input coordinates within the image data to define the user input zone. This variable input zone enables the logic circuitry 440 to accurately detect which user has provided the user input associated with the restricted action even in the event that multiple user inputs are provided simultaneously.
Based on the analysis of the neural network outputs, the logic circuitry 440 is configured to determine whether or not the potential user associated with the user input is an authorized user or, more broadly, whether or not potential suspicious behavior or users are detected. That is, if the outputs of the neural network do not match as expected, this may indicate that the user is attempting access the restricted action fraud or other unauthorized means, such as holding a picture of an authorized user in front of his or her face. In response to detecting the suspicious behavior, the logic circuitry 440 may escalate the authorization process or outright block the restricted action from being performed. Escalating the authorization process may involve one or more actions by the logic circuitry 440 and/or other devices associated with authorizing users, such as an attendant device 401 in communication with the gaming device 410. The actions may include, but are not limited to, presenting an additional authorization challenge to the user (e.g., requesting additional user input), alerting the attendant device 401, emitting audiovisual cues from the gaming device 410 indicating potential suspicious behavior, notifying an authorized user of potential fraud via text messages, email, or phone calls, and the like. These actions may deter the unauthorized user from further attempts or facilitate identification of the unauthorized user. In contrast to an unauthorized user, the system 400 is configured to enable an authorized user to initiate a restricted action via the gaming device 410 with little to no interruption.
In at least some embodiments, the logic circuitry 440 may be further configured to generate annotated image data from the image analysis. The annotated image data may be the image data with at least the addition of graphical and/or metadata representations of the data generated by the logic circuitry 440. For example, if the logic circuitry 440 generates a bounding box encapsulating a hand, a graphical representation of the boundary box may be applied around the pixels representing the hand to the image data to represent the generated boundary box. The annotated image data may be an image filter that is selectively applied to the image data or an altogether new data file that aggregates the image data with data from the logic circuitry 440. The annotated image data may be stored as individual images and/or as video files. Although the following describes the user data elements as graphical objects annotated on an image, it is to be understood that the underlying data elements may be used by the logic circuitry 440 to perform the analysis and authorization processes described herein irrespective of generating the annotated image data. Rather, the annotated image data may be used to facilitate human-comprehension of the underlying data elements generated, analyzed, and stored by the logic circuitry 440 as described herein.
In the example embodiment, the logic circuitry 440 is configured to detect three aspects of players in captured image data: (i) faces, (ii) hands, and (iii) poses. As used herein, “pose” or “pose model” may refer physical characteristics that link together other physical characteristics of a player. For example, a pose of the user 501 may include features from the face, torso, and/or arms of the user 501 to link the face and hands of the user 501 together. The graphical representations shown include a hand boundary box 502, a pose model 504, and a face or head boundary box 506, and facial feature points 508.
The hand boundary box 502 is the output of one or more neural network models applied by the logic circuitry 440. The boundary box 502 may be a visual or graphical representation of one or more underlying key user data elements. For example, and without limitation, the key user data elements may specify coordinates within the frame 500 for each corner of the boundary box 502, a center coordinate of the boundary box 502, and/or vector coordinates of the sides of the boundary box 502. Other key user data elements may be associated with the boundary box 502 that are not used to specify the coordinates of the box 502 within the frame 500 such as, but not limited to, classification data (i.e., classifying the object in the frame 500 as a “hand”). The classification of a hand detected in captured image data may be by default a “hand” classification and, if sufficiently identifiable from the captured image data, may further be classified into a “right hand” or “left hand” classification. As described in further detail herein, the hand boundary box 502 may be associated with the user 501, which is which may be illustrated by displaying a user identifier with the hand boundary box 502.
In the example embodiment, the pose model 504 is used to link together outputs from the neural network models to associate the outputs with a single player (e.g., the user 501). That is, the key user data elements generated by the logic circuitry 440 are not associated with a player immediately upon generation of the key user data elements. Rather, the key user data elements are pieced or linked together to form a player data object as described herein. The key user data elements that form the pose model 504 may be used to find the link between the different outputs associated with a particular player.
In the example embodiment, the pose model 504 includes pose feature points 510 and connectors 512. The pose feature points 510 represent key features of the user 501 that may be used to distinguish the user 501 from other players and/or identify movements or actions of the user 501. For example, the eyes, ears, nose, mouth corners, shoulder joints, elbow joints, and wrists of the user 501 may be represented by respective pose feature points 510. The pose feature points 510 may include coordinates relative to the captured image data to facilitate positional analysis of the different feature points 510 and/or other key user data elements. The pose feature points 510 may also include classification data indicating which feature is represented by the respective pose feature point 510. The connectors 512 visually link together the pose feature points 510 for the user 501. The connectors 512 may be extrapolated between certain pose feature points 510 (e.g., a connector 512 is extrapolated between pose feature points 510 representing the wrist and the elbow joint of the user 501). In some embodiments, the pose feature points 510 may be combined (e.g., via the connectors 512 and/or by linking the feature points 510 to the same player) by one or more corresponding neural network models applied by the logic circuitry 440 to captured image data. In other embodiments, the logic circuitry 440 may perform one or more processes to associate the pose feature points 510 to a particular user. For example, the logic circuitry 440 may compare coordinate data of the pose feature points 510 to identify a relationship between the represented physical characteristics (e.g., an eye is physically near a nose, and therefore the eye and nose are determined to be part of the same player).
At least some of the pose feature points 510 may be used to link other key user data elements to the pose model 504 (and, by extension, the user 501). More specifically, at least some pose feature points 510 may represent the same or nearby physical features or characteristics as other key user data elements, and based on a positional relationship between the pose feature point 510 and another key user data element, a physical relationship may be identified. In one example described below, the pose feature points 510 include wrist feature points 514 that represent wrists detected in captured image data by the logic circuitry 440. The wrist feature points 514 may be compared to one or more hand boundary boxes 502 (or vice versa such that a hand boundary box is compared to a plurality of wrist feature points 514) to identify a positional relationship with one of the hand boundary boxes 502 and therefore a physical relationship between the wrist and the hand.
In the example embodiment, the logic circuitry 440 sets 602 a wrist feature point of a pose model as the hand of interest. That is, the coordinate data of the wrist feature point and/or other suitable data associated with the wrist feature point for comparison with key user data elements associated with hands are retrieved for use in the method 600. In addition to establishing the wrist feature point as the hand of interest, several variables are initialized prior to any hand comparison. In the example embodiment, the logic circuitry 440 sets 604 a best distance value to a predetermined max value and a best hand variable to ‘null’. The best distance and best hand variables are used in combination with each other to track the hand that is the best match to the wrist of the wrist feature point and to facilitate comparison with subsequent hands to determine whether or not the subsequent hands are better matches for the wrist. The logic circuitry 440 may also set 606 a hand index variable to ‘0’. In the example embodiment, the key user data elements associated with each hand within the captured image data may be stored in an array such that each cell within the hand array is associated with a respective hand. The hand index variable may be used to selectively retrieve data associated with a particular hand from the hand array.
At step 608, the logic circuitry 440 determines whether or not the hand index is equal to (or greater than, depending upon the array indexing format) the total number of hands found within the captured image data. For the initial determination, the hand index is 0, and as a result, the logic circuitry 440 proceeds to set 610 a prospective hand for comparison to the hand associated with the first cell of the hand array (in the format shown in
The logic circuitry 440 determines 612 whether or not the wrist feature point is located within the hand boundary box of the hand from the hand array. If the wrist feature point is located with the hand boundary box, then the hand may be considered a match to the wrist and the potential user. In the example embodiment, the logic circuitry 440 may then set 614 the hand as the best hand and return 624 the best hand. The best hand may then be associated with the pose model and stored as part of a user data object of the user (i.e., the hand is “linked” to the user). Returning 624 the best hand may terminate the method 600 without continuing through the hand array, thereby freeing up resources of the logic circuitry 440 for other functions, such as other iterations of the method 600 for different wrist feature points and pose models. In other embodiments, the logic circuitry 440 may compare the wrist feature point to each and every hand prior to returning 624 the best hand irrespective of whether the wrist feature point is located within a hand boundary box, which may be beneficial in image data with crowded bodies and hands.
If the wrist feature point is not determined to be within the hand boundary box of the current hand, the logic circuitry 440 calculates 616 a distance between the center of the hand boundary box and the wrist feature point. The logic circuitry 440 then compares 618 the calculated distance to the best distance variable. If the calculated distance is less than the best distance, the current hand is, up to this point, the best match to the wrist feature point. The logic circuitry 440 sets 620 the best distance variable equal to the calculated distance and the best hand to be the current hand. For the first hand from the hand array, the comparison 618 may automatically progress to setting 620 the best distance to the calculated distance and the best hand to the first hand because the initial best distance may always be greater than the calculated distance. The logic circuitry 440 then increments 622 the hand index such that the next hand within the hand array will be analyzed through steps 610-622. The hand index is incremented 622 irrespective of the comparison 618, but step 620 is skipped if the calculated distance is greater than or equal to the best distance.
After each hand of the hand array is compared to the wrist feature point, the hand index is incremented to value beyond the addressable values of the hand array. During the determination 608, if the hand index is equal to the total number of hands found (or greater than in instances in which the first value of the hand array is addressable with a hand index of ‘1’), then every hand has been compared to the wrist feature point, and the best hand to match the wrist feature point may be returned 624. In certain embodiments, to avoid scenarios in which the real hand associated with a wrist is covered from view of the capture image data and the best hand as determined by the logic circuitry 440 is relatively far away from the wrist, the logic circuitry 440 may compare the best distance associated with the best hand to a distance threshold. If the best distance is within the distance threshold (i.e., less than or equal to the minimum distance), the best hand may be returned 624. However, if the best distance is greater than the distance threshold, the best hand variable may be set back to a ‘null’ value and returned 624. The null value may indicate to other modules of the logic circuitry 440 and/or other devices that the hand associated with the wrist is not present in the captured image data.
In the example embodiment, to initiate the method 700, the logic circuitry 440 may retrieve or be provided inputs associated with a face detected in captured image data. More specifically, key user data elements representing a face and/or head are used to link the face to a pose model representing a body detected in the captured image data. The key user data elements representing the face may include a face or head boundary box and/or face feature points. The boundary box and/or the face feature points may include coordinate data for identifying a location of the boundary box and/or the face feature points within the captured image data. The pose model may include pose feature points representing facial features (e.g., eyes, nose, ears, etc.) and/or physical features near the face, such as a neck. In the example embodiment, the inputs associated with the face include a face boundary box and facial feature points representing the eyes and nose of the face. Each pose includes pose feature points representing eyes and a nose and including coordinate data for comparison with the inputs of the face.
To initialize the method 700, the logic circuitry 440 sets 702 a best distance variable to a predetermined maximum value and a best pose variable to a ‘null’ value. Similar to the hand array described with respect to
The logic circuitry 440 then determines 706 if the pose index is equal to (or greater than for arrays with an initial index value of ‘1’) a total number of poses detected in the captured image data. If the pose index is determined 706 not to be equal to the total number of poses, the logic circuitry 440 progress through a comparison of each pose with the face. The logic circuitry 440 sets 708 the current pose to be equal to the pose stored in the pose array at the cell indicated by the pose index. For the first comparison, the current pose is stored as POSE[0]′ according to the syntax shown in
In the example embodiment, the logic circuitry 440 compares 710 the pose feature points representing a pair of eyes and a corresponding nose to the face boundary box of the face. If the pose feature points representing the eyes and nose are not within the face boundary box, the pose is unlikely to be a match to the face, and the logic circuitry 440 increments 712 the pose index such that the comparison beginning at step 708 begins again for the next pose. However, if the pose feature points are within the face boundary box, the logic circuitry 440 then calculates 714 a distance from the pose feature points and facial feature points. In the example embodiment, Equation 1 is used to calculate 714 the distance D, where left_eyep, right_eyep, and nosep are coordinates of pose feature points representing a left eye, a right eye, and a nose of the pose model, respectively, and where left_eyef, right_eyef, and nosef are coordinates of facial feature points representing a left_eye, a right_eye, and a nose of the face, respectively.
D=|left_eyep−left_eyef|+|right_eyep−right_eyef|+|nosep−nosef| (1)
In other embodiments, other suitable equations may be used to calculate 714 the distance. The logic circuitry 440 then compares 716 the calculated distance to the best distance variable. If the calculated distance is greater than or equal to the best distance, the pose is determined to not be a match to the face, and the pose index is incremented 712. However, if the calculated distance is less than the best distance, the current pose may be, up to this point, the best match to the face. The logic circuitry 440 may then set 718 the best distance to the calculated distance and the best pose variable to the current pose. For the first pose compared to the face within steps 706-718, the first pose may automatically be the assigned as the best pose because the of the initialized values of step 702. The logic circuitry 440 then increments 712 the pose index to continue performing steps 706-718 until every pose within the pose array has been compared. Once every pose has been compared, the pose index will be equal to or greater than the total number of detected poses, and therefore the logic circuitry 440 determines 706 that the method 700 is complete and returns 720 the best pose to be linked to the face.
Unlike the method 600, the method 700 does not include steps to conclude the comparison loop (i.e., steps 706-718) until every pose has been compared to ensure that an early ‘false positive’ within the pose array does not result in the method 700 ending without locating the best possible pose to link to the face. However, it is to be understood that the method 700 may include additional and/or alternative steps to conclude the comparison loop without comparing every pose, particularly in embodiments in which (i) resource allocation of the logic circuitry 440 may be limited due to number of parallel processes, time constraints, etc., and/or (ii) a reasonable amount of certainty can be achieved in the comparison loop that a pose is linked to the face similar to steps 1012 and 1014 in
The method 700 further includes protections against situations in which the body associated with the face is obscured from the captured image data, and the face is erroneously linked to a different pose. More specifically, the comparison 710 requires at least some positional relationship between the pose and the face to be in consideration as the best pose to match the face. If the body associated with the face is obscured, there may not be a pose model associated with the body in the pose array. If every pose ‘fails’ the comparison 710 (i.e., progressing directly to step 712 to increment the pose index), the best pose returned 720 by the logic circuitry 440 may still be the initialized ‘null’ value, thereby indicating a matching pose for the face has not been detected.
The methods 600, 700 of
With respect again to
The user input zone may then be compared to detected hands and/or fingers within the image data to identify which, if any, of the potential users in the image data is associated with the user input. As described in detail herein, if the potential user matching the user input does not also have a matching face, pose, and hand (in addition to another other human features detected by the logic circuitry 440), the logic circuitry 440 may determine that suspicious activity may be occurring. The logic circuitry 440 may then escalate to, for example and without limitation, issuing additional authorization or authentication challenges, notify an attendant or attendant device, and/or block the restricted action from access for a period of time.
In the example embodiment, the logic circuitry 440 detects 802 user input representing a user touching a touchscreen. More specifically, the logic circuitry 440 receives touch coordinates (e.g., for a two-dimensional touch screen, in an (x,y) format) that indicate the location of the touch on the touchscreen. These touch coordinates may be used to determine one or more actions associated with the user input, such as selection of an option displayed on the touchscreen. The logic circuitry 440 then maps 804 the touch coordinates to the pixel coordinates of a user input zone. That is, the logic circuitry 440 translates the touch coordinates from a plane defined by the touchscreen surface (x,y) to a plane defined by the pixels of the image data (u,v), and then forms the user input zone based on the translated pixel coordinates. The user input zone may include the pixel coordinates, but is primarily focus upon pixels representing a physical area in which a hand of the user providing the user input is expected to be within the image data. In the example embodiment, similar to
In addition to establishing the user input zone, several other variables are initialized prior to any hand comparison. In the example embodiment, the logic circuitry 440 sets a best distance value to a predetermined max value and a best hand variable to ‘null’. The best distance and best hand variables are used in combination with each other to track the hand that is the best match to the user input zone and to facilitate comparison with subsequent hands to determine whether or not the subsequent hands are better matches for the user input zone. The logic circuitry 440 also sets 806 a hand index variable to ‘0’. In the example embodiment, the key user data elements associated with each hand within the captured image data may be stored in an array such that each cell within the hand array is associated with a respective hand. The hand index variable may be used to selectively retrieve data associated with a particular hand from the hand array.
At step 808, the logic circuitry 440 determines whether or not the hand index is equal to (or greater than, depending upon the array indexing format) the total number of hands found within the captured image data. For the initial determination, the hand index is 0, and as a result, the logic circuitry 440 proceeds to set 810 a prospective hand for comparison to the hand associated with the first cell of the hand array (in the format shown in
The logic circuitry 440 determines 812 whether or not the user input zone (or a set of pixel coordinates representing the user input zone) coordinates are located within the hand boundary box of the hand from the hand array. If the user input zone coordinates are located with the hand boundary box, then the hand may be considered a match to the user input zone and the potential user. In the example embodiment, the logic circuitry 440 may then set 814 the hand as the best hand and return 824 the best hand. The best hand may then be associated with user input detected in the input zone and stored as part of a user data object of the user for determining the authorization status of the user for a restricted action. Returning 824 the best hand may terminate the method 800 without continuing through the hand array, thereby freeing up resources of the logic circuitry 440 for other functions, such as other iterations of the method 800 for different detected user inputs. In other embodiments, the logic circuitry 440 may compare the user input zone to each and every hand prior to returning 824 the best hand irrespective of whether the user input zone coordinates are located within a hand boundary box, which may be beneficial in image data with crowded bodies and hands.
If the user input zone coordinates are not determined to be within the hand boundary box of the current hand, the logic circuitry 440 calculates 816 a distance D between the center of the hand boundary box and the user input zone coordinates. The logic circuitry 440 then compares 818 the calculated distance D to the best distance variable. If the calculated distance D is less than the best distance, the current hand is, up to this point, the best match to the user input zone. The logic circuitry 440 sets 820 the best distance variable equal to the calculated distance and the best hand to be the current hand. For the first hand from the hand array, the comparison 818 may automatically progress to setting 820 the best distance to the calculated distance and the best hand to the first hand because the initial best distance may always be greater than the calculated distance. The logic circuitry 440 then increments 822 the hand index such that the next hand within the hand array will be analyzed through steps 810-822. The hand index is incremented 822 irrespective of the comparison 818, but step 820 is skipped if the calculated distance is greater than or equal to the best distance.
After each hand of the hand array is compared to the user input zone, the hand index is incremented to a value beyond the addressable values of the hand array. During the determination 808, if the hand index is equal to the total number of hands found (or greater than in instances in which the first value of the hand array is addressable with a hand index of ‘1’), then every hand has been compared to the user input zone, and the best hand to match the user input zone may be returned 824. In certain embodiments, to avoid scenarios in which the real hand associated with the user input zone is covered from view of the capture image data and the best hand as determined by the logic circuitry 440 is relatively far away from the user input zone, the logic circuitry 440 may compare the best distance associated with the best hand to a distance threshold. If the best distance is within the distance threshold (i.e., less than or equal to the minimum distance), the best hand may be returned 824. However, if the best distance is greater than the distance threshold, the best hand variable may be set back to a ‘null’ value and returned 824. The null value may indicate to other modules of the logic circuitry 440 and/or other devices that the hand associated with the user input zone is not present in the captured image data.
As described herein, the methods 600, 700, and 800 shown in
These inconsistencies as determined by the logic circuitry 440 may cause the logic circuitry to identify suspicious access attempts of the restricted action. A suspicious access attempt may not necessarily be an unauthorized user attempting access of the restricted action, but rather is a possibility based on the factors generated and analyzed by the logic circuitry 440. The logic circuitry 440 may then escalate the authorization process to enable authorized users to provide additional data that indicates their authorized status and/or to prevent unauthorized user from gaining access to the restricted action. For example, additional authentication or authorization challenges may be presented at the gaming device 410. The challenges may be as simple as requesting the user repeat the user input while being in clear sight of the image sensor 460, or request additional information, such as biometric data or an identification number or card. An attendant or attendant device may be notified of the suspicious behavior to enable the attendant to selectively permit or prevent the restricted action.
In the example embodiment, the logic circuitry 440, via a user input device 450, receives 902 or detects physical user input from a user and that is associated with a restricted action. For example, the user may provide the user input to initiate a wager or purchase a lottery ticket. In certain embodiments, based on the user input coordinates of the user input, the logic circuitry 440 may establish a user input zone associated with the user input. The logic circuitry 440 then receives 904 image data from one or more image sensors 460. The image data corresponds to the user input such that the image data is captured substantially near to or at the time of the user input. The image data may be received 904 as a stream of video data or as an image that is captured in response to the user input.
The logic circuitry 440 then applies 906 at least one neural network model to the received image data to classify pixels of the received image data as representing human characteristics and/or other features, such as a user input zone. The human characteristics may be categorized at least to represent faces and pose models, where the pose models include feature points representing hands and/or fingers of the poses. In at least some embodiments, the hands and/or fingers may have key user data elements separate from the pose models (e.g., hand boundary boxes). The key user data elements generated from the application 906 of the neural network models include pixel coordinate data that represent a location of the associated with human characteristic within the image data.
The logic circuitry 440 compares 908 the pixel coordinates of the key user data elements representing the human characteristics and the pixel coordinates of the user input zone to match the human characteristics together to form potential users and identify which, if any, potential user is associated with the user input. More specifically, in the example embodiment, each pose model is compared to the faces and the user input zone to identify any matches. In other embodiments in which hands and/or fingers have key user data elements separate from the pose models, the hands and/or fingers may be compared to each pose model (which may still be compared to the detected faces in the image data) and the user input zone.
Based on the comparison, the logic circuitry 440 may determine whether or not to proceed with the restricted action. In the example embodiment, in response to a pose model matching a face and the user input zone, the logic circuitry 440 permits 910 the restricted action. That is, a full set of key user data elements is detected for the user associated with the user input, and therefore is determined to be an authorized user. In at least some embodiments, additional security checks may be performed prior to permitting 910 the restricted action. For example, the user may be required to present an identification card to verify his or her identity before the restricted action is permitted 910. If, however, none of the pose models match both a face and the user input zone, the logic circuitry 440 may escalate 912 the authorization process and/or prevent 914 the restricted action from being performed. Escalating 912 the authorization process may include, but is not limited to, presenting the user with an additional authentication challenge or alerting an attendant (directly or via an attendant device).
In certain embodiments, the method 900 may be performed once for authorized users during a session of operating the gaming device 410 such that subsequent user input associated with restricted actions may automatically be permitted, thereby reducing the computational burden and resource allocation to the method 900. In such embodiments, the key user data elements may be stored for identifying the user in subsequent image data. If a different user is detected provided user input or the session is determined to have concluded, then the method 900 may be initiated for the next user input associated with a restricted action. In other embodiments, the method 900 may be repeated for each and every instance of user input associated with a restricted action.
Each of these embodiments and obvious variations thereof is contemplated as falling within the spirit and scope of the claimed invention, which is set forth in the following claims. Moreover, the present concepts expressly include any and all combinations and subcombinations of the preceding elements and aspects.
This application is a continuation of U.S. patent application Ser. No. 17/063,897, filed Oct. 6, 2020, which claims the benefit of priority to U.S. Provisional Application No. 62/911,755, filed Oct. 7, 2019, both the contents of which are incorporated by reference in their entirety.
| Number | Date | Country | |
|---|---|---|---|
| 62911755 | Oct 2019 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 17063897 | Oct 2020 | US |
| Child | 17834220 | US |
