System and method for controlling surveillance cameras

Abstract

A system and method for controlling surveillance cameras is enclosed. In the system, a surveillance camera system includes surveillance cameras configured as VoIP server endpoints. The surveillance cameras receive messages from client devices such as mobile user devices and computer workstations functioning as VoIP client endpoints. The client devices establish a voice communications channel with the surveillance cameras for receiving and controlling one or more streams of image data sent by the surveillance camera over the voice communications channel to the client device. In a preferred embodiment, the voice communications channel includes a control session based on Session Information Protocol (SIP) and a data channel that is based on Real time transfer protocol (RTP). The data channel preferably operates in half duplex mode, which limits the potential for collisions on the data channel when the surveillance cameras are transmitting their image data to the client devices.

Description

RELATED APPLICATIONS

This application is related to:

U.S. application Ser. No. 15/076,701 filed on Mar. 22, 2016, entitled “Method and system for surveillance camera arbitration of uplink consumption,” now U.S. Patent Publication No.: US2017/0278368A1;

U.S. application Ser. No. 15/076,703 filed on Mar. 22, 2016, entitled “Method and system for pooled local storage by surveillance cameras,” now U.S. Patent Publication No.: US2017/0280102A1;

U.S. application Ser. No. 15/076,704 filed on Mar. 22, 2016, entitled “System and method for designating surveillance camera regions of interest,” now U.S. Patent Publication No.: US2017/0277967A1;

U.S. application Ser. No. 15/076,705 filed on Mar. 22, 2016, entitled “System and method for deadzone detection in surveillance camera network,” now U.S. Patent Publication No.: US2017/0278366A1;

U.S. application Ser. No. 15/076,706 filed on Mar. 22, 2016, entitled “System and method for overlap detection in surveillance camera network,” now U.S. Patent Publication No.: US2017/0278367A1;

U.S. application Ser. No. 15/076,708 filed on Mar. 22, 2016, entitled “System and method for retail customer tracking in surveillance camera network,” now U.S. Patent Publication No.: US2017/0278137A1;

U.S. application Ser. No. 15/076,709 filed on Mar. 22, 2016, entitled “Method and system for modeling image of interest to users,” now U.S. Patent Publication No.: US2017/0277785A1;

U.S. application Ser. No. 15/076,710 filed on Mar. 22, 2016, entitled “System and method for using mobile device of zone and correlated motion detection,” now U.S. Patent Publication No.: US2017/0280103A1;

U.S. application Ser. No. 15/076,712 filed on Mar. 22, 2016, entitled “Method and system for conveying data from monitored scene via surveillance cameras,” now U.S. Patent Publication No.: US2017/0277947A1;

U.S. application Ser. No. 15/076,713 filed on Mar. 22, 2016, entitled “System and method for configuring surveillance cameras using mobile computing devices,” now U.S. Patent Publication No.: US2017/0278365A1.

BACKGROUND OF THE INVENTION

Surveillance camera systems using Internet protocol (IP) surveillance cameras have become increasingly common. These cameras utilize standard protocols over wired and/or wireless networks.

Image data captured by these cameras can be stored a No. of ways. Often the image data is sent for storage in possibly a local network video recording (NVR) system, which may also distribute the image data for display to users. Even more recently, surveillance cameras have been offered that include image data storage on the camera itself. The image data is then accessible by users by directly accessing the surveillance cameras.

In a typical example for accessing image data from an IP surveillance camera, a user interacts with an application on a user device such as a mobile phone. The application is typically browser or app based and enables the user to enter selection information such as the IP address of the surveillance camera as a destination IP address, username and password, and timeframe over which the image data was recorded. This information is then included in a request message, and sent over a surveillance camera network that includes the surveillance camera. In response, the IP surveillance camera either delivers a stream of the selected image data to the user device for real time display of the image data, or provides the image data for downloading to the user device.

SUMMARY OF THE INVENTION

Voice over IP (VoIP) includes technologies for the delivery of voice communications over IP based networks. Businesses have increasingly turned to VoIP as a lower cost alternative to circuit-switched technologies that use the public switch telephone network (PSTN). Technologies employed by VoIP systems include Session Information Protocol (SIP) and Real time Protocol (RTP), in examples. Communicating endpoints in VoIP sessions have traditionally included user devices such a IP telephones, laptops, and mobile phones.

It would be beneficial to provide an implementation of VoIP communications protocols for the surveillance cameras of a surveillance camera system. These protocols are very well developed, and infrastructure exists to support devices using these protocols. Moreover, the protocols enable the setup and transfer of image data from surveillance cameras to any client device that is already enabled to receive VoIP transmissions. In this way, the VoIP enabled surveillance cameras can take advantage of the same bandwidth efficiency and low cost of transmission for transfer of image data to client devices that current VoIP telephony systems leverage.

In general, according to one aspect, the invention features a system delivering image data from a surveillance camera. The system comprises a client device that establishes a voice communications channel, such as a VoIP transmission, for receiving and controlling one or more streams of image data generated by the surveillance camera.

In embodiments, the voice communications channel comprises a control session that the client device establishes with the surveillance camera, local image data storage system and/or cloud image data storage system. The client device controls the one or more streams of image data via the control session, which authorizes the client device.

In implementations, the control session is based upon the Session Initiation Protocol (SIP). The voice communications channel can further comprise a data channel over which the client device receives the one or more streams of image data from the surveillance camera. This data channel can be based upon the Real time Transfer Protocol (RTP) and might be is a half duplex data channel.

Usually, the surveillance camera will be an IP surveillance camera including an application server that establishes the voice communications channel with the client device. In a preferred implementation, the one or more streams of image data are stored within the surveillance cameras.

In one embodiment, an application server of the surveillance camera includes a SIP layer based on an implementation of the Session Initiation Protocol (SIP) for establishing a control session of the voice communications channel, for controlling the one or more streams of image data sent by the surveillance camera to the client device and an RTP layer based on an implementation of the Real Time Protocol (RTP) for establishing a half duplex data channel of the voice communications channel, for sending the one or more streams of image data to the client device.

In general, according to one aspect, the invention features a method for delivering image data from a surveillance camera. This method comprises a client device establishing a voice communications channel and receiving and controlling one or more streams of image data originating from the surveillance camera over the voice communications channel.

In general, according to another aspect, the invention features a method for enabling communications with a surveillance camera. The method comprises a client device establishing a voice communications channel with the surveillance camera, and receiving and controlling one or more streams of image data sent by the surveillance camera over the voice communications channel.

The above and other features of the invention including various novel details of construction and combinations of parts, and other advantages, will now be more particularly described with reference to the accompanying drawings and pointed out in the claims. It will be understood that the particular method and device embodying the invention are shown by way of illustration and not as a limitation of the invention. The principles and features of this invention may be employed in various and numerous embodiments without departing from the scope of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

In the accompanying drawings, reference characters refer to the same parts throughout the different views. The drawings are not necessarily to scale; emphasis has instead been placed upon illustrating the principles of the invention. Of the drawings:

FIG. 1 shows a surveillance camera system according to the present invention;

FIG. 2 is a schematic diagram of an embodiment of a surveillance camera that supports VoIP communications for control and transfer of image data from the surveillance camera to a client device, in accordance with principles of the present invention;

FIGS. 3A and 3B are flow diagrams showing methods for exemplary mobile user device and computer workstation client devices, respectively, where the client devices are configured as VoIP client endpoints that establish a voice communications channel with an IP surveillance camera configured as a VoIP server endpoint, and where the client devices accessing image data from the surveillance camera over the voice communications channel; and

FIG. 4 is a flow diagram showing a method for a VoIP client endpoint device establishing a Session Initiation Protocol (SIP) based control session of the voice communications channel with an IP surveillance camera as a VoIP server endpoint, where the control session enables the VoIP client endpoint device to send control messages for controlling the transmission of image data from the IP surveillance camera and for configuring parameters on the IP surveillance camera.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The invention now will be described more fully hereinafter with reference to the accompanying drawings, in which illustrative embodiments of the invention are shown. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

As used herein, the term “and/or” includes any and all combinations of one or more of the associated listed items. Further, the singular forms including the articles “a”, “an” and “the” are intended to include the plural forms as well, unless expressly stated otherwise. It will be further understood that the terms: includes, comprises, including and/or comprising, when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. Further, it will be understood that when an element, including component or subsystem, is referred to and/or shown as being connected or coupled to another element, it can be directly connected or coupled to the other element or intervening elements may be present.

FIG. 1 shows an exemplary surveillance camera system 10 constructed according to the principles of the present invention.

The system 10 includes IP surveillance cameras 103-1 through 103-N, a computer workstation 224, and possibly a local image data storage system 212 that communicate over a local network 210. The system also possibly includes a cloud image data storage system 310 that is accessible by the surveillance cameras 103 via a network cloud 50.

A user 60-2 holds a mobile user computing device 400 for communicating with the surveillance cameras 103 or the image data storage systems 212, 310. Each mobile user device 400 includes a display screen 410 and one or more applications 412, or “apps.” The apps 412 display information to the user 60-2 via the display screen 410 of the mobile user device 400. The apps 412 execute upon the operating system of the mobile user device 400. The apps 412 communicate with the application server 220 over a network cloud 50 via a wireless connection 264. Examples of mobile user computing devices 400 include smartphones, tablet computing devices, and laptop computers running operating systems such as Windows, Android, Linux, or IOS, in examples.

In the illustrated example, surveillance cameras 103 such as cameral 103-1 and camera2103-2 might be installed within a premises 52. Field of view 105-1 of cameral 103-1 might be directed to capture individuals 60-1 as they enter or exit through a doorway 66 of the premises, for example. Field of view 105-2 of camera2103-2 captures activity near a door 62, in the example.

In one embodiment, image data 250 captured by the camera is stored locally within the respective cameras 103. In another embodiment, the surveillance cameras 103 send their image data 250-2 over the local network 210 for storage on the local image data storage system 212. In one example, the local image data storage system 212 is a network video recorder (NVR). In yet another embodiment, the surveillance cameras 103 send their image data 250-3 over the network cloud 50 for remote storage on the cloud image data storage system 310.

VoIP communications can occur between peer VoIP endpoint devices, also known as VoIP endpoints 263. Each VoIP endpoint 263 includes a unique IP address 262 and possibly a phone No. 260. The phone No. 260 can be private to an organization or local network, or can be a public phone No., in examples. Mobile user device 400 and computer workstation 224 are configured as VoIP client endpoint devices, namely VoIP endpoint1263-1 and VoIP endpoint3263-3, respectfully. Each VoIP client endpoint 263 first establishes a separate voice communications channel 258 with a VoIP server endpoint 263, and then requests image data 250 and/or audio data from the VoIP server endpoint 263 over the associated voice communications channel 258.

VoIP server endpoints 263 can include surveillance cameras 103, the local image data storage system 212, and the cloud image data storage system 310, in examples. In the illustrated example, the VoIP server endpoints 263 are VoIP server endpoint2263-2 (camera1103-1), VoIP server endpoint2263-4 (camera4103-4), VoIP endpoint5263-5 (local image data storage system 212), and VoIP server endpoint6263-6 (cloud image data storage system 310).

In the illustrated example, mobile user device 400 is configured as VoIP client endpoint1263-1 and includes phone No. 260-1 “777-555-2001” and IP address “10.10.10.1.” In a similar vein, computer workstation 224 is configured as VoIP client endpoint3263-3 and includes phone No. 260-3 “777-555-2003” and IP address 262-3 “10.10.10.3.”

Surveillance camera1103-1 is configured as VoIP server endpoint2263-2 and includes phone No. 260-2 777-555-2002” and IP address 262-2 “10.10.10.2.” Surveillance camera2103-2 is configured as VoIP endpoint4263-4 and includes phone No. 260-4 “777-555-2004” and IP address 262-4 “10.10.10.4.” Local image data storage system 212 is configured as VoIP server endpoints 263-5 and includes phone No. 260-5 “777-555-2005” and IP address 262-5 “10.10.10.5,” Finally, cloud image data storage system 310 is configured as a VoIP server endpoint6263-6 and includes phone No. 260-6 “777-555-2006” and IP address 262-6 “10.10.10.6.”

An exemplary voice communications channel 258-1 between VoIP client endpoint1263-1 (mobile user device 400) and VoIP server endpoint2263-2 cameral 103-1 is shown. The voice communications channel 258-1 includes a control session 240 and a data channel 252. The control session 240 enables exchange of control and/or signaling information between the VoIP client endpoint 263-1 and VoIP server endpoint 263-2, while the data channel 252 enables exchange of audio and/or image data 250. To distinguish between control sessions 240 and therefore between the voice communications channels 258, each control session 240 has a unique session ID 256.

The control session 240 is preferably an IP based control session that is based upon the Session Information Protocol (SIP) but can also be implemented according to other standard and proprietary signaling protocols. Other VoIP signaling protocols include IAX, H.323, and XMPP, in examples. Control functions provided by the control session 240 include call setup functions between the endpoints 263-1/263-2 that are analogous to a traditional telephone such as the initial call request, confirmation of call request, termination of call (SIP request messages INVITE, ACK, and BYE, respectively), and ringing of a phone and busy tone (SIP response message codes 180 “Ringing” and 486 “Busy Here”) in examples. Other control functions provided by the control session 240 include commands sent from VoIP client endpoint1263-1 for requesting available resources on the cameral 103-1 such as selected frames of image data 250 and available disk space, in examples.

The data channel 252 is preferably implemented as a half duplex data channel and is preferably based upon the real-time transfer protocol (RTP). For this reason, the data channel 252 can also be referred to as an RIP session between two VoIP endpoints 263. A half duplex implementation for the data channel 252 is preferred to maximize the likelihood that image data 250 is transferred at maximum throughput with minimum loss over the voice communications channel 258. This is because when the VoIP server endpoint 263-2 is transmitting its image data of the current stream over the data channel 252 to the VoIP client endpoint 263-1, the VoIP client endpoint 263-2 cannot send RTP messages to the VoIP server endpoint 263-2 until the VoIP server endpoint 263-2 has completed transmission of the image data 250.

When configured as VoIP server endpoints 263, the surveillance cameras 103 function as video phones that automatically answer calls initiated by authenticated VoIP client endpoints 263. The authentication (e.g. authorization) is typically provided by the control session 240 portion of the voice communications channel 258. Upon establishing the authenticated control session 240, audio and/or image data is transmitted from the surveillance cameras 103 to the VoIP client endpoints 263-1/263-3 via the data channel 252. The data channel 252 is preferably operated in half duplex mode to avoid sending unwanted video and/or audio to the cameras 103 while the cameras 103 are transmitting their audio and/or image data to the VoIP client endpoints 263.

The control session 240 is preferably based upon an implementation of SIP. In this way, the VoIP client endpoints 263 can use existing SIP commands to control the streams of image data 250-1 from the surveillance cameras 103 in a manner similar to the way in which VoIP client endpoints 263 control video from VoIP server endpoints 263 of a traditional VoIP video conferencing call, in one example.

For the SIP based implementation of the data channel 252, the VoIP server endpoint 263-2 assembles the image data 250 into RTP packets for transmission over the data channel 252. In examples, the RTP based data channel 252 can support best effort delivery over UDP and/or reliable delivery over TCP. Best effort delivery is favored for several reasons. These reasons include the fact that reliable delivery often causes significant delays due to retransmission of lost frames, and that reliable delivery consumes more bandwidth than best effort delivery. Finally, because the content of frames of image data 250 from surveillance cameras typically do not change much from frame to frame, a small amount of image data loss inherent when using best effort delivery is most often an acceptable tradeoff in favor of the improved transmission speed and bandwidth utilization provided by best effort delivery.

FIG. 2 shows some of the components of an exemplary surveillance camera 103-4. In the example, the surveillance camera 103 stores its image data 250 locally and includes an integrated application server 220 as a process. Surveillance camera 103-4 is also configured as a VoIP server endpoint 263-4.

The camera 103 includes a processing unit (CPU) 138, an imager 140, a camera image data storage system 174 and a network interface 142. An operating system 136 runs on top of the CPU 138. The surveillance camera 103 saves image data 250 captured by the imager 140 to the camera image data storage system 174. Each camera 103 can support one or more streams of image data 250. The application server 220 receives and sends messages 264 via its network interface 142.

A No. of processes or software applications are executed by the operating system 136. The processes include an application server 220. The application server 220, in turn, includes a SIP based control session 240 implementation, or SIP layer 132, and an RTP based data channel 252 implementation, or RTP layer 134. Because the application server 220 enables the surveillance camera 103-4 to function as a VoIP endpoint 263, the application server 220 also maintains a phone No. 260-4 and an IP address 262-4 for the VoIP server endpoint 263-4.

The application server 220 preferably includes a SIP layer 132 based on an implementation of the Session Initiation Protocol (SIP) for establishing an IP based control session 240 of the voice communications channel 258-1, for controlling the one or more streams of image data 250-1 sent by the surveillance camera 103 to the client device 400/224. The surveillance camera 103 operates as a VoIP server endpoint 263-1, where the client device 400/224 operates as a VoIP client endpoint 263-1.

The application server 220 also preferably includes a RTP layer 134 based on an implementation of the Real Time Protocol (RTP) for establishing a half duplex data channel 252 of the voice communications channel 258-1, for sending the one or more streams of image data 250-1 to the client device 400/224.

FIG. 3A shows a method for a mobile user device 400 configured as a VoIP client endpoint 263-1 to access image data 250 from an IP surveillance camera 103 configured as a VoIP server endpoint 263.

In step 502, a mobile user device 400 establishes a Session Initiation Protocol (SIP) based control session with an IP-enabled surveillance camera 103, where the user mobile computer device and the surveillance camera 103 are VoIP/SIP endpoints 263. In step 504, the mobile user device 400 receives image data 250 over a half duplex data channel 252, where the image data 250 is included within Real time Transport Protocol (RTP) packets sent from the surveillance camera 103. Then, in step 506, the mobile user device 400 renders the image data 250 on the display screen 410 of the user mobile device 400.

According to step 508, the method tests whether the control session 240 was terminated by either endpoint (e.g. via a SIP “BYE” request message of the control session 240). If this resolves to true, the method terminates in step 510. Otherwise, the method transitions to the beginning of step 504 to process the next frame of image data 250 to be sent over the half duplex data channel 252 to the client device 400.

FIG. 3B shows a method for a computer workstation 224 configured as a VoIP client endpoint 263-1 to access image data 250 from an IP surveillance camera 103 configured as a VoIP server endpoint 263.

In step 532, a computer workstation 224 establishes a Session Initiation Protocol (SIP) based control session with an IP-enabled surveillance camera 103, where the computer workstation 224 and the surveillance camera 103 are VoIP/SIP endpoints 263. In step 534, the computer workstation 224 receives image data 250 over a half duplex data channel 252, where the image data 250 is included within Real time Transport Protocol (RTP) packets sent from the surveillance camera 103. Then, in step 536, the computer workstation 224 saves the image data 250-2 over the local network 210 to the local image data storage system 212. It can also be appreciated that the computer workstation 224 can alternatively save the image data 250-3 over the network cloud 50 to the cloud image data storage system 310.

According to step 538, the method tests whether the control session 240 was terminated by either endpoint (e.g. via a SIP “BYE” request message of the control session 240). If this resolves to true, the method terminates in step 540. Otherwise, the method transitions to the beginning of step 534 to process the next frame of image data 250 sent over the half duplex data channel 252 to the client device 224.

FIG. 4 shows a method for a VoIP client endpoint 263 device establishing a Session Initiation Protocol (SIP) based control session 240 with an IP surveillance camera 103 as a VoIP server endpoint 263 and sending control messages over the control session 240. The control session 240 is one component of a separate voice communications channel 258 established between each of the VoIP client endpoints 263 and each VoIP server endpoint 263.

In step 552, the client device 224/400 establishes a Session Initiation Protocol (SIP) based control session 240 with an IP-enabled surveillance camera 103 configured as a VoIP server endpoint 263. In step 554, the client device 224/400 sends control messages over the SIP connection/control session 240 to the surveillance camera 103. In one example, the control messages query the surveillance camera 103 to obtain the list of current resources and available functions provided by the surveillance camera 103. In another example, the control messages control the transmission of image data 250 from and configure settable parameters on the surveillance cameras (e.g. zoom setting, resolution of image data 250).

In examples, the control messages for controlling the streams of image data 250 include commands for cancelling requests for image data 250, terminating a call to tear down the voice communications channel 258, and setting parameters on the surveillance camera 103. Control messages based on the SIP request message types CANCEL, BYE, and MESSAGE types, respectively, can be utilized for these purposes, in one implementation.

In other example, the list of current resource information can include the No. of streams of image data 250 supported, the disk space available on the camera image data storage system 174, codec information of the imager 140, software versions of the SIP layer 132 and/or RTP layer 134 within the application server 220 of the surveillance camera 103. A control message based on the SIP request message types OPTIONS and/or INFO can be utilized for these purposes, in one implementation.

According to step 556, the client device 224/400 receives a SIP response message in response to the requested control commands (e.g. Success, Error). In step 558, the method tests whether the control session 240 was terminated by either endpoint (e.g. via a SIP “BYE” request message of the control session 240). If this resolves to true, the method terminates in step 560. Otherwise, the method transitions to the beginning of step 554 to process the next control message sent over the SIP based control session 252.

While this invention has been particularly shown and described with references to preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the invention encompassed by the appended claims.

Claims

1. A system delivering image data, comprising: a surveillance camera;a client device that establishes a voice communications channel for receiving and controlling one or more streams of image data generated by the surveillance camera; andan application server of the surveillance camera that includes: a Session Initiation Protocol (SIP) layer implementing SIP for establishing a control session of the voice communications channel, wherein the SIP layer automatically answers a call from the client device based on the client device being authenticated and the control session controls the one or more streams of image data sent by the surveillance camera to the client device in response to the call; anda Real Time Protocol (RTP) layer implementing RTP for establishing a half duplex data channel of the voice communications channel, for sending the one or more streams of image data to the client device.

2. The system of claim 1, wherein the client device controls the one or more streams of image data via the control session.

3. The system of claim 1, wherein the control session authenticates the client device.

4. The system of claim 1, wherein the image data of the one or more streams of image data are stored within the surveillance camera.

5. A method for dclivcringreceiving image data from a surveillance camera, the method comprising: establishing, with the surveillance camera, a voice communications channel for receiving one or more streams of image data generated by the surveillance camera;establishing, with a Session Initiation Protocol (SIP) layer of an application server of the surveillance camera, a control session of the voice communications channel in response to a call automatically answered by the SIP layer, wherein the control session controls the one or more streams of image data received from the surveillance camera and a Real Time Protocol (RTP) layer implementing the RTP for establishing a half duplex data channel of the voice communications channel;receiving, via the half duplex data channel of the voice communications channel, the one or more streams of frames of image data originating from the surveillance camera over the voice communications channel using the SIP layer.

6. The method of claim 5, further comprising the data channel referencing a session ID of the control session.

7. The method of claim 5, further comprising authenticating to the surveillance camera via the control session.

8. The method of claim 5, wherein controlling the one or more streams of frames of image data comprises the client device sending control messages to the surveillance camera via an IP based control session of the voice communications channel.

9. The method of claim 5, wherein receiving the one or more streams of frames of image data sent by the surveillance camera comprises the client device receiving the streams of the frames of image data from the surveillance camera over a data channel of the voice communications channel.

10. A method for a mobile user device to access image data from a surveillance camera, the method comprising: configuring the mobile user device as a voice over Internet protocol (VoIP) client endpoint;configuring the surveillance camera as a VoIP server endpoint that automatically answers a call from the mobile user device based on the mobile user device being authenticated;establishing, based at least in part on the call, a Session Initiation Protocol (SIP) based control session with the surveillance camera;receiving, by the mobile device, a frame of image data over a half duplex data channel, where the frame of image data is included within Real time Transport Protocol (RTP) packets sent from the surveillance camera;rendering, the mobile user device, the frame of image data on a display screen of the user mobile device; andprocessing another frame of image data to be sent over the half duplex data channel to the mobile user device by receiving the another frame of image data within Real time Transport Protocol (RTP) packets sent from the surveillance camera.