Building an IP surveillance camera system with a low-cost FPGA

3A newly introduced reference design applies FPGA-based signal processing in an inexpensive solution for wide dynamic range IP cameras with some fairly sophisticated techniques inside.

Today’s video surveillance marketplace needs higher-quality video, higher resolution, more flexibility, and new features to drive the change from cameras. By definition, HD video must be digital, so the adoption of HD standards is symbiotic with the move from to digital cameras. The higher frame rates and resolution supported by HD video standards require newer compression techniques such as H.264, which in turn require greater processing power in the camera.

The need to extract maximum image content in a wide range of lighting conditions (simultaneous high- and low-light levels) has led to surveillance cameras adopting a new class of Wide Dynamic Range (WDR) image sensors, which require within the camera, again in the digital domain.

IP Surveillance Camera Reference Design

A top-level block diagram of the IP Surveillance Camera targeting the new generation of HD (>1 MP) WDR sensors is shown in Figure 1.

Figure 1: The IP Surveillance Camera Reference Design based on the Cyclone III EP3C120 development board targets the next generation of HD WDR sensors.
(Click graphic to zoom by 1.9x)

The hardware platform for the reference design is based on the Cyclone III EP3C120 development board. Pixel data from Aptina’s WDR sensor is fed into the Image Sensor Pipeline (ISP) from Apical. The output data is YUV 4:2:0 format and written to a frame buffer in the external DDR2 SDRAM using components from Altera’s Video and suite. Video data is encoded in H.264 format using a core from EyeLytics, supporting baseline/main profile level 3 for streaming via Ethernet to be viewed on a remote host. A scatter-gather DMA controller supports Altera’s Triple-Speed Ethernet (TSE) MegaCore function for streaming the encoded video to a remote client over Ethernet.

WDR sensor

The sensor is Aptina’s 0.33" 720p60 WDR CMOS image sensor. The sensor and lens assembly are mounted on a headboard connected via an I/O transposer board to the development board.

Image sensor pipeline

WDR sensors have no on-chip image pipeline processing and output the image data in RAW/Bayer format at up to 20 bits per pixel. This data makes it difficult to connect this new generation of sensors to the ASSPs commonly used in surveillance solutions. Therefore, are an ideal choice for efficiently processing this data. The ISP includes the following functions:

·    Hot pixel removal and (spatial and temporal IP cores are available)

·    Advanced per pixel tone mapping

·    Advanced demosaic and color correction

Compared to the unprocessed image on the left of Figure 2, the image on the right shows how the Iridix tone-mapping core allows maximum detail to be extracted from a high-contrast scene. The dark areas in particular are revealed without corresponding overexposure in the bright areas.

Figure 2: Iridix dynamic range compression enables maximum detail to be extracted from high-contrast scenes.
(Click graphic to zoom by 1.9x)

Video compression

The H.264 encoder used in this design is an IP core that contains surveillance features such as multichannel support, constant quality rate control, intra/intermodes, QPEL, context-adaptive binary arithmetic coding/context-adaptive variable-length coding, and a low gate count. The core supports both main and baseline profiles.

Triple-speed Ethernet MAC

The TSE MegaCore function combines the features of a 10/100/1000 Mbps Ethernet MAC and a 1000BASE-X physical coding sublayer with an optional physical medium attachment. The development board includes a 10/100/1000BASE-T, autonegotiating Ethernet PHY with a reduced gigabit media-independent interface.

Avalon bus fabric and DDR2 frame buffer memory

A single bank of external DDR2 SDRAM with a 32-bit data bus controlled by the DDR and DDR2 SDRAM high-performance Controller MegaCore Function is used for application code and data storage, input and output frame buffers, and intermediate frame buffers for the H.264 encoder. Avalon arbitration shares are applied to each Avalon-MM bus master connection to the DDR2 memory controller, guaranteeing efficient and uninterrupted access for bursts of data to and from the H.264 encoder.

Software applications

A Nios II embedded processor is used for programming the various registers within the different modules, as well as for running the RTP stack to stream the video. Working with the Ethernet MAC module to control the ISP, the embedded processor runs the uC/OS-II real-time kernel from Micrium, NicheStack and RTP stack from InterNiche Technologies, and a video-streaming application and an embedded Web server application. The processor also handles auto exposure and auto white balance control of the ISP.

Host software

A host PC running the variable-length coding media player (or similar) is used to view the streamed video output of the reference design. The only required connection between the host and the development kit is an Ethernet cable.


When implemented on this development board, the reference design runs with an H.264 encoder core and DDR2 memory clock frequency of 150 MHz, sufficient to compress frames of 720p30 video using a H.264 baseline or main profile. The embedded processor and TSE are clocked at 125 MHz.


The latency from the sensor input of the ISP to the output of the H.264 encoder is less than two frames and results from the double buffering of the image data. A new input frame is always being written to memory while the previous frame is being encoded.

Power consumption

The total power consumption of the reference design including all of the ancillary blocks and I/O is 2.7 W. Table 1 shows the power consumption for each of the major blocks used in the reference design. The remaining 700 mW is consumed by the Avalon bus fabric, color space conversion, and I/O ancillary to the reference design.

Table 1: The IP Surveillance Camera Reference Design has a total power consumption of 2.7 W.

With features ranging from WDR image capture to IP encapsulation to integrated MegaCore functions, this IP Surveillance Camera Reference Design provides a flexible, scalable solution with rapid time-to-market capabilities.

Judd Heape is technical marketing manager in Altera’s Industrial & Automotive Business Unit. Before joining Altera, Judd spent eight years at QuickLogic Corporation in roles ranging from FAE for the South-central U.S. region to senior director of systems engineering. Prior to QuickLogic, Judd spent nine years at Texas Instruments engaging in design for DLP products and -based printing engines, including two years in Tsukuba, Japan engaging in multimedia research projects for the ‘C6x DSP. Judd has five issued U.S. patents and holds a BE in Electrical Engineering from the Georgia Institute of Technology.

Altera Corporation