How does FPGA make real-time video processing possible?
What is FPGA and where it is used
An FPGA (field-programmable gate array) is a programmable logic integrated circuit that you as a user can reprogram to perform a different task.
All elements inside the FPGA work simultaneously. You simply connect these elements inside the FPGA in the correct order to achieve the desired functionality based on the structure and elements that are already there.
You don't need to know the physical configuration of the FPGA to write for an FPGA and make an implementation.
Basic elements for work: registers (memory elements); logic gates (and, or, not).
In FPGA acronym, Gate Array is an array of gates. That is, FPGA is an array of gates that can be reprogrammed.
Let's take a closer look.
In a simplified scheme, let's examine how the program is executed using a standard processor. Imagine you wrote code, it was translated into a sequence of instructions, and this sequence is stored in memory. These instructions are then sequentially read by the processor and executed.
FPGA does not execute instructions, it is a circuit that cannot be programmed, but can be configured.
How does the circuit work on the processor example? Both simultaneously and parallel.
The processor consists of a block for reading, decoding, executing commands, and writing the result. All blocks within the processor operate simultaneously. And when you work with an FPGA, everything will work at the same time.
For example, the reading block reads the 1st instruction. At this point, the rest of the blocks do nothing. They do not receive any commands.
After that, the reading block reads the 2nd instruction and passes the result of what it read from the first one to the decode block.
At this stage, two blocks work simultaneously.
Stage 4 involves all the existing blocks working simultaneously, but performing different tasks.
Digital circuitry
Let's look at FPGA from the other side.
There is the concept of digital circuitry, and FPGA is an integral part of this field of science.
What is the difference between conventional and digital circuitry?
Conventional circuitry is working with elements - a resistor, a capacitor, an operational amplifier, etc. Conventional circuitry works with analog signals that have different value levels.
Digital circuitry works with signals limited in level and value. That is, with signals 0 and 1.
The pins are all abstract, they have no physical characteristics and the signals on them have only two possible levels : 0 or 1.
A typical FPGA operates in the 100 to 500 megahertz range. The most top-end FPGAs operate at the limit of 1 gigahertz or less.
Advantages
An FPGA is, on average, more productive in general tasks than a special-purpose processor. When CPU performance is not sufficient and there are no ready-made circuits to solve the problem, FPGAs are used.
Particularly when it comes to 100% guaranteed processing and guaranteed latency.
Unlike a powerful processor, a powerful FPGA does not require additional memory, consumes less power, and can be implemented in a compact device.
FPGA is more productive, but less flexible in terms of the implementation of some tasks. At the same time, the scope of the technology is quite wide.
What are the uses of FPGAs?
As a niche product, this is typically used in small batches in areas where high computing power and performance are required:
- video processing;
- high performance computing;
- network acceleration;
- testing and measurement;
- video processing in embedded systems;
- military, space and aircraft construction.
Let's consider some points in more detail.
- Production of control and measuring equipment
One of the most successful areas of application is the production of control and measuring equipment. Information gathered from sensors must be processed as accurately and clearly as possible. We can therefore know exactly what the delay will be between receiving a signal from any accurate sensor and processing, obtaining results from the FPGA.
- Interface converter
It is most convenient to use an FPGA when it comes to a large number of not quite standard interfaces. 4 12C, 3 SPI, and 8 UARTs, for example, can be switched between each other and transmit data between them.
With an FPGA, you can implement any digital interface at a much higher frequency than with a microcontroller. The processing and switching speeds are faster in FPGAs, since everything is done in parallel.
This guarantees both fast operation and response time.
- Digital signal processing
A signal from an antenna, for example, is pre-filtered in the analog part (matches with the antenna in the analog path), then digitized by an analog digital converter, and the reports are sent to the FPGA, where digital signal processing (Fourier transform, various digital filters and so on.) and logic processing (e.g. demodulation) begin.).
After that, the received signal can be analyzed: evaluate its characteristics, look at the signal-to-noise ratio, catch events or output to a digital-to-analog converter and play it like music.
- High-speed Ethernet interfaces (network processing of traffic packets)
This area uses FPGA to ensure lossless operation and 100% load.
When we consider the generation, for example, the FPGA is one of the few devices that can provide 100% precisely defined load
With various software tools, you can generate traffic programmatically and achieve 100% load, for example, on a 100 megabit link.
FPGAs use transceivers to process high-speed interfaces. These are special blocks.
These blocks can make a parallel interface out of a serial interface, and then the FPGA can process it.
For example, the signal that comes to us for the Ethernet standard of 1 gigabit per second, it comes with a frequency of about 1 gigahertz. This is a serial signal, we transmit gigabit, we need gigahertz.
FPGAs don't normally operate at these frequencies. Typical FPGA frequencies are 100-300 megahertz.
In order to process a signal with a low frequency changing with a frequency of gigahertz, it must be converted to a parallel format before further processing.
This is where these special hardware transceivers are used. Because they are made using a different technology, they work faster. They have two interfaces, one serial, one parallel.
- Special accelerators
These are separate boards inserted into computers to reduce calculation load and handle specific tasks associated with large projects.
FPGAs are alternatives to GPUs with graphics accelerators in this area. There are some areas where FPGAs can help reduce power consumption. In other words, abstract GPUs perform the same task faster, but consume more energy.
Despite its slower performance, the FPGA consumes so much less power that the performance to power consumption ratio is ultimately more profitable.
In addition, there are unique tasks that are not well suited to GPU architectures.
Architecture of an FPGA allows for almost infinite parallelization. It's not just about the number of cores. In this case, there are a lot of elements that are all capable of working simultaneously. In terms of their computing abilities, they are much inferior to strong processor cores (these are separate registers, adders, multipliers).
When you don't need the processing power of a core, but just need to add two numbers. For example, if you need to add a million pairs of numbers simultaneously, then you can use an FPGA to do this in one cycle. It won't work on a GPU.
- Video and Image Processing with FPGA
As a result of their structure, FPGAs are used for real-time image processing because several operations can be performed simultaneously.
In a smart camera with an integrated FPGA, the FPGA will process most of the media while the sensor streams data packets. In this case, the camera provides a stream of processed output, not a series of images.
FPGAs allow you to transcode 8K video in real time, without delay.
In areas such as video processing and machine learning, FPGAs can provide extreme hardware acceleration. For example, Blackmagic Design is known to use Xilinx FPGA during development. In the image below, taken directly from their website, you can see a Xilinx Spartan-6 FPGA powering their camera control screen.
Blackmagic 3G-SDI Shield for Arduino
FPGA is a secure way to work with data. Although there is a chance that an attacker will be able to gain control over the FPGA, they will not be able to control the system.
Therefore, in recent years, FPGAs are often used in the field of security, signal processing, and machine learning.
Innovative video processing in Callaba
Let's get back to live streaming. You can process video and adjust the adaptive bitrate literally on the fly, using the Callaba media accelerator and FPGA.
Here’s how you can do it : Next generation video transcoding via FPGA.
More news from the world of streaming in our social networks:
