High Concurrency
For distributed transcoding, each segment of video is concurrently requested by multiple AWS-LAMBDA Compute Units or AWS-EC2 Compute Units. Use AWS-LAMBDA here is because it can increase to thousands of workers at anytime and no need to wait for scaling. The original video itself is stored on AWS-S3, but AWS-S3’s concurrent request capability can’t support simultaneous requests for different segments of a video, which results in high request latency and long transcoding time, and it is not acceptable in business too, meanwhile since AWS-LAMBDA is billed on time, this brings about an increase in the cost of use. So from the perspective of transcoding speed and cost savings, we need to be able to read and slice each fragment of the video in a stable and high-concurrency manner. The solution is to deploy our own file server.
Minimum Disk Usage
At the same time, in order to reduce the amount of data transmission, we use HTTP-RANGE to access the video, so that access to video fragments only needs to specify RANGE according to time. FFMPEG also supports RANGE mode to request video fragments for transcoding. In order to support high concurrency, the file server must be deployed in multiple machines in order to withstand a large number of concurrent requests. If a video is stored in multiple copies on multiple file servers, requests for different segments of a video can be answered with different file servers. But in fact, we didn’t do this, because if each file server can support concurrent requests for the same video, then this video will be downloaded and stored multiple times, which will bring extra burden to the network bandwidth and disk storage. Instead, we only store one video in a file server, and all requests for this video will arrive at the file server. Of course, the performance of this file server must be good enough. This is equivalent to replacing a large number of low-performance machines with a small number of high-performance machines to achieve stable high concurrency. The benefits are obvious, eliminating the need to repeatedly download and store video without having to address the issue of low-performance node failure. And the high-performance node fails? Will only affect the video transcoding task on this file server. We only need to re-initiate the failed task, and its video will be processed by other file servers that are alive.
Intelligent Load Balancing
If you just deploy a file server to solve the problem of stable and high concurrency, then the problem is a bit simple. We will also face the problem of request unbalanced. If a file server is providing a request for some segment of video and a request for another video is coming, the file server resource will be tight and busy. Other file servers may be idle and resources are idle. Here, we will adopt a competitive strategy, or more accurately, a polite strategy. The video download task is not dispatched by the upper layer, but by the file server for competition or courtesy. We will monitor the video transcoding queue in real time, and can monitor the number of transcoding subtasks that need to request the file server and still waiting in the queue (the fragment size of the transcoding subtask is uniform), so the load situation of file server can be determined in the future. The file server that may be heavily loaded in the future will give up the opportunity to download new videos to a file server with a small load. After a certain period of time to adjust convergence, the load of all file servers will be more balance. This is equivalent to achieving simple load balancing, but this is different from the traditional load balancing problem.
Automatic Health Checking
What if the load on the file server itself is high and even reaches a critical value? Will cause subsequent requests to avalanche. We let the file server monitor its own resource load in real time and decide whether to download the new video immediately based on the load. The monitored load includes CPU, MEMORY, NETIO, DISK, and so on. These indicators are also reported to the master monitoring system and will be used later. If the load is high, the file server will wait until the resources are sufficient. This obviously blocks, but why is it so designed? Because we need to ensure that the high-priority completion of the task which is in transcoding is stable, it cannot be overly affected by subsequent tasks. At the same time, it does a layer of protection against the file server, so that it is not crushed by too many tasks. However, in order to alleviate the problem of excessive file server pressure, we will use more file servers for a large number of transcoding tasks to expand.
Timely Disk Recycling
The disk load problem needs to be described separately. The original video files are very large, some even reach more than 200GB. In order to increase the disk read and write rate, we use high-speed SSD disks. And because the disk is expensive, we can’t use too much disk space, so we must frequently swap in and out of the video. Here we will monitor the usage of the downloaded video in real time. If the disk space is not enough to hold the next video, we will wait and try to clear the video that is no longer accessible until there is enough disk space. All of this will be automated monitoring to ensure that the transcoding system runs stably and efficiently. We will find that it is the main goal to complete the task which is in transcoding at the maximum rate, and it will get the highest priority.
Split Main Task
Video transcoding is very CPU-intensive. That’s why we implemented this distributed video transcoding system and with the system we can finish transcoding a long video in a timely manner.
For ABR streaming, each video needs to be transcoded into different profiles with different resolutions and bitrates. For each transcoding with a specific resolution, we call it a ‘Main Task’. In order to make the transcoding highly concurrent, we split the ‘Main Task’ into many small ‘Sub Task’. Each ‘Sub Task‘ represents the transcoding task for a segment of the long video. And thus these small ‘Sub Task’ of the same or different ‘Main Task’ can then be handled concurrently and quickly on different computing units.
Trigger Sub Task
Once the main tasks are split into sub tasks, we put the sub tasks into an internal queue for next stage handling. And then the sub tasks in the queue are picked out and sent to EC2-SLAVE or AWS-LAMBDA for transcoding.
All MX’s EC2 instances can be an EC2-SLAVE for transcoding. When sending a sub task for transcoding, we check whether the existing EC2-SLAVEs are sufficient or not. If yes, the sub task will be sent to an EC2 instance for transcoding, otherwise it will be sent to AWS-LAMBDA for transcoding. In this way, we don’t need to keep a large number of EC2 instances and in the meantime, we can still make the video transcoding finished in a timely manner.