In the great ocean of Internet, while OTT platforms deliver content to customers, digital pirates are constantly trying to steal from them.
Piracy is one of the biggest challenges for any OTT provider and it becomes a catastrophe if piracy happens on the back of their content distribution networks(CDNs). Not only do they lose viewership, but also pay for CDN cost that is actually used by the pirate apps.
In this three-part blog series, let’s delve into how Disney+Hotstar (D+H) secured content from pirate applications re-streaming its content to millions of users.
Let’s first learn about the media delivery flow through CDNs
To begin, let’s first establish the concept of a CDN. A content delivery network is a server technology that strategically caches static data in close proximity to end-users.
Furthermore, these servers are optimised for delivery purposes and typically lack extensive computing capabilities. This mitigates the need for expensive compute servers by minimising associated overhead costs.
CDNs hence provide following benefits with static media data:
- Reduce load on origin servers
- Decrease latency for end users
- Enable effective audience segmentation
- Deliver large amounts of data quickly and cost-effectively
At Disney+Hotstar, CDNs are used to efficiently serve content to millions of customers worldwide.
Following is standard journey of a user trying to watch content on D+H
- User requests the D+H backend to access video data using D+H client application
- The backend authenticates the user request and issues a playback token along with a redirection request to CDN
- The client next requests the CDN edge for video data using the playback token for authentication
- CDN authenticates the request, fetches and caches the origin data near to the user
- The video is then served to the user
CDNs using the caching mechanism and lower hardware costs, optimise the content delivery flow. However, it is easier to steal from CDNs that use regular compute servers due to certain limitations.
CDNs typically have limited capabilities when it comes to playback security. Traditionally CDNs have been used to serve static website data which is inexpensive. However incase of a OTT platform, the media data is its main asset and needs to be protected.
On the serving path, CDN offers web application firewall(WAF) and static validation capability. Using this, playback tokens are assigned from the backend containing information related to a playback session. These are used to perform stateless validations on every CDN access request.
While CDNs are focused on optimising delivery speed, they typically lack a essential element — state management.
Due to this lack of state management, CDNs cannot track and remember specific users to whom they have served the data. This limitation creates a significant vulnerability that can be exploited by malicious users:
- A pirate who impersonates a legitimate user can repeatedly request data from a CDN
- Since this CDN cannot verify if this user was served data or not, it unwittingly delivers the requested content again, any number of times
- This pirate hence can very easily re-share the data to any number of malicious user directly from CDN
To address this security concern, additional measures must be implemented alongside CDNs to ensure proper user authentication and access control.
But lets first understand how big this problem actually is!
The magnitude of the problem at D+H scale becomes evident when considering the activities of pirates who exploit OTTs through CDNs vulnerabilities for their own gain. These individuals or groups create and sell modified applications which stream content for no cost directly from different OTT platforms.
Their approach involves purchasing a few legitimate subscriptions from an OTT platform to get valid user identity and playback tokens. They then distribute these through their modified applications to stream data. Shockingly, some of these pirated applications have amassed user bases consisting of millions of users.
Modified applications are an issue for the OTTs as they have to incur the CDN cost used to stream content to these apps but also lose subscribers who are using these pirate applications. These applications pose a security threat to end user as well as their data is exposed to an untrusted source.
(For the scope of this blog, we will talk about how we target content redistribution from CDN only).
A brute force approach to tackle this problem would be to scan the CDN access logs and block users that perform malicious activity.
The scale at which Disney+Hotstar operates makes it impractical to use log analysis for identifying rogue users.
For instance, consider a live match running on D+H platform with 20 million viewership.
20 millions live users requesting and fetching video data simultaneously and continuously for the whole match duration generates log data in unit of petabytes(~80 PBs/hour)
This data is geographically distributed around the globe on multiple different CDN edges
Since the match is live and duration is less(generally about 3 hours), the analysis and action needs to happen realtime
The cost of gathering this amount of data in near real time and running extensive queries on this high volume streaming data potentially outweighs the benefit. Hence, log processing could not provide a viable solution to the problem.
As discussed, a major threat to CDN are malicious users impersonating legitimate users by reusing playback tokens. None of the default CDN stacks can be utilised to tackle this problem in real time and hence a unique solution was developed using cutting edge CDN technology.
CDNs themselves are not able to provide an effective solution to the problem of unauthorised access. However, if the CDN is somehow able to communicate to an external system which is intelligent, then this system can take security decisions for the CDN.
Hypothesis: Use the CDN edge capability to make blocking API calls and paired with an intelligent and efficient validation server, the data on the CDNs can be secured from pirates.
Note: Developing capability on a CDN to make external calls is very complex and the architecture varies for different CDNs as each CDN stack is different and provides different capabilities. This topic is out of scope for the series and we will be talking more about how we utilise this capability in order to secure streaming.
CDN edges can be used to make pre-flight calls to a different validation server. This server can validate the user access request before allowing/denying it. This flow provides following advantages:
- It allows for data to be served from the CDN while simultaneously ensuring that user requests are validated
- This can potentially removes the state management limitation from the CDN by offloading the responsibility to validation server
- The validation server can profile entities interacting across different CDNs being used to serve data
- It can make decisions related to data access in real time
This is a potential solution, however before implementing this in the real world following challenges need to be addressed:
- Getting data from a trusted source: In order for the validation server to perform its job, the data needs to be authentic. Hence, data coming directly from the client is not the right source, as it might be tampered.
- Limited compute at CDN : Even though there is compute, huge CPU cycles at CDN cannot be used to profile user data. The primary job of CDN is to deliver data and that should not be affected. Essentially, edge processing should be minimal.
- Latency: Pre-flight blocking calls to the backend introduce additional latency to the content delivery flow. Balancing the need for security with delivering content in a timely manner is essential to maintain a seamless user experience.
- Multi-CDN Integration: When talking about scale at which Disney+Hotstar operates, a single CDN cannot be relied upon to make all the deliveries. And hence the solution should be able to integrate and work with multiple CDN providers.
To solve all this, D+H’s engineering team came up with an innovative solution that is able to secure media delivery while scaling to its traffic requirements and keeping the running costs to a bare minimum.
The solution works using playback tokens and Compute@Edge on CDN.
In order to validate a CDN access request, the validation server requires un-tampered user access information. This information:
- Needs to come from a trusted source
- Should not create pressure on the Edge to process data and create complex request objects for forwarding to the validation server
An existing CDN capabilities can be leveraged to achieve this objective.
CDN auth/playback tokens are already used to verify a playback sessions — these are signed tokens that cannot be tampered. Voila! we could then just pack more info in these tokens and validate them at the CDN in a fail-safe manner.
All the parameters needed by the validation server are therefore secured and added to the token from the D+H backend. The CDN edge does not need to process the request data and simply can sign-validate and forward this object to the validation server.
Even though the request is light and does not strain CDN operation, relying on validating every request on the backend might not serve to be a good design at high scale. Hence a mechanism to validate calls sparsely if needed can help in controlling CDN traffic.
One mechanism to achieve this can to have fixed windows of validation with ability to control window interval on the CDN. If a request comes to the CDN during this period, it is sent to validation server. But knowing the kind of thundering herd traffic build-ups we receive at D+H , if all the requests coming to CDN were to be validated in the same interval, it would lead to unrecoverable traffic spikes — overwhelming both the CDN edge and the validation server.
In order to build a scalable solution, instead of having fixed interval validation mechanism, validation time should be different for each playback session. Hence, CDN should make validation call at different times for each playback session.
Wait..pirates can be smart too 😉
In the previous section, we saw that validation calls might not be done for all requests during heavier loads. In this scenario, a smart pirate is still able to playback until the next validation happen.
- One solution to handle this vulnerability can be to keep the token validity short for untrusted sessions. In these cases, playback cannot be done for a long duration.
- However, with short token validity, we need a mechanism to keep the playback sessions active beyond the token expiry if there is no abuse happening. This is done by silently and randomly rotating the playback tokens with new tokens before it expires.
Now, if a pirate skips validations then their token expires and the session is terminated.
To maintain and manage multiple different CDNs used for serving D+H traffic:
- Ownership of generating and rotating playback tokens is offloaded to the validation server rather than CDN or a D+H application.
- This ensures uniformity and maintains token ownership at a single point.
- This system integrates and supports different formats of playback tokens that CDNs use
- Validations and decisions like the updated playback token, next validation interval etc are taken based on the CDN
- This design also gives us the capability to centrally track user sessions across CDNs and build more intelligence into the system.
- For this system, different CDN acts as different data sources and various adapters are used to transform them all into a common data object
All these systems and structures together form the cutting edge security layer that saves millions of unauthorised access daily for D+H.
In this blog, we talked about security challenges faced while serving content from CDN. We talked about how CDN edge compute capabilities are utilised to develop an advanced security stack for the CDN, what challenges were faced while designing this stack for scale and how the solution was made robust enough to work with multiple different CDN vendors.
In the next part, we will talk about how we leveraged this flow to create a secure concurrent device management flow.