About the customer
Sportradar is the leading global sports technology company creating immersive experiences for sports fans and bettors.
Established in 2001, the company is well-positioned at the intersection of the sports, media and betting industries, providing sports federations, news media, consumer platforms and sports betting operators with a range of solutions to help grow their business.
Sportradar employs more than 2,300 full-time employees across 19 countries around the world. It is their commitment to excellent service, quality and reliability that makes the company the trusted partner of more than 1,600 customers in over 120 countries and an official partner of the NBA, NHL, MLB, NASCAR, FIFA and UEFA. Sportradar covers more than 750,000 events annually across 83 sports.
With deep industry relationships, Sportradar is not just redefining the sports fan experience; it also safeguards the sports themselves through its Integrity Services division and advocacy for an integrity-driven environment for all involved.
The Challenge
Sportradar has worked tirelessly over several years to improve its quality and speed of execution by implementing DevOps best practices across all its product portfolio and core services. They embraced the concept that DevOps is not just a set of tools but is mostly driven by processes and culture. The culture has been incrementally changing from a monolithic one to a number of small teams (tribes) with each team taking care of specific Sportradar products.
Such a culture shift brings with it several technical challenges which required outside expertise and end-to-end automation.
One challenge was, how do you provision hundreds of AWS accounts, each using many regions and Availability Zones, whilst ensuring a common set of security best practices is followed?
Another challenge included, how do you provision hundreds of accounts in a way that their VPCs can connect privately, securely and automatically on the global Sportradar AWS network as well as with on-premise data centres?
The Solution
An executive decision was made at Sportradar to invest in an “Account Factory” solution and 56Bit was commissioned to help the company tackle the significant automation required to handle the above challenges, and thus continue to propel it to a DevOps driven future.
The chosen starting point was AWS Control Tower, a managed account factory solution. The existing AWS solution required significant improvements and additions to cater for Sportradar’s needs. 56Bit helped architect, create and test the following additions to AWS Control Tower:
- Creation of an automated discovery system that finds all AWS VPCs in multiple accounts and ensures secure connectivity is established over a global Transit Gateway network. This solution needs to integrate seamlessly with the Serverless Transit Network Orchestrator solution from AWS.
- Creation of a Gitlab CI/CD pipeline that automatically creates the Account Factory solution from scratch, including launching all the custom extensions written for Sportradar. This pipeline had to be very customizable and extendable to ensure future extensions would not require re-coding the pipeline itself.
- A CI/CD pipeline that automatically adds an account to the Sportradar AWS Organization, whilst deploying all the custom extensions, network discovery and security safeguards required. One requirement was to have the ability to create a new AWS account by simply uploading a new config file, which would trigger the pipeline, consisting of multiple Step Functions, CloudFormation stack sets, Gitlab pipelines and Python scripts, providing a fully functional AWS account to the requester.
- Migration of hundreds of accounts from the previous non-managed organization to a new AWS organization managed by Control Tower and its guardrails.
Why AWS?
AWS is Sportradar’s public cloud provider of choice. The maturity, scalability and unparalleled feature set provided by the AWS service portfolio, drove Sportradar to choose AWS for most of its workloads.
Why 56Bit?
56Bit provides peace of mind to technology-driven business through best-in-class cloud solutions. Sportradar, whose core business is totally dependent on the underlying technology required an experienced partner with profound knowledge on serverless technologies that could deliver high-quality service on time and within budget. Sportradar teamed up with 56Bit to consult, design, build and maintain this platform, working hand-in-hand with the software development team.
Solution Details – Account Factory Creation
Goal: The creation of a Gitlab CI/CD pipeline that automatically creates the Account Factory solution from scratch, including launching all the custom extensions written for Sportradar.
Solution: A Python 3.8 application that is run using Gitlab runners. This application:
- Is able to read, validate and follow a set of configuration files that drive its logic, meaning the account factory creation itself is very flexible as it is governed by JSON configuration files saved on S3.
- These files can instruct the application to create additions to the Account Factory, like Cloudformation stacks, Cloudformation stack sets, Service Control Policies at the organization level, build and upload custom applications, etc.
- All these features are deployed in 11 public AWS regions and governed by the Control Tower master account.
- The configuration files drive the application to deploy the following additions to the native Control Tower solution (allowing the following services to be deployed when new accounts are created using the Control Tower account factory):
- Sportradar’s alerting system
- Firewall Management system
- AWS GuardDuty
- AWS IAM Analyser
- AWS Inspector
- ProsperOps
- AWS Route53 DNS resolvers
- Multiple service control policies (SCPs)
- AWS SecurityHub
- Splunk
- Azure Active Directory SSO
- AWS VPCs with custom configurations
- AWS Serverless Transit Network Orchestrator (STNO) with multiple customizations to allow new accounts to join the Sportradar global AWS network.
- Lambda BGP – A custom serverless solution built by 56Bit that auto-discovers new VPCs and adds the appropriate routing to the Sportradar global AWS network.
Solution Details – Single Account Creation
Goal: To allow developers to request the creation of new AWS accounts, which in turn are created in a secure, well-orchestrated way, by extending the Account Factory features of Control Tower.
Solution: A serverless application built with AWS Step Functions, AWS Lambda Functions and other components and written in Python 3.8, that:
- Allows a requester (usually Sportradar developers working on an application) to upload a JSON config file to S3 or Gitlab, and after the request is approved, the pipeline triggers a Step Function in AWS (SQS queues and other lambda functions were also developed to ensure proper handling of multiple requests).
- This step function orchestrates the account creation by:
- Validating the configuration that the requester submitted
- Triggering an AWS Service Catalog product which creates the actual account and deploys all the Control Tower guardrails
- Constantly checks the status of the account creation process and waits for up to 1 hour before proceeding.
- Deploys a number of custom components in a controlled manner (including Python serverless applications, CloudFormation stack sets, CloudFormation stacks, SCPs, etc). These components are very dynamic since they are driven by JSON configurations.
- Handles the appropriate success / failure states that can be introduced during the process.
- Logs and alerts the process status once it is finished.
The end result is a fully managed AWS account, connected to the Sportradar global network, Sportradar’s SSO, alerting, logging, firewalling and other solutions. The requester just submits the configuration file and waits for the process to finish, which usually takes less than 1 hour to complete.
Solution Details – Transit Gateway automatic routing algorithm
Goal: Creation of an automated discovery system (internally dubbed LambdaBGP) that finds all AWS VPCs in multiple accounts and ensures secure connectivity is established over a global Transit Gateway network. This solution needs to integrate seamlessly with the Serverless Transit Network Orchestrator solution from AWS.
The need for this solution arose due to a limitation in the AWS Transit Gateway service, where automatic propagation of VPC routes stops at a region’s border. Other regions will not have automatic routing on the global Sportradar network when a new VPC is created in a separate region.
Solution: The solution involves several lambda functions built with Python 3.8 that:
- Are triggered when a new VPC is created in any region in any account.
- Grab all the required information from multiple sources, like the Transit Gateway Network Manager feature of AWS.
- Find all enabled and in-use regions on an east-west basis with the current region (where the VPC resides)
- Add the appropriate static routing to each region, ultimately directing traffic to a number of regional hubs and finally to the VPC itself.
- If a VPC is destroyed the static routes are removed.
- If need be, a report is generated of the current static routes versus what the routes should be. This will point out any configuration drift in the setup.
Solution Details – Migration to a new Control Tower managed AWS Organization
Goal: Migrate hundreds of accounts from the previous non-managed organization to a new AWS organization managed by Control Tower and its guardrails.
Solution: 56Bit was engaged to consult on a migration plan and fill any technical gaps related to this migration. The migration itself, led by Sportradar, was carried out over a few hours (a big bang approach was chosen due to many considerations) and did not result in any downtime.
All Sportradar AWS accounts, where both non-production and production workloads reside, are now running in this new organization.
The new organization is fully automated to enable the creation of new accounts that are secured by multiple guardrails, connected to the global network and connected to all the periphery services like Alerting, Monitoring, Firewalling, etc.
