AI is at the forefront of media and entertainment

Malav Shah is a Data Scientist II at DIRECTV. He joins DIRECTV from AT&T, where he worked on multiple consumer businesses – including broadband, wireless and video – and deployed machine learning (ML) models across a broad range of use cases spanning the entire customer lifecycle from acquisition to retention. Malav holds a master’s degree in Computer Science with a specialization in Machine Learning from Georgia Tech, which he uses every day at DIRECTV to help companies deliver innovative entertainment experiences by applying modern ML techniques.

Can you give an overview of your career and why you got into machine learning in the first place?

It’s been an interesting journey. During my undergrad, I actually studied information technology, so most of my courses were not machine learning initially. During my junior year, I took an artificial intelligence course where we learned about Turing machines, which intrigued me in the world of artificial intelligence. Even then, I knew I had found my calling. I started taking some extra courses outside my usual coursework and ended up taking a capstone project that built a model to predict outliers in medical diagnosis and prognosis, which got me fascinated by the power of machine learning.I did my master’s degree at Georgia Tech, specializing in machine learning, taking courses ranging from data and vision analytics to artificial intelligence courses taught by the former Google Glass tech lead Thad Stana. After graduating, I took my first role at AT&T, working in the chief data officer’s organization for about a year and a half, building acquisition and retention models for the company’s broadband products. In July 2020, I joined a new organization at DIRECTV as part of the team responsible for all data science and have a say in how we build ML infrastructure and MLOps pipelines across the organization. Being able to influence not only my team but other teams in a centralized data organization was a big motivation for joining DIRECTV.

What attracts you to your current role?

I interned at AT&T while completing my master’s degree. While the internship focused on broadband products, I was also exposed to wireless and streaming video – things I use every day as a consumer. After graduation, most of the other positions I got at the time were software engineering or ML engineering, but AT&T offered me a data scientist position. Being a data scientist and thinking about how to do research and solve problems eventually proved appealing.

This role brought directly an opportunity to take part in a video streaming journey based on DIRECTV’s nearly 30-year history. Early in my career, the opportunity to build and define new cloud tools, new infrastructure and machine learning tools was exciting. I don’t think I have access to so many levels of executives anywhere else.

How is DIRECTV’s machine learning organization structured – is there a central ML team or is most dependent on the product or business team?

Our team in DIRECTV acts as a center of excellence. Our responsibilities are twofold. The first responsibility is to help stakeholders in Marketing, Customer Experience (CX), and other teams solve problems and develop solutions. For example, we might help a marketing team build a model from scratch and deploy it into production, then hand it off to their data scientists to own – so they can have their day job while we provide ongoing support as new requirements arise Let’s update the model. The second part of our team’s work is to define the infrastructure these teams will use, making sure they have the tools and techniques they need to effectively create and deploy machine learning models. Our team is also responsible for defining best practices for ML development and deployment across the organization. To that end, we’re always looking for ways to improve our existing ML pipelines based on our strategy and goals, either by building something in-house, or by looking at what’s available in the market.

When evaluating this infrastructure, how do you evaluate whether to build or buy? The field of machine learning infrastructure has clearly changed a lot over the past few years.

This is an interesting question that came up recently when evaluating ML observability platforms like Arize. Generally, we start by looking at business value to ensure that any new functionality will actually bring value to the organization. Then, we look at how long we need the capability, how long it will take to build in-house, the capability we might build versus the vendor, and the final cost to buy or build. This evaluation process takes up quite a bit of our time, but it has proven to be effective in delivering the greatest return on investment for the business.

What is your machine learning use case?

First, DIRECTV is doing a lot of structured data modeling. For example, we work with our customer experience team to build a Net Promoter Score (NPS) critic model, which we use to provide a better experience for customers facing issues with our services. We also work with marketing stakeholders to build models around “personalized” customer offers and short- and long-term churn prediction.

Another area of ​​interest is content intelligence — not analytics, but intelligence. In the area of ​​content intelligence, building recommendation engines for the various carousels that customers see on DIRECTV products is one of our areas of focus. We are also starting to develop and see more traction in computer vision and natural language processing (NLP) models. Arize launched Image and NLP Embedding Tracking is something we may need next year as we transition to more use of unstructured data.

The media landscape has changed a lot in the past few years alone.Are you seeing an increase in things like this concept drift?

Post-pandemic consumption has certainly skyrocketed. Attrition rates have declined across the industry as people are stuck at home. For people working from home, these habits may have some staying power — not just in rural areas where satellite TV already dominates. One of the other trends in the streaming industry compared to 2019 is the historic increase in sports viewership in general (you shouldn’t really be comparing 2020 or 2021 given the compressed sports schedules and canceled events). Sports fan engagement is also becoming a big trend as more streaming services in the industry move into sports and add interactivity, such as letting people place bets on TV. With these changing consumption patterns, it becomes even more important for us to track things like concept drift and feature drift to ensure we address model performance issues immediately.

What challenges do you face after deploying your model to production? Why is this happening? Model monitoring important?

In the video industry, behavior is changing rapidly. If you notice drift after a month, it can negatively impact model performance and result in loss of business value. This is one of the main reasons I think real-time ML monitoring updates are so important in MLOps. If my model drifted this morning, I should have known it in that second. If my forecast is skewed, or there is a feature bias or some features are empty, then I don’t want to wait a week for an analyst to check it – ideally, I’d like to know the field before a week’s forecast comes out.

Models are never perfect; they will always drift as behavior changes, data changes, or source systems change. Having a centralized monitoring platform like Arize is very beneficial.

What advice would you give to someone who is first in a data science role?

One thing I advise newly graduated data scientists not to do is get obsessed with getting the perfect metric score right away.focusing on a Model metrics such as accuracy It’s important, conceptually, to focus on understanding the underlying data – what the data is doing, what the data is telling you – and making sure you understand the business impact and the problem you’re trying to solve. These fundamentals are important, but people tend to ignore them because they are too fast to try to build the best model. Instead, I’d say focus 70% to 80% of your time on everything you put into your model, because garbage in is garbage out. Most of the rest will take care of itself once you’re sure you’re not putting junk into the model.

Another piece of advice for new graduates is to keep an eye on the wave of data-centric AI tools. These could be the next big things in machine learning and are worth watching closely.

How do you collaborate with business and product owners and connect model metrics to business outcomes?

This is always happening. Whenever we create a model for any stakeholder, we meet with them regularly to make sure what we see matches what we should see in the real world. When starting a project, it is critical to make sure the requirements and data are there and that you understand the data properly. Later in the development cycle, I don’t even know what type of model to build – it could be the fourth or even fifth sprint. My approach doesn’t start with describing the type of model I want to build; I prefer to start with what business value should drive first. Having a deep understanding of the data also helps me answer nuanced questions when presenting to business executives and stakeholders.

How do you see the evolving MLOps and ML infrastructure space?

I think we are entering a very innovative era of machine learning because there are a lot of new machine learning solutions appearing across the industry every week. ML observability is a great example of a space where hundreds of things are happening. Production ML is quite different from production other applications because other applications have been around for a while – 15 or even 25 years – and they have very mature production pipelines, but for machine learning it is still relatively new. It will be exciting to see how we can make ML deployment easier and seamless, which will be a pain point for many teams. Other areas of innovation I’ll be watching closely include automated insight generation tools, data-centric AI tools, and how we can further improve the machine learning infrastructure space where everything is on the cloud.