Improve AI - Contextual Multi-Armed Bandit Platform for Scoring, Ranking & Decisions

2023-07-28T21:12:22+00:00

Introduction

Thompson Sampling is a highly effective algorithm used for solving the multi-armed bandit problem, an optimization challenge often encountered in online advertising, clinical trials, and recommendation systems. The crux of the problem is to balance exploration (trying out new options) and exploitation (choosing the best-known option) in order to maximize rewards over time. In this blog post, we will take a deep dive into Thompson Sampling, understanding its mechanics and benefits, and discussing real-world applications.

What is the Multi-Armed Bandit Problem?

Imagine a gambler at a casino, faced with multiple slot machines, each with its own unknown probability of winning. The gambler’s goal is to maximize their earnings by playing the machine with the highest win probability. However, they must first determine which machine that is, without wasting too much time and money on inferior options. This is the essence of the multi-armed bandit problem: making optimal decisions under uncertainty.

Thompson Sampling: A Bayesian Approach

Thompson Sampling is a Bayesian approach to solving the multi-armed bandit problem. It involves maintaining a probability distribution for each arm (option) based on observed rewards and using those distributions to make decisions. The algorithm follows these steps:

For each arm, maintain a prior distribution representing our belief about its expected reward.
At each time step, draw a random sample from each arm’s distribution.
Choose the arm with the highest sampled value and play it, then observe the reward.
Update the chosen arm’s distribution with the observed reward, incorporating the new information.

This approach is both exploratory and exploitative: by drawing random samples from each arm’s distribution, the algorithm is likely to choose arms with higher expected rewards more frequently, but it will still explore other arms occasionally.

Benefits of Thompson Sampling

Simple and efficient: Thompson Sampling is computationally efficient and relatively easy to implement. It only requires maintaining a distribution for each arm and updating them as new information becomes available.
Convergence to optimal arm: Over time, Thompson Sampling converges to the optimal arm, as the distributions become more accurate and the best arm is chosen more frequently.
Adaptability: Thompson Sampling can easily adapt to changes in the reward distribution or to dynamic environments, making it well-suited for online applications.
Incorporation of prior knowledge: By using a Bayesian approach, Thompson Sampling allows for the inclusion of prior knowledge or domain expertise through the choice of initial prior distributions.

Real-World Applications

Thompson Sampling has been successfully applied to various real-world scenarios, including:

Online advertising: In digital advertising, Thompson Sampling can be used to optimize ad placements, maximizing click-through rates or conversion rates by intelligently allocating resources to the most effective ads.
Clinical trials: In adaptive clinical trials, Thompson Sampling can be employed to assign patients to the most effective treatments while minimizing the number of patients exposed to inferior treatments.
Recommendation systems: Thompson Sampling can be used to recommend items or content to users, balancing the need to explore new options while exploiting the best-known recommendations.

Conclusion

Thompson Sampling is a powerful, elegant algorithm that addresses the multi-armed bandit problem by balancing exploration and exploitation in a principled, probabilistic manner. Its simplicity, adaptability, and ability to converge to the optimal solution make it a valuable tool in various fields, from online advertising to clinical trials. With its growing popularity and ongoing research, Thompson Sampling is poised to remain a cornerstone in decision-making under uncertainty.

Improve AI 8.0 - Contextual Multi-Armed Bandit Platform for Scoring, Ranking & Decisions

2023-06-08T00:00:00+00:00

We’re thrilled to introduce Improve AI 8.0, a modern, free, production-ready contextual multi-armed bandit platform that quickly scores and ranks items using intuitive reward-based training.

Multi-armed bandits and contextual bandits are corner-stone machine learning algorithms that power a myriad of applications including recommendation systems, personalization, query re-ranking, automated decisions, and multi-variate optimization.

With version 8, we’ve fully delivered on our original vision - providing a high performance, simple to use, low cost contextual multi-armed bandit platform.

Key features of v8.0 include:

Simplified APIs
90% more memory efficient XGBoost models
The reward tracker & trainer is now free for most uses
On-device scoring, ranking, and decisions for iOS and Android apps
Native Swift SDK that can rank or score any Encodable
Ranked Value Encoding for accurate scoring of String properties
Compact hash tables for reduced model sizes when encoding large numbers of string values
Balanced exploration vs exploitation using Thompson Sampling

Simple APIs

With Swift, Python, or Java, create a list of JSON encodable items and simply call Ranker.rank(items).

For instance, in an iOS bedtime story app, you may have a list of Story objects:

struct Story: Codable {
    var title: String
    var author: String
    var pageCount: Int
}

To obtain a ranked list of stories, use just one line of code:

let rankedStories = try Ranker(modelUrl).rank(stories)

The expected best story will be the first element in the ranked list:

let bestStory = rankedStories.first

Simple Training

Easily train your rankers using reinforcement learning.

First, track when an item is used:

let tracker = RewardTracker("stories", trackUrl)
let rewardId = tracker.track(story, from: rankedStories)

Later, if a positive outcome occurs, provide a reward:

if (purchased) {
    tracker.addReward(profit, rewardId)
}

Reinforcement learning uses positive rewards for favorable outcomes (a “carrot”) and negative rewards for undesirable outcomes (a “stick”). By assigning rewards based on business metrics, such as revenue or conversions, the system optimizes these metrics over time.

Contextual Ranking & Scoring

Improve AI turns XGBoost into a contextual multi-armed bandit, meaning that context is considered when making ranking or scoring decisions.

Often, the choice of the best variant depends on the context that the decision is made within. Let’s take the example of greetings for different times of the day:

greetings = ["Good Morning", 
             "Good Afternoon", 
             "Good Evening",
             "Buenos Días",
             "Buenas Tardes",
             "Buenas Noches"]

rank() also considers the context of each decision. The context can be any JSON-encodable data structure.

ranked = ranker.rank(items=greetings, 
                     context={ "day_time": 12.0,
                               "language": "en" })
greeting = ranked[0]

Trained with appropriate rewards, Improve AI would learn from scratch which greeting is best for each time of day and language.

XGBoost Model Improvements

Improve AI v8.0 is 90%+ more memory efficient for most use cases. Feature hashing has been replaced with a feature encoding approach that only uses a single feature per item property, substantially improving both training performance as well as ranking / scoring.

Ranked Value Encoding

Ranked Value Encoding is our novel approach to encoding string values in a manner that is extremely space efficient, accurate, and helps approximate Thompson Sampling for balanced exploration vs exploitation. The concept of Ranked Value Encoding is similar to commonly used Target Value Encoding for encoding string or categorical features.

With Target Value Encoding, each string or categorical feature is replaced with the mean of the target values for that string or category. Target Value Encoding tends to provide good results for regression.

However, multi-armed bandits are less concerned with the absolute accuracy of the scores and more concerned with the relative scores between items. Since we don’t need the exact target value, we can simply store the relative ranking of the string values, which saves space in the resulting model, increasing performance and lowering distribution costs.

Compact String Encoding

In conjunction with Ranked Value Encoding, rather than store entire strings, which could be arbitrarily long, Improve AI v8 models only store compact string hashes, resulting in only ~4 bytes per string for typical models.

Proven Performance

Improve AI is a production ready implementation of a contextual multi-armed bandit algorithm, honed through years of iterative development. By merging Thompson Sampling with XGBoost, it provides a learning system that is both fast and flexible. Thompson Sampling maintains equilibrium between exploring novel possibilities and capitalizing on established options, while XGBoost ensures cost-effective, high-performance training for updated models.

Get Started Today

Improve AI is available now for Python, Swift, and Java. Check out the Quick-Start Guide for more information.

Thank you for your efforts to improve the world a little bit today.

Making Decisions with Ranker

2023-03-19T00:00:00+00:00

In this tutorial, we’ll walk through how to use the Ranker class to make decisions, specifically choosing a discount to offer from a list of discounts. We’ll also need to track the decision with the RewardTracker and reward it for the amount of profit if a purchase is made. In this way, the Ranker will learn to rank the most profitable discount first.

Using the Ranker class

First, let’s create an instance of the Ranker class.

from improveai import Ranker

ranker = Ranker(model_url)

Ranking items

To rank items, you can call the rank method of the Ranker class. This method accepts a list of items and an optional context.

discounts = [.10, .15, .20, .25, .30]
ranked_discounts = ranker.rank(discounts)

Selecting the highest-ranking discount

Now that we have the ranked discounts, we can choose the highest-ranking discount.

best_discount = ranked_discounts[0]

Tracking the decision with RewardTracker

Next, we need to track our decision using the RewardTracker class.

Creating a RewardTracker instance

First, create a RewardTracker instance with the required arguments.

from improveai import RewardTracker

tracker = RewardTracker("discounts", track_url)

Tracking the decision

Now we can track our decision using the track method of the RewardTracker class. This method accepts the selected item, a list of candidates, and an optional context.

reward_id = tracker.track(best_discount, discounts)

Rewarding the decision

Once a purchase is made, we can add a reward to the decision using the add_reward method of the RewardTracker class. This method requires a reward value and a reward_id that was obtained when tracking the chosen discount.

reward = profit_amount
tracker.add_reward(reward, reward_id)

Improve AI will automatically balance exploring new discounts with doubling down on the discounts expected to be most profitable. Over time and as more decisions are made, the model will learn to choose the optimal discount.

Conclusion

In this tutorial, we covered how to use the Ranker class to make decisions, such as choosing a discount from a list of discounts. We also learned how to track the decision with the RewardTracker and reward it based on the profit.

Getting Started

Improve AI is available for Python, Swift, and Java. See the Quick-Start Guide to learn more.

Thank you for your efforts to improve the world a little bit today.

Simple Re-Ranking of SQL Queries with Machine Learning

2023-03-19T00:00:00+00:00

In this tutorial, we will demonstrate how to use the Scorer and RewardTracker classes to update a ‘score’ column for a number of products in a ‘products’ table. The ‘score’ column will be used to learn an optimized ranking for these products, allowing for extremely fast and simple product recommendations. By sorting the queries based on the score column, we can provide a highly efficient product recommendation system.

Step 1: Initialize Scorer and RewardTracker Instances

First, you need to create instances of the Scorer and RewardTracker classes.

from improveai import Scorer, RewardTracker

# Initialize Scorer instance
scorer = Scorer(model_url)

# Initialize RewardTracker instance
reward_tracker = RewardTracker("products", track_url)

Step 2: Query Products from Database and Score Them

Next, you should query the products from your database and score them using the Scorer instance. Scores are updated periodically via a CRON job.

import your_database_module as db

# Query products from the database
products = db.query("SELECT * FROM products")

# Score the products using the Scorer instance
scores = scorer.score(products)

# Update the 'score' column for each product in the products table
for product, score in zip(products, scores):
    db.update("UPDATE products SET score = %s WHERE product_id = %s", (score, product["product_id"]))

Step 3: Sort Products by Score and Display Them

When querying the list of products, sort them descending by score.

# Query products sorted by score
sorted_products = db.query("SELECT * FROM products ORDER BY score DESC")

# Display the sorted products
for product in sorted_products:
    print(product)

Step 4: Track Product Selection and Purchase

When the user selects a product to browse, track the product using the RewardTracker. If the product is purchased, track a reward for the amount of profit generated.

def on_product_selected(selected_product, products):
    # Track the selected product using the RewardTracker
    reward_id = reward_tracker.track(selected_product, products)

    return reward_id

def on_product_purchased(profit, reward_id):
    # Track a reward for the amount of profit generated
    reward_tracker.addReward(profit, reward_id)

Now, you can use the on_product_selected and on_product_purchased functions to track product selection and purchase events in your application.

To wrap up, we have shown you how to use the Scorer class from the improveai module to update the ‘score’ column of a products table, sort the products descending by score, and track product selection and purchase events using the RewardTracker class. By implementing these functionalities, you can improve your product recommendation system by leveraging the Scorer and RewardTracker classes.

Getting Started

Improve AI is available for Python, Swift, and Java. See the Quick-Start Guide to learn more.

Thank you for your efforts to improve the world a little bit today.

Optimize Any Python, Swift, or Java Object with Reinforcement Learning

2022-11-14T00:00:00+00:00

Improve AI is a machine learning platform for making apps self-improving, meaning they optimize their own data structures and variables to improve revenue and conversions.

With Improve AI v7.2, you can now optimize the variables of any Java, Swift, or Python object with reinforcement learning. It’s like an exponentially faster form of A/B testing.

The new optimize() method finds the best combination of variable values given current conditions.

Optimized objects are created immediately, on the fly, with zero network latency.

Optimize Any Object

Improve AI can optimize any object or JSON-encodable dictionary in Swift, Java, or Python to find the best combination of variables given current conditions.

As an example, let’s optimize a pop up discount offer in an iOS app. The goal is to maximize expected revenue by assigning the best combination of variable values.

First, we’ll create an Offer type with four variables: title, description, discount, and buttonText:

struct Offer: Codable {
    var title: String
    var description: String
    var discount: Float
    var buttonText: String
}

Next we’ll create a mapping from each variable to possible values.

Here the values are hardcoded but they could easily be loaded from a database or configuration file:

titles = ["Special Offer!", "Limited Time!"]
descriptions = ["New and improved.", "Great features." "Huge value."]
discounts = [0.1, 0.2, 0.3, 0.4]
buttonTexts = ["Try Free", "Subscribe", "Continue"]

To generate an optimized Offer, simply call optimize() with a map of the variables and the Offer type.

offer = offersModel.optimize(["title": titles, "description": descriptions, "discount": discounts, "buttonText": buttonTexts], 
                             Offer.self)

The best combination of variables is selected by the machine learning model and the result is returned as a new Offer object.

That’s like A/B testing on steroids.

Contextual Optimization

Unlike A/B testing, optimize() also considers the context of each variable assignment. On iOS and Android, the context automatically includes country, language, OS, device, time of day and many other attributes. (No personally identifiable information is ever used, transmitted, or stored)

With contextual optimization, different versions of the object may be returned for different contexts. This allows the object’s variables to adapt dynamically to it’s environment, enabling deep optimization, personalization, and more.

Custom context can also be provided via given():

offer = offersModel.given(["churned": true])
                   .optimize(variableMap, Offer.self)

In this example, given the user has churned, create an Offer that will maximize expected revenue.

Perhaps churned users need a larger discount? Perhaps they just need a nudge with the right message? With optimize() you don’t have to guess - it will automatically learn the best Offer for each context.

Contextual optimization, which is often infeasible with A/B testing, is trivial with this new capability.

Simple Training

optimize() is easily trained with reinforcement learning.

if (purchased) {
    offersModel.addReward(profit)
}

With reinforcement learning, positive rewards are assigned for positive outcomes (a “carrot”) and negative rewards are assigned for undesirable outcomes (a “stick”).

When rewards are business metrics, such as revenue or conversions, it will automatically optimize those metrics over time.

Getting Started

Object optimization is available immediately for Python, iOS, and Android/Java. See the Quick-Start Guide to learn more.

Thank you for your efforts to improve the world a little bit today.

Self Improving Apps: The Origin of a Radical Idea

2022-11-11T00:00:00+00:00

I’ve been working on Improve AI for many years, but by 2017 the idea had begun to gel enough that I was ready to talk about it.

Much has changed since this original talk, but the grand vision of building apps that automatically improve their own metrics remains unchanged.

Getting Started

Improve AI is available now for Python, iOS, and Android. See the Quick-Start Guide to learn more.

Thank you for your efforts to improve the world a little bit today.

The NEW 20 lines of code that will beat A/B testing every time

2022-04-05T00:00:00+00:00

In 2012, Steve Hanov wrote the popular and controversial blog post “20 lines of code that will beat A/B testing every time” that brought the previously academic idea of multi armed bandit algorithms into the awareness of the larger developer community.

Here were his original 20 (actually 16) lines of “code”:

def choose():
    if math.random() < 0.1:
        # exploration!
        # choose a random lever 10% of the time.
    else:
        # exploitation!
        # for each lever, 
            # calculate the expectation of reward. 
            # This is the number of trials of the lever divided by the total reward 
            # given by that lever.
        # choose the lever with the greatest expectation of reward.
    # increment the number of times the chosen lever has been played.
    # store test data in redis, choice in session key, etc..

def reward(choice, amount):
    # add the reward to the total for the given lever.

What he’s describing here is a very simple form of reinforcement learning known as the Epsilon Greedy algorithm. Much of the controversy surrounding this post was that it greatly oversold the simplicity of deploying such an algorithm in production.

Now after years of work and many thousands of lines of code, I’m pleased to present a new “12 lines of code that will beat A/B testing every time” that is robust, scalable, and proven in large scale production:

from improveai import Ranker, RewardTracker

ranker = Ranker(model_url)
tracker = RewardTracker(model_name, track_url)

def choose(variants):
    best = ranker.rank(variants)[0]
    decision_id = tracker.track(item=best, candidates=variants)
    return (best, decision_id)

def reward(amount, decision_id):
    model.add_reward(amount, decision_id)

Unlike Steve’s code that envisioned querying simple statistics from something like Redis, Improve AI loads a trained machine learning model locally into the process so that decisions are made immediately with zero network latency.

Improve AI is also a contextual multi-armed bandit, which means it can make use of contextual data such as language, time of day, and screen resolution to make the best decision. This makes it simple to implement recommender systems, personalization, app optimization, and more.

Is it finally time to stop wasting conversions and throw away A/B testing for good?

Improve AI is available now for iOS, Android, and Python.

Improve AI - Easily Optimize Your App with Reinforcement Learning

2022-04-05T00:00:00+00:00

Optimize and personalize your apps with fast AI decisions that get smarter over time. Decisions are made immediately, on-device, with no network latency.

The heart of Improve AI is the which() statement. which() is like an AI if/then statement.

greeting = greetingsModel.which("Hello", "Howdy", "Hola")

which() uses logic stored in a machine learning model to make the best choice. Models are easily trained with reinforcement learning.

if (success) {
    greetingsModel.addReward(1.0)
}

With reinforcement learning, positive rewards are assigned for positive outcomes (a “carrot”) and negative rewards are assigned for undesirable outcomes (a “stick”).

which() chooses the option that provides the highest expected reward given the current conditions.

When rewards are business metrics, such as revenue, conversions, or user retention, the decisions will optimize to automatically improve those metrics over time.

That’s like A/B testing on steroids.

However, unlike A/B testing, Improve AI uses context to personalize each decision. On iOS and Android, the context automatically includes language, OS, device type, day of week, time of day, app version, and many other attributes. (No personally identifiable information is ever used, transmitted, or stored)

With this context, for a Spanish speaker the greetings model will learn to choose Hola.

Custom context can also be provided via given():

greeting = greetingsModel.given({"language": "cowboy"})
                         .which("Hello", "Howdy", "Hola")

Given the language is cowboy, the variant with the highest expected reward is Howdy and the model will learn to make that choice.

The combination of given(), which(), and addReward() make it simple to implement personalization, recommendations, dynamic In App Offers, and more.

Example: Personalizing an Upsell Offer

Improve AI is powerful and flexible. which() accepts any JSON encodeable object including strings, numbers, booleans, lists, and dictionaries.

For a dungeon crawler game, say the user is purchasing an item using an In App Purchase. Improve AI can choose an additional product to display as an upsell offer during checkout. With a few lines of code, a model can be trained that will learn to optimize the upsell offer given the original product being purchased.

product = { "name": "red sword", "price": 4.99 }

upsellOffers = [{ "name": "gold", "quantity": 100, "price": 1.99 },
                { "name": "diamonds", "quantity": 10, "price": 2.99 },
                { "name": "red scabbard", "price": 0.99 }]

upsell = upsellModel.given(product)
                    .which(upsellOffers)

The product to be purchased is the red sword. Notice that the variants are dictionaries with a mix of string and numeric values.

The rewards in this case might be any additional revenue from the upsell.

if (upsellPurchased) {
    upsellModel.addReward(upsell.price)
}

While it is reasonable to hypothesize that the red scabbord might be the best upsell offer to pair with the red sword, it is still a guess. Any time a guess is made on the value of a variable, instead use Improve AI to decide.

Replace guesses with AI decisions.

Example: Performance Tuning

I used to write a lot of video streaming code. The initial motivation for Improve AI came out of my frustrations with attempting to tune video streaming clients across heterogenious networks.

I was forced to make guesses on performance sensitive configuration defaults through slow trial and error. My client configuration code maybe looked something like this:

config = { "bufferSize": 2048,
           "videoBitrate": 384000 }

This is the code I wish I could have written:

config = configModel.which({"bufferSize": [1024, 2048, 4096, 8192],
                            "videoBitrate": [256000, 384000, 512000]})

This example decides multiple variables simultaneously. Notice that instead of a single list of variants, a dictionary mapping keys to lists of variants is provided to which. This multi-variate mode jointly optimizes all variables for the highest expected reward.

The rewards in this case might be negative to penalize any stalls during video playback.

if (videoStalled) {
    configModel.addReward(-0.001)
}

Improve AI frees us from having to overthink our configuration values during development. We simply give it some reasonable variants and let it learn from real world usage.

This is an enormously powerful tool and I’m sure you’ll come up with some very interesting uses. Look for places where you’re relying on guesses or an executive decision and consider instead directly optimizing for the outcomes you desire.

Improve AI is available now for iOS, Android, and Python. See the Quick-Start Guide to learn more.

Thank you for your efforts to improve the world a little bit today.

– Justin Chapweske

Improve AI - Contextual Multi-Armed Bandit Platform for Scoring, Ranking & Decisions

Introduction

What is the Multi-Armed Bandit Problem?

Thompson Sampling: A Bayesian Approach

Benefits of Thompson Sampling

Real-World Applications

Conclusion

Improve AI 8.0 - Contextual Multi-Armed Bandit Platform for Scoring, Ranking & Decisions

Simple APIs

Simple Training

Contextual Ranking & Scoring

XGBoost Model Improvements

Ranked Value Encoding

Compact String Encoding

Proven Performance

Get Started Today

Making Decisions with *Ranker*

Using the Ranker class

Ranking items

Selecting the highest-ranking discount

Tracking the decision with RewardTracker

Creating a RewardTracker instance

Tracking the decision

Rewarding the decision

Conclusion

Getting Started

Simple Re-Ranking of SQL Queries with Machine Learning

Step 1: Initialize Scorer and RewardTracker Instances

Step 2: Query Products from Database and Score Them

Step 3: Sort Products by Score and Display Them

Step 4: Track Product Selection and Purchase

Getting Started

Optimize Any Python, Swift, or Java Object with Reinforcement Learning

Optimize Any Object

Contextual Optimization

Simple Training

Getting Started

Self Improving Apps: The Origin of a Radical Idea

Getting Started

The NEW 20 lines of code that will beat A/B testing every time

Improve AI - Easily Optimize Your App with Reinforcement Learning

Example: Personalizing an Upsell Offer

Example: Performance Tuning

Making Decisions with Ranker