reward_tracker.py module

class RewardTracker

Bases: object

Tracks items and rewards for training updated scoring models. When an item becomes causal, pass it to the track() function, which will return a reward_id. Use the reward_id to track future rewards associated with that item.

property MODEL_KEY: str

Key under which model name is stored in track request dict / body

Returns:

Track request body key storing model name

Return type:

str

property MESSAGE_ID_KEY: str

Key under which message ID is stored in track request dict / body

Returns:

Track request body key storing message ID

Return type:

str

property ITEM_KEY: str

Key under which item is stored in track request dict / body

Returns:

Track request body key storing best variant

Return type:

str

property CONTEXT_KEY: str

Key under which context is stored in track request dict / body

Returns:

Track request body key storing givens

Return type:

str

property REWARD_KEY: str

Key under which reward is stored in reward request dict / body

Returns:

Track request body key storing reward value

Return type:

str

property REWARD_ID_KEY: str

Key under which reward ID is stored in reward request dict / body

Returns:

Track request body key storing decision ID

Return type:

str

property API_KEY_HEADER: str

Key for request headers storing an API key

Returns:

Track request headers key storing API key

Return type:

str

property PAYLOAD_FOR_ERROR_KEY: str

user info dict key storing track request body which caused an error

Returns:

user info dict key storing track request body which caused an error

Return type:

str

property REQUEST_ERROR_CODE_KEY: str

user info dict key storing track request error code

Returns:

user info dict key storing track request error code

Return type:

str

property ITEMS_COUNT_KEY: str

Key under which candidates / items count is stored in reward request dict / body

Returns:

Track request headers key storing variants count

Return type:

str

property SAMPLE_KEY: str

Key under which sample is stored in reward request dict / body

Returns:

Track request headers key storing sample

Return type:

str

property threaded_requests: bool

Boolean flag indicating whether requests should be executed with ThreadPoolExecutor. If True requests are non-blocking and executed within threads. If False requests are blocking and executed consecutively.

Returns:

should requests to AWS track endpoint be non-blockng

Return type:

bool

__init__(model_name, track_url, track_api_key=None, _threaded_requests=True)

Create a RewardTracker for a specific model.

Parameters:

  • model_name (str) – Name of the model, such as “songs” or “discounts”, which either makes the decisions or which decisions are being rewarded

  • track_url (str) – The track endpoint URL that all tracked data will be sent to.

  • track_api_key (str) – track endpoint API key (if applicable); Can be None

  • _threaded_requests (bool) – flag indicating whether requests to AWS track endpoint should be non-blockng / executed within sub-threads. True by default

property model_name: str

Name of tracked model. Model name must pass “^[a-zA-Z0-9][w-.]{0,63}$” regex

Returns:

Name of the tracked model

Return type:

str

property track_url: str

Improve AI AWS track endpoint URL

Returns:

Improve AI track endpoint URL

Return type:

str

property api_key: str

AWS track endpoint API key (if applicable); Can be None

Returns:

track endpoint API key

Return type:

str

_get_track_body(item, num_candidates, context, sample)

Helper method to create track body. Used by RewardTracker’s track() and track_with_sample()

Parameters:

  • item (object) – tracked variant

  • num_candidates (int) – number of variants

  • context (dict) – givens for this decision

  • sample (object) – sample to be tracked

Returns:

body for _post_improve_request()

Return type:

dict

_get_sample(item, candidates)

Randomly selects a sample from candidates excluding item.

Parameters:

  • item (object) – the best of candidates

  • candidates (list or tuple or np.ndarray) – collection of candidates

Raises:

AssertionError – if len(candidates) == 1 it is nto possible to select a sample

Returns:

sample from candidates

Return type:

object

track(item, candidates=None, context=None)

Tracks the item selected from candidates and a random sample from the remaining items. If len(candidates) == 1 there is no sample.

Parameters:

  • item (object) – any JSON encodable object chosen as best from candidates

  • candidates (list or tuple or np.ndarray) – collection of items from which best is chosen

  • context (object) – any JSON encodable extra context info that was used with each of the item to get its score

Returns:

reward_id of this track request or None if an error happened

Return type:

str or None

track_with_sample(item, num_candidates=None, context=None, sample=None)

Tracks the item selected and a specific sample.. Provided sample is appended to track request (in contrary to track(…) where sample is randomly selected from candidates).

Parameters:

  • item (object) – any JSON encodable object chosen as best from candidates

  • num_candidates (int) – total number of candidates, including the selected item

  • context (object) – any JSON encodable extra context info that was used with each of the item to get its score

  • sample (object) – a random sample from the candidates

Returns:

reward_id of this track request or None if an error happened

Return type:

ste or None

add_reward(reward, reward_id)

Add reward for the provided reward_id

Parameters:

  • reward (float or int) –

    the reward to add; must be numeric (float, int ro bool), must not be

    None, np.nan or +-inf

  • reward_id (str) – the id that was returned from the track(…) / track_with_sample(…) methods

Returns:

message ID

Return type:

str

_is_valid_message_id(message_id)

Check if message_id is a valid Ksuid

Parameters:

message_id (str) – checked message_id

Returns:

True if message_id otherwise False

Return type:

bool

do_post_improve_request(payload_json, headers)

Execute Improve POST to track_url with provided payload and headers

Parameters:

  • payload_json (str) – improveai body dumped to JSON

  • headers (dict) – request headers

post_improve_request(body_values, message_id=None)

Posts request to AWS tracker endpoint. In case of an exception during POST process does not break the main thread, just prints a warning with an error message.

Parameters:

  • body_values (dict) – dict containing request body. Bodies differ for track() and add_reward()

  • message_id (str or None) – ksuid of a given request

Returns:

message id of sent improve request

Return type:

str