Skip to content

Commit

Permalink
Release 4.2.0 (2024-10-19)
Browse files Browse the repository at this point in the history
### Features

* [BYOB] Add udf_execution metrics for generateBid
* add proto field to enable buyer debugging for eligible request
* adding aws inference dashboard
* Enable B&A experiments in GCP by splitting traffic
* enable buyer debugging for eligible request
* Implement Data Version Header for KV Server Clients
* Implement v2 bidding support - add branch selection
* Implement v2 bidding support - add kv_buyer_signals_adapter
* Implement v2 bidding support - update kv dependency
* output packer logs to stderr on AMI creation failure
* Pass data version to GenerateBidsRawRequest
* Support model deletion in model store
* Update Kv client to match the latest contract

### Bug Fixes

* Accumulate js_execution.errors_count metric
* Copy common signals instead of move in BuildProtectedAudienceBidRequest
* Fix changelog generation
* GCP demo terraform outputs url
* Prevent a potential read-after-delete bug in the inference benchmark
* start_bidding runs inference default js file
* typo in bidding_service_integration_test and buyer_code_wrapper

Bug: 374379327
GitOrigin-RevId: fe009c936554bbaa97d98a8823c1e9766de78b86
Change-Id: I5dc590ef23daa175d2edb5ac156916165871f963
  • Loading branch information
lukedeng95 committed Oct 24, 2024
1 parent 09e0c1f commit 532e293
Show file tree
Hide file tree
Showing 189 changed files with 5,676 additions and 1,692 deletions.
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,7 @@ repos:
)$

- repo: https://github.com/DavidAnson/markdownlint-cli2
rev: v0.13.0
rev: v0.7.1
hooks:
- id: markdownlint-cli2
name: lint markdown
Expand Down
34 changes: 30 additions & 4 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,38 @@

All notable changes to this project will be documented in this file. See [commit-and-tag-version](https://github.com/absolute-version/commit-and-tag-version) for commit guidelines.

## 4.1.0 (2024-10-10)
## 4.2.0 (2024-10-19)


### ⚠ BREAKING CHANGES
### Features

* [BYOB] Add udf_execution metrics for generateBid
* add proto field to enable buyer debugging for eligible request
* adding aws inference dashboard
* Enable B&A experiments in GCP by splitting traffic
* enable buyer debugging for eligible request
* Implement Data Version Header for KV Server Clients
* Implement v2 bidding support - add branch selection
* Implement v2 bidding support - add kv_buyer_signals_adapter
* Implement v2 bidding support - update kv dependency
* output packer logs to stderr on AMI creation failure
* Pass data version to GenerateBidsRawRequest
* Support model deletion in model store
* Update Kv client to match the latest contract


### Bug Fixes

* Accumulate js_execution.errors_count metric
* Copy common signals instead of move in BuildProtectedAudienceBidRequest
* Fix changelog generation
* GCP demo terraform outputs url
* Prevent a potential read-after-delete bug in the inference benchmark
* start_bidding runs inference default js file
* typo in bidding_service_integration_test and buyer_code_wrapper

## 4.1.0 (2024-10-10)

* All sellers and buyers must update the js_num_workers parameter to udf_num_workers in terraform
* Sellers must wait for all the integrated buyers to update to 4.1.0 before updating to accomodate serialization changes in the GetBidsRequest.

### Features

Expand Down Expand Up @@ -39,6 +64,7 @@ All notable changes to this project will be documented in this file. See [commit
* SFE sends updateIfOlderThanMs data to client
* update start_bidding with minimal defaults and local file flag


### Bug Fixes

* BYOB generateBid should only parse first bid from binary response
Expand Down
8 changes: 4 additions & 4 deletions WORKSPACE
Original file line number Diff line number Diff line change
Expand Up @@ -87,11 +87,11 @@ http_archive(

http_archive(
name = "service_value_key_fledge_privacysandbox",
# commit d186bcf8d643888fac81784f9c91ffb2fb48791c 2024-04-19
sha256 = "53521403ffbe18a306417b8e7b41c2a13127ba0892ad921229861a451ad2045e",
strip_prefix = "protected-auction-key-value-service-d186bcf8d643888fac81784f9c91ffb2fb48791c",
# commit bdadee7c80dd84197f9253d4fd92c310f457be00 2024-09-26
sha256 = "6f91715f5ac946b2c5a9c4536f8e7deebf74b73e5b75e93163fd0e6276731ac1",
strip_prefix = "protected-auction-key-value-service-bdadee7c80dd84197f9253d4fd92c310f457be00",
urls = [
"https://github.com/privacysandbox/protected-auction-key-value-service/archive/d186bcf8d643888fac81784f9c91ffb2fb48791c.zip",
"https://github.com/privacysandbox/protected-auction-key-value-service/archive/bdadee7c80dd84197f9253d4fd92c310f457be00.zip",
],
)

Expand Down
24 changes: 24 additions & 0 deletions api/bidding_auction_servers.proto
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,11 @@ message ProtectedAudienceInput {
// Clients can indicate whether K-Anonymity is enforced for the traffic.
// By default, this would be false, i.e. K-Anonymity is not enforced.
bool enforce_kanon = 9;

// Clients can indicate whether this request can be used for debug.
// By default, this would be false, i.e. cannot be used for debug.
// Currently Android request with ad-id is eligible.
bool prod_debug = 10;
}

// ProtectedAuctionInput is generated and encrypted by the client,
Expand Down Expand Up @@ -128,6 +133,11 @@ message ProtectedAuctionInput {
// Clients can indicate whether K-Anonymity is enforced for the traffic.
// By default, this would be false, i.e. K-Anonymity is not enforced.
bool enforce_kanon = 9;

// Clients can indicate whether this request can be used for debug.
// By default, this would be false, i.e. cannot be used for debug.
// Currently Android request with ad-id is eligible.
bool prod_debug = 10;
}

// Grouping of data pertaining to protected app signals.
Expand Down Expand Up @@ -976,6 +986,10 @@ message GetBidsRequest {
// Clients can indicate whether K-Anonymity is enforced for the traffic.
// By default, this would be false, i.e. K-Anonymity is not enforced.
bool enforce_kanon = 16;

// A boolean value which indicates whether this request can be used for debugging.
// current eligible request: Android request with ad-id
bool is_debug_eligible = 17;
}

// Encrypted GetBidsRawRequest.
Expand Down Expand Up @@ -1226,6 +1240,13 @@ message GenerateBidsRequest {
// Clients can indicate whether K-Anonymity is enforced for the traffic.
// By default, this would be false, i.e. K-Anonymity is not enforced.
bool enforce_kanon = 12;

// Optional. Value of the data version header from the buyer kv server.
// This is passed through to the buyer reporting function.
uint32 data_version = 13;

// If BFE is sampling this request for debug.
bool is_sampled_for_debug = 14;
}

// Encrypted GenerateBidsRawRequest.
Expand Down Expand Up @@ -1327,6 +1348,9 @@ message GenerateProtectedAppSignalsBidsRequest {
// Clients can indicate whether K-Anonymity is enforced for the traffic.
// By default, this would be false, i.e. K-Anonymity is not enforced.
bool enforce_kanon = 12;

// If BFE is sampling this request for debug.
bool is_sampled_for_debug = 13;
}

// Encrypted GenerateProtectedAppSignalsBidsRawRequest.
Expand Down
2 changes: 1 addition & 1 deletion builders/.pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ repos:
- markdown

- repo: https://github.com/DavidAnson/markdownlint-cli2
rev: v0.13.0
rev: v0.7.1
hooks:
- id: markdownlint-cli2
name: lint markdown
Expand Down
30 changes: 30 additions & 0 deletions builders/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,36 @@

All notable changes to this project will be documented in this file. See [commit-and-tag-version](https://github.com/absolute-version/commit-and-tag-version) for commit guidelines.

## 0.71.0 (2024-10-18)


### Dependencies

* **deps:** Pin AmazonLinux2023 Nitro CLI versions
* **deps:** Update AmazonLinux base images to 2024-10-01
* **deps:** Upgrade pre-commit and pylint

## 0.70.0 (2024-10-10)


### Features

* add awk, crane and jq to build-* images for the oci_pull credential helper

## 0.69.1 (2024-09-19)

### Bug Fixes

* Use digest to ensure deterministic environment for build system


## 0.69.0 (2024-09-15)


### Features

* Pin build-debian ubuntu base image from 20.04 to focal-20240530

## 0.68.1 (2024-08-21)

### Bug Fixes
Expand Down
2 changes: 1 addition & 1 deletion builders/images/build-amazonlinux2/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.

FROM amazonlinux:2.0.20240412.0
FROM amazonlinux:2.0.20241001.0

COPY /install_apps install_golang_apps install_go.sh generate_system_bazelrc .bazelversion /scripts/
COPY get_workspace_mount /usr/local/bin
Expand Down
2 changes: 1 addition & 1 deletion builders/images/build-amazonlinux2023/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.

FROM amazonlinux:2023.5.20240722.0
FROM amazonlinux:2023.5.20241001.1

COPY /install_apps install_golang_apps install_go.sh generate_system_bazelrc .bazelversion /scripts/
COPY get_workspace_mount /usr/local/bin
Expand Down
16 changes: 13 additions & 3 deletions builders/images/build-amazonlinux2023/install_apps
Original file line number Diff line number Diff line change
Expand Up @@ -46,9 +46,19 @@ function install_python() {
}

function install_nitro() {
local -r arch="$1"
local -r -A extensions=(
[amd64]="x86_64"
[arm64]="aarch64"
)
local -r ext="${extensions[${arch}]}"
if [[ -z ${ext} ]]; then
printf "Unrecognized or unsupported architecture for nitro: %s\n" "${arch}" &>/dev/stderr
return 1
fi
dnf install -y \
"aws-nitro-enclaves-cli-1.3.*" \
"aws-nitro-enclaves-cli-devel-1.3.*"
"aws-nitro-enclaves-cli-1.3.1-0.amzn2023.${ext}" \
"aws-nitro-enclaves-cli-devel-1.3.1-0.amzn2023.${ext}"
}

function install_gcc() {
Expand Down Expand Up @@ -91,7 +101,7 @@ fi

install_python
dnf_update
install_nitro
install_nitro "${BUILD_ARCH}"
install_misc
install_clang
install_golang "${BUILD_ARCH}"
Expand Down
5 changes: 4 additions & 1 deletion builders/images/build-debian/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,10 @@
# See the License for the specific language governing permissions and
# limitations under the License.

ARG BASE_IMAGE=ubuntu:20.04
# Use fixed manifest digest to ensure reproducible.
# https://hub.docker.com/layers/library/ubuntu/focal-20240530/images/sha256-85c08a37b74bc18a7b3f8cf89aabdfac51c525cdbc193a753f7907965e310ec2
# ubuntu v20.04, focal-20240530
ARG BASE_IMAGE=ubuntu@sha256:fa17826afb526a9fc7250e0fbcbfd18d03fe7a54849472f86879d8bf562c629e

# ignore this hadolint error as BASE_IMAGE contains an image tag
# hadolint ignore=DL3006
Expand Down
5 changes: 4 additions & 1 deletion builders/images/build-debian/install_apps
Original file line number Diff line number Diff line change
Expand Up @@ -68,13 +68,15 @@ function install_misc() {
bsdmainutils \
ca-certificates \
chrpath="0.16-*" \
libcurl4="7.68.*" \
curl="7.68.*" \
file="1:5.*" \
gawk="1:5.*" \
gettext="0.19.*" \
git="1:2.25.*" \
gnupg="2.2.*" \
google-perftools="2.*" \
jq="1.6-*" \
libcurl4="7.68.*" \
locales="2.31-*" \
lsb-release="11.1.*" \
openssh-client="1:8.2*" \
Expand All @@ -85,6 +87,7 @@ function install_misc() {
wget="1.20.*" \
xz-utils="5.2.*" \
zip="3.0-*"

if [[ -n ${INSTALL_LOCALE} ]]; then
printf "\nSetting locale to: %s\n" "${INSTALL_LOCALE}"
locale-gen "${INSTALL_LOCALE}"
Expand Down
7 changes: 7 additions & 0 deletions builders/images/install_golang_apps
Original file line number Diff line number Diff line change
Expand Up @@ -40,10 +40,17 @@ function install_bazelisk() {
rm -rf /bazel_root/*
}

function install_crane() {
go install github.com/google/go-containerregistry/cmd/[email protected]
CRANE="$(go env GOPATH)"/bin/crane
ln -s "${CRANE}" /usr/bin/crane
}

function cleanup() {
cd /
go clean -cache -modcache
}

install_bazelisk
install_crane
cleanup
4 changes: 2 additions & 2 deletions builders/images/presubmit/install_apps
Original file line number Diff line number Diff line change
Expand Up @@ -82,8 +82,8 @@ function install_docker() {
function install_precommit() {
/usr/bin/python3.12 -m venv "${PRE_COMMIT_VENV_DIR}"
"${PRE_COMMIT_VENV_DIR}"/bin/pip install \
pre-commit~=3.7 \
pylint~=3.1.0
pre-commit~=4.0 \
pylint~=3.1
"${PRE_COMMIT_TOOL}" --version

# initialize pre-commit cache, which needs a git repo (a temporary will suffice)
Expand Down
2 changes: 1 addition & 1 deletion builders/tests/data/hashes/build-amazonlinux2
Original file line number Diff line number Diff line change
@@ -1 +1 @@
57ca4f2381a0fc193b0476663171c4d339b6ef66c0d1f1c24bb3f48d368b38ab
566ebf3a2f4bc0a9675b51ee68de67bf0ea6d90c1fbbb9dcd9096002a24ed365
2 changes: 1 addition & 1 deletion builders/tests/data/hashes/build-amazonlinux2023
Original file line number Diff line number Diff line change
@@ -1 +1 @@
59a82d2db8173784b0b49959c9f82ead6c2e6da78a6be21cdc78520aa43741e3
9cde9b8ca775243a0093c486e229f3fa885bead7e12afc3bdd1408a919076e7e
2 changes: 1 addition & 1 deletion builders/tests/data/hashes/build-debian
Original file line number Diff line number Diff line change
@@ -1 +1 @@
c194dafd287978093f8fe6e16e981fb22028e37345e20a4d7ca84caa43f0d4c0
877727ca542620501a2822cc32424fe5387cc37f09012a08a8cfdc86881c39a5
2 changes: 1 addition & 1 deletion builders/tests/data/hashes/presubmit
Original file line number Diff line number Diff line change
@@ -1 +1 @@
560a5a1726e7b6fd2a507f72f2563eb3938d153a7d4b4aada575a7fe772873b0
59738b43c28ecd1582112ce61fdbb697aa4952636b99955fdecdcc624bd245fe
2 changes: 1 addition & 1 deletion builders/version.txt
Original file line number Diff line number Diff line change
@@ -1 +1 @@
0.68.1
0.71.0
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,7 @@ module "buyer" {
TELEMETRY_CONFIG = "" # Example: "mode: EXPERIMENT"
ENABLE_OTEL_BASED_LOGGING = "" # Example: "true"
CONSENTED_DEBUG_TOKEN = "" # Example: "123456"
DEBUG_SAMPLE_RATE_MICRO = "0"
TEST_MODE = "" # Example: "false"
BUYER_CODE_FETCH_CONFIG = "" # Example:

Expand Down Expand Up @@ -165,9 +166,10 @@ module "buyer" {
MAX_ALLOWED_SIZE_DEBUG_URL_BYTES = "" # Example: "65536"
MAX_ALLOWED_SIZE_ALL_DEBUG_URLS_KB = "" # Example: "3000"

INFERENCE_SIDECAR_BINARY_PATH = "" # Example: "/server/bin/inference_sidecar_<module_name>"
INFERENCE_MODEL_BUCKET_NAME = "" # Example: "<bucket_name>"
INFERENCE_MODEL_BUCKET_PATHS = "" # Example: "<model_path1>,<model_path2>"
INFERENCE_SIDECAR_BINARY_PATH = "" # Example: "/server/bin/inference_sidecar_<module_name>"
INFERENCE_MODEL_BUCKET_NAME = "" # Example: "<bucket_name>"
INFERENCE_MODEL_CONFIG_PATH = "" # Example: "model_config.json"
INFERENCE_MODEL_FETCH_PERIOD_MS = "" # Example: "60000"

# TCMalloc related config parameters.
# See: https://github.com/google/tcmalloc/blob/master/docs/tuning.md
Expand Down
6 changes: 6 additions & 0 deletions production/deploy/aws/terraform/modules/buyer/service.tf
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,12 @@ module "buyer_dashboard" {
region = var.region
}

module "inference_dashboard" {
source = "../../services/dashboards/inference_dashboard"
environment = var.environment
region = var.region
}

module "buyer_app_mesh" {
# Only create if using service mesh
count = var.use_service_mesh ? 1 : 0
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -582,7 +582,7 @@ resource "aws_cloudwatch_dashboard" "environment_dashboard" {
"type": "metric",
"properties": {
"metrics": [
[ { "expression": "SEARCH(' service.name=\"bidding\" deployment.environment=\"${var.environment}\" Noise=(\"Raw\" OR \"Noised\") MetricName=\"js_execution.duration_ms\" ', 'Average', 60)", "id": "e1", "label": "$${PROP('Dim.service.name')} $${PROP('Dim.deployment.environment')} $${PROP('Dim.Noise')} $${PROP('Dim.service.instance.id')}" } ]
[ { "expression": "SEARCH(' service.name=\"bidding\" deployment.environment=\"${var.environment}\" Noise=(\"Raw\" OR \"Noised\") MetricName=\"udf_execution.duration_ms\" ', 'Average', 60)", "id": "e1", "label": "$${PROP('Dim.service.name')} $${PROP('Dim.deployment.environment')} $${PROP('Dim.Noise')} $${PROP('Dim.service.instance.id')}" } ]
],
"timezone": "UTC",
"region": "${var.region}",
Expand All @@ -594,7 +594,7 @@ resource "aws_cloudwatch_dashboard" "environment_dashboard" {
"showUnits": false
}
},
"title": "js_execution.duration_ms [MEAN]"
"title": "udf_execution.duration_ms [MEAN]"
}
},
{
Expand All @@ -605,7 +605,7 @@ resource "aws_cloudwatch_dashboard" "environment_dashboard" {
"type": "metric",
"properties": {
"metrics": [
[ { "expression": "SEARCH(' service.name=\"bidding\" deployment.environment=\"${var.environment}\" Noise=(\"Raw\" OR \"Noised\") MetricName=\"js_execution.errors_count\" ', 'Average', 60)", "id": "e1", "label": "$${PROP('Dim.service.name')} $${PROP('Dim.deployment.environment')} $${PROP('Dim.Noise')} $${PROP('Dim.service.instance.id')}" } ]
[ { "expression": "SEARCH(' service.name=\"bidding\" deployment.environment=\"${var.environment}\" Noise=(\"Raw\" OR \"Noised\") MetricName=\"udf_execution.errors_count\" ', 'Average', 60)", "id": "e1", "label": "$${PROP('Dim.service.name')} $${PROP('Dim.deployment.environment')} $${PROP('Dim.Noise')} $${PROP('Dim.service.instance.id')}" } ]
],
"timezone": "UTC",
"region": "${var.region}",
Expand All @@ -617,7 +617,7 @@ resource "aws_cloudwatch_dashboard" "environment_dashboard" {
"showUnits": false
}
},
"title": "js_execution.errors_count [MEAN]"
"title": "udf_execution.errors_count [MEAN]"
}
},
{
Expand Down
Loading

0 comments on commit 532e293

Please sign in to comment.