Gradient Coordination for Quantifying and Maximizing Knowledge Transference in Multi-Task Learning

Multi-task learning (MTL) has been widely applied in online advertising and recommender systems. To address the negative transfer issue, recent studies have proposed optimization methods that thoroughly focus on the gradient alignment of directions or magnitudes. However, since prior study has proven that both general and specific knowledge exist in the limited shared capacity, overemphasizing on gradient alignment may crowd out task-specific knowledge, and vice versa. In this paper, we propose a transference-driven approach CoGrad that adaptively maximizes knowledge transference via Coordinated Gradient modification. We explicitly quantify the transference as loss reduction from one task to another, and then derive an auxiliary gradient from optimizing it. We perform the optimization by incorporating this gradient into original task gradients, making the model automatically maximize inter-task transfer and minimize individual losses. Thus, CoGrad can harmonize between general and specific knowledge to boost overall performance. Besides, we introduce an efficient approximation of the Hessian matrix, making CoGrad computationally efficient and simple to implement. Both offline and online experiments verify that CoGrad significantly outperforms previous methods.

Personalized Inter-Task Contrastive Learning for CTR&CVR Joint Estimation

In most of advertising and recommendation systems, multi-task learning (MTL) paradigm is widely employed to model diverse user behaviors (e.g., click, view, and purchase). Existing MTL models typically use task-shared networks with shared parameters or a routing mechanism to learn the commonalities between tasks while applying task-specific networks to learn the unique characteristics of each task. However, the potential relevance within task-specific networks is ignored, which is intuitively crucial for overall performance. In light of the fact that relevance is both task-complex and instance-specific, we present a novel learning paradigm to address these issues. In this paper, we propose Personalized Inter-task COntrastive Learning (PICO) framework, which can effectively model the inter-task relationship and is utilized to jointly estimate the click-through rate (CTR) and post-click conversion rate (CVR) in advertising systems. PICO utilizes contrastive learning to integrate inter-task knowledge implicitly from the task representations in task-specific networks. In addition, we introduce an auxiliary network to capture the inter-task relevance at instance-level and transform it into personalized temperature parameters for contrastive learning. With this method, fine-grained knowledge can be transferred to improve MTL performance without incurring additional inference costs. Both offline and online experiments show that PICO outperforms previous multi-task models significantly.

AdaSparse: Learning Adaptively Sparse Structures for Multi-Domain Click-Through Rate Prediction

Click-through rate (CTR) prediction is a fundamental technique in recommendation and advertising systems. Recent studies have proved that learning a unified model to serve multiple domains is effective to improve the overall performance. However, it is still challenging to improve generalization across domains under limited training data, and hard to deploy current solutions due to their computational complexity. In this paper, we propose a simple yet effective framework AdaSparse for multi-domain CTR prediction, which learns adaptively sparse structure for each domain, achieving better generalization across domains with lower computational cost. In AdaSparse, we introduce domain-aware neuron-level weighting factors to measure the importance of neurons, with that for each domain our model can prune redundant neurons to improve generalization. We further add flexible sparsity regularizations to control the sparsity ratio of learned structures. Offline and online experiments show that AdaSparse outperforms previous multi-domain CTR models significantly.

Towards Personalized Bundle Creative Generation with Contrastive Non-Autoregressive Decoding

Current bundle generation studies focus on generating a combination of items to improve user experience. In real-world applications, there is also a great need to produce bundle creatives that consist of mixture types of objects (e.g., items, slogans and templates) for achieving better promotion effect. We study a new problem named bundle creative generation: for given users, the goal is to generate personalized bundle creatives that the users will be interested in. To take both quality and efficiency into account, we propose a contrastive non-autoregressive model that captures user preferences with ingenious decoding objective. Experiments on large-scale real-world datasets verify that our proposed model shows significant advantages in terms of creative quality and generation speed.

CREATER: CTR-driven Advertising Text Generation with Controlled Pre-Training and Contrastive Fine-Tuning

This paper focuses on automatically generating the text of an ad, and the goal is that the generated text can capture user interest for achieving higher click-through rate (CTR). We propose CREATER, a CTR-driven advertising text generation approach, to generate ad texts based on high-quality user reviews. To incorporate CTR objective, our model learns from online A/B test data with contrastive learning, which encourages the model to generate ad texts that obtain higher CTR. To alleviate the low-resource issue, we design a customized self-supervised objective reducing the gap between pre-training and fine-tuning. Experiments on industrial datasets show that CREATER significantly outperforms current approaches. It has been deployed online in a leading advertising platform and brings uplift on core online metrics.