arXiv Statistics @arxiv_stats@qoto.org

We revisit the problem of estimating $k$ linear regressors with self-selection bias in $d$ dimensions with the maximum selection criterion, as introduced by Cherapanamjeri, Daskalakis, Ilyas, and Zampetakis [CDIZ23, STOC'23]. Our main result is a $\operatorname{poly}(d,k,1/\varepsilon) + {k}^{O(k)}$ time algorithm for this problem, which yields an improvement in the running time of the algorithms of [CDIZ23] and [GM24, arXiv]. We achieve this by providing the first local convergence algorithm for self-selection, thus resolving the main open question of [CDIZ23]. To obtain this algorithm, we reduce self-selection to a seemingly unrelated statistical problem called coarsening. Coarsening occurs when one does not observe the exact value of the sample but only some set (a subset of the sample space) that contains the exact value. Inference from coarse samples arises in various real-world applications due to rounding by humans and algorithms, limited precision of instruments, and lag in multi-agent systems. Our reduction to coarsening is intuitive and relies on the geometry of the self-selection problem, which enables us to bypass the limitations of previous analytic approaches. To demonstrate its applicability, we provide a local convergence algorithm for linear regression under another self-selection criterion, which is related to second-price auction data. Further, we give the first polynomial time local convergence algorithm for coarse Gaussian mean estimation given samples generated from a convex partition. Previously, only a sample-efficient algorithm was known due to Fotakis, Kalavasis, Kontonis, and Tzamos [FKKT21, COLT'21].

In the NFL draft, teams must strategically balance immediate player impact against long-term value, presenting a complex optimization challenge for draft capital management. This paper introduces a framework for evaluating the fairness and efficiency of draft pick trades using norm-based loss functions. Draft pick valuations are modelled by the Weibull distribution. Utilizing these valuation techniques, the research identifies key trade-offs between aggressive, immediate-impact strategies and conservative, risk-averse approaches. Ultimately, this framework serves as a valuable analytical tool for assessing NFL draft trade fairness and value distribution, aiding team decision-makers and enriching insights within the sports analytics community.

Interference occurs when the potential outcomes of a unit depend on the treatment of others. Interference can be highly heterogeneous, where treating certain individuals might have a larger effect on the population's overall outcome. A better understanding of how covariates explain this heterogeneity may lead to more effective interventions. In the presence of clusters of units, we assume that interference occurs within clusters but not across them. We define novel causal estimands under hypothetical, stochastic treatment allocation strategies that fix the marginal treatment probability in a cluster and vary how the treatment probability depends on covariates, such as a unit's network position and characteristics. We illustrate how these causal estimands can shed light on the heterogeneity of interference and on the network and covariate profile of influential individuals. For experimental settings, we develop standardized weighting estimators for our novel estimands and derive their asymptotic distribution. We design an inferential procedure for testing the null hypothesis of interference homogeneity with respect to covariates. We validate the performance of the estimator and inferential procedure through simulations.We then apply the novel estimators to a clustered experiment in China to identify the important characteristics that drive heterogeneity in the effect of providing information sessions on insurance uptake.

Classification is a fundamental task in supervised learning, while achieving valid misclassification rate control remains challenging due to possibly the limited predictive capability of the classifiers or the intrinsic complexity of the classification task. In this article, we address large-scale multi-class classification problems with general error rate guarantees to enhance algorithmic trustworthiness. To this end, we first introduce a notion of group-wise classification, which unifies the common class-wise and overall classifications as special cases. We then develop a unified algorithmic framework for the general group-wise classification that consists of three steps: Pre-classification, Selective $p$-value construction, and large-scale Post-classification decisions (PSP). Theoretically, PSP is distribution-free and provides valid finite-sample guarantees for controlling general group-wise false decision rates at target levels. To show the power of PSP, we demonstrate that the step of post-classification decisions never degrades the power of pre-classification, provided that pre-classification has been sufficiently powerful to meet the target error levels. Additionally, we further establish general power optimality theories for PSP from both non-asymptotic and asymptotic perspectives. Numerical results in both simulations and real data analysis validate the performance of the proposed PSP approach.

The saddlepoint approximation to the likelihood, and its corresponding maximum likelihood estimate (MLE), offer an alternative estimation method when the true likelihood is intractable or computationally expensive. However, maximizing this approximated likelihood instead of the true likelihood inevitably comes at a price: a discrepancy between the MLE derived from the saddlepoint approximation and the true MLE. In previous studies, the size of this discrepancy has been investigated via simulation, or by engaging with the true likelihood despite its computational difficulties. Here, we introduce an explicit and computable approximation formula for the discrepancy, through which the adequacy of the saddlepoint-based MLE can be directly assessed. We present examples demonstrating the accuracy of this formula in specific cases where the true likelihood can be calculated. Additionally, we present asymptotic results that capture the behaviour of the discrepancy in a suitable limiting framework.

As demand for Large Language Models (LLMs) and AI agents rapidly grows, optimizing systems for efficient LLM inference becomes critical. While significant efforts have targeted system-level engineering, little is explored through a mathematical modeling and queuing perspective. In this paper, we aim to develop the queuing fundamentals for LLM inference, bridging the gap between queuing and LLM system communities. In particular, we study the throughput aspect in LLM inference systems. We prove that a large class of 'work-conserving' scheduling algorithms can achieve maximum throughput for both individual requests and AI agent workloads, highlighting 'work-conserving' as a key design principle in practice. Evaluations of real-world systems show that Orca and Sarathi-serve are throughput-optimal, reassuring practitioners, while FastTransformer and vanilla vLLM are not maximally stable and should be used with caution. Our results highlight the substantial benefits queuing community can offer in improving LLM inference systems and call for more interdisciplinary developments.

The Unit-Lindley is a one-parameter family of distributions in $(0,1)$ obtained from an appropriate transformation of the Lindley distribution. In this work, we introduce a class of dynamical time series models for continuous random variables taking values in $(0,1)$ based on the Unit-Lindley distribution. The models pertaining to the proposed class are observation-driven ones for which, conditionally on a set of covariates, the random component is modeled by a Unit-Lindley distribution. The systematic component aims at modeling the conditional mean through a dynamical structure resembling the classical ARMA models. Parameter estimation in conducted using partial maximum likelihood, for which an asymptotic theory is available. Based on asymptotic results, the construction of confidence intervals, hypotheses testing, model selection, and forecasting can be carried on. A Monte Carlo simulation study is conducted to assess the finite sample performance of the proposed partial maximum likelihood approach. Finally, an application considering forecasting of the proportion of net electricity generated by conventional hydroelectric power in the United States is presented. The application show the versatility of the proposed method compared to other benchmarks models in the literature.

Chronic kidney disease (CKD) is a global health challenge characterized by progressive kidney function decline, often culminating in end-stage kidney disease (ESKD) and increased mortality. To address the limitations such as the extended trial follow-up necessitated by the low incidence of kidney composite endpoint, the eGFR slope -- a surrogate endpoint reflecting the trajectory of kidney function decline -- has gained prominence for its predictive power and regulatory support. Despite its advantages, the lack of a standardized framework for eGFR slope estimand and estimation complicates consistent interpretation and cross-trial comparisons. Existing methods, including simple linear regression and mixed-effects models, vary in their underlying assumptions, creating a need for a formalized approach to align estimation methods with trial objectives. This manuscript proposes an estimand framework tailored to eGFR slope-based analyses in CKD RCTs, ensuring clarity in defining "what to estimate" and enhancing the comparability of results. Through simulation studies and real-world data applications, we evaluate the performance of various commonly applied estimation techniques under distinct scenarios. By recommending a clear characterization for eGFR slope estimand and providing considerations for estimation approaches, this work aims to improve the reliability and interpretability of CKD trial results, advancing therapeutic development and clinical decision-making.

Cohort studies of the onset of a disease often encounter left-truncation on the event time of interest in addition to right-censoring due to variable enrollment times of study participants. Analysis of such event time data can be biased if left-truncation is not handled properly. We propose a semiparametric sieve likelihood approach for fitting a linear regression model to data where the response variable is subject to both left-truncation and right-censoring. We show that the estimators of regression coefficients are consistent, asymptotically normal and semiparametrically efficient. Extensive simulation studies show the effectiveness of the method across a wide variety of error distributions. We further illustrate the method by analyzing a dataset from The 90+ Study for aging and dementia.

Reliable machine learning and statistical analysis rely on diverse, well-distributed training data. However, real-world datasets are often limited in size and exhibit underrepresentation across key subpopulations, leading to biased predictions and reduced performance, particularly in supervised tasks such as classification. To address these challenges, we propose Conditional Data Synthesis Augmentation (CoDSA), a novel framework that leverages generative models, such as diffusion models, to synthesize high-fidelity data for improving model performance across multimodal domains including tabular, textual, and image data. CoDSA generates synthetic samples that faithfully capture the conditional distributions of the original data, with a focus on under-sampled or high-interest regions. Through transfer learning, CoDSA fine-tunes pre-trained generative models to enhance the realism of synthetic data and increase sample density in sparse areas. This process preserves inter-modal relationships, mitigates data imbalance, improves domain adaptation, and boosts generalization. We also introduce a theoretical framework that quantifies the statistical accuracy improvements enabled by CoDSA as a function of synthetic sample volume and targeted region allocation, providing formal guarantees of its effectiveness. Extensive experiments demonstrate that CoDSA consistently outperforms non-adaptive augmentation strategies and state-of-the-art baselines in both supervised and unsupervised settings.

Sparsified Learning is ubiquitous in many machine learning tasks. It aims to regularize the objective function by adding a penalization term that considers the constraints made on the learned parameters. This paper considers the problem of learning heavy-tailed LSP. We develop a flexible and robust sparse learning framework capable of handling heavy-tailed data with locally stationary behavior and propose concentration inequalities. We further provide non-asymptotic oracle inequalities for different types of sparsity, including $\ell_1$-norm and total variation penalization for the least square loss.

Research on chemotherapy, heart surgery, and vaccines has indicated that the risks and benefits of a treatment could vary depending on the time of day it is administered. A challenge with performing studies on timing treatment administration is that the optimal treatment time is different for each patient, as it would be based on a patient's internal clock time (ICT) rather than the 24-hour day-night cycle time. Prediction methods have been developed to determine a patient's ICT based on biomarker measurements, which can be leveraged to personalize treatment time. However, these methods face two limitations. First, these methods are designed to output predictions given biomarker measurements from a single tissue sample, when multiple tissue samples can be collected over time. Second, these methods are based on linear modelling frameworks, which would not capture the potentially complex relationships between biomarkers and a patient's ICT. To address these two limitations, this paper introduces a recurrent neural network framework, which we refer to as LassoRNet, for predicting the ICT at which a patient's biomarkers are measured as well as the underlying offset between a patient's ICT and the 24-hour day-night cycle time, or that patient's dim-light melatonin onset (DLMO) time. A novel feature of LassoRNet is a proposed variable selection scheme that minimizes the number of biomarkers needed to predict ICT. We evaluate LassoRNet on three longitudinal circadian transcriptome study data sets where DLMO time was determined for each study participant, and find that it consistently outperforms state-of-the art in both ICT and DLMO time prediction. Notably, LassoRNet obtains a median absolute error of approximately one hour in ICT prediction and 30 to 40 minutes in DLMO time prediction, where DLMO time prediction is performed using three samples collected at sequential time points.

This paper presents a student-led activity designed to explore the use of statistical software in academic research across economics, political science, and statistics. Students reviewed replication files from major journals and repositories, gaining hands-on experience with reproducible workflows while contributing to cross-disciplinary datasets. Web-scraped metadata and student data collection, together covering more than 10,000 papers, reveal clear disciplinary patterns: Stata remains dominant in economics, while R is increasingly popular in political science and is the standard in statistics. Within the social sciences, a growing number of articles also use multiple software platforms within a single manuscript. Students reported increased understanding of academic workflows and greater awareness of software diversity in quantitative research. The activity is easy to adapt across course levels and disciplines, and we offer suggestions for follow-up assignments that reinforce key concepts in reproducibility and data fluency. The resulting insights into current software practices are also valuable for instructors seeking to align their teaching with evolving trends in research.

This paper aims to develop practical applications of the model for the highly technical measure-valued populations developed by the authors in \cite{FanEtal20}. We consider the problem of estimation of parameters in the general age and population-dependent model, in which the individual birth and death rates depend not only on the age of the individual but also on the whole population composition. We derive new estimators of the rates based on the use of test functions in the functional Law of Large Numbers and Central Limit Theorem for populations with a large carrying capacity. We consider the rates to be simple functions, that take finitely many values both in age $x$ and measure $A$, which leads to systems of linear equations. The proposed method of using test functions for estimation is a radically new approach which can be applied to a wide range of models of dynamical systems.

This paper investigates sequential change-point detection in reconfigurable sensor networks. In this problem, data from multiple sensors are observed sequentially. Each sensor can have a unique change point, and the data distribution differs before and after the change. We aim to detect these changes as quickly as possible once they have occurred while controlling the false discovery rate at all times. Our setting is more realistic than traditional settings in that (1) the set of active sensors - i.e., those from which data can be collected - can change over time through the deactivation of existing sensors and the addition of new sensors, and (2) dependencies can occur both between sensors and across time points. We propose powerful e-value-based detection procedures that control the false discovery rate uniformly over time. Numerical experiments demonstrate that, with the same false discovery rate target, our procedures achieve superior performance compared to existing methods, exhibiting lower false non-discovery rates and reduced detection delays.

Network pruning is used to reduce inference latency and power consumption in large neural networks. However, most existing methods struggle to accurately assess the importance of individual weights due to their inherent interrelatedness, leading to poor performance, especially at extreme sparsity levels. We introduce Hyperflows, a dynamic pruning approach that estimates each weight's importance by observing the network's gradient response to the weight's removal. A global pressure term continuously drives all weights toward pruning, with those critical for accuracy being automatically regrown based on their flow, the aggregated gradient signal when they are absent. We explore the relationship between final sparsity and pressure, deriving power-law equations similar to those found in neural scaling laws. Empirically, we demonstrate state-of-the-art results with ResNet-50 and VGG-19 on CIFAR-10 and CIFAR-100.

We review the literature on algorithms for estimating the index space in a multi-index model. The primary focus is on computationally efficient (polynomial-time) algorithms in Gaussian space, the assumptions under which consistency is guaranteed by these methods, and their sample complexity. In many cases, a gap is observed between the sample complexity of the best known computationally efficient methods and the information-theoretical minimum. We also review algorithms based on estimating the span of gradients using nonparametric methods, and algorithms based on fitting neural networks using gradient descent

Tangent approximation form a popular class of variational inference (VI) techniques for Bayesian analysis in intractable non-conjugate models. It is based on the principle of convex duality to construct a minorant of the marginal likelihood, making the problem tractable. Despite its extensive applications, a general methodology for tangent approximation encompassing a large class of likelihoods beyond logit models with provable optimality guarantees is still elusive. In this article, we propose a general Tangent Approximation based Variational InferencE (TAVIE) framework for strongly super-Gaussian (SSG) likelihood functions which includes a broad class of flexible probability models. Specifically, TAVIE obtains a quadratic lower bound of the corresponding log-likelihood, thus inducing conjugacy with Gaussian priors over the model parameters. Under mild assumptions on the data-generating process, we demonstrate the optimality of our proposed methodology in the fractional likelihood setup. Furthermore, we illustrate the empirical performance of TAVIE through extensive simulations and an application on the U.S. 2000 Census real data.

The gut microbiome plays a crucial role in human health, yet the mechanisms underlying host-microbiome interactions remain unclear, limiting its translational potential. Recent microbiome multiomics studies, particularly paired microbiome-metabolome studies (PM2S), provide valuable insights into gut metabolism as a key mediator of these interactions. Our preliminary data reveal strong correlations among certain gut metabolites, suggesting shared metabolic pathways and microbial co-metabolism. However, these findings are confounded by various factors, underscoring the need for a more rigorous statistical approach. Thus, we introduce microbial correlation, a novel metric that quantifies how two metabolites are co-regulated by the same gut microbes while accounting for confounders. Statistically, it is based on a partially linear model that isolates microbial-driven associations, and a consistent estimator is established based on semi-parametric theory. To improve efficiency, we develop a calibrated estimator with a parametric rate, maximizing the use of large external metagenomic datasets without paired metabolomic profiles. This calibrated estimator also enables efficient p-value calculation for identifying significant microbial co-metabolism signals. Through extensive numerical analysis, our method identifies important microbial co-metabolism patterns for healthy individuals, serving as a benchmark for future studies in diseased populations.