Wenjie Sun (孙文杰)

The guidance of scaling laws has increased the resource demands of modern large language models (LLMs), yet it remains questionable whether these models utilize resources effectively under a fixed budget. Previous research has proved superposition as a key contributor to loss. By leveraging the Neural Feature Ansatz, we extend superposition from parameter space to gradient space and define it as neural interaction. We find that under a fixed budget, good generalization is usually accompanied by efficient neural interactions, and the model can be placed in an efficient interaction interval by adjusting its depth-width ratio. Our findings reveal that the depth-width ratio influences resource utilization efficiency and thereby affects generalization, providing insights into model shape initialization and the understanding of model generalization mechanisms.

Polysemy has long been a major challenge in Mechanistic Interpretability (MI), with Sparse Autoencoders (SAEs) emerging as a promising solution. SAEs employ a shared encoder to map inputs to sparse codes, thereby amortizing inference costs across all instances. However, this parameter-sharing paradigm inherently conflicts with the MI community's emphasis on instance-level optimality, including the consistency and stitchability of monosemantic features. We first reveal the trade-off relationships among various pathological phenomena, including feature absorption, feature splitting, dead latents, and dense latents under global reconstruction-sparsity constraints from the perspective of training dynamics, finding that increased sparsity typically exacerbates multiple pathological phenomena, and attribute this trade-off relationship to amortized inference. By reducing reliance on amortized inference through the introduction of semi-amortized and non-amortized approaches, we observed that various pathological indicators were significantly mitigated, thereby validating our hypothesis. As the first step in this direction, we propose Local Amortized SAE (LocA-SAE), a method that groups polysemantically close latents based on the angular variance. This method is designed to balance the computational cost of per-sample optimization with the limitations of amortized inference. Our work provides insights for understanding SAEs and advocates for a paradigm shift in future research on polysemy disentanglement. The code is available at https://github.com/wenjie1835/sae-amortization.

Electrocardiography (ECG) plays a central role in cardiovascular diagnostics, yet existing automated approaches often struggle to generalize across clinical tasks and offer limited support for open-ended reasoning. We present DiagECG, a novel framework that integrates time-series (TS) and language modeling by enabling large language models (LLMs) to process 12-lead ECG signals for clinical text generation tasks. Our approach discretizes continuous ECG embeddings into symbolic tokens using a lead-independent encoder and quantization module. These symbolic representations are then used to extend the LLM’s vocabulary with ECG-specific tokens, allowing the model to handle both ECG and natural language inputs in a unified manner. To bridge the modality gap, we pretrain the model on an autoregressive ECG forecasting task, enabling the LLM to model temporal dynamics using its native language modeling capabilities. Finally, we perform instruction tuning on both ECG question answering and diagnostic report generation. Without modifying the core model, DiagECG achieves strong performance across tasks while maintaining generalization to out-of-distribution settings. Extensive experiments demonstrate the effectiveness of each component and highlight the potential of integrating symbolic ECG representations into LLMs for medical reasoning.

Sparse Autoencoders (SAEs) have emerged as a predominant tool in mechanistic interpretability, aiming to identify interpretable monosemantic features. However, how does sparse encoding organize the representations of activation vector from language models? What is the relationship between this organizational paradigm and feature disentanglement as well as reconstruction performance? To address these questions, we propose the SAEMA, which validates the stratified structure of the representation by observing the variability of the rank of the symmetric semipositive definite (SSPD) matrix corresponding to the modal tensor unfolded along the latent tensor with the level of noise added to the residual stream. To systematically investigate how sparse encoding alters representational structures, we define local and global representations, demonstrating that they amplify inter-feature distinctions by merging similar semantic features and introducing additional dimensionality. Furthermore, we intervene the global representation from an optimization perspective, proving a significant causal relationship between their separability and the reconstruction performance. This study explains the principles of sparsity from the perspective of representational geometry and demonstrates the impact of changes in representational structure on reconstruction performance. Particularly emphasizes the necessity of understanding representations and incorporating representational constraints, providing empirical references for developing new interpretable tools and improving SAEs.

Innovation is a key driver of modern economic and technological development. Correct and equitable identification of innovation is essential for promoting market competitiveness and ensuring the optimal allocation of resources. Existing research on innovation evaluation mainly focuses on qualitative or quantitative evaluation of the results, while ignoring potential biases in the application process. This work investigates an unexplored issue in the field of innovation evaluation: Whether the technicality of the title of an application affects its degree of attention in the review process? The key lies in two aspects: how to evaluate the technicality of the title and how to quantify this effect. To achieve this goal, we combine the term extraction schemes and causal inference techniques by modelling the fairness detection task in a causal diagram, and propose a novel framework called CauseTerML. The framework can be applied to fairness detection in a variety of application scenarios. Extensive experiments on a real-world patent dataset validate the effectiveness of CauseTerML.

Accurate estimation of the State of Health (SOH) of lithium-ion batteries (LIBs) is of significant importance for the utilization of electrical devices powered by batteries, maintenance of battery energy storage equipment, and economic considerations for battery storage applications. Data-driven methods have gained increased attention due to their simple modeling and real-time learning capabilities. However, these methods have been constrained by cumbersome feature engineering and limited model transferability. Considering the emergence of time series foundation models, this study fine-tunes TimeGPT using cycling data from 143 LIBs with six different cathode materials to enhance the model’s accuracy and generalization capability in SOH estimation. Experimental results demonstrate that the fine-tuned model can adapt to various LIBs and operating conditions, exhibiting strong adaptability and transferability. After 100 steps of fine-tuning, the model maintains an RMSE below 1.06% and MAE below 0.59% across different Test sets, achieving an average reduction of 21.55% in RMSE and 13.55% in MAE compared to zero-shot inference. This research treats LIBs SOH estimation as a time series predication downstream task, validates the feasibility of fine-tuning, and provides a new solution for developing next-generation battery management systems.

Sodium-ion batteries (SIBs) have shown great promise as an alternative to lithium-ion batteries (LIBs) due to abundant sodium resources. Accurate and universal state of charge (SOC) estimation is essential for building an effective battery management system (BMS) for these emerging batteries. However, traditional SOC estimation methods for LIBs cannot be directly applied to SIBs due to significant differences in charge–discharge mechanisms and electrochemical characteristics. To address challenges with SIBs, this study proposes a novel framework integrating deep learning models. The BiLSTM is implemented to learn patterns from current and voltage time series. Additionally, the N-BEATS network extracts high-level features without manual feature engineering to mitigate the low sensitivity of SOC to voltage. By combining strengths of both networks, the fused model enhances SOC prediction robustness. Specifically, the proposed model is trained under various operating conditions and evaluated on both training and untrained datasets. Experiments demonstrate the fused model reduces root mean square error (RMSE) by 11.24% and 74.44% compared to individual N-BEATS and BiLSTM networks. The SOC estimation achieves mean absolute error (MAE) and RMSE below 0.30% and 0.39%, respectively. This research can inform the development of effective BMS for practical applications of SIBs, paving the way to the application of the new battery type.