Mechanistic Interpretability of Transformer Attention Heads in Indonesian Fiscal Policy Discourse: Decoding Internal Representations and Latent Biases

Zia Ul Rehman Zafar; Dedi  Gunawan; Kashif  Ali Abdul Wahid Alias; Ahmad Hassan; Adnan Nadeem

Mechanistic Interpretability of Transformer Attention Heads in Indonesian Fiscal Policy Discourse: Decoding Internal Representations and Latent Biases

Authors

Zia Ul Rehman Zafar Universitas Muhammadiyah Surakarta
Dedi Gunawan Informatics Engineering, Faculty of Communication and Informatics, Universitas Muhammadiyah Surakarta
Kashif Ali Abdul Wahid Alias Department of Computer Science, University of Debrecen, Hungary
Ahmad Hassan Department of Economics and Management, odisee University of Applied Science, Belgium
Adnan Nadeem Department of Social Science, University of Okara, Pakistan

Keywords:

Mechanistic Interpretability, Attention Entropy, Bias Sensitivity Index, Indonesian Fiscal Policy

Abstract

Transformer-based language models are increasingly used in economic text analysis, yet their internal reasoning remains insufficiently interpretable, particularly in multilingual policy contexts. This study examines the mechanistic interpretability of attention heads in analyzing Indonesian fiscal policy discourse using a bilingual dataset comprising 47 YouTube transcripts and 23,500 news articles collected between 2018 and 2025. A fine-tuned IndoBERT model is implemented for fiscal stance classification, achieving strong performance with accuracy of 0.91, precision of 0.89, recall of 0.88, and F1-score of 0.885, ensuring a reliable basis for interpretability analysis. To uncover internal model behavior, an integrated framework combining attention entropy, head importance scoring, and Bias Sensitivity Index (BSI) is applied. Results show a consistent decline in attention entropy across layers from 2.85 to 1.62, indicating a shift from distributed contextual processing to focus semantic representation. Head-level analysis reveals functional specialization, with key attention heads reaching importance scores up to 0.25. Ablation experiments further demonstrate measurable prediction shifts, reflected in elevated BSI values, confirming sensitivity to specific heads associated with fiscal risk terms. These findings indicate that transformer models encode structured economic reasoning while exhibiting latent bias, highlighting the need for interpretability-driven approaches in AI-based policy analysis.

Downloads

Download data is not yet available.

References

S. Bahoo, J. W. Goodell, R. Rhattat, and S. Shahid, “Artificial Intelligence in Economics Research: What Have We Learned? What Do We Need to Learn?,” Journal of Economic Surveys, vol. 39, no. 5, pp. 2194–2214, 2025, doi: 10.1111/joes.12694.

O. Jaulín-Méndez and A. Ramos, “Becoming green: Decomposing the macroeconomic effects of green technology news shocks,” Energy Economics, vol. 157, 2026, doi: 10.1016/j.eneco.2026.109228.

G. I. F. de Paiva et al., “Leveraging Large Language Models for time series forecasting: A systematic literature review,” Knowledge-Based Systems, vol. 343, 2026, doi: 10.1016/j.knosys.2026.115938.

Y. Li and B. Zhang, “Can analyst textual opinions predict stock returns?,” Journal of Banking and Finance, vol. 187, 2026, doi: 10.1016/j.jbankfin.2026.107688.

S. Bi, “A Political Economy Analysis of Modern State Development in Southeast Asia,” Risks, Resilience and Interdependency: Developing Countries in the Age of Uncertainties, pp. 59–88, 2025, doi: 10.1007/978-981-96-7167-0_3.

J. K. Gummerus, D. M. O’Loughlin, C. Kelleher, and C. von Koskull, “All is not well: value codestruction and consumer illbeing in service systems,” European Journal of Marketing, vol. 60, no. 3, pp. 753–775, 2026, doi: 10.1108/EJM-04-2024-0280.

J. Chen, Y. Fu, S. Ma, Q. Chen, W. Zhang, and J. Huang, “Research on the supply-demand balance evaluation and driving mechanism of community public service facilities,” PLOS ONE, vol. 20, no. 5 May, 2025, doi: 10.1371/journal.pone.0322109.

K. Houhala, T. Korhonen, T. Laine, and V. Salminen, “Reasonable Living as a Proactive Customer Value: Feeling Heard, See, and Understood,” Applied Human Factors and Ergonomics International, vol. 176, pp. 1–14, 2025, doi: 10.54941/ahfe1006298.

P. Mikalef, I. O. Pappas, J. Krogstie, and M. Giannakos, “Big data analytics capabilities,” Information & Management, vol. 55, no. 3, pp. 263–276, 2018.

A. Susanto, P. W. Handayani, and Y. G. Sucahyo, “Digital transformation in Indonesian SMEs,” Procedia Computer Science, vol. 161, pp. 962–969, 2021.

M. H. Davies and M. Schröder, “A simple model of internal and external balance for resource-rich developing countries,” Australian Journal of Agricultural and Resource Economics, vol. 69, no. 1, pp. 178–199, 2025, doi: 10.1111/1467-8489.12589.

T. Loate and N. Viegi, “Can monetary and fiscal policy account for South Africa’s stagnation?,” Applied Economics, vol. 58, no. 11, pp. 2027–2042, 2026, doi: 10.1080/00036846.2025.2473109.

M. Teti̇k, “Do macroeconomic drivers fuel bubbles in emerging markets? Insights from GSADF and machine learning,” Acta Oeconomica, vol. 76, no. 1, pp. 142–160, 2026, doi: 10.1556/032.2026.00177.

M. M. Bojaj, “Dynamic effects of blockchain on financial markets: evidence from TVP-Bayesian VAR with a connectedness approach,” Empirical Economics, vol. 68, no. 5, pp. 2159–2197, 2025, doi: 10.1007/s00181-024-02696-8.

A. Ciunova-Shuleska, N. Palamidovska-Sterjadovska, T. Jakimova, and B. Trenovski, “Understanding Monetary Policy Communication in a Small and Open Economy Under an Exchange Rate Anchor: The Case of North Macedonia,” Eastern European Economics, vol. 63, no. 2, pp. 204–236, 2025, doi: 10.1080/00128775.2024.2304212.

E. Beqiraj, Q. Cao, R. De Haas, and R. Minetti, “Reprint of: Global banking and macroeconomic stability. Liquidity, control, and monitoring,” Journal of International Economics, vol. 156, 2025, doi: 10.1016/j.jinteco.2025.104133.

M. Ristić Cakić, B. Kalaš, J. Đurović Todorović, and M. Đorđević, “Economic drivers of tax revenue: insights from European economies,” Post-Communist Economies, vol. 37, no. 5, pp. 499–515, 2025, doi: 10.1080/14631377.2025.2461929.

I. González, J. A. Montecino, and J. E. Stiglitz, “Equity prices, market power, and optimal corporate tax policy,” European Economic Review, vol. 176, 2025, doi: 10.1016/j.euroecorev.2025.105039.

G. Rubbaniy, S. Ali, S. Abdennadher, and C. Siriopoulos, “Financial Market Determinants of Dynamic Herding in North American Energy Equity Market,” International Journal of Finance and Economics, vol. 31, no. 1, pp. 742–762, 2026, doi: 10.1002/ijfe.3165.

H. Adamu, M. A. Azmi Murad, and N. A. Nasharuddin, “A hybrid stacked ensemble learning framework for multilabel text emotion detection,” Scientific Reports, vol. 16, no. 1, 2026, doi: 10.1038/s41598-026-38172-9.

S. Shalawati, A. H. Nasution, W. Monika, T. Derin, A. Onan, and Y. Murakami, “Beyond BLEU: GPT–5, Human Judgment, and Classroom Validation for Multidimensional Machine Translation Evaluation,” Digital, vol. 6, no. 1, 2026, doi: 10.3390/digital6010008.

A. D. A. Wicaksono, “Classifcation of Social Media Posts X For Mental Health Symptoms Identification Using NLP Techniques and Transformers,” Journal of Computer Science, vol. 22, no. 1, pp. 244–259, 2026, doi: 10.3844/jcssp.2026.244.259.

R. H. Gusmita, A. F. Firmansyah, H. M. Zahera, and A.-C. Ngonga Ngomo, “ELEVATE-ID: Extending Large Language Models for End-to-End Entity Linking Evaluation in Indonesian,” Data and Knowledge Engineering, vol. 161, 2026, doi: 10.1016/j.datak.2025.102504.

A. Muzakir and U. Suriani, “Improved Performance of Cyberbullying Detection in Indonesian Using Sentence-Level Semantic Expansion,” International Journal on Informatics Visualization, vol. 10, no. 1, pp. 313–320, 2026, doi: 10.62527/joiv.10.1.4504.

Y. Tanaka and D. Suhartono, “INTEGRATING INDOBERT WITH CNN, BI-LSTM, AND ATTENTION FOR IMPROVED INDONESIAN HOAX CLASSIFICATION,” ICIC Express Letters, Part B: Applications, vol. 17, no. 3, pp. 223–230, 2026, doi: 10.24507/icicelb.17.03.223.

S. Dillah, L. T. Syahyaningsih, D. F. Surianto, N. A. E. Budiarti, and D. D. Andayani, “Integrating triplet loss with Paraphrase-Tuned SBERT for Enhancing Semantic Meaning Representation in Indonesian Text,” International Journal of Fuzzy Logic and Intelligent Systems, vol. 26, no. 1, pp. 35–46, 2026, doi: 10.5391/IJFIS.2026.26.1.35.

T. G. Desantoro, L. B. Prasetyo, and M. I. Lubis, “Media Framing of Human-Orangutan Interactions in Kalimantan: A Topic Modelling and Sentiment Analysis Approach,” Jurnal Manajemen Hutan Tropika, vol. 32, no. 1, pp. 63–72, 2026, doi: 10.7226/jtfm.32.1.63.

M. E. Rosadi, P. N. Andono, A. Z. Fanani, and A. Marjuni, “Token-aware Multi-source Attention for Indonesian Named Entity Recognition,” International Journal of Intelligent Engineering and Systems, vol. 19, no. 4, pp. 198–214, 2026, doi: 10.22266/ijies2026.0430.12.

D. Mahmoudinia, “The asymmetric effects of monetary policy, currency crises and oil price on food inflation in Iran: an application of quantile regression,” Macroeconomics and Finance in Emerging Market Economies, vol. 19, no. 1, pp. 176–198, 2026, doi: 10.1080/17520843.2021.2005926.

P. Sadorsky, “Modeling volatility and conditional correlations between socially responsible investments, gold and oil,” Economic Modelling, vol. 38, pp. 609–618, 2014, doi: 10.1016/j.econmod.2014.02.013.

B. Dalheimer et al., “Stocks and shocks: Assessing the relative roles of public and private inventories in buffering rice price volatility in the Philippines,” Food Policy, vol. 139, 2026, doi: 10.1016/j.foodpol.2026.103052.

M. Nahiduzzaman, S. K. Sarker, B. C. Kuri, B. K. Dhar, P. P. Roy, and R. Karim, “Navigating environmental vulnerability and resource dependence: Toward equitable and sustainable growth pathways in resource-rich economies,” Resources Policy, vol. 111, 2025, doi: 10.1016/j.resourpol.2025.105768.

T. Bareith and I. Fertő, “Monetary policy, exchange rates and food price dynamics: navigating heterogeneity in non-Eurozone countries,” British Food Journal, pp. 1–17, 2026, doi: 10.1108/BFJ-07-2025-0975.

T. Bareith and I. Fertő, “Monetary policy and food price inflation: the case of Hungary,” Journal of Agribusiness in Developing and Emerging Economies, vol. 16, no. 1, pp. 65–82, 2026, doi: 10.1108/JADEE-10-2023-0251.

Y. Li, B. Zhao, and M. Che, “Modeling long-run global agricultural price dynamics: a trend-cycle decomposition and unobserved components approach,” Frontiers in Sustainable Food Systems, vol. 10, 2026, doi: 10.3389/fsufs.2026.1751456.

K. Szafranek, P. Macias, D. Stelmasiak, and A. Błażejowska, “Is Big Data a Big Help? Evidence From Nowcasting Food Inflation During Covid-19 and Wartime,” Journal of Forecasting, vol. 44, no. 8, pp. 2346–2363, 2025, doi: 10.1002/for.70013.

M. Balcilar and O. Usman, “Global Energy Market Uncertainty and Exchange Rate Pass-Through to Food Inflation,” Energy Research Letters, vol. 7, 2026, doi: 10.46557/001c.125871.

D. Neděla and A. Kresta, “Applicability of Intraday Entropy for Trading During Regular Market Hours,” Computational Economics, 2026, doi: 10.1007/s10614-026-11347-2.

E. Ouassou and P. Tamasiga, “Food Price Shock Transmission Through SME Exporting Networks: A Bayesian GVAR Approach,” Global Food Value Chains: An In-depth Exploration of International Trade Dynamics, pp. 167–197, 2026, doi: 10.1007/978-3-032-11623-9_8.

R. Saputra, D. Devianto, F. Yanuar, N. Hila bt Zainuddin, and H. Yozza, “Dynamic impact of macroeconomic factors and government expenditure shocks to food inflation in Indonesia during and after COVID-19,” presented at the E3S Web of Conferences, 2025. doi: 10.1051/e3sconf/202567710003.

R. L. Aor and A. A. Salisu, “DISAGGREGATED ECONOMIC COMPLEXITY AND INFLATION IN OECD COUNTRIES,” Buletin Ekonomi Moneter dan Perbankan/Monetary and banking economics bulletin, vol. 29, no. 1, pp. 53–76, 2026, doi: 10.59091/2460-9196.2572.

Mechanistic Interpretability of Transformer Attention Heads in Indonesian Fiscal Policy Discourse: Decoding Internal Representations and Latent Biases