Human vs AI: Who Is Accountable When AI Makes Financial Decisions?

A Governance Challenge in the Age of Algorithmic Decision-Making

 

Abstract

Using Artificial Intelligence (AI), in financial decision making has provided substantial efficiencies in credit scoring, portfolio management, fraud detection etc., however it has also produced a fundamental governance dilemma: who is accountable for AI-driven decisions that result in adverse outcomes? This paper analyzes the accountability gap created by algorithmic decision-making in finance from a legal, operational and ethical perspective. It argues that current accountability frameworks designed for human decision makers are insufficient for AI systems. The study proposes a hybrid accountability model combining human oversight, model governance and regulatory alignment to ensure that AI is deployed responsibly.

  1. Introduction

AI systems have been rapidly adopted by financial institutions for automated decision-making related to high-risk matters such as loan approvals, automated trading, and fraud detection. By implementing these AI systems, they are able to achieve higher levels of efficiency, scalability and prediction. However, there is a major governance problem that arises with the use of AI systems; namely, who will be held accountable if an AI system makes a poor or incorrect financial decision? This is especially important today due to the increasing complexity of our global financial systems and their interconnectivity. The failure of an AI driven decision could result in both financial loss at the firm level and potentially create a risk to the overall stability of the financial system (Financial Stability Board, 2017). As such, accountability regarding AI has become a primary concern in financial regulation and governance.

  1. Nature of AI-Driven Financial Decisions

The major difference between AI systems and the traditional rule-based systems is how they function. This is due to the fact that they generate their output based on:
• Inferring results using data and not determinative or absolute rules
• Generating probabilistic outcomes instead of definitive or final decisions
• Learning and adapting continuously
The above mentioned differences will also create significant transparency issues with the traceability of a decision’s reasoning when an organization uses a complex model like deep neural network. Therefore, auditing this type of system can be very complicated (López de Prado, 2018).

  1. The Accountability Gap

3.1 Diffusion of Responsibility

There are several actors involved with an AI system. For example, developers, data scientists, managers, users – all of whom have some role in its development and operation. As a result, there may be a “diffusion of responsibility” which makes it difficult to assign full blame to one actor for the outcome(s) generated by the AI system.

This multiplicity of actors also leads to increased complexity of governance structures and decreased effectiveness of traditional means of assigning accountability (Varian, 2019).

3.2 Opacity and Explainability Challenges

In many cases ML models lack “explainability,” i.e., they do not provide clear explanations of how a particular decision was made; e.g. why a bank granted credit approval on a loan; or why a trade action was taken. The lack of explainability is a significant factor in limiting the ability to hold someone accountable for that decision. In areas subject to regulatory oversight, the need to demonstrate transparent decision-making processes is critical to maintaining public trust.

Lack of explainability has been cited as one of the primary barriers to AI adoption in the banking industry (European Banking Authority, 2021).

3.3 Legal Ambiguity

The legal framework of existing laws is based on a model of human agency. With the introduction of autonomous decision making, through an AI system, there will likely be new questions raised about liability. Questions are already being asked:
• Can institutions be held liable for decisions made with AI?
• What does negligence mean when it comes to deploying an AI?

Many jurisdictions have yet to address these questions, and they will continue to develop as regulations continue to develop.

  1. Case Dimensions of Accountability

4.1 Credit Scoring and Lending

The use of AI-based credit scoring could also lead to bias, which would then result in discrimination. It has been found through research that algorithmic decisions can unintentionally perpetuate inequality if an algorithm is trained using historically biased data (Barocas & Selbst, 2016).

4.2 Algorithmic Trading

The use of automated trading systems increases speed and decreases human involvement. While automated trading systems may help to increase operational efficiency by processing information faster than a human being; however, the same is true regarding increased volatility within the markets.

An example of this was demonstrated during the Flash Crash of 2010 which indicated how an automated system could potentially cause a disruption of an entire system.

4.3 Fraud Detection Systems

Fraud detection using artificial intelligence (AI) based systems will improve how accurately fraudulent activity is detected. There are also concerns about how false positive and false negative errors may be introduced with AI-based fraud detection systems. If a system incorrectly classifies a legitimate transaction as illegitimate then the business or entity responsible could suffer financially or experience some form of reputational damage. Therefore, the concern is who will ultimately be held accountable for how well an organization uses a fraud detection system.

  1. Limitations of Traditional Accountability Frameworks

The conventional frameworks for assessing traditional financial accountability have relied on three key elements:
• A clearly identifiable owner of the decision-making process
• The ability to trace back through each step to understand how each decision was made
• Static rule based systems that remain unchanged over time
As a direct consequence of the complex and dynamic nature of AI systems, there exists no current framework for governance to effectively address the financial risks associated with them (Financial Stability Board, 2017).

  1. A Hybrid Accountability Framework

6.1 Human-in-the-Loop Oversight

In spite of the increasing use of AI in various aspects of the financial industry, human oversight is still necessary when making critical decisions regarding an organization’s finances.

6.2 Model Risk Management (MRM)

A well-designed model risk management system will include model validation/stress testing, continuous model performance monitoring as well as detection of model drift. These practices are consistent with many of the long-standing risk management practices used within the financial services industry (López de Prado, 2018).

6.3 Explainable AI (XAI)

SHAP values and feature importance analysis provide additional transparency allowing stakeholders to better understand how models generate results. Transparency has become an increasing number of regulatory requirements in the financial services industry, particularly in banking (European Banking Authority, 2021).

6.4 Clear Allocation of Responsibility

There should be clear definitions of who is accountable:
• Developer – Design and Data Integrity of Model
• Management – Decision of Deployment
• Institution – Overall Accountability
These clearly assigned accountabilities help reduce the ambiguity associated with accountability.

6.5 Regulatory Alignment

Regulatory bodies have established frameworks regarding governance of AI that will focus on transparency, fairness and risk management. Financial institutions need to comply with these emerging regulations when it comes to implementing their AI systems.

  1. Emerging Market Perspective

Countries like Bangladesh present unique challenges to accountability due to:
• Limited regulatory framework governing AI
• Lower digital infrastructure maturity
• Lack of experts in governance of AI
In this environment, financial institutions must establish pro-active governance processes, regardless if there are no explicit regulatory mandates.

  1. Case Studies: AI Accountability in Bangladesh’s Financial Sector

8.1 AI in Digital Credit Scoring and Lending

Rapidly expanding digital lending in Bangladesh has compelled banks and fintechs to develop artificial intelligence (AI)-based credit scoring systems. In this regard, institutions such as BRAC Bank and mobile money services like bKash have increasingly turned toward alternative data sources (i.e., transaction history, mobile use) for assessing borrowers’ creditworthiness. Alternative data is used by these institutions to determine how much a potential borrower can afford to borrow (Bangladesh Bank, 2023; International Finance Corporation, 2020).

Use Case

AI models evaluate:

    • Transaction frequency and volume
      • The mobile user’s behavior with mobile wallets
      • Payment patterns from micro-loan repayment
      Alternative data allows lenders to extend financial inclusion to individuals who were previously unbanked (International Finance Corporation, 2020).

Accountability Challenges

    1. Opacity of Decision-Making
      Lenders have little ability to explain to their customers why they receive approval or denial for a loan, which creates problems around transparency (European Banking Authority, 2021).
    2. Algorithmic Bias
      Algorithms that are trained using past financial data could be biased against lower income or rural populations and could create even greater inequalities within society based upon those same characteristics (Barocas and Selbst, 2016).
    3. Data Privacy Risks

Using alternative data creates issues concerning whether customers provide adequate consent and if there is an appropriate level of ethics surrounding the data being collected (Financial Stability Board, 2017).

Implication

Although the advancement of technology such as Artificial Intelligence (AI) is rapidly changing many aspects of the way banking is performed and processed, the responsibility for assuring the continued fair treatment of customers, protecting consumers and complying with federal regulation falls squarely on the shoulders of institutions. Institutions have also taken proactive roles as evidenced by the central bank of Bangladesh’s role overseeing digital financial service and its focus on the welfare of consumers (Bangladesh Bank, 2023).

8.2 AI in Banking Risk Management

Many large commercial banks in Bangladesh, such as Eastern Bank Ltd. and City Bank Ltd., have started using machine learning (ML) techniques in their credit risk assessment and early warning systems as part of a broader digital transformation effort (Asian Development Bank, 2021; Bangladesh Bank, 2023).

Use Case

Currently, AI systems are used for the following applications:

    • To predict loan default
    • To identify high-risk borrowers
    • To monitor portfolio quality

All of these applications utilize algorithms that are based on the combination of data from financial ratio analysis, behavioral indicator analysis, and macro-economic variable analysis.

Accountability Challenges

    1. Model Risk and Drift

Economic conditions (e.g. inflation, exchange rates) can lead to changes in performance of models. The Asian Development Bank reports that adverse effects from changes in economic conditions can result in a loss of predictive accuracy for models (Asian Development Bank, 2021).

    1. Over-Reliance on Automated Decisions

Concerns have been expressed about the potential over-reliance by credit officers on the output generated by AI-based models for decision-making purposes. A report issued by López de Prado states that there is a fear that credit officers may rely so much on the output generated by AI-based models that they do not adequately validate the accuracy of the decisions generated by those models. (López de Prado, 2018).

    1. Regulatory Compliance

Commercial banks operating in Bangladesh are required by law to engage in prudent credit risk management. However, since the use of artificial intelligence (AI) is relatively new in banking, regulatory bodies still need to develop regulations governing the use of AI. At present, therefore, there is no clearly defined framework for applying current regulations to AI-based models. (Bangladesh Bank, 2023).

Implication

In order to minimize some of the accountability risks associated with the growing use of AI technologies by commercial banks, commercial banks should implement comprehensive Model Risk Management (MRM) Programs which include validation activities, continuous monitoring and sufficient documentation to demonstrate compliance with regulatory requirements.

8.3 AI in Fraud Detection and AML Systems

The Financial Institutions and Mobile Financial Service Providers have been using Artificial Intelligence (AI), as it has become a key tool to assist in detecting fraudulent transactions and in Anti-Money Laundering (AML) Compliance (Financial Stability Board, 2017).

Use Case

Use Cases include the use of AI models to identify and evaluate transactional anomalies, network relationships, and behavioral patterns. These evaluations can result in the immediate identification of suspicious activity.

Accountability Challenges

There are three major challenges that account for the lack of accountability when utilizing AI for AML/transactional fraud purposes.

    1. False Positives
      This occurs when legitimate transactions are inaccurately identified and flagged as fraudulent which negatively impacts the customer’s experience (European Banking Authority, 2021).
    2. False Negatives
      This occurs when the AI system fails to correctly identify fraudulent transactions which causes financial losses and regulatory consequences for the institution (Financial Stability Board, 2017).
    3. Explainability Issues
      Explainability challenges exist because there is limited ability to explain why certain transactions were flagged as suspicious resulting in further complications regarding AML compliance.

Implication

Although institutions will continue to bear responsibility for ensuring accurate detection of potential transactions that could be considered as fraudulent, institutions are also responsible for protecting their customers from potentially fraudulent transactions, and for ensuring they adhere to all applicable AML regulations.

8.4 Fintech Innovation vs Governance Gap

As Bangladesh continues to grow its Fintech ecosystem through increased mobile penetration and digital financial services, the country still lacks the necessary standards and guidelines governing how AI is adopted and used. As such, the country remains at a significant disadvantage due to this “governance gap” (Asian Development Bank, 2021 & IFC, 2020).

Key Observations

Based on research conducted into how Fintech innovations are being utilized within Bangladesh, several important observations were made.

    • The rapid growth of the adoption of AI technologies by financial institutions in Bangladesh was occurring without the presence of validated and standardized evaluation frameworks.
    • Little or no regulatory guidance existed regarding the accountability for algorithmically generated decision-making processes.
    • A number of skill gaps exist related to the governance and auditing of AI systems.

Systemic Risk Perspective

Unless properly overseen by appropriate authorities, widespread utilization of AI in the financial sector could create systemic risk. Some of these systemic risks could include mispricing of credit risk, lending decisions that contain systemic biases against certain demographic groups and consumers’ loss of confidence in the integrity of financial institutions. The aforementioned concerns mirror some of the warnings issued globally about the potential risks associated with AI technology adoption in financial institutions (Financial Stability Board, 2017).

8.5 Synthesis: Accountability in the Bangladesh Context

All of these case study analyses have commonality across the cases and include:

    1. Institutions Retain Ultimate Responsibility

AI does not remove the responsibility of banks and other financial institutions from being held accountable (Financial Stability Board, 2017).

    1. Transparency Remains a Critical Gap

The vast majority of all AI systems are unable to provide an adequate level of explanation to users and to regulators as reported by European Banking Authority (European Banking Authority, 2021).

    1. Governance Frameworks Are Underdeveloped

Bangladesh has less formalized regulatory frameworks for the use of AI compared to most developed countries (Asian Development Bank, 2021).

    1. Human Oversight Is Essential

Hybrid decision making systems which combine humans with machine learning systems will be needed to maintain accountability (López de Prado, 2018).

  1. Key Insight

The use of AI does not decrease accountability; it changes how accountability is determined and distributed among stakeholders.

Responsibility is transferred from the individual who made the decision to those who designed the model, validated the model and governed its development and implementation.

  1. Conclusion

The integration of AI into the world’s financial services creates a highly complex accountability issue. While AI can enhance productivity and improve predictability related to financial decision-making it can create issues relative to attributing responsibility to individuals or entities involved in financial decision-making processes.

This paper indicates that accountability should never be given to machines or artificial intelligence and instead must be integrated into a broad based governance structure that includes human oversight and monitoring, model risk assessment and regulatory compliance.
The long term success of AI in finance will be significantly influenced by whether institutions are able to effectively integrate AI into their businesses while ensuring they take responsibly for the design, validation, testing, deployment, operation and maintenance of AI systems.

References

Asian Development Bank (2021) Digital financial services in Asia and the Pacific. Manila: ADB.

Bangladesh Bank (2023) Guidelines on ICT security and digital financial services. Dhaka: Bangladesh Bank.

Barocas, S. and Selbst, A.D. (2016) ‘Big data’s disparate impact’, California Law Review, 104(3), pp. 671–732.

European Banking Authority (2021) Report on machine learning in financial services. Paris: EBA.

Financial Stability Board (2017) Artificial intelligence and machine learning in financial services. Basel: FSB.

IFC (2020) Digital financial services and fintech in Bangladesh. Washington, DC: International Finance Corporation.

López de Prado, M. (2018) Advances in Financial Machine Learning. New York: Wiley.

Varian, H.R. (2019) ‘Artificial intelligence, economics, and industrial organization’, in The Economics of Artificial Intelligence. Chicago: University of Chicago Press.