Revisited: Machine Intelligence in Heterogeneous Multi Agent Systems Kaustav Jyoti Borah1, Department of Aerospace Engineering, Ryerson University, Canada email: [email protected] Rajashree Talukdar2, B. Tech, Department of Computer Science and Engineering, SMIT, India. Abstract: Machine learning algorithms has been around for decades and employed to solve various sequential decision-making problems. While dealing with complex environments these algorithms faced many key challenges. The recent development of deep learning has enabled reinforcement learning algorithms to calculate the optimal policies for sophisticated and capable agents. In this paper we would like to explore some algorithms people have applied recently based on interaction of multiple agents and their components. This survey provides insights about the deep reinforcement learning approaches which can lead to future developments of more robust methods for solving many real world scenarios. 1. Introduction An agent is an entity that perceives its environment through sensors and acting upon that environment through effectors [1]. Multi agent systems is a new technology that has been widely used in many autonomous and intelligent system simulators to perform smart modelling task. It aims to provide both principles for construction of complex systems involving multiple agents and mechanisms for coordination of independent agents’ behaviors. While there is no generally accepted definition of “agent” in AI [2], In case of multi agent systems all agents are independent of each other such a way that they have access independently to the environment. Hence, they need to adopt new circutance, Therefore, each agent should incorporate a learning algorithm to learn and explore the environment. In addition, and due to interaction among the agents in multi-agent systems, they should have some sort of communication between them to behave as a group. However, software agents get great importance in academia and commercial applications with help of internet technology. Based on the heterogeneity of agent systems, multi agent systems can be further subdivided into two categories 1) Homogeneous and 2) Heterogeneous. Homogeneous MAS include agents that all have the same characteristics and functionalities, while heterogeneous MAS include agents with diverse features that means the heterogeneous systems are composed of many subagents or sub components which are not uniform throughout. It is a distributed system contains many kinds of hardware and software working together in cooperative fashion to solve a critical problem. Example of Heterogeneous systems are smart grids, computer networks, Stock Market, Power systems distributions in aircraft etc. A good point about multi agent learning is that, agent performance improves every moment. Multi agent learning systems is a well-known matter of learning through interactions with their agents and environments. Many of algorithms developed in machine learning can be transferred to settings where there are multiple, interdependent, interacting learning agents. However, they may require modification to account for the other agents in the environment [3,4]. In our paper, we focussed on different machine learning techniques. The paper organises as section 3 describes diffreneces of single and multiple agent, section 4 describes the environments in multi agent system, section 5 describes general heterogeneous system architecture, section 6 machine learning techniques followed by section 7 applications and future work. 2. Ongoing Research This research is carried out during my PhD studies so far has been focused primarily on multiagent systems and machine learning. 3. Single agent vs multi agent systems 1) When there is only one agent defined in the system environment, it is known as single agent system (SAS) Single agent system interacts only with the environment. 2) SAS is a totally centralized system. 3) Centralized systems have one agent which decides everything like decision and action, while other acts as remote slaves. 4) "single-agent system'' can be called as complex, centralized system in a domain which also allows for a multi-agent approach. 5) A single-agent system might still have multiple entities - several actuators, or even several robots. However, if each entity sends its perceptions to and receives its actions from a single central processor, then there is only a single agent: the central process. 6) Multi Agent systems have more than one agent which interacts itself and ith the environment to give a better suggestion. 7) Multi agent system differ from single agent system most significantly in that the environment's dynamics can be determined by other agents. In addition to the uncertainty that may be inherent in the domain, other agents intentionally affect the environment in unpredictable ways. Thus, all multiagent systems can be viewed as having dynamic environments. 8) Multi-agent systems can manifest self-organization as well as self-direction and other control paradigms and related complex behaviors even when the individual strategies of all their agents are simple. 9) In central agent system all models are a single as a self, that section completes all single agent and multi agent approaches. 10) Multi agent systems are implemented in computer simulations stepping the system through discrete "time steps, they communicate using weighted request matrix. 4. Multi Agent System Environments Agents can operate in many different types of environments. The main categories are summarised below, in mutually excluding pairs, based on the definitions provided by [30] [31] 1. Static environments: the environment in which the agent operates reacts only based on the agent's action. 2. Dynamic environments: the environment can change without input from the agent, to potentially unknown states. 3.Fully observable environments: the full state of the environment is available to the agent at each point in time. 4.Partially observable environments: parts of the state of the environment are unobservable to the agent. 5. Deterministic environments: an action taken in the current state fully determines the next state of the environment. 6. Stochastic environments: an action taken in the current state can lead to different states. 7. Stationary environment: a stationary environment does not evolve over time and has a predefined set of states. 8.. Non-stationary environment: a non-stationary environment evolves over time and can lead an agent to previously unencountered states. 5. General Heterogeneous Multi agent architecture Figure 1: General Heterogeneous Architecture The figure here shows general multi agent scenario for heterogeneous communicating agents. These agents are communicating to each other to fulfill a global goal. In the figure above, we have “m” group of heterogeneous agents where A= {A1, A2, … Am} are the group of agents. Inside group 1, we assume that all agents are homogeneous. All the arrow mark with blue colours shows a communication link between the agents. 6. Machine learning Techniques Multi Agent systems is a well-known subfield of Artificial Intelligence area and make significant impact on creating and solving very complex framework problems. Well there is no specific definition of agent in multi agent systems hence as stated earlier we consider agents as entity with goals, action and domain knowledge in the environment. They will act as a “behaviour”. Although the ability to consider coordinating behaviors of autonomous agents is a new one, the field is advancing quickly by building upon pre- existing work in the field of Distributed Artificial Intelligence (DAI) [2]. Machine learning perspective is always distinguished by what kind of reward or return that critic can provide as a feed back to learner. There are three main techniques [5] that were discussed widely earlier was 1) Supervised Learning 2) Unsupervised learning and 3) Reinforcement learning. In supervised learning, the critic provides the correct output of the system where as in unsupervised learning, no feedback is at all guaranteed. While in reinforcement learning, a new method is introduced termed as actor-critic techniques which is applied to the learner’s output which provides an ultimate reward. In all three cases, the learning feedback is assumed to be provided by the system environment or the agents themselves. The use of multiagent reinforcement learning algorithms has attracted attention because of its generality and robustness. Those techniques have been applied in both stationary and non-stationary environments. 6.1 Supervised Learning It is a machine learning technique that can learn a function that maps an input to an output based on example input-output pairs. It infers a function from labeled training data consisting of a set of training example. Supervised learning considers each example as a pair that consists of an input and a desired output. Supervised learning-based algorithm analyzes the training data and produces an inferred function, which can be used for mapping new examples. Optimal scenario will allow for algorithm to determine the perfect class labels for critical and unseen instances. This requires the learning algorithm to generalize from the training data to unseen situations in a "reasonable" way. There are lots of complexity arises when multiple agents interact each other, hence supervised learning algorithms cannot be applied directly because they typically assume a critic that can provide the agents with the “correct”