contents| the problem | autonomous agents | how to achieve common solutions among autonomous agents | a deliberate agent architecture with norms | to sum up |

the problem...

• Multiple agents in common environments will have to face problems posed
  • by a finite world (e.g., resource scarcity), and therefore
  • by social interference.
• These are problems of social interdependence. A problem is common if it is shared and if it requires a common solution: one plan, containing multiple actions for a unique end, and if agents are interdependent within it, that is, are all necessary to carry it out.
• This creates the necessity for agents to design and set up common solutions to any of these problems. That is why solutions (whether spontaneous or deliberately issued) are effective only to the extent that they are adopted by the majority, even the vast majority, of the agents.
• How to obtain that complementary but autonomous agents adopt these solutions, if they do not actually perceive the problem nor understand the solution? There is no easy answer to this question, as social scientists know (see the free-riding problem).

autonomous agents?

• Self-sufficient: By definition, impossible!
  • Self-interested: what does it mean?
    • Selfish
      • Have own criteria to filter external inputs
      • Beliefs: they may not perceive/understand
        • Problem (cognitive biases)
        • As common (")
        • Solution (")
        • Complementarity (") -
• Goals: they accept requests when these are useful for some of their goals
  • BUT: difference between interests and goals...

  

The problem of risk and problem perception is crucial, and involves the issue of cognitive bias. It is not possible to dwell on this issue, here, but there is a lot of cognitive psychological modelling on biases in risk perception which ought to be considered. Beliefs will select goals: in fact, the formation of a new goal is a belief-based process. Agents which do not perceive a problem will not form the goal to solve that problem! On the other hand, agents may perceive the problem, but be not motivated to solve it: this is due to the difference between interests and goals: agents may know what their interests are, but still do not form goals corresponding to their interests. For example people may know that not to smoke is in their interest, but they still do not form the goal to quit smoking!

how to achieve common solutions among autonomous agents?

There are several scientific approaches which provide an answer to this question, and which are currently implemented. At least two of them are very well-known:

• bottom-up: emergent, spontaneous processes among adaptive agents:

  One classical way is by means of spontaneous evolutionary processes among adaptive and learning agents (usually implemented on evolutionary algorithms). In such a case, solutions are emergent effects of agents adapting to their (social) environment. Agents are happy because they spontaneously do well.

• top-down: designed, incentives and sanctions modifying the preferences of rational agents

  • Acquisition of solutions
  • Enforcing mechanisms
  • Violation

The complementary approach is to induce rational agents to accept by means of complex systems of incentives and sanctions to accept (institutionally) designed solutions. This approach accounts for the adoption of designed solutions, policies, norms, laws, etc. on the side of rational agents. It also allows for violation and therefore flexibility in compliance. Agents undergo sanctions and are loaded with heavy calculations!

BUT...

But both these solutions have drawbacks.

• Evolutionary processes and Adaptive Agents: socially acceptable outcomes are doubtful! As we will see, it is far from obvious that adaptive and learning agents (based upon evolutionary algorithms) allow social processes to "evolve" spontaneously in a socially desirable direction.
• Rational agents require unrealistic conditions (specified, severe and certain sanctions). On the other hand, the complementary solution, to design complex systems of incentives and sanctions to induce rational agents to act according to the global benefit, is effective only to the extent that sanctions or incentives are specified, and certainly applied. But these are highly unrealistic conditions!

Let us analyse these solutions with some detail.

bottom up: adaptive agents

• Analogy between biological evolution and social processes

  Adaptive agents (e.g., genetic algorithms) and learning agents (e.g., classiifier systems) are based upon the assumption that fitter agents and fitter strategies tend to spread through the population via genetic reproduction and/or via social imitation. In substance, agents adapt to the fitter strategies, and by adapting themselves let society evolve for the better.

• But there are two non-demonstrated statements here

  - First, agents imitate fitter agents and fitter strategies. But how do agents know what is fit, and what is not fit? "How do individuals get information about average fitness, or even observe the fitnesses of other individuals?" (Chattoe, 1998) - Secondly, by imitating fit strategies and fit agents, socially desirable outcomes spread. How to make sure that what propagates is what is socially desirable, without the intervention of some deus ex machina (the programmer)? This is rather optimistic...

top down: rational agents

• Socially desirable effects are deliberately pursued through incentives and sanctions.
• Incentives and sanctions induce rational agents to act according to global interest.
• Rational agents take decisions by calculating the subjective expected value of actions according to their utility function.
• A rational decider will comply with the norm if utility of incentive (or of avoiding sanction) is higher than utility of transgression.

Given an agent agi and a set of alternatives for action Ai = a1, ..., an, the value of each alternative (taking into account its costs) per its relative probability of occurrence will be compared. That which yields the maximum utility (including the alternative "don't act") will be put to execution.

It is easy to see how this calculation can be extended to policies, norms, laws. Suppose agents are informed about the norm (they are provided with criteria to recognise norms). How will they decide to comply with it? To simplify matters, we assume the value of compliance to be equal to the incentive (or sanction). The value of transgression is always positive since a normative action is by default inconvenient. If an incentive is lower than the value of transgression, a rational decider will not comply with the norm, unless the probability of incentive is lower than the complementary probability (to not receive incentive, or to undergo sanctions).

effects

• Rational deciders will violate a norm ni as soon as one or more of the following conditions applies:
  • Sanctions are not imposed: an incentive-based decider will certainly violate a norm if no sanction is expected to follow from violation, since by definition in absence of incentives norm compliance is individually irrational.
  • Sanctions are expected but are not specified: in such a condition a rational decider will either infer the specification of sanctions, or will not take any decision.
  • The sanction for violating ni is lower than the value of transgression with equal probability of application of the sanction (1/2).
  • The sanction for violating an incompatible norm is higher. This aspect of norm-based decision-making is important especially in societies of growing complexity, where the set of norms tends to increase, and conflicts among norms become more likely.
  • The sanction (negative incentive) for violating the norm ni is not or rarely applied: pi tends to 0. Since the utility of norm compliance, as seen above, is equal to the value of incentive (or sanction) per its relative probability of occurrence (taking into account the utility of transgression), obviously with a probability proximate to zero, the utility of incentive is also nullified.
• Fast decline in norm compliance is likely to follow from any of the above conditions. Even with a moderately convenient value of transgression, a rational decider is likely to violate the norm. Consider that both the probability and entity of sanctions may be inferred by observing others' behaviour: the more others violate, the less likely and/or severe the sanction is expected to be.

• With a homogeneous society of sanction-based deciders, a fast decline or even a collapse in norm compliance is likely to follow from any of the above conditions. The inconvenience of norm compliance will be detected sooner or later by all members of the society. Consequently, their decisions will rapidly converge on norm violation, unless the external source of sanctions monitors the behavioural effects of agents' decisions and takes efficient measures of norm enforcement, by either intensifying the application of sanctions or augmenting their entity.

• Then, what?

Top-Down and Bottom-Up

• Top down
  • agents acquire S
  • agents decide to accept S
  • have S even if they do not apply it
• Bottom up
  • infer S from others
  • communicate S to one another
  • control each other.
• Difference from previous approaches:
  • S is represented as a specific object in the mind
  • Can travel from one mind to the other.

  Of course, S can be a policy, a norm, etc.

but how to tell S?

• Sanction
  • may be sufficient (but unnecessary) for acceptance
  • but it is insufficient (and unnecessary) for recognition.

believed obligations

• May be insufficient for acceptance
• But necessary & sufficient for recognition!

this requires..

What is needed is a mentally represented solution or a norm. Why? Because if solutions are mentally represented as specific objects, links between these representations and other mental states are established

• This requires a cognitive deliberative agent:
  • To communicate, infer, control: meta-level representation (beliefs about representations of S). Then agents can not only accept or reject them.
  • But also know that solutions exist even when they do not execute them infer from and communicate to one another about solutions and possibly induce others to accept them. To decide whether to accept/reject a believed: meta-meta-level representation (decision about belief about representation of S).

When the content of representation is a Norm, we have Deliberative Normative Agents. This requires a cognitive and deliberative agent. Let us see each solution with some detail.

deliberative normative agents

Our objective is a norm-autonomous agent; i.e., an agent

• Are able to recognize the existence of norms: able to know that a norm exists in the society and that it is not simply a diffuse habit, or a personal request, command or expectation of one or more agents;
• Can decide to adopt a norm for different reasons (different meta-goals): able to adopt this norm impinging on its own decisions and behaviour
• Can deliberately follow that norm in the agent's behaviour
• or violate it in specific cases: able to deliberatively violate a norm in case of conflicts with other norms or, for example, with more important personal goals; of course, such an agent can also accidentally violate a norm (either because it ignores or does not recognize it, or because its behaviour does not correspond to its intentions).
• Can react to violations of the norm by other agents

To adopt a norm does not necessarily imply to follow it. The concept of 'adopting a norm' means that the agent decides to generate goals and plans on the basis of its belief that there is such a norm and that it also concerns the agent itself. These are the 'normative beliefs' and the generated goals are the 'normative goals'. However, although I have adopted the norm and generated the corresponding goals, these goals will not necessarily always be preferred to my other active goals: they will not necessarily become one of my intentions and determine my external behaviour. Thus, although I adopt a norm I can deliberatively violate it.

Deliberative Normative Agents

• Can reason about norms
• Can communicate norms
• Can negotiate about norms

This implies

• Have norms as mental objects
• Have different levels of representations and links among them. In this architecture norms are mental representations entering the mental processing, interacting in several ways with beliefs, goals, and plans and thus are able to determine the agent's behaviour. Next, the deliberative normative agent architecture is described globally.

a deliberative agent architecture with norms (Castelfranchi et al., 1999)

DESIRE I (Brazier et al., 1999)

see original version for diagram

This is the representation of the DESIRE architecture as developed initially by the Dutch group, plus the normative components which have been added thanks to the collaboration between our two groups.

The generic agent model further includes the components Cooperation Management, Maintenance of History, and Agent Specific Tasks. These components have been omitted in the current architecture for briefness; they can be added when needed. Furthermore, the current model includes the component Maintenance of Society Information; this component is not part of the generic agent model as presented in [7]. This component is added specifically for dealing with society properties such as norms.

Information about aspects external to the agent can be received by communication or by observation (perception). If the agent decides that information is valuable, it is stored. In storing information within the agent a distinction is made according to the content of the information, abstracting from the source of information. Information about the world (the agent's world model) is stored in the component Maintenance of World Information. Information about other agents (the agent's agent models, sometimes also called acquaintance models) is stored in the component Maintenance of Agent Information. Finally, information about the society as a whole (the agent's society model) is stored in the component Maintenance of Society Information.

DESIRE II

Main Components:

Comunication

• Agent Interaction Management, Perception
• World Interaction Management,
• Maintenance of Agent Information = agent models
• Maintenance of World Information = world models
• Maintenance of Society Information = social world models
• Own Process Control = mental states & processing: goal generation & dynamics, belief acceptance, decision-making.

Information about aspects external to the agent can be received by communication or by observation (perception)

The generic agent model further includes the components Cooperation Management, Maintenance of History, and Agent Specific Tasks. These components have been omitted in the current architecture for briefness; they can be added when needed. Furthermore, the current model includes the component Maintenance of Society Information; this component is not part of the generic agent model as presented in the generic architecture. This component is added specifically for dealing with society properties such as norms.

DESIRE III

Representation at different meta-levels

1) Object Level : Beliefs such as (society 1) you_ought_to_drive_on _the_right

2) Meta Level: communicated by (has norm (self, you_ou ght_to_drive_on _ the_right), positive assertion, agent_B)

3) Meta Meta level: Adopted_own_process_goal(y ou_ought_to_drive _on _the_right)

Society norms are also explicitly represented as a specific type of society information; for example, 'you ought to drive on the right' or 'be altruistic' as a society norm can be stored in the component Maintenance of Society Information.

The three maintenance components (Maintenance of World Information, Maintenance of Agent Information, and Maintenance of Society Information) are all at the object level.

Meta-level: In processing incoming and outgoing information (by communication or observation), the process events involved are represented at a meta-level, with reference to the information involved. For example, ifanother agent B has communicated that it has you_ought_to_drive_on _the_right as a norm, then at the input interface of the agent this is represented at a meta-level. The positive_assertion denotes the illocution. Note that to explicitly represent norms as mental concepts (in contrast to, e.g., norms as constraints outside the mental processing) makes it possible to have communication about norms. Via incoming communication, information is transferred to the component Agent Interaction Management, in which the content information is extracted from the communication information. If however, agent B communicates that you_ought_to_drive_on_the_right is a norm in society1, and the agent B is considered reliable, then an object level representation is derived, that society 1 has a given norm. For example, if agent B is a police agent, it may be considered reliable automatically.

Meta-meta-level. Since norms have an effect on the control of the agent's internal processes, within Own Process Control also a meta-meta-level is used to explicitly represent information used to reason about control on the agent's own internal processes.

DESIRE IV

Own Process Control

see original version for diagram

Before, it was shown in an example how the society norm you_ought_to_drive_on_ the_right as communicated by a police agent leads to an agent's belief that the norm you_ought_to_drive_on_the_right is a society norm in society1, stored in the component Maintenance of Society Information. Via the information links Norm Management, which is at the meta-level within Own Process Control, receives as input belief(has_norm(society1, you_ought_to_drive_on_the_right), pos). In case the agent considers itself as belonging to society1, it can decide to try to conform to such a society norm, using the knowledge

if belief(has_norm(society1, N:NORMS), pos)

and belief(belongs_to(self, society1), pos)

then adopted_norm(N:NORM)

For example, using the beliefs belief(has_norm(society1, you_ought_to_drive_on_ the_right), pos) and belief(belongs_to(self, society1), pos) it can derive: adopted_ norm(you_ought_to_drive_on_the_right). This output of Norm Management is transferred to input of the component Strategy Management. This component, which is at the meta-meta-level, uses the representation: adopted_own_process_goal( you_ought _ to_drive_on_the_right). The information link normative meta-goals specifies that semantically speaking an adopted norm corresponds to a goal on the agent's own internal processes, i.e., a meta-goal or own process goal. The component Goal Management is composed of two sub-components: Goal Generation (where candidates for goals are generated) and Goal Selection (where a choice is made from the candidate goals). In a similar manner Plan Management is composed of Plan Generation and Plan Selection. Action generation and selection is part of plan generation and selection.

DESIRE V

• Own Process Control Components information
• Norm Management (Norms Belief)
• Strategy Management (Candidate Goals)
• Goal Management (Selected Goals)
• Plan Management (Plan Chosen) (Action)

Incoming information enters the NM component, where a candidate norm may be accepted as such, or not (brackets are used to indicate a possible but not necessary outcome).

A candidate norm may be acceptedb and give rise to a normative belief. This will give input to the SM component. This will process the norm and possibly (but again not necessarily) give rise to a candidate normative goal (e.g., to drive on the right). This will enter the GM component and possibly be selected as an actual goal. Only if this selected goal, after entering the PM component, selects or generates a plan, the norm will be finally put to execution.

DESIRE VI:

Norms and goals

• non-adopted norms:
  • useful for coordination (predict the behaviour of the other agents)
• adopted norms:
  • impact on goal generation; among the possible 'sources of goals' -> normative goals
  • impact on goal selection by providing criteria about how to select among existing goals; e.g., preference criteria.

Impact of Norms on Goals: In this architecture adopted norms play two main roles in goal determination, according to the composition of the component Goal Management introduced in Section 2.4 (in terms of the components Goal Generation and Goal Selection). Norms have impact on goal generation; goals that do not derive from desires or wishes: what we should/have to do, not what we would like/wish to do. Thus norms are among the possible 'sources of goals' together with bodily needs, emotions, wishes and desires, and others. By generating goals, norms provide also a target for reasoning.

Norms have impact on goal selection by providing criteria about how to manage and select among existing goals; in particular preference criteria. They push us to prefer one goal over another. For example, the norm 'be altruistic!' or the norm 'obey to norms!' do not specify and provide the agent any specific behaviour or goal (like 'you ought to stop for a red traffic light!'), they provide a criterion about what goal should be preferred.

DESIRE VII:

Norms and plans

• Norms may generate plans
• Norms may select plans
• Norms may select actions

Eg: the norm "be kind to colleagues" may lead to a preferred plan to reach a goal within an organisation.

In general, goals have a crucial impact on the process of plan generation and selection. In the case of a deliberative normative agent, where norms have an impact on the goals that are generated and selected, in an indirect manner norms have impact on plans as well. In addition to this impact, also a direct impact is possible, especially in cases where norms indicate more that certain actions are not done, than that they indicate certain goals. For example, the norm 'always use the most friendly, least aggressive means to achieve your goals' refers to properties of actions and plans instead of specific types of goals. Similar to the previous section, in our architecture adopted norms have two types of direct impact in Plan Management:

to sum up

• Adaptive agents: fit = socially acceptable?
• Rational agents are good enough if sanctions are severe, effective, and certain. Otherwise, collapse in compliance...

with deliberative agents

• Acquisition of norms online
• Communication and negotiation (social monitoring and control)
• Flexibility:
  • agents follow the norm whenever possible
  • agents violate the norm (sometimes)
  • agents violate the norm always if possible
• But graceful degradation with uncertain sanctions!

work in progress

DESIRE is used for simulation-based experiments on the role of deliberative agents in distributed social control

• Market with heterogeneous, interdependent agents
  • Make contracts in the morning (resource exchange under survival pressure)
  • Deliver in the afternoon (norm of reciprocity)
  • Violations can be found out and
    • The news spread through the group
    • The event denounced
    • Both.
• Objectives:
  • What are the effects of violation? (Under given enviornmental conditions, norms may be non adaptive)
  • When and why agents react to violation?
  • What are the effects of reaction?

DESIRE is currently used as a platform for agent-based simulation, in order to test the role of deliberative agents in the bi-directional process mentioned above. The environment of the simulation is exclusively social. Agents with different and possibly complementary resources may need to effectuate exchange in order to obtain all resources required for survival. But since transactions are not simultaneous, agents are also induced to break the contract when they have received the resources which they need.

The objectives are more than one. Not only to check the effects of otherrs reactions to cheat. But also, to observe the effects of violation : this is not always disfunctinal, for example with severe caresty, variance in resource distribution might guarantee the survival of a share of the population, which otherwise would extinguish.

what to do next

• DESIRE is complex
  • Computationally: too few agents. Can simpler languages implement meta-level representations?
  • Mentally: too much deliberation
    • Emotional/affective enforcement? NB -> E (moral sense, guilt) -> NG -> NA.
    • Emotional shortcut others' evaluations -> SE (shame) -> NA implicit norms implicit n-goals.
    • Affective computing! But integrated with meta-level representations.

DESIRE Assumes more deliberation than actually performed by nastural agents, and probably more than needed.

In fact, emotions allow sometimes the automatic activation of goals, and the generation of normstive goals. In this sense in presence of normative beliefs, moral emotions may act as enforcing mechanisms, and spare complex conscious decision-making. Furthermore, sometimes social emotions, like shame, allow shortcut even in inferring norms. Others' evaluations may a ctg as implicit prescriptions and activate the goal to show that the groups' values are shared.