Grand challenges for ambient intelligence and implications for design contexts and smart societies

3.1.Human control vs. automation

The first design trade-off concerns the current shift towards more or even complete automation of previously (partially) human operator-controlled activities. Smart devices and underlying algorithms are gaining ground in controlling processes, services and devices as well as the interaction between devices and humans. Humans are increasingly removed from being the operator, supervisor or at least being in charge and thus from being in control. The problems caused by the ‘smart-everything’ paradigm can be categorized in three problem sets: A) Inability and error-prone behavior. B) Rigidity, and C) Missing transparency and traceability. Since a more elaborate description is provided in [81], we mention here only a few examples.

Error-prone behavior or inability of AI or other algorithmic approaches can be observed in many areas despite manifold promises. A major problem is the unresolved dependency of supervised machine learning on having appropriate, unbiased, and sufficient training data of high quality. As a consequence, differences in training data and algorithmic constraints result in very different results/predictions, although they are supposed to provide the same “right” answer [42]. The high expectations towards autonomous driving are disappointed by failures, for example, in recognizing speed-limit signs or being fooled by so-called ‘scam stickers’ [27]. Why are autonomous cars driving too fast although they are supposed to make traffic safer as in the recent deadly accident caused by an Uber car [54]? Moreover, during the phase of having only level 2 to 4 capabilities (which will be the standard for a long time), according to the SAE [69] classification11, wrong detection information might result in unjustified legal consequences for the human drivers because they will be still liable for damages. A practical example: the car system sends (incorrect) messages about discrepancies (driving at a speed limit correctly obeyed by the driver which is higher than a lower speed limit wrongly identified by the car) to the police or insurance company. This might result in a fine or increase in insurance premium, although based on wrong information identified and sent by the vehicle.

Rigid behavior is another problem. Users and customers experience it when confronted with fully automated call centers or on-line shops without humans involved. It needs only small deviations from the standard routine or process and the system cannot handle the requests. The problem is that customers are going to lose control and be completely at the mercy of companies and their algorithms with no recourse. Hotel booking systems repeat recommendations for hotels in cities which are not relevant anymore. Customers of on-line shops are confronted with the same category of items just bought, although one does not need multiple items in this category at the same time.

While the first two problem sets might be remedied (partially) by progress in the field, missing transparency and traceability and incomprehensible decisions are and will stay with us as an essential problem. Assuming further “progress”, AI-based behavior will increasingly become non-transparent and in-comprehensible to observers. Being untraceable implies also that there are no reproducible outcomes and a lack of liability. People are already now confronted with the problems and lack of transparency, as demonstrated in the financial domain with high frequency trading or decisions on creditworthiness. When nobody can trace the underlying argumentation or mechanisms we have a really serious problem. A more detailed discussion of these issues and additional references can be found in [81]. There are some attempts to address these problems [4,5]. The General Data Protection Regulation (GDPR) [24,25] of the European Union (EU), effective since May 2018, highlights also some of these issues. For example, the GDPR requires to provide users with an explanation or the rationale of a decision made by the underlying algorithms and provides citizens with the right to opt-out or to make different decisions.

Furthermore, it seems obvious that these challenges are intricately related with ethical issues. So, it is no coincidence that the world’s largest professional engineering community IEEE engaged in the definition of a standards body on “Ethically Aligned Design” (EAD). The efforts of the IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems (A/IS) (https://ethicsinaction.ieee.org) and its recommendations for EAD are discussed in detail in Section 6.

The problems and their implications for the welfare as well as a fair and democratic treatment of people and society at large require changing the design approach. We argue for a ‘people-empowering smartness’ instead of a ‘system-centric, importunate and automated smartness’ [81,85]. People should not only be in control, but they should “own the loop”. People should be empowered by smart capabilities and thus be able to make more informed and mature decisions. Thus, the design goal is that “smart spaces make people smarter”. This people-oriented, empowering smartness approach is in line with the AmI vision. Obviously, there is a caveat to being in control at all levels of the process and making all the decisions because people have limitations on the amount of data they can process. But they should be in control of the trade-off between the degree of automated (pre)processing and aggregation of data vs. the degree of human intervention and decision making. The degree of system automation must be configurable by the user/citizen. But to be put into the position at all of making this trade-off requires that the options and the type of balance must be anticipated as a design objective and carefully prepared by the system designers in the first place.

3.2.Privacy vs. smartness

It is no surprise that there is a tricky trade-off between maintaining privacy and providing smartness. A smart system can potentially be smarter with more data about the person requesting a smart service [79]. The trade-off decision should be under the control of the respective persons. They should be able to make the decision which data are provided and for which purpose and for how long these data should be accessible and available to the system afterwards. The challenge is now to find the right balance which – again – requires transparency about which data are really necessary for providing the smart service. This is also in line with the requirement of the EU-GDPR [25] being in effect since May 2018.

The problem was and still is (despite GDPR), that people are not asked for their permissions to collect and process their personal data on a specific basis and providing the rationale for doing so. The technology aspects of data collection and processing in AmI environments are discussed in more detail in Section 5, when we describe how data science can inform the design of AmI environments.

Currently, people do not have the choice to decide and make the trade-off decision between smartness and privacy themselves but are confronted with serious privacy infringements [78,79].

Solving the problem is not trivial, especially when considering public spaces in urban environments, where different AmI environments with different properties and permissions might overlap due to undefined boundaries between them. This is partly also caused by the unobtrusive design of embedding sensors and actuators in the real/hybrid environment, part of the original AmI vision and instantiated by the Disappearing Computer approach [75,76,82]. People are usually not aware of being monitored and which data are collected about them and their context. How can they take control, if they are not provided with the transparency and the options to opt-in or to opt-out to a specific data collection effort?

Of course, we cannot ignore the fact that people provide data also on a voluntary basis to all kind of institutions and service providers. Unfortunately, many of them are not aware of the implications when their personal data are stored on servers in a foreign country where only very weak or no legislation exists to protect their rights and their data. This shows again that much more information and education is necessary about the implications. People should have the freedom making their own decisions, but these should be informed decisions and people should be in control over their data. To remedy the difficult situation, a ‘privacy by design’, respectively ‘privacy by default’ approach is proposed [80,81].

Finally, it should be mentioned that in real life situations the two trade-off conditions described before are confounded, because they coexist in parallel. Therefore, one must address combinations of trade-offs which makes things even more complex. This is elaborated in more detail in [81]. Nevertheless, we argue strongly that these trade-offs should be made accessible as part of a people-/citizen-centered design approach of current and future AmI environments.

4.1.The design of AmI experiences as spatial communication interfaces

In his liner notes for the “Music for Airports” album,22 Brian Eno defined the term “ambient music” as a form of “environmental music suited to a wide variety of moods and atmospheres”. He related the objective of creating such a musical composition to the creation of an environmental situation which induces certain emotional responses to the listeners experiencing this situation. The term “ambience” is an important aspect of the concept of “ambient intelligence”, which stresses the environmental character of these systems and the experiences they evoke.

The concept of ambience in AmI [1] implies that computation becomes non-obtrusively integrated into everyday objects and spaces. In a talk on Everyware in 2009, Greenfield33 asserted that in the case of an (appropriately designed) AmI experience, information processing colonizes the environment of everyday life and the design of the experience “dissolves in behavior”. Computation is also perceived as dissolving in behavior as well as into the physical environmental context. Consequently, the physical environment becomes a medium for supporting interaction between the user and the AmI functionality, hence the interface becomes spatialized.

Mark [50] has acknowledged the significance of physical space as a characteristic of pervasive computing. Most conventional computer interfaces take neither physical space nor the presence or identity of human beings into account. However, as computation gradually becomes part of everyday physical space, the spatial context within which interaction between humans and computation takes place radically changes from a fairly static single-user, location-independent world to a dynamic multi-user situated environment. The physical location of the interface to the computation now becomes relevant. Networked pervasive computation is embedded in the environment as in the Disappearing Computer approach [82,83] and communicates multimodal content which dynamically changes as a result of user interaction. Thus, computation is moved from the center of our attention to the periphery, the area just outside focal attention [51] and added to static spatial elements, forming a coherent whole that offers an enhanced environmental experience [67]. The spatial context within which interaction takes place comprises of both computation and all physical environmental stimuli that may be involved in the process of interaction. This context is a spatial interface.

The term spatial interface44 characterizes human computer interfaces that utilize space as a context for supporting navigating within and interacting with information. Since humans use spatial organizing principles in their daily lives, they are used to and skillful in navigating space and communicating easily within space. It is therefore often appropriate to employ a spatial distribution of information as a means for organizing interaction with information and certain applications in a functional, well-structured and meaningful manner. Therefore, a physical environment enriched with pervasive and ubiquitous computation, an ambient intelligence, may be considered as a type of spatial interface, a hybrid (physical and digital) spatial configuration, where computation expands in the physical space.

Humans utilize various modalities during direct human-human communication. The implementation of multiple modalities in HCI results in interfaces with reduced cognitive load [68]. Furthermore, the use of other senses besides vision may accelerate user adaptation [48]. Any environmental experience is multimodal. Apart from visual sensory input, all other sensory inputs like the perception of auditory, olfactory, thermal, and tactile input, and the sense of proprioception all contribute to the establishment of a sense of space [34]. This approach is also in agreement with Hall’s conception of personal and social space [38]. Therefore, we suggest that the development and use of multimodal interfaces results in spatial interfaces affording an enriched and more complete spatial experience.

When considering the relation between the user, the computation, and the environment within which interaction takes place in the case of an AmI experience, we could suggest that pervasive and ubiquitous computation and linked media communicate information to the user in various modalities. This information escapes the representational context of the limiting two-dimensional space of a screen and is projected onto and manifested via the activity of the technological artifacts located in the physical environment. The activity of these artifacts shapes the context of human beings in a rather implicit manner which is not only attributed to human-machine communication but may also be attributed to machine-to-machine or environment-machine communication.55

Communication systems embody and integrate the functions of a communication interface, a series of transmission channels and an organizational infrastructure. Biocca and Delaney [11] define a communication interface as the interaction of physical media, codes, and information with the user’s sensorimotor and perceptual systems. As suggested earlier, an important characteristic of the particular interfaces that this section deals with is their environmental character. Following Biocca and Delaney’s definition, the spatial interface to an AmI environment could be considered a communication interface that engages the human sensorimotor channels into a vivid communication experience and that also affords an environmental experience [16]. Accordingly, we may use the term spatial communication interface to characterize the type of interface experienced by AmI users. Designing such a communication interface implies the design of the way in which interaction occurs among physical media, codes, and information on the one hand and the user’s sensorimotor and perceptual systems on the other hand, as well as the appropriate environmental context, media displays, representations and other actuators, which function as a framework wherein this interaction occurs.

However, as earlier suggested, in AmI environments, computation is perceived as dissolving in behavior as well as into the physical environmental context. The computer “disappears” as a “visible” distinctive device, either physically due to being integrated in the environment or mentally from our perception [75,76], thus providing the basis for establishing a calm technology as it was envisioned by Mark Weiser [89] and realized in multiple projects of the Disappearing Computer Initiative [82]. A main challenge is that “users” are often not fully aware of the interaction options provided in an AmI environment. A related implication is that they receive no feedback about wrong or inadequate user input or even system failures.

This new constellation requires a rethinking of the notion of “affordances” [33,34,56–58] in this new type of environments. Affordances were introduced by Gibson ([34], p. 36) as the relationship, the set of possible actions, between an object of an environment and a living organism that may act upon this object ([58], p. 123). Norman [57] has appropriated and extended this concept to the world of design. He has stressed the significance of designing affordances which, when perceived, may inform the user of which actions can be performed on an object and how they may be performed by the user. When this design objective is successfully achieved, the designed artifact may communicate its purpose and functionality to the user.

Norman also suggests that media have special properties which may enhance and constrain their usage. A communication medium may not be a physical object, but it still has affordances ([58], pp. 123–124). We may communicate the affordances of these media by appropriately designing the form of the media objects integrated within the AmI environment and the way in which this form may be transformed over time via interaction with the user. When this design goes wrong, we may have a lack of information (hidden affordances) or wrong information (false affordances) [19] being communicated from the media object and perceived by potential users. Gaver [32] also stresses the fact that the perception of theses affordances is partly determined by the observer’s culture, social setting, experience and intentions.

In rethinking the new constellation in AmI environments, where users are often not provided with direct clues for interacting with the embedded, invisible computational devices, one must extend the notion of affordances. Streitz et al. [83] proposed the notion of “inherited affordances” for coping with such challenges in integrated smart environments, based on the design experiences with their interactive “Roomware^®” environments [87].

4.2.On intelligence and non-verbal communication with/in the ambient intelligence

The concept of intelligence in AmI implies that the computational aspect of the environment supports some form of intelligent interaction. Intelligent behavior [1] involves four system elements: context awareness, personalization, adaptivity and anticipatory behavior, in which the AmI environment can extrapolate behavioral characteristics and generate pro-active responses. Additionally, this system intelligence must be compliant with societal conventions.

In order to achieve the above, the AmI experience has to somehow initiate and maintain bidirectional communication of meaning with the user. This could either be:

This categorization of implicit and explicit communication elements is adopted by van de Ven et al. [88]. Mark [50] has also acknowledged the fact that pervasive computation is implicit.

Schmidt [71] has discussed “Implicit Human–Computer Interaction” (iHCI), where the user offers implicit input and receives implicit output. Implicit input refers to actions and behaviors of the user, which are not considered primarily as interaction-initiating, but are perceived as such by the system. Implicit output, similarly, refers to output, which occurs as a result of the reception and processing of implicit input. Implicit output is seamlessly integrated with the environment and supports the user’s task. Essentially, the system detects subtle communicational cues inherent in the behavior of a human through the use of appropriate devices. After processing these data, the system reaches some conclusions about the user’s state and the task to be accomplished and may subtly act on the environment towards increasing the possibility of the user successfully completing the task.

Kaptein et al. [44] used explicit measures of users’ tendencies to comply with distinct persuasive strategies as well as implicit, behavioral measures of user traits for implementing persuasion profiling, as a method for personalizing the persuasive messages used by a system to influence its users.

From a communicational perspective, a person’s experience of reality is altered by an additional layer of mediation that is placed between the user and the environment. This layer may have an impact on the users’ conception of the computer and their behavior within such an enhanced environment [67]. Reeves & Nass [64] propose their theory of media equation, according to which humans interact with media technologies as if they were human. Computers are viewed as a social medium [25] and even as potential interlocutors or “social actors” [55]; humans tend to attribute to them abilities and traits they do not have (e.g. intelligence) and are willing to interact with them in the same way as they do with other humans. We could then suggest that the user may partly perceive the AmI experience as a process of communication with an artificial, human-like entity, where implicit communication may be mostly prevalent.

Rizopoulos [66] analyses the potential relation between non-verbal communication and spatial interaction in the context of spatial interfaces and suggests that: from a communicational perspective, iHCI is based on the perception of communication signals produced and transmitted by the human without her intention and which “reveal” her internal state. Implicit communication is largely embodied, since the body is closer to the unconscious and is more difficult to consciously control [19].

Non-verbal communication is often the way for providing input when the implicit HCI paradigm is adopted [71]. This form of communication entails the information which is communicated through the user’s perceptual channels, in a non-verbal manner. There are elements of non-verbal communication (prosodic) which relate to the verbal message [8]. Other elements of non-verbal communication are independent of the message: i.e. paralinguistic signals [8], which refer to the manner in which the message is communicated (i.e. tone, style and intensity of voice, speech). Non-verbal communication relates to the embodied aspect of communication and consists of three main categories: a) tacesics (the study of bodily touch between humans), b) proxemics (the interpersonal distances which are kept for negotiating our personal space and territories) [38] and kinesics (the analysis of the bodily movements and of the meanings related to them) [66]. Argyle [8] also explains that non-verbal elements of communication have the following functions: a) they express emotions, b) they communicate interpersonal attitudes, c) they accompany and support speech, d) they support self-presentation and e) they play a prominent role in rituals of social behavior. Indeed, some non-verbal signals stand for emotions, attitudes or experiences which are not easily expressible in words.

Recapitulating the main arguments presented in this section, we could suggest that a part of the communication of information between the user and an AmI experience is implicit. Users may perceive this experience as a process of communication with an artificial, human-like entity, but since this communication is partly implicit, it may escape the user’s attention and although it may “reveal” her internal state, it may be based on the perception of communication signals produced and transmitted by the user without their intention. It should be clarified here that a part of the communication of information between the user and an AmI experience may also be explicit and this serves the functionalist objectives of interacting with the AmI experience to achieve an application task.

When implicit communication is adopted in an AmI experience, non-verbal communication is often the way for providing input for both the user and the system. We could then inform the process of designing implicit communication elements in an AmI experience by learning from the manner in which non-verbal communication signals are exchanged in social interaction amongst humans. It is necessary however to go through a systematic design research process of abstracting the ways in which non-verbal signals are communicated in human social interaction and adapting these ways to the specific characteristics of the output devices through which this implicit communication signals will be presented in an AmI context. Devices which were or can be used for this purpose are: ambient displays [90], ambient light smart artefacts as, e.g., the Hello.Wall [60,85], multisensory output devices, motors and other kinetic effectors, other material artifacts (possibly utilizing smart materials) the formal characteristics of which may be transformed via interaction and/or via the transformation of environmental parameters.

Of course, explicit, linguistic or representational elements may also be communicated within an AmI environment. McCullough [52] discusses various ways in which information may be embedded onto the elements of an environment or communicated to users via appropriate multimodal media displays: epigraphs, adhesive electronics (creating links between the digital and the physical context), cultural tagging, frames, screens, urban screens, etc.

As we conclude this section, we should also consider that “every course of action depends in essential ways upon its material and social circumstances” [86]. Humans often act on impulse and adapt to these circumstances, achieving intelligent action. Contexts in communication are not preset; rather, they are co-constructed by the participants. Communication should not be viewed as the process of information exchange, but as the process of the exchange of meanings and interpretations of the situations the actors are involved in [65].

5.1.Data structure and organization

It has been stated before that 80% of data science is effectively data cleaning. Despite the large potential value of all the data that is currently collected, it still proves hard to tie data together, transform it into usable formats, and disclose the data without infringing privacy or violating ethical norms. Health data provides a prime example: while the randomized controlled trial (RCT) is regarded as the pinnacle of evidence in the medical sciences [35], one would be inclined to believe that the estimates of effectiveness of different treatments that originate from RCTs can be further refined by looking at their effects “in the field”. Theoretically such data are readily available; hospitals store the treatment and outcome combination for each disease for each patient, and often insurance companies will have direct records of the costs efficiency of treatments. However, combining data from hospitals, let alone merging the health outcomes with health care costs, has proven notoriously difficult. Our failure to easily combine data, to disclose data with privacy and security guarantees, and to analyze data resulting from multiple source currently limits the value of data.

AmI could make large contributions to this existing problem: as AmI devices continuously collect data in the field [6], AmI researchers and practitioner could actively contribute to creating standard for data sharing and merging. They could be on the fore-front of developing methods to deal with missing data, and – continuing its focus on contextual factors – could create standards that allow us to not only collect data of the primary processes in play (e.g., disease treatments and outcomes), but also the context in which the process played out. As of now however most AmI prototypes collect diverse types of data without a focus on standards for data sharing and data portability.

5.2.Embracing uncertainty

Even if we manage to share the data originating from diverse sources, we will still need to change our fundamental view of the world and the evidence data brings to the table. AmI applications often take intelligence – in the form of elaborate decision rules that depend on the current context and user – as a given when developing and evaluating applications. In reality, however, such deterministic rules, when derived from finite data, will always contain inferential errors. And, if we believe that the user and the context matter – a position that is strongly held in healthcare with its recent focus on personalized medicine [35] – the data that informs the actions of AmI technologies will inherently be very limited: we will never know with full certainty what the best action is for the current user in the current situation.

This has several consequences: first of all, we should actively model this uncertainty; despite a contemporary focus on point estimation in most of the data science literature effective methods for uncertainty quantification have been developed over the last decades and should not be ignored [62]. Second, if our uncertainty is too large to make a decision, we should inform the user, or perhaps actively illicit user input.

5.3.Learning sequentially

Not only should we embrace uncertainty and make it transparent to users – refraining from making decisions when the uncertainty is too large – we should also actively consider how our actions reduce future uncertainty. AmI technologies will always be imbedded in their environment and they can learn from actively interacting with their environment.

If an AmI technology is trying to make the best choice for a given user in a given context based on limited information, it is abstractly solving a decision problem called the contextual multi-armed bandit problem (MAB) [53]. The MAB problem is easily motivated in a health context: given two different treatments (or actions) a=1 and a=2, and given some outcome (or reward, r) dependent on these actions, we aim to maximize our outcome over sequential choices of our actions ∑t=1Trt. If initially we do not know the outcomes of the different actions, we will need to explore the options and learn about the outcomes. However, as our knowledge of the outcomes give the actions grows, we would like to exploit and choose the action we believe serves the current patient best [49].

AmI technologies that actively learn from interacting with their environment will need to solve this exploration-exploitation trade-off: since for one specific user, in a specific context, no deterministic rule can be available, we need to balance making choices that inform our future choices, with utilizing the knowledge we currently have. Deterministic rules are asymptotically suboptimal to address this problem. Here embracing uncertainty is key: if we can properly quantity the uncertainty of the outcomes given our actions we can actively explore uncertain outcomes. A large literature that examines effective strategies, or policies to balance exploration and exploitation has emerged (see, e.g., [3]), and AmI researcher should embrace this sequential learning view on the word: AmI technologies should actively seek information to inform their future decisions.

5.4.Considering the future value of data

Once we approach making intelligent decisions based on data as a sequential problem in which uncertainty is abound, we quickly encounter the following question: can we use the data that we collected using one specific policy to choose our actions to evaluate what would have happened if we had used another policy? This question is particularly relevant in a health-care setting: does the data originating from a randomized clinical trial in which the patient population is assumed to be homogeneous (e.g., each unit has the same probability of receiving a treatment) allow us to evaluate an alternative policy that selects treatments based on user characteristics [45]?

Emerging answers to this question have direct implications for AmI technologies: it turns out that using data generated by a specific policy to evaluate “what if” questions is possible as long as the probabilities of receiving a treatment conditional on the user and context characteristics are known. This so-called propensity score can subsequently be used to counterbalance the effect of the policy that generated the data and allows us to obtain unbiased estimates of the performance of alternative policies [10]. Interestingly, such counterbalancing is impossible if probabilities are 0 or 1; this again highlights the importance of embracing uncertainty [91]. Policies that consist of deterministic decision rules generate data that is effectively useless for the evaluation of alternative strategies. AmI technologies, when interacting with their environment, should at the very least store the probabilities of the actions they took at each point in time to generate data that is valuable for re-use. Obviously, any data collection, storage and processing by AmI technologies must comply with the GDPR [25] introduced and discussed before in Section 3.1.

5.5.Understanding mechanisms that generate data

A very specific version of the “what if” question abound: we often have access to observational data – thus data that originated from a policy unknown to us – and we want to use it to evaluate alternative policies. For example, we might have data considering the outcomes of two treatments for a specific disease as administered in a hospital, and we want to know what the outcomes will be for future patients if we select one of the treatments.

This question is in general not solvable: there is no guarantee that the observational data originating “in the field” allow one to properly estimate the causal effect of the treatment. For example, a naive comparison of the survival rates for breast-cancer patients receiving chemotherapy or not based on observational data in the Netherlands would lead one to conclude that chemotherapy negatively affects survival rates. However, this conclusion is fully confounded by the severity of the tumor: only women with a severe tumor receive chemotherapy. If we had known the propensity scores in this case, we would have concluded that – given the fact that these were 1 for those with severe tumors and 0 for those with mild tumors – the observational data was useless to evaluate another scheme of administering treatments.

Luckily, recent advances in our study of causal inference based on observation data have greatly improved our understanding of this problem. Effective methods to estimate propensity scores [29] and to uncover causal structures [59] now exists. AmI technologies that use existing data in their reasoning should actively incorporate these methods to prevent erroneous decisions. As AmI (and AI) applications are becoming more and more prominent in highly impactful areas such as healthcare (see also Section 7 of Gams et al. [31]), a proper understanding of the (causal) mechanisms that generate our data is of increasing societal importance.

5.6.Make transparent decisions based on data

A final topic that has recently emerged in data science that should resonate with AmI researchers is the topic of transparency and fairness. As more and more decisions that affect individuals are made based on data, there is a growing need for methods that a) allow individuals to understand why a decision was made, and b) control the feasibility of a decision in terms of fairness and avoid possible discrimination. Both areas are active areas of study: on the one hand, researchers are working actively on making decisions of black-box machine learning models transparent to their users [22,47]. These methods should be incorporated into AmI technologies that autonomously make decisions that affect end-users. This work indirectly highlights a benefit of explicit, rule-based, processing methods: rule-based methods are often easy to understand for users. On the other hand, there is also a growing community of researchers that focusses on developing algorithmic and technical solutions to ensure fair and discrimination aware data science (e.g., [37]): these methods should also be embraced by AmI researchers.

As we have seen, contemporary developments in data science are likely to affect the AmI research field. Most notably of these is the recognition of the uncertainty contained in our data, and the sequential and interactive nature of data collection and decision-making. Whatever we do based on data is likely to generate new data, and this should affect our decisions. This view does not merely influence AmI: developments in data science and neighboring disciplines are currently challenging the use of RCT’s for the collection of knowledge in the health sciences [45] and have transformed decision making in online marketing (for examples, see [3]). As our views regarding the utility and value of data are constantly changing, the AmI vision, in which interaction with complex environments based on continuously collected data is key, should embrace these changes. This is especially important since our abilities of monitoring the environment and context (see, e.g., Prati et al. [61] for an overview of the state-of-art) are rapidly increasing and hence our ability of making meaningful decisions based on sensor information is likely to improve strongly in years to come.

6.1.Managing ethics of autonomous systems

The IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems66 (or Ethically Aligned Design – EAD) is an attempt to map out the territory of ethics in artificial intelligence, and to offer actionable recommendations to its constituents of engineers and software designers.

No doubt, there is a real need for, and public interest in, governing artificial intelligence [30]. From autonomous vehicles to virtual assistants, Autonomous and Intelligent Systems (A/IS) are impinging on every aspect of life along a multitude of vectors. It is no secret that engineers and designers who actually build artificial intelligences should consider ethics as a formal part of system development. The real problem is how to go about it in practice.77

EAD is both an initiative and a document under active construction and versioning.88 EAD documents are a collaborative effort to which many people contribute. The stated goals of the initiative are twofold, first, to advance (and moderate) a public discussion on ethics in A/IS, and second, to create a standards and associated certification program that will enable and guide artificial intelligence research and development in practice. The current second version of the EAD document includes a comprehensive overview of relevant references from multiple perspectives on existing and new questions on ethics within artificial intelligence. The document covers the domains of affective computing, mixed reality, well-being, personal data, methodologies, safety, superintelligence, autonomous weapons and economics.

Ethical design is not news to the ambient intelligence community. Ethical concerns are inscribed into the conception of ambient intelligence research ab initio. For over a dozen years [14,23,26,43], ambient intelligence research has addressed ethics of IT systems operating in everyday life. However, AmI has not been able to move ethical design from research contexts into ethical design in the wild, at scale, and has even been forced to see good intentions altered beyond recognition. The intelligent home compromised by its own data harvesting appliances is but one prominent case in point.

What the current EAD initiative offers beyond ambient intelligence’s experimental contributions are implementation-oriented guidelines for considering ethics specifically of advanced A/IS for an entrepreneurial global context. With A/IS systems now operating in the wild at scale, this shift in scope is significant. The next subsection describes the EAD initiative in more detail.

6.2.From inspiration to recommendation

The general principles the EAD initiative subscribes to are inspiring: human beneficence as a superset of human rights, the prioritization of benefits to humanity and the natural environment, and the mitigation of risks and negative impacts. With these principles, EAD proceeds across multiple sections to elaborate on the significance of AI ethics in the areas mentioned above.

The initiative clearly struggles to reconcile its ambitions with the vast territory it maps out. The approach the initiative takes is informed by its ultimate goal, namely making artificial intelligence and its application as A/IS manageable. With this ulterior motive in mind, each section of the document contains – as the examples below illustrate – practical recommendations that serve as the glue between the topic overviews and proposed implementations. The goal of the following paragraphs is to understand the logic of the initiative’s argumentation through a critical reading of select sections of the current version of the document.

6.3.Recipes are not good enough

Applied ethics generally concerns itself with concepts of good and bad conduct. A/IS, as engineering in general, considers the increase of efficiency as desirable in its own right. But efficient solutions need not be ethically sound solutions. Certainly the history of warfare offers copious examples to the point.

With the preferential positioning of efficiency and solutions-oriented methodologies, EAD risks skewing the interpretation of applied ethics in A/IS from a moral to a requirements management problem.

For example, the section on Transparency rightly points out that “transparency is important because it provides a simple way (for stakeholders) to understand what the system is doing and why” (EADv2, p. 30). The corresponding recommendation then stresses the need to “develop new standards that describe measurable, testable levels of transparency”. As an example of this testable transparency, the text mentions a care robot with a why-did-you-do-that-button one can activate to have a robot explain an action it just performed. While this button might make getting a response from a robot easier, it certainly does not guarantee that the response is helpful. Imagine the robot indifferently stating it “did what is was programmed to do” when asked for an explanation. What is missing in this recommendation are the deeper dimensions of a transparent explanation such as context, and the ability to question the result delivered by the robot. EAD offers only a formal version of transparency, ‘transparency lite’, adequate maybe to satisfy legal requirements, but not even close to bona fide transparent action.

6.4.Take the long road

There is some help from other sources. The idea of algorithm impact auditing, part of ongoing efforts from media scholars [70], legal scholars [21,73], and institutes [4] is a case in point. Algorithm impact auditing seeks to make algorithms accountable. Auditing includes the concept of disputability, allowing the public not only to see what an algorithm is doing but to dispute its outcome. Auditing also implicitly considers the effects of code in the real world, including pathologies of scaling. Evaluating A/IS in the lab on small sample data is not the same thing as running A/IS in the messy world on data from millions of people.99 Side effects are much more likely to occur in complex environments, and much harder to counter with optimization approaches. Expanding the reach of algorithm control to the level of accountability [20] is important, legally and politically, as enforceable action is only available from large-scale structures charged with upholding the interests of the public. In this regard, the European Union’s General Data Protection Regulation [25] and its formulation of enforceable, individual rights is an important attempt to apply policy level intervention at least on personal data.

A/IS are complex socio-technical systems composing computers, sensors, data, databases, multi-author algorithms with various levels of autonomy running continuously in remote locations. Then: time constraints, patches not applied, deadlines looming, people under stress in the workplace, etc., etc. These (and many other) intertwined factors contribute to how an A/IS behaves – and fails – in the real world.

Indeed, the term failure hardly catches the many dimensions along which outcomes can deviate from expectation. Even the space of technical failures is vast. Learning systems can go bad simply because of improperly formulated goals, fragile or miss-specified objective functions [7] which an algorithm might try (and succeed) to optimize. Making A/IS safe, let along ethically aligned is a complex undertaking into which research is far from complete.

To be clear: The issue is not only that ethically aligned algorithms in A/IS are not ready for mass production, but that the scope of the challenge itself has not been adequately established. Ethically aligned interventions require more than a purportedly ethically aligned algorithm. The recent “success” of a state of the art school bus routing algorithm designed to increase equity for Boston public school students1010 is a case in point. The algorithm met its goals of reconfiguring bus start times and cutting transportation costs, but created a political disaster as affluent sections of the city lost established advantages and protested against the algorithmically proposed changes.

Instead of suggesting broad recommendations at this point in time, EAD could call for and support open research into ethics of A/IS in the wild. For example, EAD could suggest and coordinate experiments by which to test and evaluate ethically aligned design concepts, much like it has been proposed for A/IS safety [7]. To these experiments, monitored and evaluated by multi-disciplinary teams, one could add continuously updated field reports from A/IS failures in the real world, operating at scale. Together these sources could constitute a compendium on safety, failures and ethically aligned experiments in A/IS. Importantly, such a compendium should be publicly available. Bringing known problems within A/IS from closed board rooms into public view is one way to increase trust in A/IS, a central concern of the EAD initiative.

At least in the autonomous vehicle industry recent accidents [54] have increased the pressure for safer solutions, making autonomous vehicles a good candidate for the approach outlined above. Safety, as opposed to ethics, is directly linked to economic imperatives. Liability lawsuits in response to lax safety provisions may just be the most effective entry point into making harm-adverse A/IS a reality. Then, with safer systems under development, the most effective tested safe algorithms could be used as basis for ethically sensitized algorithms. This would give the engineering community a robust departure point from which to consider the hard non-engineering elements of A/IS ethically aligned design, namely policies and politics, business and global culture.

A cautious iterative approach is not a Luddite retreat. Rather it suggests carefully building a path towards ethical design in AI/S while one can still afford to make mistakes. After all, the currently deployed A/IS are proficient mostly in specific domains. But the coming realm of Artificial General Intelligence, “an intellect that is much smarter than the best human brains in practically every field” [15] might be much less forgiving. Casting Artificial General Intelligence as an extension of A/IS, and not an unrelated alien creature, is a good move on the part of EAD; it allows one to take control of the development of Artificial General Intelligence. In principle, at least.

One candidate recommendation offered for countering malicious Artificial General Intelligence is the safe-by-design (EADv2, p. 79) approach. As above, details on precisely how safe-by-design systems might operate in critical situations are missing. For example, how would safe-by-design prevent an armed autonomous drone from optimizing a reward function of minimizing public disturbance by simply picking the most effective action, killing protesters, even though the drone was never explicitly programmed to do so? Likewise, the statement “teams working on developing Artificial General Intelligence should be prepared to put significantly more effort into AI safety research as capabilities grow” (EADv2, p. 77) offers little help and even less solace. Instead of debating the merits and drawbacks of Arkin’s ethical governor [9], for example, the recommendations prefer uplifting messages, to wit: “Adopt the stance that superintelligence should be developed only for the benefit of all of humanity” (EADv2, p. 82).

As the historian Yuval Harari reminded his audience at the World Economic Forum 2018, it took societies millennia to learn how to organize something as simple as the ownership of land [39] through an evolving set of concepts on contracts, fences and city walls etc. How can one expect to robustly organize the ownership of endless global data streams, let alone the super intelligence that will process and learn from them so quickly? It is too early to craft recipes. A long view is required. No one can afford to be guided by naïve hopes; no one can afford not to learn from past mistakes. Ambient intelligence was early to the game of ethically aligned design but did not succeed in bringing the concept to industrial scale. The intelligent home is in danger of been compromised by its own data harvesting appliances; the smart city a victim of greedy data collection marketing business models whose moto Bruce Sterling aptly described as “information about you wants to be free to us” [74].

The time is ripe for ethically aligned design; done carefully without the shortcuts the EAD initiative proposes. Whichever rules of AI management will be agreed upon next should be understood as provisional. It is important to anticipate mistakes and remain adaptive; more adaptive yet than the new superintelligences under construction.

7.1.Beyond ‘smart-only’ cities and societies

While the analyses and recommendations have general applicability, it is useful to apply them to the domain of current and future urban AmI-environments. Currently, one can observe an increasing hype indicated by the label ‘smart cities’. Sterling [74] even asks for “Stop saying smart cities”. As shown in several examples in the preceding sections, there is a need to move beyond ‘smart-only’ cities by putting a different set of requirements and design goals in the first place. One could use a rephrasing of smart: “smart, but only if cooperative and humane”. In accordance with the design trade-offs mentioned before, the overall goal of designing and realizing future or refurbishing existing cities should be to build humane, sociable and cooperative hybrid cities reconciling people and technology by providing a balance between human control and automation as well as privacy and smartness [77,78,81]. This implies that we need to foster and enable the following actions and requirements for designing and building AmI applications in the context of smart urban environments:

This list of actions and requirements applied to future urban environments point to a promising prospect, but only if they are taken into account and affect the manner in which these systems are structured and realized in the implementations. One must be aware that there are severe risks caused by different goals and value systems of the different stakeholders in our society – requiring a discussion of pros and cons – on the way to a humane and cooperative smart urban society. Therefore it is important that the proliferation of AmI systems in the urban and social realm is proactively evaluated by a meticulous and adaptable approach at the level of policy making and governed by an appropriate legal framework that will safeguard these policies.

7.2.Claims for future developments

While the application domains of cities and urban societies will play an increasingly important role due to living in the urban age, we can also abstract and formulate some claims in a more general fashion. So, we anticipate the following developments:

Ambient Intelligence values, objectives and methods can play a major role in achieving the goal of reconciling people and technology in a future ‘smart’ society, hopefully a beyond ‘smart-only’ society. An important aspect, but no guarantee, is the actual implementation of the design trade-offs and the ethical considerations described and discussed before. But we must keep in mind, that the AmI approach is only one perspective and not at all a comprehensive solution for all the problems cities and society are facing today and will be so even more in the future. Beyond the role of AmI-based technologies, there is a wide range of important issues, including socio-economic, ecological, sustainability and political aspects.