Forschungsseminar

Development of a Hierarchical Approach for Multi-Agent Reinforcement Learning in Cooperative and Competitive Environments

Robert Pfeifer

Wed, 11. 9. 2024, 1/368 and https://webroom.hrz.tu-chemnitz.de/gl/jul-2tw-4nz

Multi-Agent Reinforcement Learning (MARL) enables multiple agents to learn purposeful interactions in a shared environment by maximizing their own cumulative rewards. As a subject of current research MARL is getting applied in various fields, such as robotics, traffic control, autonomous vehicles and power grid management. However, these algorithms faces challenges including partial observability, non-stationarity of the environment, coordination among agents and scalability issues when incorporating a multitude of agents. This thesis explores Hierarchical Reinforcement Learning (HRL) as a potential solution to address these challenges, using the Goal-Conditional Framework to decompose complex tasks into simpler sub-tasks, which facilitates better coordination and interpretability of behavior. The Goal-Conditional Framework learns this decomposition automatically in an end-to-end fashion by assigning the respective tasks to a hierarchy of policies. The higher-level policy proposes a goal vector either in a latent state representation or directly in the state space while operating on a potentially slower time scale. The lower-level policy receives the goal as part of its observation space and obtains an intrinsic reward based on how well it achieves the goal state. Only the top-level policy receives the reward signal from the environment. The thesis implements the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm proposed by (Lowe et al., 2020) and extends it for hierarchical learning using either a centralized or decentralized manager policy. It investigates the performance of this approach in various environments, focusing on diverse learning aspects such as agent coordination, fine-grained agent control, effective exploration and directed long-term behavior. Additionally, the thesis explores the influence of architecture, time horizon and intrinsic reward function on final performance, aiming to gain a deeper understanding of the possibilities and challenges associated with Goal-Conditional hierarchical algorithms.

A Survey of Reservoir Computing for Control Tasks

Maximilian Titz

Wed, 11. 9. 2024, 1/368 and https://webroom.hrz.tu-chemnitz.de/gl/jul-2tw-4nz

Reservoir Computing has seen a noticeable increase in popularity in recent years. Reservoirs aim to leverage the complex behaviors that naturally emerge in high-dimensional, dynamical systems in response to an input. In doing this, they constitute a simple way to circumvent the vanishing/exploding gradient problem that arises when training recurrent neural networks. This makes them effective for tasks that require time series processing. Additionally, they significantly reduce the number of parameters that need training. Despite these advantages, the training of reservoirs presents unique challenges. These include, for example, finding good hyperparameters used to initialize the network. This seminar delves into recent research papers that utilize reservoir computing for control tasks. The primary goal is to showcase and compare different approaches, highlighting how reinforcement learning techniques, such as Q-learning and Proximal Policy Optimization, are used with reservoir computing. Furthermore, it explores novel methodologies, such as optimizing the hyperparameters of a reservoir using Meta-Learning and the integration of random weight Convolutional Neural Networks with reservoir computing to process video data. By examining these diverse strategies, the seminar aims to provide an overview of the current state of RC and its potential for future advancements in the field.

Entwicklung von KI-Modellen zur Adressdatenextraktion: Herausforderungen, Methoden und Empfehlungen zur Bewältigung von Vielfalt und Kontextfehlern

Erik Seidel

Tue, 27. 8. 2024, 1/368 and https://webroom.hrz.tu-chemnitz.de/gl/jul-2tw-4nz

Die präzise Erkennung postalischer Adressen stellt eine zentrale Herausforderung dar, insbesondere, wenn Modelle auf die Erkennung anderer Entitäten, wie beispielsweise Personen, trainiert werden. Die dabei entstehenden Konflikte haben Einfluss auf deren Identifikation. Ein Beispiel hierfür ist die Verwechslung bei Namen wie "Professor Richard Paulik" mit "Professor Richard Paulik Ring". Aber auch die Unterscheidung zwischen Personen- und Ortsnamen sind eine Herausforderung. Zentrale Probleme stellen Komplexität, Diversität (Vielfalt) und Ambiguität (Mehrdeutigkeit) der Entitäten dar. Ausreichend annotierte Daten von Adressen oder Personen sind nur schwer zu finden. Die Arbeit untersucht, welche Ansätze sich am besten für die Erkennung von Adressen eignen. Im Fokus stehen Transformer Modelle. Für das Problem der Datenbeschaffung wird der Einsatz synthetisch erstellter Daten betrachtet. Das Ziel ist eine möglichst breite Vielfalt an Beispielsätzen für das Training und die Evaluation zu erreichen. Es werden verschiedene Metriken und Werkzeuge vorgestellt, mit dessen Hilfe sich Modelle und Daten bewerten und optimieren lassen.

Vergleich und Verifikation von Methoden der eXplainable Artificial Intelligence anhand der Re-Identifikation von Personen

Simon Schulze

Tue, 13. 8. 2024, 132 and https://webroom.hrz.tu-chemnitz.de/gl/jul-2tw-4nz

Die Verifikation von Methoden der eXplainable Artificial Intelligence (XAI) ist unabdingbar, um die Vorhersagen komplexer, undurchsichtiger Modelle des maschinellen Lernens (ML) interpretier- und nachvollziehbar zu machen und die Korrektheit der Erklärungen zu testen. Besonders in kritischen Infrastrukturen ist eine Prüfung der eingesetzten Algorithmen durch XAI-Verfahren notwendig, um das Vertrauen in diese zu stärken und deren Richtigkeit nachzuweisen. Durch die zusätzliche Bestätigung der Korrektheit der XAI-Aussagen wird dies zunehmend untermauert. Im Rahmen dieser Bachelorarbeit werden drei Modelle des ML, eine Support-Vektor-Maschine (SVM), ein Multi-Layer Perzeptron (MLP) und eine Gated Recurrent Unit (GRU), eingesetzt. Eine Anwendung dieser erfolgt auf dem von Smerdov u. a. [SME+20] vorgestellten eSports-Sensor-Datensatz, in welchem zahlreiche biometrische Attribute einzelner Spieler enthalten sind. Mit Hilfe von XAI-Methoden sollen die relevantesten Eigenschaften der einzelnen Spieler analysiert werden. Eingesetzt werden dazu Local Interpretable Model-Agnostic Explanations (LIME), Shapley Werte, Gradients, Gradients-SmoothGrad (Gradients-SG) und Layer-Wise Relevance Propagation (LRP). Deren Aussagen werden mit Hilfe verschiedener Vergleichs- und Verifikationsmetriken - die Größe der Schnittmenge der relevantesten Attribute, die Stabilität von XAI-Methoden, deren deskriptive Genauigkeit, deskriptive Spärlichkeit und maximale Sensitivität - auf ihre Richtigkeit kontrolliert. Es wird gezeigt, dass sich die Aussagen verschiedener XAI-Verfahren unterscheiden und somit eine individuelle Korrektheitsprüfung dieser notwendig ist. Da biometrische Merkmale laut Payne u. a. [PAY+23] unter Umständen einen signifikanten Aufschluss über individuelle Muster und Charakteristiken einzelner Personen geben, besteht die Annahme, dass diese durch XAI-Methoden abgebildet werden können. Schlussendlich erfolgt daher eine Wiedererkennung der Individuen ausschließlich auf Basis der erhaltenen XAI-Ergebnisse. Es kann erfolgreich bestätigt werden, dass dies in einem bestimmten Rahmen möglich ist.

Investigation of Time Series Based AI Models for Autonomous Driving Function Validation

Vincent Kühn

Mon, 5. 8. 2024, 1/367 and https://webroom.hrz.tu-chemnitz.de/gl/jul-2tw-4nz

Bevor autonome Fahrfunktionen freigegeben werden, müssen sie während der Entwicklung ausgiebig getestet werden. Dies geschieht in der Regel in einer Simulationsumgebung, da in den frühen Phasen der Entwicklung keine geeigneten Fahrzeuge zur Verfügung stehen und eine Erprobung in der realen Welt aus verschiedensten Gründen nicht möglich ist. Abhängig von der Simulationsumgebung und den Testwerkzeugen kann die anschließende Bewertung sehr langsam sein. Um diese Testauswertungen zu beschleunigen und effizienter zu gestalten, werden verschiedene moderne Algorithmen zur Zeitreihenanalyse ausgewählt und verglichen. Im Einzelnen handelt es sich um Zeitreihenklassifizierungsmodelle zur schnellen und korrekten Klassifizierung von Testszenarien als bestanden oder nicht bestanden und um Zeitreihenprognosemodelle zur Gewinnung zusätzlicher Erkenntnisse über das zu testende Fahrverhalten.

IntelEns - a Reinforcement Learning based Approach for automated Time Series Forecasting

Robin Wagler

Tue, 16. 7. 2024, 1/368 and https://webroom.hrz.tu-chemnitz.de/gl/jul-2tw-4nz

Creating time series predictions and forecasts is a highly specialised process. Therefore, it is hard to automate it. On the other hand, there is a huge need for such predictions as they can tell information about the future. This thesis proposes a model called Intelligent Ensemble (IntelEns), which should be able to automate the time series forecasting process. IntelEns combines a classical forecasting ensemble with reinforcement learning (RL). The idea is to train different base learners (regression models), which create individual predictions. A RL-agent learns how to combine these individual predictions at every time step based on the actual observation of the time series. The result is an ensemble prediction for every time step. The RL-agent should learn which model to use, when and how intense. As the RL algorithms learn through trial and error, no human supervision is needed, and the forecasting could be automated. This thesis investigates intensively the properties and abilities of the proposed model. A prototype was built using Python. Experiments are done on scientific time series data with good patterns and a real-world dataset from Porsche's aftersales domain. The strengths and weaknesses of the proposed approach are presented, and the question of whether an RL-based ensemble could be used to automate the time series forecasting process will be answered.

Deep Learning for Recognition of Type Labels on Heating Systems in 3D Scenes

Goutham Ravinaidu

Thu, 4. 7. 2024, https://webroom.hrz.tu-chemnitz.de/gl/jul-2tw-4nz

This presentation will cover the background and working principles used in tackling the problem statement and present the achieved results. This thesis aims to detect the type labels present on the heating systems from a video sequence and extract the text present on those labels. The various object detection models used to localize the labels, the use of image quality analysis to ensure quality, and the utilization of various OCR models used in extracting the text information. This presentation also shows the results of the completed pipeline of the thesis with the combined use of object detectors, quality control and OCR models. I will finally discuss the results and provide a conclusion.

Optimizing Neural Network Architectures for Fail-Degraded Computing Scenarios in Automated Driving Applications

Diksha Vijesh Maheshwari

Fri, 28. 6. 2024, 1/368 and https://webroom.hrz.tu-chemnitz.de/gl/jul-2tw-4nz

Neural networks (NNs) are commonly used in automated driving applications. With shrinking transistor size for denser integration, integrated circuits become increasingly susceptible to permanent faults. These permanent faults can lead to a complete loss of functionality, which is unacceptable in safety-critical applications. The thesis introduces a robust Fail Degradation-Aware Quantization (FDQ) strategy coupled with sensitivity prediction algorithms to maintain high algorithmic performance in presence of permanent faults. The FDQ method improves the optimization process by incorporating losses for both higher and lower precision to compute optimal quantization step sizes for model parameters and activations. Additionally, the thesis investigates various sensitivity prediction algorithms that identify the critical components of the model, enabling strategic computational remapping of sensitive features to insensitive features to enhance model robustness and accuracy. Experimental results with ResNet18 and VGG16 models on CIFAR-10 and GTSRB dataset showcase the effectiveness of integrating FDQ with sensitivity-driven task remapping in hardware accelerators, significantly improve the robustness of NNs against permanent faults, ensuring sustained functionality and accuracy under adverse conditions. Specifically, at the highest fault injection error rate of 6.20% for the ResNet18 model on CIFAR-10 dataset, the FDQ method along with sensitivity remapping improved the accuracy by 10% from 76.60% to 87.07%.

Observational Learning: Neue Konzepte und exemplarische Realisierung eines Agenten innerhalb einer virtuellen Umgebung

Leon Kolberg

Mon, 17. 6. 2024, 1/367a and https://webroom.hrz.tu-chemnitz.de/gl/jul-2tw-4nz

Während uns Menschen 'Lernen durch Beobachtung' intuitiv gelingt, ist das programmieren und trainieren von künstlicher Intelligenz teuer und zeitaufwendig. In dieser Arbeit stelle ich Verfahren zum Lernen durch Beobachtung für Künstliche Intelligenzen vor und präsentiere eine eigene exemplarische Implementation mit einem Neuronalen Netz und Q-Learning in einer virtuellen Realität. Die Ergebnisse dieser Arbeit zeigen nicht nur die Machbarkeit von Observational Learning in einer virtuellen Umgebung, sondern sie zeigen auch Vorteile, Nachteile sowie Risiken, die abgewogen werden müssen.

Funktionsweise des Softwareframeworkes Lava im Vergleich zu ANNarchy

Andreas Schadli

Thu, 13. 6. 2024, 1/368 and https://webroom.hrz.tu-chemnitz.de/gl/jul-2tw-4nz

Diese Bachelorarbeit befasst sich mit dem Softwareframework LAVA, das für neuromorphe Hardware, insbesondere den Loihi-Chip von Intel, entwickelt wurde. Um LAVA mit dem Neurosimulator ANNarchy zu vergleichen, wurden verschiedene Experimente durchgeführt. Dabei wurden Beispielsimulationen in beiden Softwares implementiert und anhand folgender Fragen verglichen: Ist es möglich, die Simulationen in beide Softwares zu übertragen? Sind die Simulationsergebnisse identisch oder gibt es Unterschiede? Wie unterscheiden sich die Arten und Weisen, in denen die Simulationen implementiert wurden? Welche Laufzeitunterschiede bestehen? Die Experimente beleuchten die unterschiedlichen Implementierungsstile, und die Ergebnisse zeigen, dass beide Softwares ähnliche Resultate liefern können und dass LAVA deutlich längere Ausführungszeiten als ANNarchy benötigt.

Investigation of Reward-Guided Plasticity in Recurrent Neural Networks for Working Memory Tasks

Max Werler

Thu, 16. 5. 2024, 1/368 and https://webroom.hrz.tu-chemnitz.de/gl/jul-2tw-4nz

A variety of recent works have indicated that Perturbation Learning provides a valid approach to train the recurrent weights of an artificial recurrent neural network. In this context, Miconi (2017) demonstrated how the thereby induced deflections of the excitation of the neurons can be captured based on information that is locally available to synapses in a Hebbian manner and how these can be integrated into a reward-guided weight update rule such that it characterizes a biologically plausible training algorithm that is capable to solve cognitive tasks. However, his learning architecture suffers from a significant flaw that effectively can prohibit learning progress for some random initial conditions and disrupt a successfully converged network towards an error level like it was observed before training. In this thesis, we investigate these scenarios intending to find the underlying network properties that cause this undesired behavior. As low intrinsic activity and imbalance between excitation and inhibition were detected to strongly correlate with these phenomena, related learning rules as well as a different input weight initialization scheme have been proposed and evaluated. While our results show that we were able to enhance the speed as well as reliability of the initial network convergence greatly, the possibility for a sudden deterioration of a temporarily successful network remains existent.

Comparison of two motor learning models of the basal ganglia and cerebellum

Christoph Ruff

Thu, 2. 5. 2024, 1/368 and https://webroom.hrz.tu-chemnitz.de/gl/jul-2tw-4nz

A model of motor learning, which was developed here on the professorship by Baladron et al [1] will be compared with a motor learning model from Todorov et al [2]. Both models use a model of the basal ganglia (BG) and the cerebellum (CB). The BG chooses a certain action and reaches to that location, while the CB fine-tunes the reached location to come closer to the target or adapt the movement to an altered location. Both models have a different structure and therefore function differently. During the seminar I will give a closer overlook on how these two models differ and what are advantages/disadvantages of them. The tasks with which these two models were trained differ as well. As part of my internship, I trained the model from Baladron et al [1] with the tasks of Todorov et al [2] to see which parameter values and adaptations are necessary to get a similar result and if it even is possible to train the model with these tasks. The results of this implementation will be presented as well.

Neuromorphic Computing as a Low Power, Minimal Footprint Solution for Dynamical System Control

Valentin Forch

Thu, 18. 4. 2024, 1/368 and https://webroom.hrz.tu-chemnitz.de/gl/jul-2tw-4nz

In cooperation with the Research Center for Materials, Architectures and Integration of Nanomembranes (MAIN) we develop a neural network architecture that will enable the control of autonomous modular micro robots carrying CMOS chiplets. Realizing a neural network controller on this scale poses multiple challenges: the machine must run on a minimal energy budget, minimal memory footprint, and can only be build on top of a low-level instruction set. Further, these machines should in principle be able to adapt to changing environments without a complete re-training.
To answer these challenges, we start by optimizing recurrent spiking neural networks for simple motor control benchmarks in an evolutionary framework. The networks possess a simplified integrate-and-fire neuron model and ultra low resolution synapses. We further reduce the memory footprint by optimizing only subsets of the network connectivity matrix and by controlling multiple network weights through singular bits. By controlling the neural network and resulting motor activity, we find well-performing solutions that minimize energy consumption. Lastly, we introduce a novel approach for optimizing agents through highly parallelized swarm evolution.

ANNarchy User Forum

Helge Ülo Dinkelbach, Julien Vitay

Thu, 11. 4. 2024, 1/368 and https://webroom.hrz.tu-chemnitz.de/gl/jul-2tw-4nz

Topics will be:

1. ANNarchy 4.7.3
• Introduction to the features and extensions since 4.7.0
2. Spotlight features and discussion
• Float-precision handling in ANNarchy
• Auto-tuning in ANNarchy, the new default?
3. Future developments
4. Resources overview
5. Open forum
• Removal of the MagicNetwork
• Discussing the future network interface and handling
• Discussing your feature requests and ideas