Under the emergence of the Internet of Things (IoT), tremendous amounts of data have been generated in a distributed manner from IoT devices, e.g., smart sensors, smartphones, cameras, etc. To effectively learn the patterns among such distributed data, distributed learning/federated learning frameworks have been explored to effectively utilize the distributed/decentralized data resource. However, the current distributed learning systems over IoT face inevitable challenges, mainly categorized into security, privacy, compatibility, and efficiency. For instance, first, the success of most existing learning systems relies heavily on a central server to coordinate the learning process. Second, traditional distributed learning requires the data on different devices to be in the same type/dimension. Third, the learning efficiency of the existing frameworks is another major concern. This work aims to systematically design a secure and fully decentralized learning method over IoT devices that enables the system to learn patterns from heterogenous data, i.e., data in different types/dimensions. This work will also deploy the developed method on a decentralized platform, i.e., blockchain, and design corresponding system protocols to eliminate the need to use a central server as in traditional distributed learning. On the other hand, the proposed system aims to exploit potential system vulnerabilities, develop attack and defense mechanisms, and theoretically analyze the system's reliability. Moreover, this work lays the groundwork for system research in dense IoT applications supported by decentralized learning. It generates preliminary experimental data necessary to develop an independent and competitive research agenda.<br/> <br/>This work will extend the traditional distributed/federated learning into a more general and practical scenario under IoT, where various types of data and IoT devices exist in the system. This work will also enhance the reliability of learning performance when a certain portion of the data in the network is attacked/malicious. Additionally, this work is potentially transformative as it may help generate innovative and secure decentralized deep learning techniques for numerous applications, e.g., smart cities, smart homes, and mobile health, since the proposed system can effectively utilize heterogeneous resources. It could also have significant impacts on research in transforming the existing centralized or distributed smart applications into a fully decentralized manner with security and performance guarantees. To address the education and social aspects, this project aims to provide students with resources to pursue advanced degrees and careers in STEM. This work plans to engage students from underrepresented groups as a part of a continuous effort to broaden participation in computing and develop advanced graduate-level courses to introduce new trends and cybersecurity challenges in future decentralized artificial intelligence over IoT.<br/><br/>This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.