Inspired by the boom of the consumer IoT market, many device manufacturers, new start-up companies and technology behemoths have jumped into the space. Indeed, in a span of less than 5 years, we have experienced the manifestation of an array of solutions for the smart home, smart cities and even smart cars. Unfortunately, the exciting utility and rapid marketization of IoTs, come at the expense of privacy and security. Online and industry reports, and academic work have revealed a number of attacks on IoT systems, resulting in privacy leakage, property loss and even large-scale availability problems on some of the most influential Internet services (e.g. Netflix, Twitter). To mitigate such threats, a few new solutions have been proposed. However, it is still less clear what are the impacts they can have on the IoT ecosystem. In this work, we aim to perform a comprehensive study on reported attacks and defenses in the realm of IoTs aiming to find out what we know, where the current studies fall short and how to move forward. To this end, we first build a toolkit that searches through massive amount of online data using semantic analysis to identify over 3000 IoT-related articles (papers, reports and news). Further, by clustering such collected data using machine learning technologies, we are able to compare academic views with the findings from industry and other sources, in an attempt to understand the gaps between them, the trend of the IoT security risks and new problems that need further attention. We systemize this process, by proposing a taxonomy for the IoT ecosystem and organizing IoT security into five problem areas. We use this taxonomy as a beacon to assess each IoT work across a number of properties we define. Our assessment reveals that despite the acknowledged and growing concerns on IoT from both industry and academia, relevant security and privacy problems are far from solved. We discuss how each proposed solution can be applied to a problem area and highlight their strengths, assumptions and constraints. We stress the need for a security framework for IoT vendors and discuss the trend of shifting security liability to external or centralized entities. We also identify open research problems and provide suggestions towards a secure IoT ecosystem.