The HouseReader project was conceived as a research initiative, focusing on the applicability of models, multimodal Large Language Models (LLMs), and artificial intelligence algorithms. This research primarily involves computer vision for interpreting interior videos in residential-type assets.

Before delving into the specifics of the project, it's crucial to underscore the significance of generative artificial intelligence's consolidation as a work tool. This advancement has accelerated and unlocked capabilities previously unachievable just a year ago, particularly following the launch of ChatGPT and its accessibility to end users.

Beginning with a profile like mine, grounded in technology but not deeply versed in low-level, highly technical, and full-stack developments, the use of assistants or chatbots such as ChatGPT marks a significant advancement in developing a concept, idea, or project unlike any other tool available to date..

The utilization of modern assistants and chatbots significantly offers a "superpower" that goes beyond just saving time on known, repetitive, and daily tasks. This superpower extends further, enabling humans to tackle challenges, tasks, or projects that were previously beyond their initial reach or areas of expertise and knowledge.

I want to emphasize a crucial point: While it might seem commonplace today, the effective use of AI chatbots is, and will continue to be, a decisive factor in executing personal or professional projects. Additionally, these tools offer invaluable learning opportunities for the end user.

We are discussing the enhancement of capabilities when facing a challenge, conducting market research, addressing problems, developing concepts, proposing functionalities, evaluating alternative approaches, and seeking solutions to achieve specific goals. This often demands extensive technical knowledge in areas like programming languages, infrastructure, DevOps, security, networks, marketing, design, and more. Typically, ideating and executing a project of this nature would require the involvement of several specialized and dedicted profiles.

The project was ultimately carried out entirely by myself, utilizing spare time during weekdays and weekends over the past two months. The only assistance I had was the GPT chatbot, which served as a faithful companion throughout this journey. Without the aid of ChatGPT and similar assistants, it's not just that the project would have been delayed by months or even a year; it would have been outright impossible for an individual with my profile to execute it within a comparable timeframe.

In conclusion, a potent synergy has emerged: the use of artificial intelligence for the development of AI projects. This combination has arrived and is here to stay.

Over the past 12 months, a series of presentations and publications have showcased multimodal Large Language Models (LLMs) capable of interpreting videos or image sequences. Trained with billions of parameters, these models merge vision encoders with both open and proprietary large language models. Their objective is to provide end users with advanced instruction and reasoning capabilities.

Promising outcomes have been noted in all computer vision models and tools tested so far, particularly in short-duration videos where multimodal LLMs effectively respond to basic questions about the scene. However, their performance notably diminishes when processing longer videos and striving for a higher degree of response reliability. Additionally, using more advanced proprietary models could significantly raise the cost.

Further complicating matters are additional uncertainty factors, like processing end-user recordings of their home interiors, which can be up to five minutes long, in various formats and qualities. There's also the potential for unintentional overlapping or repetition of spaces. The aim is to interpret spaces, layouts, and count and identify objects, rendering the task highly complex.

The sole solution to this challenge involves combining several pre-existing models and logics to develop a comprehensive algorithm. This algorithm is capable of reliably interpreting a given video for a specific space, using only the user's video as input, without any additional indications or parameters.

HouseReader, by executing its video analysis algorithm, is capable of:

• Providing a general description of your home.
• Determining the distribution of spaces and rooms that make up the space.
• Estimating the cost value of the identified elements inside.
• Detecting potential risks related to:

  • Rapid deterioration of the property.
  • Lifestyles of the people living in the home.
  • Potential risks of fire, combustion, ventilation, pests.
• Compiling a census of household objects, with images and labeling.
• Generating a final report for the user with a complete summary of the analysis.

This project is scalable to various fields, types of spaces, and functionalities, although the current algorithm primarily focuses on the residential domain.

Multimodal LLMs for video analysis can be classified based on how they process the video content. The two main types are:

• LLM Frame-by-Frame: These models process each frame of the video individually. This can be efficient for tasks such as object and person recognition, but may be less accurate for tasks requiring an understanding of the video context, like video description or video segmentation.
• LLM Stream-Based: These models process the video as a continuous stream. This can be more efficient for tasks that require an understanding of the video context, but may be less accurate for tasks requiring the recognition of specific objects and people.

In addition to these two main types, there are also other types of multimodal LLMs for video analysis, such as Hybrid LLMs, Attention-based LLMs, and Deep Learning LLMs.

2023 has been the year with the most advancements in the publication of models and visual assistants in history. Some examples of the most relevant models that have emerged in this area:

After conducting multiple tests with various conversational models for video, using recordings ranging from 1 to 5 minutes, none have been able to extract information with 100% accuracy regarding the number of rooms in the house, the detail of the elements in each room, their location or distribution, etc. Some models, like LLaVA, are capable of correctly answering some of these questions, but the conclusion is that conversational video models that fully understand their content still have a vast field for development.