When navigating Indian roads, especially for newcomers or those unfamiliar with road rules and regulations, understanding India’s road rules and regulations becomes pivotal. Lack of knowledge about these guidelines while driving can lead to potential risks and violations. Thus, for a safe and compliant driving experience, it’s imperative to familiarize oneself with India’s road laws before hitting the road. With an increasing number of vehicles on the roads and diverse infrastructural complexities, ensuring adherence to road rules becomes paramount for enhancing safety and efficiency in transportation.
This case study delves into applying advanced technology to tackle these critical issues. Specifically, it explores the integration of a Knowledge Base within Dialogflow—an AI-powered conversational platform—to disseminate comprehensive information regarding India’s road regulations, while concurrently addressing concerns related to traffic control, offenses, penalties, and procedural intricacies.
Improving road safety and compliance with traffic laws are imperative for reducing accidents and ensuring smoother traffic flow. By employing innovative technological solutions like Dialogflow’s Knowledge Base, this study endeavors to enhance public awareness and accessibility to essential information. This could potentially reduce violations and contribute to a safer road environment across the country.
The primary aim of this study is to assess the efficacy of generative AI models using Dialogflow’s Knowledge Base feature as a platform for disseminating information on India’s road rules and regulations, aiming to create a more accessible and user-friendly resource for individuals seeking information about India’s traffic laws.
The methodology involves using Dialogflow’s capabilities to structure a knowledge base that encompasses India’s road rules and regulations. By employing the capabilities of Dialogflow’s knowledge base, we created a generative agent. The Dialogflow CX console offers a selection of diverse generative AI models. Different foundational models were utilized by us to enhance the agent’s ability to understand and respond to user queries in a natural and informative manner. By harnessing the power of generative AI within Dialogflow CX, a user-friendly AI chatbot is constructed. This chatbot enables anyone to easily inquire about any aspect of Indian road rules and regulations and receive comprehensive answers to their queries. We tested the AI chatbot with a set of predetermined questions related to Indian road rules and regulations by utilizing each available generative AI model in the evaluation process. We utilize Google Cloud Storage for assessment, wherein the stored data consists solely of unstructured PDFs, with no inclusion of website links or any structured data. We asked it different questions about traffic rules like driving when mentally or physically unfit to drive, Offences relating to licences, Driving dangerously, offences, etc. We explore the impact of individual generative AI models on improving AI chatbot performance in responding to user queries.
The list of supported generative AI Models in Dialogflow CX Console are:
1. text-bison@001
2. text-bison@002
3. text-bison@001 tuned (conversational)
4. text-bison@001 tuned (informational)
5. Gemini-pro
The table summarizes the key details of each model for an easy comparison.
| Model Name | Description | Key Characteristics | Language Support |
|---|---|---|---|
| text-bison@001 | A larger model with enhanced capabilities for text generation and language understanding. | More comprehensive knowledge base, potential for greater accuracy and fluency. | Available in all supported languages. |
| text-bison@001 tuned (conversational) | Fine-tuned for engaging and coherent conversations. | Prioritizes natural language flow, conversational style, and engaging responses. | Only English is supported at the moment. |
| text-bison@001 tuned (informational) | Fine-tuned for providing concise and informative answers to questions. | Focuses on accuracy, factual correctness, and clarity in responses. | Only English is supported at the moment. |
| text-bison@002 | A newer generation model with further improvements in accuracy and fluency. | Enhanced capabilities for language understanding and text generation. | Available in all supported languages. |
| Gemini Pro | Designed to handle natural language tasks, multi-turn text and code chat, and code generation. | Excels in complex tasks, code understanding, and flexibility for multi-turn interactions. | Available in all supported languages. |
The name of the PaLM 2 Text Bison large language model is text-bison. It’s a foundational model that excels across diverse natural language tasks, including entity extraction. The text-bison@001 model has several use cases, including:
Based on evaluations by utilizing the text-bison@001 model and focusing on specific aspects of traffic regulations, we draw the following conclusions about the AI chatbot’s effectiveness,
text-bison@001 tuned(informational) a tuned version of the PaLM (Pathway Language Model) model, specifically optimized for informational tasks. Ideal for tasks requiring direct and concise information retrieval. Might be more prone to errors when dealing with ambiguous or unclear prompts, as it prioritizes factual accuracy.
An evaluation of the AI chatbot, utilizing text-bison@001tuned(informational) and specific questions, resulted in the following conclusions,
text-bison@001 tuned (conversational) is better suited for tasks involving dialogue and creative text generation. Specializes in engaging in natural, open-ended conversations. select text-bison@001(conversational) for applications demanding natural conversations and creative text generation. For additional information, see Scenarios to use model tuning.
To assess the performance of our AI chatbot, we conducted targeted evaluations employing the text-bison@001(conversational) model and a set of predefined questions. Our findings are summarized as follows,
text-bison@002 is a powerful foundational model within the PaLM 2 family, adept at handling a wide range of natural language tasks.
Key takeaways:
Through targeted evaluations with the text-bison@002 model and pre-defined questions, we gained valuable insights into the AI chatbot’s capabilities. Here are the key takeaways.
The Gemini Pro foundation model is a powerful language model known for its expertise in comprehending and generating language. It supports both single-turn prompt and response interactions and continuous, multi-turn conversations, even extending to tasks involving code understanding and generation.
Limitations of the Gemini Pro model:
The gemini pro model’s performance in answering pre-defined questions related to Indian road rules and regulations provides the following insights into the AI chatbot’s strengths and areas for improvement.
To evaluate the outcomes of individual Generative AI Models in conjunction with a Knowledge base in the Dialogflow CX console, navigate to the Generative AI sub-tab within Agent Settings after selecting the Knowledge base Agent in your GCP project.
Check the link for guidance if you haven’t made the Dialogflow knowledge base yet.
By presenting these results within the Knowledge Base Dialogflow framework, we intend to enhance accessibility, compliance, and understanding of India’s road regulations. These findings serve to underscore the potential and effectiveness of various models in facilitating easier access to crucial information within the realm of Indian road laws.
The evaluation criteria encompassed parameters such as language comprehension, contextual accuracy, response coherence, and the models’ capacity to present information pertinent to Indian road rules within the Knowledge Base Dialogflow environment.
We measured the accuracy of several language models by running them through a pre-defined set of questions, just like the ones used for Gemini Pro. We kept track of all the different answers Gemini Pro gave to each question.
Gemini-pro Model Results
Q1: Speed limits for different types of vehicles on Indian roads?
Ans: The speed limits for different types of vehicles on Indian roads are determined by the State Government or an authorized authority. These limits are set to ensure public safety and convenience and may vary depending on the type of road or bridge. Additionally, the National Highways Authority of India or other authorized agencies may impose speed limits on national highways.
Q2: Legal blood alcohol limit for driving in India
Ans: The legal blood alcohol limit for driving in India is 30 mg. per 100 ml. of blood.
Q3: What are the guidelines for using mobile phones while driving in India?
Ans: According to the Motor Vehicle Act, 2019, it is illegal to use handheld communication devices while driving in India.
Q4: Any fine for use on a mobile phone while vehicle driving?
Ans: Yes, there is a fine for using a mobile phone while driving a vehicle. For the first offense, the fine is two thousand rupees. For a second or subsequent offense, the fine is ten thousand rupees.
Q5: What are the consequences for traveling without a proper pass or ticket in a stage carriage according to the Act?
Ans: Whoever travels in a stage carriage without having a proper pass or ticket with him or being in or having alighted from a stage carriage fails or refuses to present for examination or to deliver up his pass or ticket immediately on a requisition being made therefore, shall be punishable with fine which may extend to five hundred rupees.
Q6: What are the repercussions for allowing unauthorized individuals to drive vehicles as per the Act’s provisions?
Ans: The Motor Vehicles (Amendment) Act, 2019 states that allowing unauthorized individuals to drive vehicles is punishable with imprisonment for a term that may extend to three months, or with a fine of five thousand rupees, or with both.
Q7: If we fail to give free passage to the emergency vehicle then?
Ans: According to the Act, failing to allow free passage to emergency vehicles, such as fire service vehicles or ambulances, while driving a motor vehicle is an offense punishable by imprisonment for up to six months, a fine of ten thousand rupees, or both.
After testing multiple questions on each model, we’ve identified specific strengths and weaknesses for each, detailed in the table below.
| Sr.No | Model Name | Strengths | Weaknesses |
|---|---|---|---|
| 1. | text-bison@001 Model | • Responds to user queries from the given knowledge base. | • Inconsistent accuracy, often failing to provide accurate answers when information is available in the knowledge base. • Lack of precision and reliability in responses. |
| 2. | text-bison@001 tuned (informational) Model | • Offers brief information in response to user queries. | • Limited performance with NLP-based inquiries. • Inability to effectively address follow-up questions. |
| 3. | text-bison@001 tuned (conversational) Model | • Its skill in understanding detailed legal information proves its ability to grasp important details highlighted in the marked sections. • Generally improved responses compared to text-bison@001. |
• Inaccuracy in some responses despite relevant data in the datastore. • Tendency to provide extraneous information occasionally. |
| 4. | text-bison@002 Model | • Commendable performance in answering various questions. • Adequately addresses follow-up questions. |
• Occasional failure to provide answers already in the knowledge base. • Addresses the follow-up question with more details |
| 5. | Gemini Pro Model | • Generally superior performance compared to other models. • Proficient in addressing follow-up questions with additional specifics. |
• Inability to respond to some user queries despite relevant data in the knowledge base. |
In comparison to the other models, the Gemini Pro emerged as a superior model in certain aspects. However, occasional instances were noted where even Gemini Pro failed to respond to user queries, indicating room for improvement.
To enhance the models’ performance, it is recommended to focus on refining their accuracy, addressing challenges in handling follow-up questions, and minimizing occasional failures. Fine-tuning the models based on user feedback and incorporating additional contextual data may contribute to improved outcomes.
