Developing an AI-enabled mobile app for visually and hearing impaired to identify the denomination of Indian currency notes

At the project’s outset, team Daffodil presented a technical proposal, project management strategies, software requirement specifications, high-level & low-level design documents, and user acceptance test cases to RBI. These documents were reviewed and approved by their technical team, allowing the development process to commence.

Image Dataset Training

To counter the usage of the app in poor internet connectivity areas, the app was required to work in the offline model. Since the app would not be able to learn by itself due to its offline nature, it was crucial for the Daffodil team to train it specifically for offline usage.

They compiled a dataset containing 150,000 images of Mahatma Gandhi series banknotes in the denomination of Rs. 10, 20, 50, 100, 200, 500, and 2,000 to facilitate this training. Whenever a user scans a note, the app compares the scanned image of the note with 150,000 images in the dataset and provides accurate results in the offline mode or even in low lighting conditions.

Advanced Testing Techniques

Additionally, the development plan incorporated security protocols such as Vulnerability Assessment & Penetration Testing (VAPT) and Static Application Security Testing (SAST) to secure the app against vulnerabilities in both its performance and code.

The mobile app developed by team Daffodil enables the visually and hearing impaired to perform the following actions:

• Recognize the denominations of Mahatma Gandhi Series and Mahatma Gandhi (New) series banknote (Rs. 10, 20, 50, 100, 200, 500, and 2,000) by scanning either the front or backside/part of the note, even in the case of half-folded notes, at various holding angles, and under different lighting conditions (including normal, low light, and daylight).
• Identify the denomination through an audio notification in Hindi/English and non-sonic mode such as vibration (suitable for those with vision and hearing impairment).
• Enable seamless navigation of the mobile application using voice-activated controls such as Siri and the Talkback feature, if supported by the device and operating system, ensuring accessibility to all application features.
• Identify the denomination of the banknotes without requiring an internet connection. The app operates offline and automatically adjusts the smartphone flashlight settings to accommodate varying lighting conditions.

Enhanced Language Accessibility

In response to diverse linguistic needs, our solution features voice commands and voice overs in 11 regional languages. When users launch the app, they are presented with an intuitive menu displaying language options, each one being clearly announced. Users can effortlessly choose their preferred language using simple verbal instructions.

Let’s consider a native Punjabi speaker using this application. They have the flexibility to opt for Punjabi as their preferred language setting. As they scan a currency note, the app delivers a clear and audible announcement of the denomination value in Punjabi. This inclusive feature is designed to accommodate users from diverse regions, allowing them to seamlessly navigate the app in their chosen language.

Improved Accessibility with Guided Audio Support

To enhance accessibility further, the application includes a greeting audio file that guides visually impaired users through its features and usage instructions. However, integrating this audio file was challenging due to its large size. For maintaining uninterrupted offline functionality, it was imperative to uphold a minimal application size.

Our team adeptly resolved this issue through meticulous optimization, allowing the application to offer a seamless user experience while maintaining its compact design and essential features.

Vibration Mode for Hearing & Visual Impairment

For people with hearing and visual impairments, the app has predefined number of vibrations for different denominations; one vibration for Rs. 5, two vibrations for Rs. 10, three for Rs. 20, four for Rs. 50, five for Rs. 100, six for Rs. 200, seven for Rs. 500 and eight for Rs. 2,000. If the app fails to recognize the denomination, it triggers a prolonged vibration and prompts the user to rescan the note.

Community-Centric Testing and Validation

To guarantee the accessibility and usability of the mobile application for visually and hearing impaired, we took a hands-on approach to validate our solution. Our team engaged directly with individuals from these communities, carefully selecting participants for extensive testing and User Acceptance Testing (UAT). This validation process ensured that the digital platform was not only functional but genuinely easy to use, enhancing the experience for all users.

Their firsthand experiences and feedback affirmed that the application is easy to navigate and operate, reinforcing our commitment to creating technology that is accessible and beneficial for everyone.

Flexible Interaction Options for Partial or Color-Blind Users:

Acknowledging the varied needs of our users, the application includes an additional feature “No impairment”, catering to partially or color-blind individuals. This feature allows users to opt out of the TalkBack functionality, providing a simpler and more flexible interaction method, regardless of their visual capabilities.

Community-Centric Testing and Validation

To guarantee the accessibility and usability of the mobile application for visually and hearing impaired, we took a hands-on approach to validate our solution. Our team engaged directly with individuals from these communities, carefully selecting participants for extensive testing and User Acceptance Testing (UAT). This validation process ensured that the digital platform was not only functional but genuinely easy to use, enhancing the experience for all users.

Their firsthand experiences and feedback affirmed that the application is easy to navigate and operate, reinforcing our commitment to creating technology that is accessible and beneficial for everyone.

Support and Feedback Setup

The app simplifies user interaction by enabling support requests and feedback submissions via SMS or missed calls, both seamlessly embedded within the app and managed with expertise by Team Daffodil in the backend. To streamline this process, a dedicated dashboard was developed, allowing the support team to monitor and address reported issues and suggestions. These valuable inputs are utilized to continually enhance the application’s functionality.