Помощник (для слепых) Description
How the “assistant” works:
The user launches the application, confirms the appropriate permissions for the application to access the camera, microphone, etc. Afterwards, the application launches itself as a service, which allows you to exit the main screen of the application, keeping it constantly active.
In service mode, the application waits for the user's key voice commands. So, if the user says “assistant, what do you see?” - the rear camera is automatically activated and the process of recognizing objects caught in the camera lens will begin, with voice information to the user (for example, “table in the center, refrigerator a little to the left, chair a little to the right”).
As a result, the user can “see” the objects in front of him through the “eyes” of the application and determine their approximate location.
The process of recognition and voice information occurs in real time until the user says “stop assistant.”
Upon receiving such a command, the application will stop recognizing and continue waiting for further commands.
Basic voice commands:
Assistant, can you hear me? (to make sure the service is running)
Assistant, what time is it | what time is it? (will report the current time)
Assistant, what day is it today? (will tell you the day of the week and date)
Assistant, battery charge (will tell you the battery level of the smartphone)
Assistant, tell me the news (tells the 5 main news at the moment)
Assistant, first | second | third | fourth | fifth news (tells the news in detail)
Assistant turn on radio Zvezda | Russia (includes playing an Internet stream of a radio station)
Weather Assistant (will tell you about the weather in the user’s locality, as well as the forecast for the next two days). Important: the exact geoposition of the user is not determined, only the locality in which the user is located is determined. No GPS required.
Assistant, what do you see (will launch an activity with the camera and begin to recognize objects falling into the lens of the rear camera)
Assistant, stop (will stop the current action - stop playing the radio, stop recognizing objects, stop telling the news)
An important point is that all the functionality of the application (except for requesting current news and radio broadcasting) works completely locally (without the need for Internet access). No information is transferred outside the smartphone or stored on it.
Plans:
1. search for sponsors for the development of the project.
2. collecting a dataset taking into account domestic realities, marking it, followed by training the model to recognize much better and more objects (for example, slippers, an open or closed door, an open or closed window, a switch, curtains, an iron, an ironing board, a washing machine, matches, bread, a loaf and much more).
3. expanding the functionality of interaction with the user by voice (adding the ability to make calls to the desired contacts using voice commands (for example, “assistant, call Elena” - the application will find the contact in the contact list and dial the number), (adding the ability to communicate with the “assistant” on abstract topics.).
4. Adding available radio stations.
5. Improving the quality of voice command recognition.
6. Correction of detected errors.
IMPORTANT:
1. To work with the “assistant”, it is recommended to use handsfree or headphones. This is necessary so that the “assistant” can continue to “hear” the user while speaking information about objects, news, playing the radio, etc.
2. By default, for Russia, a female voice is set in the smartphone settings. To work comfortably with the “assistant”, it is recommended to change the speech synthesis settings in your smartphone by choosing a more convenient voice.
3. Some phone models additionally require permission to display pop-up windows. This permission is necessary to be able to launch the camera and begin recognizing objects, even if the application is not in the foreground or the smartphone screen is turned off. It is also recommended to disable smart battery optimization mode.
The user launches the application, confirms the appropriate permissions for the application to access the camera, microphone, etc. Afterwards, the application launches itself as a service, which allows you to exit the main screen of the application, keeping it constantly active.
In service mode, the application waits for the user's key voice commands. So, if the user says “assistant, what do you see?” - the rear camera is automatically activated and the process of recognizing objects caught in the camera lens will begin, with voice information to the user (for example, “table in the center, refrigerator a little to the left, chair a little to the right”).
As a result, the user can “see” the objects in front of him through the “eyes” of the application and determine their approximate location.
The process of recognition and voice information occurs in real time until the user says “stop assistant.”
Upon receiving such a command, the application will stop recognizing and continue waiting for further commands.
Basic voice commands:
Assistant, can you hear me? (to make sure the service is running)
Assistant, what time is it | what time is it? (will report the current time)
Assistant, what day is it today? (will tell you the day of the week and date)
Assistant, battery charge (will tell you the battery level of the smartphone)
Assistant, tell me the news (tells the 5 main news at the moment)
Assistant, first | second | third | fourth | fifth news (tells the news in detail)
Assistant turn on radio Zvezda | Russia (includes playing an Internet stream of a radio station)
Weather Assistant (will tell you about the weather in the user’s locality, as well as the forecast for the next two days). Important: the exact geoposition of the user is not determined, only the locality in which the user is located is determined. No GPS required.
Assistant, what do you see (will launch an activity with the camera and begin to recognize objects falling into the lens of the rear camera)
Assistant, stop (will stop the current action - stop playing the radio, stop recognizing objects, stop telling the news)
An important point is that all the functionality of the application (except for requesting current news and radio broadcasting) works completely locally (without the need for Internet access). No information is transferred outside the smartphone or stored on it.
Plans:
1. search for sponsors for the development of the project.
2. collecting a dataset taking into account domestic realities, marking it, followed by training the model to recognize much better and more objects (for example, slippers, an open or closed door, an open or closed window, a switch, curtains, an iron, an ironing board, a washing machine, matches, bread, a loaf and much more).
3. expanding the functionality of interaction with the user by voice (adding the ability to make calls to the desired contacts using voice commands (for example, “assistant, call Elena” - the application will find the contact in the contact list and dial the number), (adding the ability to communicate with the “assistant” on abstract topics.).
4. Adding available radio stations.
5. Improving the quality of voice command recognition.
6. Correction of detected errors.
IMPORTANT:
1. To work with the “assistant”, it is recommended to use handsfree or headphones. This is necessary so that the “assistant” can continue to “hear” the user while speaking information about objects, news, playing the radio, etc.
2. By default, for Russia, a female voice is set in the smartphone settings. To work comfortably with the “assistant”, it is recommended to change the speech synthesis settings in your smartphone by choosing a more convenient voice.
3. Some phone models additionally require permission to display pop-up windows. This permission is necessary to be able to launch the camera and begin recognizing objects, even if the application is not in the foreground or the smartphone screen is turned off. It is also recommended to disable smart battery optimization mode.
Open up
