amused
Savant is Digital Working Memory: Search, Browse, Video Replay, & Copy/Paste for anything you've ever seen on your screen! Savant / "Savant Recall" from YCHacks is going Open Source! Our old website is stored (here) with Demo V.2, and Demo V.1 can be seen on Devpost (here). I received permission from Mark (Zhengkan) after many months. <3
You can find the repository here to start working with it: https://bitbucket.org/theluxury/savant
Eventually would like to create more of a Walk-through, and your conversations will really help, so feel free to reach out to me especially if you are an Open Source developer. (This is in Obj.C Cocoa for Mac OS X).
What was the initial inspiration for Savant?: While developing and designing, we work across many different active windows, tabs, and programs. After stepping away to get a sandwich, it's easy to have the familiar feeling of "What was I just doing?" We felt there was a compelling need for instant replays to jog your short-term memory. Savant helps get users back into the flow after an interruption. It is definitely no fun feeling lost. Knowing your way around your own workflow would give you the confidence to stay on a winning trajectory.
Team story, and where YOU fit in: Core team of Savant is Paul Musgrave, Mark Wang, then myself Kasey Robinson who was secondary (now leader of Open Source community fielding questions), and I also want to include Nick Hain who did some great work demo-ing to judges at YC Hacks 2014. Then Paul and Mark got to YC W15 interview, and we (did not make it into the batch). I was very disappointed, but for other reasons, thankful of this. Same goes for when we went to Kevin and Julia Hartz looking for funding - they were right to assume this is a great project, great team, but might not be far enough along nor sound enough to stay together - and suggested a plan B. Our plan B happened: that Paul Musgrave went back to finish his CS degree, and Mark and I became serious in our personal relationship. Then Savant was on the perpetual back burner. We need YOUR help to bring this technology back to life. And I want to hear from you.
Here is the patent pending for the technology!
Method and apparatus for enabling a searchable history of actions taken on an electronic device.
Abstract
A method and apparatus for enabling a searchable history of actions taken on an electronic device is described. The method may include capturing media data corresponding to what is shown on the electronic device's screen. The method may also include capturing user input on the electronic device. The method may include transmitting the captured data to a server computer system, allowing the server computer system to perform one or more recognition processes on the capture data to add to a history of the actions taken by the user. The method may also include transmitting a query of the user to the server computer system to initiate a search of the history of the actions, and receiving results relevant to the query that include data indicative of the actions in the history of the user.
Description
Field of the Invention
The present invention relates generally to the field of logging on electronic device, and more particularly, to enabling a searchable history of the actions a user has taken on said electronic device.
Description of Related Art
Users perform a multitude of actions on an electronic device, and often times it can be useful to be able to recall said actions. A relevant example would be your browser history, where you can search for the website you have been too. However, while such technologies exist for individual apps, there currently does not exist a method to record and query said actions on a device level. This ability to record and query a user's actions across all apps would give a fuller history of how the user interacts with his/her device.
If such a technology were to exist, it would allow the user to be able to utilize his previous device actions as a sort of searchable database, in order to get a history of the actions he has taken and to query previous actions he took that the user now wishes to recall.
Summary of the invention
A method and apparatus for enabling a searchable history of actions taken on an electronic device is described. According to an exemplary method, data is collected from an electronic device that corresponds to a user's inputs on the device. Such data could include: videos or images of the changes in the screen as the user interacts with the device, the keystrokes of the user on the keyboard, the mouse events of the user, the physical location the user presses on touch screen devices, or any other media or hardware input events.
The data is then processed and stored on a server computer system: either the original device, or on another device with the data being transmitted by means on a network. One or more recognition processes is run on the recorded data, and the data is added to a history database of the user, so that the data may be indexed and made searchable. Example of such processes could include: optical character recognition being performed on media to identify text in said media, image recognition on media, processing of level of change in the media to indicate when there are high levels of activity on the device, indexing of the keystrokes by syntax, context, and characters, indexing of any and all of these records by contextual cues (such as recent levels of activity from the user on the device), and anything else that could be useful in making the data useful and searchable.
The user then may use a search engine, either on the original device or another, to query for specific keywords or similar images that exist in this database. For example, if the user is a programmer and wishes to write code for a functionality he had previously implemented in another program, but has since forgotten the exact code, he can search for relevant keywords, and be able to find the exact steps he took in writing the code. While app specific search features already allow users to do some version of this within specific apps, our invention provides two significant advantages. The first advantage is that our invention will provide a “playback” feature that allows the user to actually see the actions he took, and not just the end result of the finished documents. The second advantage is that the system level recording of our method allows the user to search through all previous history to find his results, whether he himself wrote them in an app or if he found them on the internet.
Once relevant results have been queried and found, the user might have the ability to reconstruct the events surrounding the result. This will be done by a mixture of video, reconstructing video from screenshots, and replay of the actions taken on the device (keystrokes, mouse events, etc).
In addition, there might be a timeline feature, where the user can filter his queries by time interval, or choose to browse through his history by time instead keywords or images.
The purpose of our invention is to allow the user to have a searchable history of his actions taken on a device, and to have the ability to playback said actions through some combination of media and recorded actions on said device or possibly another device. Below is a list of possible uses for this invention which could be useful (this list is by no means exhaustive):
To reorient himself with a task he was doing if he gets distracted.
To reorient himself with what he was doing at work, after he went home last evening.
To recall how to perform certain tasks that he had done in the past.
To search through device history to find specific pieces of information.
To show others, either as a way to get advice on how to improve productivity, or as a way to demonstrate his abilities.
To catalog his own actions on the device, so that he may know how much time he spends on what actions, to become more productive.
To search for his actions on one device on another, so that he can retain the benefits of app specific search features even when working on another device.
These functions described above can be implemented in digital electronic circuitry, in computer software, firmware or hardware. The techniques can be implemented using one or more computer program products. Programmable processors and computers can be included in or packaged as mobile devices. The processes and logic flows can be performed by one or more programmable processors and by one or more programmable logic circuitry. General and special purpose computing devices and storage devices can be interconnected through communication networks.
Some implementations include electronic components, such as microprocessors, storage and memory that store computer program instructions in a machine-readable or computer-readable medium (alternatively referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, read-only compact discs (CD-ROM), recordable compact discs (CD-R), rewritable compact discs (CD-RW), read-only digital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a variety of recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.), magnetic and/or solid state hard drives, read-only and recordable Blu-Ray® discs, ultra density optical discs, any other optical or magnetic media, and floppy disks. The computer-readable media can store a computer program that is executable by at least one processing unit and includes sets of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as is produced by a compiler, and files including higher-level code that are executed by a computer, an electronic component, or a microprocessor using an interpreter.
While the above discussion primarily refers to microprocessors or multi-core processors that execute software, some implementations are performed by one or more integrated circuits, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some implementations, such integrated circuits execute instructions that are stored on the circuit itself.
As used in this specification and any claims of this application, the terms “computer”, “server”, “processor”, and “memory” all refer to electronic or other technological devices. These terms exclude people or groups of people. For the purposes of the specification, the terms display or displaying means displaying on an electronic device. As used in this specification and any claims of this application, the terms “computer-readable medium” and “computer-readable media” are entirely restricted to tangible, physical objects that store information in a form that is readable by a computer. These terms exclude any wireless signals, wired download signals, and any other ephemeral signals.
To provide for interaction with a user, implementations of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user.
Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a backend component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such backend, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), an inter-network (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).
The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data (e.g., an HTML page) to a client device (e.g., for purposes of displaying data to and receiving user input from a user interacting with the client device). Data generated at the client device (e.g., a result of the user interaction) can be received from the client device at the server.
It is understood that any specific order or hierarchy of steps in the processes disclosed is an illustration of example approaches. Based upon design preferences, it is understood that the specific order or hierarchy of steps in the processes may be rearranged, or that all illustrated steps be performed. Some of the steps may be performed simultaneously. For example, in certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.
The previous description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects. Thus, the claims are not intended to be limited to the aspects shown herein, but are to be accorded the full scope consistent with the language claims, wherein reference to an element in the singular is not intended to mean “one and only one” unless specifically so stated, but rather “one or more.” Unless specifically stated otherwise, the term “some” refers to one or more. Pronouns in the masculine (e.g., his) include the feminine and neuter gender (e.g., her and its) and vice versa. Headings and subheadings, if any, are used for convenience only and do not limit the subject disclosure.
A phrase such as an “aspect” does not imply that such aspect is essential to the subject technology or that such aspect applies to all configurations of the subject technology. A disclosure relating to an aspect may apply to all configurations, or one or more configurations. A phrase such as an aspect may refer to one or more aspects and vice versa. A phrase such as a “configuration” does not imply that such configuration is essential to the subject technology or that such configuration applies to all configurations of the subject technology. A disclosure relating to a configuration may apply to all configurations, or one or more configurations. A phrase such as a configuration may refer to one or more configurations and vice versa.
The word “exemplary” is used herein to mean “serving as an example or illustration.” Any aspect or design described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects or designs.
All structural and functional equivalents to the elements of the various aspects described throughout this disclosure that are known or later come to be known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the claims.
More documentation is to come, such as process plans + brainstorming, top feedback pos + neg, the YC application + other applications, what we tried + what we would have liked to try, and then the next direction we can go in. Perhaps we can get a nice clean Github repository going - or a new website.
Thanks for your patience and support! Thanks for choosing Savant! :)
