- A GitHub Security Lab researcher says AI workflows helped find and report 24 Android vulnerabilities, including flaws in the OsmAnd and Wikipedia apps.
- The researcher says the AI often rated severity wrongly and returned low-value findings, so each result should be reviewed by someone who knows mobile apps.
- Ask your team who confirms severity of AI-found flaws and how many analyst hours review takes. This is GitHub's report on its own tool, from one researcher's Android work.
GitHub says its AI workflows help find complex security flaws faster. Its new report also shows that people still had to decide which findings mattered. Finding a flaw is only half the job. The other half is judging it.
What GitHub's researcher did
A researcher at GitHub Security Lab built a set of automated AI workflows, which GitHub calls taskflows, to audit Android apps. They run on the lab's open source Taskflow Agent. The researcher reports 24 vulnerabilities found and reported so far.
The workflows do not simply ask an AI model to find bugs. They break the job into steps. One step sorts the places where outside data can enter an app, so a mobile app is judged as a mobile app. Another step gives the model a checklist of common Android weakness types to consider.
The researcher then combined a strict prompt with a broad one, across several runs. The strict one keeps obvious bugs from being missed. The broad one lets the model look for less obvious ones.
Two flaws that show the stakes
The first example is OsmAnd, a navigation app with over 10 million downloads on Android. The report describes a screen in the app that any other app on the phone can call. An attacker app could use it to change OsmAnd's settings without the user noticing.
The researcher says this could send map-tile coordinates and route origins and destinations to an attacker's server. The attacking app would need no special permissions. In plain terms, a location-tracking route was open.
The second example is the Wikipedia Android app. A bug in how the app reads web addresses let a link that only looked like Wikipedia load an attacker's page. Chained with a second flaw, the researcher describes it as an account takeover. The report says the exposed cookies are long-lived.
Where the AI needed a human
This is the part executives should read closely. The researcher says the AI returned low-severity issues even when told not to. Some findings needed unlikely conditions that would almost never occur in real use.
The model also got severity wrong. A file-access flaw confined to an app's external storage is a minor issue. The model often missed such limits unless it was told to build a proof of concept.
Even then it could be wrong. In one case, the app's internal data overrides the external data an attacker can write. Then there is no vulnerability at all. The researcher's conclusion is direct: each finding should be reviewed by a researcher who knows mobile apps.
The researcher expects false positives to fall as models improve. That is an expectation, not a measurement. The report gives no count of how many raw findings were discarded.
Finding got faster; judging still needs people
The lesson here is that AI widened the search, and people still made the final call. A smoke alarm that also sounds for burnt toast is only useful if someone checks the kitchen. The tool raises the alarm. A person decides what deserves a fix.
The work also has running costs. The report says a run on a medium-sized repository might take an hour or two. It uses premium model requests and can consume many tokens. Then someone must read what comes out.
The report does not say how long that review takes. Two caveats also apply. The report comes from GitHub about its own tool. And it covers one researcher's work on Android apps, including OsmAnd and Wikipedia. It does not show how the method performs on a company's private code.
Questions to put to your team
First, when a tool or vendor reports AI-found flaws, who confirms severity before anyone acts? Ask what proof they require.
Second, what does review cost? Ask for the analyst hours spent per confirmed finding, alongside the model and licence spend.
Third, does any app your company ships or depends on expose screens or links to other apps? The OsmAnd and Wikipedia flaws both grew from that kind of opening. The researcher says the types of flaws found were where an experienced security researcher would expect them.
The lesson: AI can lengthen the list of findings, but people still decide which ones matter.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: GitHub.
