The Hacker Mind Podcast: Ghost Token
What if an GPC project OAUTH access token wasn’t deleted? This could expose databases to bad actors. Tal Skeverer from Asterix discusses his DEF CON 31 presentation GhostToken: Exploiting Google Cloud Platform App Infrastructure to Create Unremovable Trojan Apps.
<iframe style="border-radius:12px" src="https://open.spotify.com/embed/episode/7Ff0ZmxbxDZJqK7DU2QBbR?utm_source=generator" width="100%" height="352" frameBorder="0" allowfullscreen="" allow="autoplay; clipboard-write; encrypted-media; fullscreen; picture-in-picture" loading="lazy"></iframe>
When you delete something -- say a word or a sentence in a doc -- i’s not really deleted. For example you can undo it -- there’s usually a button -- and the deleted item magically reappears. So it must be stored somewhere because you can literally undelete the last several items you deleted. Or maybe it wasn’t really deleted at all. Hmm.
The same is true with the document itself. You can delete it, and in Windows, what that really means is that the document is removed from the master boot record -- so it’s no longer indexed. And there too you can restore something in the trash before it is permanently deleted. But permanently deleted? Not really. And that is why in digital forensics it is possible for investigators to go back and reconstruct deleted files so long as those blocks of memory aren’t overwritten with new material.
There are some caveats. First, if the file is written over with new data, it can still be -- albeit not easily -- recovered. And second, you can get military-grade scrubbers that write 1s and 0s hundreds of times over the deleted file. That file isn’t coming back any time soon.
Most people don’t go to great lengths to delete a file. They just delete it. And in most cases it’s not a big deal.
But what happens when you delete a developer project in GPC and that project has an OAUTH access token? What if that token didn’t really get deleted.
In a moment we’ll hear about a presentation given at this year’s DEF CON 31 about a GHost Token found in Google’s Cloud Platform. It’s one of those cases where you think something’s deleted, but in this case it could have opened the door all the data in a deleted app.
[music]
VAMOSI: Welcome to The Hacker Mind, an original podcast from the makers of Mayhem Security. It’s about challenging our expectations about the people who hack for a living. I’m Robert Vamosi and in this episode I’m discussing what happens when an access token for an app doesn’t get deleted and how, within 30 days, you can gain full access to the deleted app’s content just because the ghost token is still available.
[music]
VAMOSI: A few years ago I worked at an Identity and Access Management company. IAM isn’t sexy but it’s very important. These are the tools that give you access to the apps and services you need. When you sign in using Google, there’s a whole back end process that unlocks services for you to use. Often that means there’s a token. A lot of attention has been paid to setting up the token. But what happens on the other end? What happens when that app is deleted. So I reached out to someone who found out the answer to that question in a presentation: GhostToken: Exploiting Google Cloud Platform App Infrastructure to Create Unremovable Trojan Apps
SKVERER: So my name is Tal Skverer. I'm currently the research team lead at Asterix.
VAMOSI: Asterix sits at the intersection of security and identity management.
SKVERER: So basically, nowadays, large organizations manage users using all different kinds of platforms and solutions, so that it will be easy to if someone joins your company, you just with one click of a button you give them access to all the platforms you use and when they live. You make them live and make sure they have no access and slix wants to solve this specific problem for non human entities. So usually employees and others you know, people that join your company and get access to your resources are called the humanities, and there are plenty of non-human entities that live in your organization, which can be very difficult to manage. So stuff like credentials and access keys and automatic services. I think the classic example for someone I guess, you know, people listening to this are more on the technological side of developers. So let's imagine that one of your developers creates an access key personnel access token and puts it in your CI CD process. And then when they leave the company, you, you, you need to decide whether to maybe, you know, move the access and then you risk your production falling apart because this automatic process will stop working, or maybe not do it and then your risk is simply still having access to your code. So this is a little bit of pitch.
VAMOSI: So in this episode we’re going to be talking about something called OAUTH
SKVERER: That's interesting. So it sounds like you've taken this approach asking about oauth first. So I guess I was, like the original name for it was the open optimization framework. So it's not a standard thing. They decided to call it a framework, at least the second version of it. And it's a protocol that allows you know, self pouty entities to access your data. On your behalf. You give someone else access to your data in some allowance provider, and to be more or less specific. Let's take your Gmail account, and you want someone to maybe view your calendar and summarize meetings that you have or maybe summarize your Docs and Google Docs. So you're giving them access only to this specific thing, Google Docs, and then they can do whatever they want with this data, but they don't have access to send emails, for instance. So this is the original problem that the auth graders wanted to solve.
VAMOSI: And while we’re defining things, let’s unpack GPC for those who might not know.
SKVERER: Yeah, so one of the services provided by Google is the Google Cloud Platform, which is Scholten, everyone to GCP. So if anyone in the audience is familiar with Azure or AWS, it's the same thing on the Google creator. And it's a cloud service where they offer plenty of services for developers and creators to do, you know, create machines and creative ematic processes? Use cloud compute resources, like let Google Compute stuff for you to handle your keys? Anything that you want to upload to the cloud? And, and not host on premise yourself?
VAMOSI: My understanding is that there are different ways that you can delete an OAuth token. And there's a way in which it doesn't really get deleted in GPC is that sort of a very high level what we're talking about here with the ghost token.
SKVERER: this is a good, a good high level of the general issue on this in the lead to this vulnerability that we find Yeah.
VAMOSI: So how did this come up? I know vulnerabilities present themselves in weird ways. I imagine there's a good story there.
SKVERER: Yeah, exactly. So I think this may be the top of the leaderboard for where the four way lessons to find a vulnerability but actually just, you know, I didn't even do a lot of security research and cloud research. To find this specific issues. And actually, this ghost accountability completely didn't come from my research. So our research teams does a lot of routine security checks on our customers' environments. And while we were looking at some one of the environments of our customers, we found something weird found an app, like an OS app, whose name was smell some gibberish identifier. Usually we see names like the developer name their app, someone in the past, and now we've seen just some identifier that is meaningless. And we want them. We were wondering, what's going on? So our best guess was that maybe this happens when the developer deletes there. And then maybe the Google back end doesn't know what to take the name from. So it just defaults to show the identifier. Yeah. Yeah, so so this is why we initially even thought about looking at this whole deletion process in Google religion of auth apps and, you know, take a closer look and find out if there was anything weird going on during this process.
VAMOSI: So is it because the app didn't take care of its tokens? Or is it Google's responsibility to say the app isn't there that were there for the token doesn't need to be there?
SKVERER: It is 100% Google’s responsible. So they needed to make a decision, right, if the owner of the app leaves out? Now, the logical thing to do is to immediately revoke all tokens that the app has, and I guess when they decided whether to make this decision or not, they thought to themselves, well, you know, the developer deleted the app, they have no way to use the app again. So it doesn't really matter what happens to the tokens. They can stay alive, they can die. We don't really care. And actually the reason why I thought your vote is interesting is that you have to ask why, why is it almost even connected to GCP. So we have GCP on one hand here, which is like the cloud services that Google provides, and we have AWS on the other side, and there's a question on why they're even combined together. So let me tell you that about six years ago, if you needed to create an app in Google, as a developer, if you wanted to create an app for Google users to use and all of that, and then you go to some page movement, and you say, Hi, my name is Tal. I'm a developer. My app name is a very nice app. And then you'd get the ID and secret value that you needed to use in the OAuth protocol later, and that was easy. And I've actually never been able to find the exact exact time that it's happened, but sometime in the past about six years ago, Google decided to combine this process of creating an app with the GCP platform. And now I'm going to make a speculative statement, b ut my guess is they said, Well, we have a large cloud platform for developers. So maybe if we make them, create them in this cloud platform, maybe they want to use other services that we provide in GCP. We can make it easier on them, right? The whole infrastructure is already there. They can just turn it on with a few clicks. And once this decision has been made, maybe then the original Google developer decided what happens when an app is deleted, wasn't involved in this idea. And then things fell, we fell between the cracks and this goes to a vulnerability that was created back then.
[music]
VAMOSI: So you've come upon this oddly named token. And what did you do next? Did you try to explore what it was related to or, or what it could allow you to do?
SKVERER: At first, we just wanted to verify our speculation. We speculated that this happened when an app was deleted. So creating an app, installing it on ourselves, like on a fake user, and then deleting the app and seeing what happens to the tokens. Now I will mention the way that we deliver the app, so an app in Google is, as I said, contained in GCP. And the way it's contained is you have to first create a project inside GCP. And within the project, you create your app and give it a name you tell Google who the developer gives other details. So you actually can go to the project and specifically the deleted the app. So, once we did that, we saw that this oddity of the name becoming the ID has been replicated. The same thing happened. And so actually we will find it and we will discuss it in the Monza amongst ourselves. Well is this? Does this explain the behavior that we've seen in the customer environment? And then while we were discussing this, we wondered, what would happen if we deleted the whole project and not just the app itself? So an app is contained within a project. So what happened? What would happen if we needed a prod? And once we did the length of the project, the same thing happened. The client ID sorry, the name of the app became the ad and all the tokens stopped working like we did when they deleted the app directly. But what we saw is that you can restore like the project isn't immediately getting deleted, it enters like a limbo state, they call it a pending deletion state where we have 30 days, 30 days to regret leaving the project and install it. Maybe you have some important stuff in this box and it contains a lot of things. It can contain a huge table and a lot of data that you may accidentally have someone in your organization that needs it and you want to install it. So it makes sense that they offer this feature but we notice that once you install the project that was deleted before all the tokens that the other users have given to this app that belong to this project, start walking again, so they will invalid while the app was deleted. While the project was the latest, sorry, but when we started it, they then other tokens started walking again. And actually at this point, we were like, Okay, this is weird, but we're not sure if there's a vulnerability here. So, like, what's going on? And when we inspected what the user sees, while this whole thing happens. So remember, this whole deletion is stealing concepts for the developer of the app. So the developer of the app decides to do that. And when we looked at it from the user's point of view, we noticed that while the app is in the pending deletion state, then the user cannot see it just doesn't appear everywhere. And if the user cannot see it, the user can delete it. Revoke the access, though, there isn't one nice page in Google where you as a user, you can see all the apps that you've given access to and once the app is in pending deletion, then it doesn't appear in this page and you can't do anything about it. So it's, it's gone.
VAMOSI: So from the user's point of view, it's gone. But from the cloud, the token still exists.
SKVERER: Yeah, I can still exist and it belongs to the developer, which at any point in time, they can install the object, access the user data using the token that you already have, and then delete the project. Again, it's all processes and in a matter of seconds, and the developer can continuously do that, in order to steal the users data without the user being able to remove the app.
VAMOSI: Man but wait, so restore it anytime within 30 days?
SKVERER: Yeah, right. But I just said it's a matter of seconds. So every every Thursday there is the developer who let's now stop calling an attacker because it's evil. In random shots to second script that we started the project X has died and then deleted again. And once you do the restore and deletion the 30 days refresh, they start again. Got it. Oh, in fact, it's infinite. It's forever. Not limited by the time.
VAMOSI: So the way in which you will impact the end user is that any data that you had in that app could be exposed to a bad actor?
SKVERER: Exactly. Yeah. So at any time this app can be used to access the user data without the user being able to say something about it to say, now I don't want to give this app access anymore. I want to remove it.
VAMOSI: And there's really no way for the end user to even know this is going on without looking at the Google page, which of course, we all have opened on our Chrome browser for constant review.
SKVERER: Right? Actually, even if the user does open this page, it is actually they need to do it. During the attacker, you know, the developer restoring the app and deleting it which is a matter of seconds. So unless you are constantly on this page and constantly refreshing aid in order to find this, you know, org app that you have there. You will never be able to catch it right right and it will be access
VAMOSI: So you came upon this token with an ID number. Were you able to trace it back to the application?
SKVERER: Yeah, so actually, the identifier that you're talking about is a unique identifier for the application. So actually for this specific application, we don't know its name. We have some ways in which like in the product that Asterix offers, we have a ways to enrich this you know identifier that means nothing into more data we can we can tell you about who created is what's what company they belong. to, what access do they have, stuff like that, because as an identity platform, Asterix needs to provide all this information to you. So it can decide whether to remove this app for moving exec organization or keep it so we can actually say more about IDs like that. The issue is that when we Okay, so at this point, we understand that we have an ability here and it actually has been open for six years at least. And our most pressing concern was first to report about it to go well, and the second pressing concern is to find out whether any of our customers have been affected. Like was Did someone ever exploit this vulnerability. And the issue is that developers delete apps. It happens naturally, maybe on rare occasions. But feel it happens. And there was no way to identify between apps the tip and min openly related and honestly the later on and apps that were abused in this deletion or restore method and exploited this vulnerability.
[music]
VAMOSI: So one of the things we should understand is the role of OAUTH in all of this. OATH started as a framework for identity and access management. With OAUTH 2 is more of a standard, however there are a lot of loopholes. Perhaps this is where the vulnerability started, in the gray areas.
SKVERER: I want to touch on the first point that you've asked, like how is OATH related to all these things? So actually, I'm totally talking in hindsight. So after that, we discovered the body and the spine was fix. I wanted to write something about it and you've read my blog. And during this work, I was starting to ask myself why the hell did it happen? What happened here? Why is all related to GCP? And what happened with the project deletion what happens to tokens, and obviously once we found it, this is an aside. Once we found it, we immediately went to other platforms such as slack, Azure or AWS and try to see if the same thing happened there. So what happens when the developer deletes an app? What happens to the tokens, and actually everyone handles and that is handled this well. But during this process, I was thinking, what happened.
VAMOSI: So Tal wanted to see what was happening around the time that OAUTH was created; what was thinking behind how they do things, and how various vendors do things related to OAUTH.
SKVERER: I actually went back to the OAUTH RFC that went out in 2012. So more than a decade. ago, and I read through it and in the past, I've read through it a couple of times. I've been in the web hacking scene for a while now. But this time I pay special attention to whether like, what does that tell you about managing apps? So what happens when a developer wants to register an app? And what happens when the developer doesn't want the app anymore and wants to delete it? And specifically, what happens to such apps in relation to how well the users can manage apps that they've given access to? So I call him the user management page, the page where the end user can see all the apps they've given access to and I found nothing.
VAMOSI: Oh, that’s not good.
SKVERER: I found that the RFC doesn't really talk about it. And specifically, only in the case of what happens when you want to register an app. And we can go to the RFC now and I will actually read exactly what it says maybe maybe in a moment, but it makes fairly says, Well, if a developer wants to register an app, they go on the on the server, which means well, you know, in our case, and they asked to, to create an app and they get in return, they gave Google two technical values needed for the protocol. And then there was a third point that says, any other value will be provided as requested by the server by Google. And then in parentheses, they say things like the developer name, the email address, etc.
VAMOSI: Okay, that leaves a lot to interpretation to the vendor.
SKVERER: And I read it and was like, this is not the standard No, this leaves a lot of a lot of things open this fall when Google wants to implement this. This OAUTH framework, this leaves a lot of things to happen to them they can do whatever they want, they can combine the after discussion with GCP is not a problem with that according to their say, and do whatever they want. And, and in fact, this is an issue that at least in Asterix, if I put the vulnerability aside for now, we run into this issue for a lot of platforms. Like let's say we see an app installed on the customer's environment. And we were questioned about this app. Who belongs to what they do with the data? What should we hav e the behavior look like? And usually, we don't have this information. This version is very limited. We can usually maybe try to find if the app is public in some marketplace, and then connect it to the developer there. But in a lot of platforms, we can do that. And then this app is just there. We can say anything interesting about it. And it's a problem because like and the owner of the app needs to know whether to remove it or not remove it and we can say anything about it. So so yeah. So in hindsight, I realized this, this was like, the root cause lies somewhere more than not defined enough in this area that actually causes the ghost token probability.
VAMOSI: Oh, sure. Blame OATH. Yeah.
SKVERER: Yeah, no, actually, to blame. I was following the creation of it when I was pretty young. back then. But it was, I was following the creation of it. And actually, there was some interesting discussions back there, even if we look at the historic history pages and what went down. So there was two versions of OAUTH, one was released in 2010, I think and then two years later, the second version, and within the first version of the second version, many of those worked on the first version, and then they presented the developers side. You know, the one that commonly creates apps, they actually designed and they didn't work on the second version and there's still people that specifically followed one of them that was very famous like them for grading us. And he said that he couldn't agree with the wishes of the corporations. So there is a lot of what I'm looking for here … politics. It's made of politics but there was a push from the corporations to do things their way.
VAMOSI: In any standards organization you have representations of corporations, vendors, who had an interest. Obviously they want the standard to reflect how they are solving a given problem -- hence, their product will become the defacto standard to use. But in ceding so much to the corporations you end up with some interesting and -- sometimes-- dangerous loopholes.
SKVERER: And for instance, I think one of the most appalling thing as a developer in our is the fact that you need to go if you want to create an app that works for Azure AWS schoolgirls slack, Salesforce, I can continue the naming, platform sale, you need to go to each individual platform and register your app there and realize, find out how they want you to register your app. And it's going to be different on all these platforms. Instead of you know, just doing this once and everyone will know your app. So you need to do this individually. This is a classic example. They want the power for themselves in this case.
[music]
VAMOSI: So I'm gonna go back to the very beginning. How again, did you happen upon an ID? I mean, a token with an ID as opposed to a name? How did you discover that initially again?
SKVERER: So we just do, we make routine checks on our customers' environments, to identify apps and services and any non-human identities that could pose a threat to the organization. Maybe they have IO pipe permissions, or they've been used in a weird way. Let's say they, they, we analyze how they look on a normal day, maybe give them a baseline behavior. And then when they trigger an anomalous behavior, we can go and tell their customers well, you have an app that has been behaving in a weird way. Maybe you want to delete it, or you have an app here that is developed by an individual and not a company. Maybe you don't have any agreement with this individual, but they actually have access to your entire organization's Drive account. For instance, this is what we do. And to do this, we need to routinely check and analyze environments like Google environments of customers. And in one of these environments, we found this app whose name was actually the ID of the app. So this triggered the whole thing.
VAMOSI: So there's a way in which you can search for services that are connected to your environment and so forth. And in this process, you found a token that didn't have a name, it had an ID. There's no way of knowing that that has been deleted in the last 30 days or not. It's just luck on your part that you found one or that you then created your own example where you could see that process. Is that right?
SKVERER: Exactly, exactly right. Okay. So it's only the fact that we tried to replicate the same behavior we created. I want an example where this happens because we wanted to explain it to ourselves, like what's going on?
VAMOSI: Right, so just the mere discovery of a token with an ID number isn't like a high threat to the environment. It's bad, but it doesn't have to be like, you know, someone is actively exfiltrating data with this token. Organizations are constantly, you know, hooking into services and they don't necessarily know this app was created by a single individual and not an organization or a company.
SKVERER: Right? Yeah. So they have to have some sort of problem to even have visibility on this. Let's say you're a security officer in an organization and you're aware of this problem. You're aware that maybe your users are giving access to these apps or services or other companies, and you want to be in control of that. And the ways that, you know, platforms, such as Google give you access to information is severely limited.
VAMOSI: Maybe a specific example here?
SKVERER: I guess we can maybe mention that as if you if your organization uses Google workspace. This is the name that Google gives like the infrastructure that allows you to manage an organization and like, like sort of an Active Directory organization inside Google. And so as an admin in the Google workspace environment, you could actually, you could have actually found apps that were deleted back then. So the controls that Google gave to admins inside Google workspace did allow admins to review apps within organizations and their apps within the organization. And actually remove apps that behave in a weird way, like having this ideas name. And so this was actually a possibility. But the problem is that admins in Google workspace didn't know what this meant; they saw an app with a weird name. I guess the first thought is not let's remove it because it poses a risk, but it was technically there. So I want to say that as as to highlight the importance of giving good admin controls in these platforms.
VAMOSI:Right. I wondered about Tal’s experience working with Google on this. I understand it was reported in June of 2022.
SKVERER: Right. So we recorded this a few. I mean, I think a day after we discovered it and realized that it's invulnerable. So we walk real world report about the soul process to Google and we let them know and they responded, I think in August, and they said, It's the famous Nice catch, which verifies this as a vulnerability that take take may take into account and they open a ticket and we walked someone together with them on possible solutions. And actually in the first place, we had an interesting discussion in which we realized that they didn't understand the severity of his problem. So we talked about this whole 30 days you actually asked the same question they did. So in the first place, they thought okay, so the scope of the attack is very limited because after 30 days, the product is deleted, and then the attacker is dead, the attacker cannot do anything. And they didn't realize that every time we restore the ledger to the project was 30 days, start a new one and we actually will illustrate during a discussion about another thing and I was like, it sounds like they don't understand the severity of how critical this vulnerability is. And I made sure that these points get across and once they heal as they say, they change like the same. Thank you very much. We understand now, which is the priority of this and let's work on a solution together.
VAMOSI: But I guess I'm a little concerned, you notified them in June and then they got back to you in August. I guess that's a normal timeframe because they have to do their due diligence they have to think about like is this really impactful? Because I'm sure they get a lot of bug reports filed and they need to vet them.
SKVERER: Yeah, exactly. So actually during this time, we got in touch with them and they said really swamped with reports. We, we got in a time where a lot of other reports have come through and they didn't get to view our product yet. But actually, within the three months, 90 days, they usually give an actually more interesting thing, the fact that they took a lot of time to fix it once they were aware of it. So they only fixed it in April, April of next year. But I think I'm not holding against them because it's an issue in the deep core aspects of how apps are creating Google. And I think a lot of time went into deciding how to solve it. Because it's not as simple. I mean, I would say once an object has been deleted, go and revoke all the tokens that they have received. Maybe I view I'm viewing it in a simplistic way and it's way more complicated because there's a lot of you know, infrastructures and architecture involved that I don't understand and and maybe it's way faster than I imagined or the end and then to solve it by showing the user apps that have been delivered. Right? This is the way to actually solve it in the end. But this whole weirdness happening while the app is delivered still happens today, only now and users can actually do something about it. So however that is patched now.
VAMOSI: And by patched that means that while the ID name is still on the token, however end users should now be able to remove it from their management console.
SKVERER: I sorted this thing out. The only thing that has changed is that users can do something about it. They can see apps that the name has changed and and they want to remove their access and they can just do it. They don't disappear from the management page before you can see all the apps that you can get access to right. So end users can just go there and move the access.
Exploring The Best SBOM/SCA Tools: Docker Scout
Add Mayhem to Your DevSecOps for Free.
Get a full-featured 30 day free trial.
