The Hacker Mind Podcast: Gaining Persistence On Windows Boxes

When we hear about bad actors on a compromised system for 200+ days, we wonder how they survived for so long. Often they hide in common misconfigurations. 

From her talk at SecTor 2022, Paula Januszkiewicz, CEO of Cqure, returns to The Hacker Mind and explains how a lot of little configuration errors in common Windows tools and services can open the door to persistence on a system for bad actors and what sysadmins can do to mitigate these. She’ll also be presenting again at RSAC 2023 in April.

Vamosi: Whenever there’s a data breach or an attack, I look at how long the bad actor was active on the compromised network.  Estimates vary greatly, with some security vendors claiming dwell time is as low as 11 days with ransomware while others claim dwell time can be as high as 200 days or more with more sophisticated attacks. 

So how does this happen? 

If you’re running edge detection, if your scanning your networks, even occasionally rebooting your servers these activities will remove some running malware, yet the bad actors somehow return and remain persistent. Hence we talk alot about Advanced Persistent Threats. These APTs have somehow found a way to bypass most security tools, hence their persistence. 

So I thought it would be good to look into how this persistence might be achieved. Special coding tricks? Stealth malware. Magic? Maybe. But it turns out there are services and tools within Windows that when misconfigured or mishandled can help the bad actors. Perhaps these should be your first targets for pen testing and internal red and blue teams. 

So in a moment we’ll hear how bad actors can persist. And we’ll learn how we can better detect these APTs.  I hope you’ll stick around.

[MUSIC]

Welcome to the Hacker Mind, an original podcast from ForAllSecure. It’s about challenging our expectations about people who hack for a living.

I’m Robert Vamosi, and in this episode we’re talking about ways in which bad actors can manipulate legitimate tools to gain persistence on a site so they can steal data or encrypt it for ransom. 

[MUSIC]

VAMOSI: In Episode 49 I talked with Kyle Hanslovan about Living off the LAnd, about bad actors using existing tools to pivot inside a network, how they don’t need to upload their own-- they’re already there on the system.  In this episode we’re going to go deeper into this idea, and for that I turned to an expert on Windows. 

JANUSZKIEWICZ: Paula Januszkiewicz. I'm the CEO of Cqure and Cqure Academy, also an MVP and regional director. And at the end, a cybersecurity architect specialist expert, and someone who basically enjoys what I do.

VAMOSI: Paula is a Microsoft MVP.  Microsoft describes it as “a global program of recognized technology experts and community leaders who actively support technical communities through unique, innovative, and consistent knowledge sharing.” In other words, Microsoft has confirmed Paula’s expert status.  And Paula’s company, that’s the letter C with qure, Cqure specializes in pen testing., threat hunting, and training.

JANUSZKIEWICZ: Cqure is a company that I established almost 15 years ago, and I started only by myself. And right now we have a team that's almost 50 and we deal with cybersecurity services. So basically, we deliver custom penetration tests. We do forensics incident response, but also in general cybersecurity consulting. Being on the good side and also on the bad side. And secure Academy. It is an educational part of secure where we have over 40 custom trainings, online and offline. And also the easier already advanced ones, the ones that are happening regularly or the ones that are only happening once per year. So anything that we deliver there is of course devoted to cybersecurity.

VAMOSI: SecTor, Canada’s largest security conference held in Toronto each year, Paula gave the talk last October was entitled “Adventures in the Underland: Uncommon Hacker’s Persistency Methods and Countermeasures”

JANUSZKIEWICZ: Yes, absolutely. So, in general, the idea of the target sector was to show what are the ways how malware can simply be persistent or malicious actors can be persistent in the infrastructure and at the end, of course, there are many conversations about points of entry. So that could happen for example, through phishing through fingers, main misconfigured things very well in the rubble. But also the very important parts to pay attention to is how hackers actually stay in the infrastructure and how is it possible that for so many days, because depending on which starts we look into, but usually it's around 200 days, IBM says so FBI says so and so on, hackers actually remain on notice. So what's wrong with our methodology for the discovery and that is what was like this major flavor for the session.

VAMOSI: The bad actors can enter through phishing attacks, but the question is where can they hide on your system. How do they achieve persistence? OFten network systems are misconfigured and that often leads to breaches. At least this is the common complaint I hear from pen testers.

JANUSZKIEWICZ: As well, configuration is an important part and one thing is of course, just the general securing of infrastructure. So what do we do to make sure that we are resistant against the different threats? So things like Application Whitelisting starting with the simple things, ending up with privileged access management, making sure that we are monitoring anything that executes and then not only practices but also network connections, the pipes and Wi Fi queries and all these things, for example, that, for example system has on the list, and that's one thing but the second thing of course, is in general, the capability to spot that event, connect the dots, and being able to identify that all these activities that we see actually, at the end, make the attack.

VAMOSI: So obtaining user credentials or finding a flaw in the authentication, that gets you inside.  Of course once the bad actor gets inside your network, that’s when the persistence begins. 

JANUSZKIEWICZ: But once you are there, then you need to figure out the way to hide yourself. So that could be for example, through our hiding services. So creating a service and setting up permissions to the service in a way that is not visible by listing by the vast majority of the tools but again, depends what tool we use. On the other hand, it could be that that persistence might be through getting access to some kind of data. It also might be, for example, by hiding yourself in terms of permissions. So you can for example, use it or that G permissions, generic permissions, or you can also use the advanced permissions which is not really hiding, but when to first look when someone looks into properties of the file or folder, then you're going to see that for example, that group does not really have a read access, but actually it's read data which is a granular advanced so called permission, that only gives you actually a possibility to open a file, and so on. So it's all about this little details that are making the persistence or hiding, eventually possible.

VAMOSI: Except there are a lot of little things like this.

JANUSZKIEWICZ: Yes. And that's what makes the job interesting.

[MUSIC]

VAMOSI: So, given that we know the attack vectors, as a pen tester, I wonder if Paula and her team have any favorite GoTos, spots that they look at first when analyzing a system for the very frist time. 

JANUSZKIEWICZ: Oh, you know, I think it all depends on the tool that we use. But in general, when you think about this, like one thing to definitely start with, that's going to be just to try out runs and ultra runs. It's a Sysinternals tool. It's maybe even quite an obvious choice, but it's this first thing to check for. Atheros immediately is going to give you the different nuances and different unusual situations on the spot. Of course, there's a way you are still able to hide yourself. But that will at least allow you to find the low hanging fruits,

VAMOSI: Sysinternals is a set of free utilities. Originally created by Bryce Cogswell and Mark Russinovich, it was sold to Microsoft rebranded Windows Sysinternals.  The sysinternals autoruns utility, provides a comprehensive view of auto-starting locations of any startup monitor, shows you what programs are configured to run during system bootup or login, and when you start various built-in Windows applications like Explorer and media players. It’s comprehensive. That’s how it’s supposed to work. But there are ways that a bad actor can use these tools created for good to find running services that they can then leverage for bad.

12:06 JANUSZKIEWICZ: Absolutely. There are so many places that are allowing us to do it. Mainly, for example, anything really that's out runs, therefore, we've got an ultra runs tool from Sysinternals that allows you to discover anything that's just out running. So surviving the reboot. So it's going to be a service. It's going to be for example, part of the parts of the Winlogon it's going to be scheduled tasks that might be running just after you reboot. Or it might be for example, some different plugins that you've got for Outlook. So in general for office or for the browser, and it might also be the parts of the registry that will load things automatically. So there's so many places that actually are out running. But luckily for us, that's good news. Absolutely. The vast majority of these places, it's no one and it's discovered by the tools. So it's just a matter of whether we are allowed to run the code that we don't know. And we've got, for example, let's say a service that could be a malicious service, that's going to be of course Ultra running after the reboot. But if we will not allowed to run that service or an executable that's not digitally signed, are we going to have a well established monitoring system? This will just not happen, but that's why I was saying that outruns is just like the first handy tool that we should reach in order to check basically what's out there out running,

VAMOSI: A lot of the attacks today are around gaining user credentials. Why? Well, why hack your way through a complicated backdoor method when you can waltz right in the front door as a fully credentialed user and then escalate individual privileges from the inside. 

JANUSZKIEWICZ: Of course the value of access might be due to miss configuration. It might be through the connection of the different situations. Yeah. So the user might be, let's say just that user, but at the same time user might have right access to the folder where the service is running, and the server is gonna be running as a local system. So a hacker is able to do misconfiguration, able to escalate to a local system, and then maybe extract hashes and maybe someone is not using labs or maybe they are so they are randomizing administrators passwords or not. So it's I look when I look at these kinds of possibilities, also for persistence, I always imagine like a decision tree. So there's a like, you can do this or that. And when you do this, you can do that and that. So it's all about skills at the end and knowledge of a hacker or someone that's trying to reveal hacker steps.

VAMOSI: In her talk, Paula cites some of the crazy examples of potential abuse she’s seen in her years of pen testing. For an example there’s the case where granting permissions to a file is actually pretty complicated. In other words, there are instances where granting access and then seeking to deny that access later results in the user actually retaining their access. Where deny doesn’t mean deny.

JANUSZKIEWICZ: Yeah, so that's actually a very nice concept that I really liked. Because when you've got to, in simple words, when you've got an item, so an object, so that could be file or folder, let's say and when you go to the properties, you see a simple window that's going to show you that administrators users this group, that group has that kind of an access. And the question is that when for example you have on one item, even coming from inheritance, two types of permissions which are allowed to deny, the question would be what is the effective access that we can guess from that window that users are gonna have? And most of the people answer that it's going to be denied because of the previously gained knowledge that deny takes precedence. But the thing is, and that's what I'm really like we're we like to do is to break this level. I was gonna call it a myth, because that's actually not true. So in windows allow and deny they are on the same level. It's just a matter of the order. And that is very nicely seen when you display for example, a security descriptor so the SD and when you go, for example, you enter PowerShell, and you do get ACL to the object, pipe format list, you're gonna see in the bottom of the output, the SDL for the security descriptor definition language. And over there, it's just a piece of text. It's a flat, one liner. That describes you. What are the permissions for the folder, or file and then you were able to evaluate it from left to right and whatever it's on the left side, overrides whatever it's on the right side. And if the allows gonna be before deny, then allows gonna simply override deny so if there's like a direct battle against denial, it's not the denial that actually wins. It's simply depending on the order.

VAMOSI: Is there a recognized mitigation for that?

JANUSZKIEWICZ: Well, there are good practices that definitely we should follow. For example, one of them should be to avoid denial, as simple as this so that the pure it becomes, the better it is to predict what's going to happen next. And of course, it's not that ideal in the enterprise. Because if you gotta, let's say, file server, and that file server has its own history, and you are taking these files and you are moving them to the other structure, then some of the things might be inherited. And then at the end, there's quite a mix that happens. After that kind of operation depends, of course, how we're going to do it. So one thing is going to be avoiding it. And another one is going to be to read or focus when you assign permissions on role based access. So it's better to not create a security group, let's say HR and say everybody from HR can do it. But the other giome this person, that person cannot, but rather we should do it in a way that we create a group which says, all the people that can access that item, and then we add whoever we think is supposed to be or it's an HR, it's all the people that supposed to have access that item. And then whoever is not part of the group just simply does not have access to the item. And, of course, it's very hard to implement and it's hard to manage, but that kind of a role based access is the one that can nicely mitigate issues that we are talking about.

VAMOSI: Another example from her talk is one involving an unquoted service path. The danger here is having spaces in the command line, the order of operations, and how those spaces get filled and executed by default.

JANUSZKIEWICZ: Basically, service paths should be quoted and it should be quoted in quotes, because whenever we are loading the service executable, the SCM soda loader for the services it searches first of all, for that first part. I'm told that there's a space sign. So let me put it put it maybe from the other side, if there is a path that is like C, Program space files, and then you've got service, space, axis and then you've got at the end services dot exe, or file dot exe, or just to make it more pure than that file dot exe. It's also obviously the final executable that should be loaded. But unfortunately, when that part of that we just mentioned it's not in quotes, what that service loader is going to do, it's going to first reach for the C program. And normally we know it's program files, but it's going to actually go to the program and it's going to add exe to it. And it's going to try to load and search for a C program that the XE even though it sounds quite silly, but that's unfortunately how it is. And if it doesn't find a program for the exam, then it's going to go to a program file, and then it's going to go to the service. And we said service access. So in this case, it's going to search for service dot exe. And if it will not find it, then it's gonna go to that my file or file dot exe. So if that service path would be in quotes, then it will go directly to that file dot exe. Instead of searching for C program that exe that's why quite often malware is actually called program that exe. And we might be wondering where it comes from. It actually comes from that case with the service.

VAMOSI: Again, are there legitimate reasons for someone to do that? I mean, it seems like Microsoft or somebody might have addressed this well known flaw at some point in time. Perhaps there's coding reasons why they needed it to be otherwise.

JANUSZKIEWICZ: Absolutely. So it's really I don't know the reason for that one, but it's a known thing, and should that be addressed? Definitely. On the other hand, it's a kind of like, you know, this little thing that it's easy to fix. So, I believe that there are some higher priorities than that. But definitely, like if we are thinking about making the system perfect, then that should be one of the things to be addressed.

VAMOSI: Here again is something that should be easy to fix. So, given that’s not high up on Microsoft’s to do list, it's then a matter of then teaching people that they need to put this in quotes and that would mitigate some of the problems that we just discussed.

JANUSZKIEWICZ: Absolutely. So you know that the thing is that this is such a tiny thing that sometimes by managing an enterprise environment that might just get lost in the whole space of more serious tasks that we have to do. And we're gonna offer an example, our academy when we teach, usually during the first day, these kinds of principles, what does it mean to have a secure operating system? Quite often we see that someone has like 20 years of experience working on it, or even sometimes in security, but they don't know that. And, of course, we are wondering why this is the case. And that's just because it's not in the books

VAMOSI: True. A lot of infosec’s knowledge is either tribal -- passed on from one person to another - or can be found in books. A lot of us don’t have Computer Science degrees, so if you’re like me, we’ve learned what we know in a book or from direct experience. To Paula’s point, there’s a lot of direct experience yet to be documented for the future

JANUSZKIEWICZ:. So we can find it, of course, in some presentations in our, you know, podcasts like this and so on. But it's not really like an official knowledge that's served out there that says, usually have a strong password. It should be served in a similar way. It's just that it's a very tiny detail that probably it's hard to put through every single presentation, every single article and so on.

[MUSIC]

VAMOSI: Just as a tool such as those provided in Windows Sysinternals can be used both ways, information itself can be used both ways. Such as hearing how an unquoted service path a bad actor might hear this podcast and be thinking how they can use it if they didn't already know this already. Then again, these aren’t secrets. You can Google them as well as I can.

JANUSZKIEWICZ: Absolutely. So all these things can be always used in both ways. A pen tester is technically a person that does similar things to the hacker. Of course, we have different goals sometimes in different ways. But at the end, the vast majority of the steps, depending on the situation it's better similar. And the same story with our tools. For example, some of our tools are actually recognized by various antivirus services, not because they contain a virus, but because it's considered to be a hacking tool, which we definitely use for our pandas. But could that be used in a wrong way in wrong hands? Absolutely.

VAMOSI: In addition to tools from Microsoft and others, pen testers, well, they  have their own set of tools that they use on a job. Paula’s company, Cqure has their own. They have their own versions of popular tools such as Mimikatz.

JANUSZKIEWICZ: Yeah, so Mimikatz, of course, it's originally written by Benjamin Delpy and also in cooperation with Benjamin we created our own version of Mimikatz, which has some extensions. So that's one thing but in the end, we have actually over 200 tools that we have in house, that everything we share are in public. Some of the things we do because we think it's very useful for the community to use it. But some of the things it's a script, it's some very particular tools in a particular situation and so on. So it's historically grown to know how to use it during the past.

VAMOSI: Cqure has also offered some of their own proprietary tools to the public for other pen testers to use. 

JANUSZKIEWICZ: Actually, I think one of our favorites, it's going to be a CQ Secrets Dumper. It's historically also one of the oldest tools and it's written by our developer, Augusto Jetski. And basically that tool can do a bunch of things. The one thing is that you can just simply extract the password from the service accounts, which today does not sound very outstanding. But let's say 10 years ago, that was quite useful and relatively rare, but also the tool is very useful because it has multiple functions. For example, it can display you a boot key, which you get from the system hide from the registry, and the root key you can use to extract the passwords from the services offline, as well as hashes as well as different values that the operating system uses for as a secret. And that tool we find extremely useful also for different cryptographic information, why we need to technically decrypt something like for example, the cook the database, or get the hashes or get the secret for the group managed service account and these kinds of things.

VAMOSI: Another example from the talk is using a way to get the certificate off of a login and then just simply adding the certificate to a browser page to the cookie.

JANUSZKIEWICZ: I was planning during the presentation with two certificate or certificate related demos. One of them was actually with the use of a publicly available tool which is called Evil Jinx. And it is very, actually easy to deliver because evil Jinx allows us to replicate the website of the login for the office 365. And we presented somehow through the phishing activity to the user. So you send a phishing email, it has to go through this. And then the user clicks, for example, to the website. For example, when we are not available on teams, we get an email saying, Hey, someone wanted to contact you on teams. And we see that part of the message. So someone clicks on that. And then we are taking the person to the fake website to the fake website that looks like an office 365 login. After that, we collect all the information also depending on the situation, we collect the information within the multi factor authentication, and we then later collect the user's cookie. And then we redirect the user to the original website. But now having a user cookie where we are able to do so we can submit it to the browser and this is how we are able to log in to that website. So this is an attack that uses multiple factors of order. One thing is that of course, the cookie that we have a possibility to collect. And the second part is that the user of course has to ask to be interacted with us. So this is an attack for multi-factor authentication.

VAMOSI: Another example, is extracting that certificate from the domain controller.

JANUSZKIEWICZ: So it could also be one of my favorite subjects out there. I was also presenting a deck keynote that is related to the work down data protection API. So that was that situation when I was extracting that certificate from the domain controller, allowing us to permanently wipe the domain from the earth to decrypt various cryptographic information that's stored or created by users. And, and one of the things around that is that this is the fact and we have to deal with this. And through that, we have a complete forever persistence in infrastructure to get access to the cryptographic blobs, but it's all a matter of who can execute what kind of software and with what kind of privileges. Therefore monitoring was executed in our organization, and seriously controlling that privilege access and who is allowed to log on where and for what, it's actually something that we need to definitely step towards too.

VAMOSI: Wait. You can gain persistence to a domain?

JANUSZKIEWICZ: Absolutely, in many different ways, of course, but one of the ways I was describing was through extracting the certificate, which we call actually a backup private key. And that's something that our team, as, as far as we know, the first in the world did the full reverse engineering of the cryptographic API. So the data protection API in Windows and by extracting that certificate from the domain, you get a persistence that you really cannot cancel. And the whole cryptographic data of all of the users is the only way to cancel a persistence. If you really want to cancel it is just to set up a new domain from scratch. 

VAMOSI: Wow. That’s pretty extreme. Spin up a new domain periodically just so you can be sure no one retains certificates.

JANUSZKIEWICZ: Okay. pretty drastic, I know. Yeah. But considering that we can already say that, let's say every domain has been already compromised. And I really like to say that because if there is one person that had some kind of privileged access to the domain, in times when monitoring wasn't that popular, so let's say five, six years ago, or even two three years ago, yeah. Then basically this person could have already extracted the certificate. So simply we cannot trust the on prem domain that has been once accessed by a privileged, uncontrolled user.

VAMOSI: That’s pretty basic. If someone has superadmin access, once they leave, you really should scrub down the network. Not that they would act maliciously after they leave, but rather, you don’t want someone taking over their dormant accounts.

JANUSZKIEWICZ: The rest is just the monitoring and prevention to do that. To do the math.

[MUSIC]

VAMOSI: In Episode 35, Paula shared a great story about a ransomware attack where the decryptor didn’t work. She and her team had to do technical support, figure out why the decryptor didn’t work, in order to release files that otherwise would have remained encrypted, even after the victim paid the ransom.  Is there another story?

JANUSZKIEWICZ: Especially, I would say that almost every week, there is some kind of a story that happens in our team. Maybe not with that kind of a scale like I was presenting at the keynote, because actually that's truly one of my favorite projects right now. But there were many, many cases like this, for example, we had a case with our customer that it's not that detailed. Because it's a relatively fresh case. But we had a case or the customer that actually was buying software and using software from the software vendor. And I would say kind of classic but also quite exciting. And then it appeared that that software was just broken in the process. So again, it's a vendor problem. And we analyzed that situation because we've been acquired deep inside, and then it appears that there's actually too many people that on the vendor side that had access to that certificate. So it was not really maybe that much reason we were hackers attacking a vendor or weather vendor being hooked into their customers network. So a very strange case. Ongoing but also showing that we need to somehow control obviously the software that we are using from the vendors, not necessarily completely trust them to be able to verify their protest. We all know that it's not always possible. But if we again have a well established monitoring on our side, and if we monitor privilege access, then even though that kind of a threat is going to be brought through the vendor, we can still minimize the impact that that kind of software could have. It's not an easy case because every organization has its own flavors, like this one for example. But but definitely a very interesting story to deal with and not solved yet.

VAMOSI: Persistence doesn’t just apply to the bad actors. Persistence also applies to the good actors as well. And given that you've got these solutions that are tools that are out there, you need a human, you need a pen tester to come in and is there any sort of guidance on that? Like how often should you be running a pen test and even when you run a pen test, I imagine you scope it. You can't possibly scope everything.

JANUSZKIEWICZ: So at the end, at the end, it's designed to last a certain amount of days, and ideally for the organization and for the sake of the project is going to be to run the Panthers endlessly. So that you are regularly tested upon any change upon any certain time periods. So if for example, six months passes, just as a CTRL, pencil pen test, you do the check. But of course, I'm saying what would be nice to make sure that we are continuously spotting all the different misconfiguration holes that potentially could appear because someone did something suddenly in the infrastructure. But of course this is not happening. And that's why that kind of activity even got renamed a little bit and we call it more of a threat hunting and organizations every certain time it's a relatively new type of a service. They actually do the threat hunting. One thing is to check and avoid independence. What kind of holes are there? What hackers can get in and so on. But the second part is to focus more on the review of the systems that are already there full privilege to verify whether there is possibly some kind of a leftover after the attack or maybe there's an attack going on. So these two activities are of a similar kind about they have different goals as they're more interested in one over the other right now.

VAMOSI: So, should everyone be running threat hunting exercises or a pen test.

JANUSZKIEWICZ: It's still kind of like pentesting because organizations are more focusing on what can happen from the outside. And if it's about inside, they somehow manage it. It's not perfect, of course. And I think that threat hunting in general should be a part of the mandatory activity for every organization, for example, every year, but of course, it's all a matter of budget. So if you've got a choice, what would you choose? Well, just to clarify, what are you going to be hacked from the outside or maybe even from the inside to support all these kind of low hanging fruit versus just to review all of the servers searching for potential signs of activity, which might not even have happened yet. So long story short, if you've got a choice it's more efficient for the money that you need to spend within a year.

VAMOSI: Additionally Paula recommends that you monitor your services and registry and all of that to make sure that the stuff isn't abused. 

JANUSZKIEWICZ: There are many good practices, obviously out there, but the major generic focus is on monitoring identity in the organization, monitoring what executes preventing the unknown stuff from executing and also connecting the dots or correlating different events, and then classifying them as attacks as threats and then having an appropriate reaction to it. So that's kind of a big role of a CRM system. That's kind of a big role of any kind of whitelisting solutions, smart ones. And that's something that doesn't really solve the problem, but it minimizes the risk to the point that we just stop looking interesting for a hacker and we're just becoming a waste of time. And then just hacker is going to choose someone else, right. So that's one of the major goals I would say,

VAMOSI: I’d really like to thank Paula for her second appearance on the show.  If you missed her talk at SecTor in October 2022, Paula will be speaking in April 2023 at RSAC 2023 in San Francisco. I encourage you to see her live. Also, as I said, you should check out Episode 35 where Paula talks about digital forensics. Want more? Checkout the Cqure Academy to learn more about information security. We’re here to help each other learn about security.