Next-Gen AppSec Series: Secrets Done Right: Code, Container Images, Env Vars, and Process Memory

I’m the director of DevSecOps Solution Strategy for CyberArk from Seattle, and basically looking forward to talk to you all today about App Security.

Awesome. And Evan, I have to ask you, what’s your favorite coffee shop drink is?

Oh, I’m a huge fan of Whiskey Barrel Aged Coffee. Two, actually, either The Morning Dram or Starbucks actually has one too.

Evan, you’re up in Seattle, right?

Yes.

Yeah, good. So Evan gets to choose the coffee today.

Awesome. Mike, introduce yourself please.

Hi everyone. Mike Larkin. I’m CTO and co-founder of Deepfactor, along with Kiran. I’ve been in the industry a long time, done a lot of security, virtualization, lots of low level programming. Be good to talk to everyone today about secrets.

And Mike, your favorite coffee?

Oh, I’m not a coffee connoisseur, but just in case, I actually have two different coffees here. I know you can’t see it, but two different coffees for the talk today, so I’m good to go. I think this one is actually a maple bourbon, so it’s kind of like what Evan likes. Maple bourbon flavor today.

Awesome.

I want to try that.

So guys, the topic for today’s Next-Gen AppSec Series episode is secrets. Everybody knows about some form of secrets or the other that is hiding in various parts of the application. Could be hiding in your code, could be hiding in your container images or could be hiding at runtime in your environment variables that are being exchanged between various processes of your application. In this episode we’re going to talk about why is it a big deal to think about secrets and plan for it as part of your AppSec initiative. How can bad secrets being in clear text spread over these resources that you have hurt you, especially if they’re mismanaged? What is the right way to manage them and how do you detect and make sure that you set up these practices so that your organization gets in the habit of healthy healthily managing secrets? All right, so we’re going to get started with Evan. Evan, tell us what are secrets?

So secrets are effectively any information that can be used to define a application, and I’ll take a little bit of time with this. One kind of key point to it is that from a human perspective, we can hide ourselves because we have our brains as secure storage. Applications aren’t so lucky. Computers by design are meant to have shared store. So the way that the application identifies itself is generally with some form of token, whether it’s a certificate, whether it’s a password, an API key, anything along those lines. That’s really what a secret is, and that’s kind of what organizations really need to protect because that secret represents the identity of the application.

Got it. What does it mean by mismanaging a secret and how can it hurt you?

Well, if I have it just sitting out there, so again, because it represents the identity of the application, if someone can take it and there’s nothing attached to that secret to bind it per se to the application, which there really isn’t too much out there to be able to do that, then what winds up happening is that secret can be used by malicious actors to compromise the identity of the application, mimic the application, and then basically use or gain access to any of the data that the application has. Now at face value, that can seem either really bad or really simple, but it’s actually really bad. And the reason for such is if you think about any application, it has full read-write access to the schema of whatever backend system it’s accessing just because of its business need, it’s going to need to constantly read and write data to respond information to customers. So that basically gives you carte blanche to all of the application data.

Right, right. That’s awesome. Well, it’s not awesome.

It’s scary is what it is. It’s horrifying actually.

Exactly. Yeah. Mike, tell us about PII versus secrets. Is it the same thing? Is it a different thing?

Yeah, that’s a good question, Kiran. And that was actually something that the three of us were talking about last time we met. I’ll get my take and I kind of want to get Evan’s take on this as well. So when we were talking about secrets and what we’d like to talk about today for everybody, I immediately jumped to PII. I’m like, oh, PII are secrets, right? I think there’s a distinction between PII, a credit card number or a date of birth or an address or a phone number or something. I think there’s a difference between that and what Evan’s talking about, which is like application, authorization, tokens and secrets.

So although PII can be considered secret information, Evan, where do you draw the line between what the difference between PII and secrets? I mean, personally, I think that you have to protect all of these things, but the way that you protect these things are different. For example, it’s probably pretty unlikely that you would see PII checked into a GitHub repo or something, but you might see secrets checked into a GitHub repo, and I think that’s something we’ll probably want to touch on later in the discussion. But PII appears in different places and I think there’s a different technique to protecting that, than there is to enter application secrets. What do you think? Am I on target there or am I not?

I would say that the secrets give you access to the PII. So playing off what I said earlier, if it’s some kind of healthcare application, it’s going to have the patient data in there, but that application is using a secret to be able to authenticate to that database to be able to do it. I’ll muddy the water a little bit in a way by saying that the secrets are actually the application PII. So if we were to look at personally identifiable information for an application, the secret would be a example of that PII. Again, kind of tying back to what I said, if you steal the secret for an application to do something, it’s basically application identity theft.

Right? Makes sense. You mentioned the magic word healthcare there. I don’t know if anybody else that’s in the conferences has had this happen to them, but a neighbor of mine got that infamous note in the mail that said they were a victim of the MOVEit breach a couple of weeks ago. So yeah, it’s out there.

I got two of those in the span of a week myself, and I was just sitting there going, “Really?” It wouldn’t be nice if they actually told you the details of everything that was compromised. They just say, “Well, some of your information leaked.”

“What information?”

Your hat size.

Yeah, exactly.

Awesome. All right, so let’s get to, in the spirit of the coffee shop style conversation, talk about some of the craziest places you’ve ever seen secrets being stored? Evan, let’s start with you.

Okay, so I’m going to give two examples, human example and application example. From a human example, this is kind of showing my age, but my favorite one was I was standing in a data center and I literally saw a local admin password sitting on a whiteboard, like, “Hey, if you wanted to access any server in the data center, here you go, it’s just there on the whiteboard.” And I was like, “Well, that’s convenient.” And then from an application perspective, it wasn’t removed from the comments. So it wasn’t in the code, it was just sitting there, username and password in case you need it to cut and paste here.

Nice.

And honestly, that can be fairly common just because hey, you want to just have something in there when you’re troubleshooting the application and maybe you don’t want to go to an AKV or an AWS SM initially, so you just have it there and you forget to take it out. So that I would say is more accidental than it would be malicious. How about you, Mike?

Yeah, I was giving a lecture a few years ago at San Jose State to a bunch of students, a bunch of master students, and it was funny in my security class, and what we were talking about was not really secret management, but just be careful what you check into your repo and whatnot, comments and things. And there was a group of two or three students at the back of the room that were just kind of giggling and laughing, and I’m like, “What’s going on there?” And they said, “We’ve been there. We’ve done that.” And I said, “Well, what happened?” And I said, “If you leak your secrets.” As you mentioned Evan, it’s the application’s personal identity. “What can you do with that as an attacker if you got ahold of that?” And they’re like, “Well, we got off lucky, because the only thing that the attacker used our secrets for was to create 50,000 AWS VMs to do Bitcoin mining.”

And I said, “Yeah, you did get off lucky. That’s just a cash write off. Imagine if they had stolen your data or defaced your application or whatever.” But yeah, so not a specific example of where they were kept, but a specific example of somebody that got owned by that. As for the craziest place I’ve ever seen secrets, and it’s funny, it’s not that crazy because actually it’s pretty common unfortunately, is when you have application tokens that are being passed around between components in an application or when you’re launching an application, you’ll typically find those things passed into containers and passed into applications through environment variables or files that are in the repo or whatnot. And you would think that that’s crazy, but it’s not actually that uncommon. So I think you have to watch out for places that tokens are kept in memory as well as ephemerally and other places.

So there’s-

Go ahead, Kiran. Sorry.

No, go ahead, go ahead.

Yeah, I was going to say, it’s funny how many, I have some friends who are dads, and preference for secrets is always environment variables. Just because, hey, I don’t have to worry about what platform I’m dealing with. I prefer go, so it’s just os.getm, name of the environment variable, and I’m done. You can’t get any easier than that. But the problem is is that you can run print ENV on just about any Unix host, and there’s all your environment variables laid out for you, unless you’re running in a separate shell.

This is a good point you bring up, even if you’re running in a separate shell, not many people realize how easy it is for a process running as a user to get access to the environment variable blocks of any other processes running as that user. I mean, actually it’s pretty trivial, and it’s interesting that we’re going down this path talking about memory and such, because it’s my opinion that there’s not a lot of developers that are intentionally saying, “Well, I’m going to use this because it’s insecure.” It’s just that people don’t know, let’s call it lost tribal knowledge from the last 40 years or something that unless you’re really, really careful about putting things in environment variables, you may think that they’re completely unique to a process and that actually is not the case.

So environment variables, process memory stuff, I know we’ll talk about this a little bit later in the conversation, but these are areas you definitely have to watch out for. And for junior developers or folks that are just coming into the workforce and understanding how to build robust enterprise applications, you really have to educate folks and you really have to take them by the hand and say, “You don’t do this, and here’s why.” And all it takes is one of those episodes like I was describing earlier, where somebody got your token or got your secrets and then all of a sudden what can happen? A lot of bad stuff.

So let’s talk about how can an attacker leverage mismanaged secrets? Obviously, we all know secret leaking is bad, but let’s double click a little bit and walk through the attacker. If the attacker is able to get his answer, her hands on your secret, then how can they cause damage?

Oh, in slow data exfiltration.

Yep.

That’s the easiest way to do it.

Yeah, it kind of depends, Kiran, I think, on what the attacker wants. So if you’re an attacker, put your attacker hat on for a second. What is my goal? Ultimately, most likely, unless I’m just kind of goofing around, my goal is probably going to be some kind of financial gain, and how do I get that? How do I accomplish that with a secret that I stole from you? Well, as I mentioned earlier, if it’s an AWS token or a Google Cloud token, for example, I can use it to spin up a bunch of compute resources and mine Bitcoin hypothetically. That’s something you could do. The other thing I could do is like what Evan said is I can treat your data as value and exfiltrate it in a way that allows me to get the data so that I can hold it for ransom, not necessarily encrypting the data or ransomware, although you could do that too. It’s more along the lines of I monetizing my access to your data as financial incentive for me to do nefarious things. What else can you do? I mean, you can do application defacement, there’s corporate espionage. There’s all sorts of ways that an attacker could gain money from owning access to your secrets. Evan, do you have any examples of secret stealing turning into monetary gain for an attacker?

I mean, yeah, usually those are not things you can talk about too much, but I would-

Hypothetically.

I would tend to say you pretty much covered it in terms of what goes on there, but I think to the extent of it, when you bring that up, most people think of financial institutions because it just automatically translates to money. But one of the things that I tell most of the customers that I talk to is that it doesn’t matter what you do. I distinctly remember one example where it could be as simple as retail or any other organization where, “Hey, we can’t be impacted by this.” Well, actually, yes you can, because every company is a software company, and the code that you write is intellectual property that belongs to your organization, and some of that may be proprietary and revenue generating, and just that IP being lifted could be sold for financial gain by the hackers, or to your point, corporate espionage to be able to learn what you are doing as opposed to your competitors.

It could be anybody. And this is a thing that people need to understand. It’s not just a bank, it’s retail, it’s pharma, it’s especially pharma if you think about it, it’s anything that’s out there, if their information is lifted, it could be used maliciously, and the odds are it’s stored somewhere that you would need a secret to be able to get to. It’s not something where it’s just, “Hey, it’s sitting on the disk of the server.” That’s not there. It’s in some data store that needs to be securely accessed.

As organizations, obviously generally speaking, nobody’s intending for these secrets to be out in the wild and open for everyone to use, but how do we end up getting into these situations, both from a code perspective as well as from a runtime perspective?

So honestly, and I said this earlier, I think a lot of it is more human error than maliciousness. You’ll see people, like I said earlier, I’m writing code, I have some place to put them, but I don’t want to deal with that yet. So I just hard code the credential and then I comment it out, and then I leave it there, because I forget to delete it, and then I either push it to a GitHub repo or it worst case scenario, it just gets compiled into a build and it’s just kind of sitting there, but where it is actually code and it’s just a variable. So other things are they sit in configuration files, Kubernetes manifests. So we talked about environment variables. If you think about it, if I’m deploying a container, I have an ENV statement in my manifest, the ENV statement is going to define all of my environment variables. There it is out in the open, and the odds are it’s not encrypted. So not only is it existing as in variable in the container that we just discussed, it’s also existing in the manifest that’s being loaded.

Yeah, I think-

Go ahead. Sorry.

I was going to say, I think it’s also important to say that as you mentioned, it’s usually not malicious and it’s usually, let’s call it accidental. I’ve actually seen cases where there’s let’s call it a secret in air quotes that gets put into a piece of code or something in public or an environment variable like you’re saying, where legitimately at the beginning of that secrets’ life, it did not protect anything that was vital. But what happened is nobody else knew that, hey, the secret that you have in your code that allows you access to this resource, even though now it’s test data or whatever. Now, somebody didn’t know that and they started putting production data in the same place. So now the secret that I didn’t know about suddenly has access to data that it shouldn’t. So it’s just generally speaking, yeah, just don’t do it.

That’s a common one actually, something that we even saw. Again, speaking about harmless mistakes is that I’ve seen where the production and the QA password were the same, and someone ran the cleanup script that they used for QA against production and replaced all the prod data with temp data.

All right. So all of these things lead to secrets being in these unexpected or unintended places. Now let’s talk about what are some best practices around detecting these secrets? And let’s start with code and containers first, Evan, and then I’ll switch to runtime, Mike.

Yeah, so when we look at that, obviously the first place to start is where the code’s being written. You want to make sure that you’re using some form of Git scanner, like TruffleHog or GitGuardian or GitLeaks or any of those products that’s going to find them, again, human error. I accidentally left it in the code, let me at least get flagged that I made a mistake and then I can remove it before it becomes an actual problem. The second thing is you move up the train if you want to make sure and look at it in your build processes to see you kind of have some code scanning just to make sure there’s nothing kind of floating around in there that shouldn’t be, it’s defense in depth.

You want to be able to layer yourself that you’re catching it at every single stage to be able to make sure that nothing gets through because something, hey, for whatever reason, how you’re looking for it, the secret may, because most of them you can do high entropy and things of that nature. Maybe they just did it in a way where it didn’t come up as a secret for one thing, and then you run a scanner down the line and then it picks it up. So by repeatedly moving through the processes and scanning it every layer, you’ll be able to figure it out. And then Mike, I’ll turn that over to you because the final line of defense is runtime.

Yeah, yeah, I think efinitely nodding in agreement with everything you said. I think one thing we haven’t really touched on yet, maybe this kind falls into the runtime camp is. As Kiran was asking, I have secrets, you have to have them in order for an application to do useful things. So what are some of the best practices around managing these? I think first and foremost is don’t have them stored in, let’s call chronic or long-lived places. Make everything as ephemeral as you can. So environment variables, we’ve been saying, don’t do that, don’t do that. But there may be a situation where you have to do that. I mean, there’s just with the way the application is built. So in that case, you can actually inside the process, read the secret blank out the secret so that if somebody externally with the same privilege were to look into the process’s memory and looking at the environment block, what are they going to see? They’ll see an empty string or no value or garbage or whatever. So when you use the secret, as soon as you’re done with it, wipe it out. So hopefully that means that you don’t need something that you need a token for through life of the process. If you do-

100%.

Yeah, if you do, then probably my advice would be to use something like a vault, and we didn’t really talk about vaults yet, but a vault is a piece of software that’s created specifically for managing secrets. And these are built by people who ostensibly know how to manage secrets. So we have HSMs and things for managing key and key material. You can get vaults even built into something like Kubernetes for managing secrets where you can get access to the secret. Then when you’re done, you release the secret, you blank it out and you’re done. So that at least adds a level of, it adds another barrier for an attacker because now they may have a toehold in your application may be able to do something, but if they can’t read the secret because they need to perform a transaction with the vault, it becomes a lot more difficult for them to create that kind of remote code execution. Not impossible, but more than code.

Even then, and I want to reiterate what you said, because it’s important there, even if you are using a vault.

Yeah, no, absolutely.

You have to make sure that you dispose of your secrets. Memory lingers, and you have to make sure that you appropriately dispose of it, you override it because of how sensitive it is when you have it in, I would say, I’ll use the analogy, when you log out of your payroll vendor or your bank, what does it always tell you to do? Close the browser window. It’s the same thing with secrets. Make sure that you dispose of the variable content as soon as you’re done using it. It’s highly sensitive.

And that goes for environment variables, but it also goes for just, let’s call them in the runtime terminology, stack-based local variables. You have variables that contain secret material on your stack and in your heap and other places. And just like it’s possible for a process with appropriate credentials to be able to read in another process’s environment block, you can also attach to the process using something like a P-trace or something like that, to be able to just read the process’s memory directly. That’s a little bit more of an advanced attack, but the data’s there, it’s in memory. And if I am user A and running and I’m attacking a process that’s also owned by user A then in theory the operating system’s going to let me do that. Now again, there’s some mechanisms you can use to put in place to prevent that. You can turn off those kinds of debugging extensions, the kernel and whatnot. But how many teams are doing that? I mean, how many teams are going to that? Going to those extremes, and you may not even have the ability to make those manipulations in certain managed environments. You may be stuck with whatever the cloud service provider decided that you needed and that may or may not match up with your organization’s security policy.

So for those of you that joined a late, we covered so far, what are secrets? How can mismanaged secrets come back and hurt you? What are some crazy places where secrets could be hiding that you may not have thought of? What are some obvious places where secrets could be hiding such as your code, your container images, and in memory in your environment, variables, et cetera? How do you get into this position? What can an attacker do to leverage these kinds of mismanaged secrets? And then we’re talking about detection of these secrets using scanning and code on the left, observing runtime on the right, et cetera. Now let’s move from detection to protection. So how do we make sure that we protect our enterprises from mismanaged secrets?

I think we kind of touched on that a little bit already in the past few minutes. Kind of just to reiterate, secrets in memory. I think I’ve been summing it up best. Always dispose your secrets when you’re done with them. I mean, if that limits the blast radius, it contains the amount of damage that can be done and the amount of that can be done amount, the amount of time your application is vulnerable. So if I need a secret to invoke a particular API in another service in my application, get the secret, invoke the API, destroy the secret. That solves of some of these problems. Evan, I don’t know if you want to chat about the secrets in code and secrets and whatnot, aside from don’t do it.h,

Well, I mean you kind of have to do it. Because the application has to identify itself. How do you do it securely and have to dispose? But the other thing that you need to do is how you grab those secrets and managing the application identities. I think one thing that is just an oversight a lot is that applications that we had a statement many years ago from a marketing, it was applications are people too. And what it meant was that applications have identities just like we do, and they access stuff just like we do. They’re not sentient and they don’t have secure storage like our brains, but they’re basically doing the same thing, and it’s up to us to be able to manage it. I’ll also say that applications are only as altruistic as their creators. So that also goes for how well the app is written.

And so because secrets are necessary, it’s making sure the application has an appropriately managed well-defined identity that does not exist just as a token that you use some form of vault. And then the underlying authentication or identity methodologies that exist in systems. Kubernetes provides you the ability to define a pod. You can use that definition of a pod to be able to authenticate a running workload and be able to say, “Okay, I know who you are because of the information that I have.” All the cloud providers have the various different IAM type ways of authenticating as well. Granted, it’s the compute, not the workload, but it’s still something better than just, Hey, here’s my API token just sitting there out in the open.

Yeah. Awesome. So summarizing, use a vault, scan your code regularly, probably part of the CI pipeline, disposable secrets. Scan your containers regularly. Again, part of the CI pipeline. When you use secrets in memory, make sure that you run your runtime scanners to detect environment variables that are hiding in your secrets. And if you use secrets at runtime, make sure that you have good practices from a code review perspective to make sure that the developer is disposing off a secret that is being used. Right? Anything else to add?

I’ll throw one more thing in there, just kind of simple from a security perspective. It’s something I heard several years ago and it’s become kind of a mantra that I live by. It’s almost impossible to completely prevent somebody from getting in. The goal from a security perspective is not to say, “Hey, look, I’m going to stop everybody.” That’s just not happening. The goal is realistically to make it as difficult as possible, because the more difficult you make it for a malicious actor, the more time you have to be able to detect them before they’re actually able to do something malicious. So by buying yourselves more time as security professionals, what you’re doing is that you’re giving the hacker more time to be caught and more time to prevent something going on. So doing all of these things will literally make their lives, excuse me, a lot harder in terms of being able to get access to customer data because you’re following these processes. And because of that, they have to be more bold and expose themselves more to be able to try to get access to it, so it becomes easier to prevent.

Yeah, it’s like beware of a dog or protected by ADT kind of signs that you throw in your house. Awesome. Thank you so much, guys. I think this was a pretty informative episode number three. Evan, I’d like to give you a couple of minutes to explain to the audience what CyberArk does in this area, and if you can intro cyber arc to the

Audience. Sure. So as you can see to my left here, we are the identity security company. And one of the things that we focus on is what was discussed earlier, which is application identity. It is the ability to effectively manage the identities of applications, understand what those applications have access to, and control how those applications get, what secrets they need to be able to do their jobs. And of course, most importantly, provide an audit around what they’re doing as well.

Got it. So members of the audience, if you need help managing your secrets or coming up with a strategy around it, tooling around it, talk to Evan. And I’d like to also introduce, for those of you that don’t know, is a next generation application security company that is powered by Runtime. Our flagship product is a runtime reachability product that essentially looks in the memory of your running applications, Kubernetes environments, et cetera, to identify which vulnerable components are actually being used by your application, loaded in memory, et cetera. So we’re able to essentially take your thousands of software composition analysis vulnerabilities found in your open source scans or container scans, and narrow them down significantly to what truly matters and represents the true risk to your organization.

Check it out. And of course, because we look inside the memory of running applications, we also look for secrets being used in your environment, variables and things like that. So if that’s of interest, check out. I wish you all a very happy holidays. Thank you so much for joining our episode number three. Stay tuned. I think Virginia is going to drop some helpful information on the screen for you guys to take away in case you are interested in some holiday reading or some helpful links here, so thank you. Thank you, Evan-

Thanks again.

… and thank you Mike for-

Thanks everyone.

… joining us in this episode.