In this article, we are going to talk about the security approach, guidelines and best practices that Lemma engineering teams follow.
These guidelines are not meant to be exhaustive, but rather a starting point, a common denominator of basic security needs to “lock the front door” for any software development project.
Efficient engineering teams don’t need to spend much time to align and operationalize these principles and can instead invest thinking power into more sophisticated practices or domain-specific security considerations for the given application complexity, scale and roadmap.
We have compiled the following categories that we’ll describe below:
- Industry Standards
- Infrastructure
- Architecture
- Application
Industry Standards
First and foremost, each engineer must be familiar with the official security best practices from the partners or vendors the team works with.
For example, any web app hosted on AWS must follow their official recommended best practices for security, identity and compliance.
Additionally, we have a baseline responsibility to stay up to date with industry standards for security, such as OWASP’s top 10. This means not just being familiar with the top level threats (such as “Cryptographic Failures”) but also their root causes (such as “hard-coded secrets” but also potentially more elusive ones like “Insufficient Entropy”.)
Because our own list focuses on the fundamentals, you will find overlap between these sources, but nonetheless, we recommend reviewing them carefully and individually, as the level of detail and specific recommendations for each one varies.
Security Philosophy: Shift-left
“Shift-left” for Security teams means moving focus on security to earlier stages of the development process. This is similar to operations teams moving infrastructure code into the project at earlier stages.
We design for security in the earliest stages, before any implementation takes place, following best practices for secure software development as well as incorporating peer review not just as a post-implementation gate, but also at the solution design stage.
As part of our “Shift-left” approach, we employ dynamic and static scanning tools (for example, Sonarqube) in our continuous integration and delivery process. These tools reveal potential vulnerabilities before code can be merged or deployed and allows teams to prioritize and resolve them early.
Infrastructure
Open ports
Access to infrastructure must be restricted based on the principle of least privilege, which sets out that, by default, all ports must be closed on firewalls or similar security appliances and only ports necessary for an application to work should be opened.
When certain ports need to be open in non-production environments, ensure they are closed in production environments.
Access control mechanisms should be documented for every project.
Most projects will typically require the following ports to be open:
- 80 (HTTP)
- 443 (HTTPS)
- 22 (SSH) where needed, not to all traffic.
The following configuration options should be in /etc/ssh/sshd_config (or equivalent):
Evaluate the following actions to further restrict SSH access beyond key authentication:
- Close port 22 when not in use. This may be practical for smaller projects or those where the infrastructure rarely uses SSH.
- Use Single Packet Authorization. At the time of writing this, fwknop is our tool of choice.
- Use Port Knocking. SPA is preferred, however, port knocking is acceptable.
Another widely prescribed practice is using an alternative port for SSH access. This is not discouraged, however, it is not necessarily a solution since it is quite trivial to scan remote hosts for open ports.
SSH Keys
Note that the configuration snippet listed above disables password authentication for SSH. This should be the case for all projects regardless of their size and complexity. SSH access, when necessary, should only be granted via public/private keys and whenever possible, the Ed25519 format should be utilized. Read more about SSH key formats.
Credentials and Permissions
Access to systems must be configured using Role Based Access Control where users are assigned roles and no single user is granted discrete or extraordinary permissions. Always refer to the principle of least privilege and consider the proper tradeoffs between security and productivity when implementing restrictions.
Other Considerations
- Always use AWS IAM roles and not root credentials. Consider managing IAM objects with Terraform. It has resources for writing IAM resources as code. And therefore the configurations may be checked into source control. As well, it has commands that allow you to audit the state of your IAM configuration to help ensure it remains as intended.
- Minimize the use of passwords to strictly necessary and default to public/private key authentication when necessary.
- When passwords are used, make sure Two Factor Authentication is configured as mandatory for all users.
- Passwords or credentials, even when encrypted, should never be stored in a code repository.
- Follow the best practices listed on Lemma’s Password and Security policies for the creation of safe passwords.
Database configuration
A database server should never be exposed to a public network interface. If this can't be avoided, the team should create an AllowList of approved source IP addresses and deny access to any IP not on the list.
Although most databases come with sensible security defaults, there are exceptions. The team must always research what the recommended security options are for all databases used in the project. It is crucial that this research also takes into account which version of the database is installed on the server, as that may change things drastically.
As an illustrative example, as of version 3.2, Redis provides a protected mode by default, but prior to that version, Redis defaults to open the 6379 port in all network interfaces.
Data Encryption at Rest
All stored data should be encrypted at rest. If this is not possible, the team must document why and explain the risks and tradeoffs taken to justify this limitation. It is imperative that our clients understand why we would forego a security best practice like this one and that we have clear and thorough documentation supporting that decision.
When working within AWS, we recommend using Server-Side Encryption for S3, databases and snapshots encryption for RDS, and EBS Encryption for EC2.
Data Encryption in Transit
Data that is transmitted over the wire (from system to system or from system to user) must be encrypted following industry standards. In most cases this means securing network connections using TLS 1.2+ and ensuring that all HTTP communication occurs via HTTPS (the only valid usage of HTTP is to automatically redirect a user from an HTTP to an HTTPS connection).
If appropriate in the context of the project, we recommend using the Let's Encrypt service as a free solution to simplify the process of adding valid SSL certificates.
Data must be encrypted in transit on every connection including those considered "secured" such as a Local Area Network or a VPC. In fact, when data needs to be transmitted over the internet (such as between cloud providers or availability zones of the same cloud provider) consider implementing a VPN connection between entry points and transmitting all data via the VPN.
VPN connections must be authenticated with client and server certificates and never with passwords.
Architecture
Secure architecture considerations are too varied and nuanced depending on the details of the architecture. We feel it would be too generic and abstract or, on the other hand, too prescriptive to provide specific guidelines for architecture. Instead, this section will describe the fundamental principles that our teams leverage for designing secure software architectures.
Isolation
“Each service must be an autonomous piece of the overall application puzzle. A microservice needs to be able to be deployed, maintained, modified, scaled and retired without affecting any of the other microservices around it.” - Gavin Kenny of IBM
Isolation prevents changes to or vulnerabilities in one service from impacting another service.
Defense in Depth
Defense in depth is a security best practice in which there isn’t just one security control layer (like a firewall) but rather a multi-layered system of controls for all components of an architecture.
For sensitive services (accounts, billing, etc) it’s imperative to ensure that each service is properly secured individually as well as the architecture as a whole.
API Gateways
Service Oriented Architectures must have a single point of entry that handles all external requests to block direct access to individual services within the architecture.
Rate Limiting
All services must have rate limiting mechanisms to prevent against forms of brute-force attacks and DDoS.
Rate limiting mechanisms can be as simple as limiting the number of requests allowed per minute, or they can be arbitrarily complex and robust, changing the rate limiting approach based on user type and behavior. The strategy to be used in each case must be carefully reviewed and planned by the engineering team with our client.
Granular Access Control
Each service must have the minimum required level of access for anything that it interacts with. For example, a microservice designed for logging may only need WRITE access to a database, while different services may only need READ access from the same database.
Threat Modeling
It is also recommended that teams document data flow diagrams for each service in the architecture and use that to define a threat modeling diagram for the architecture. The data flow diagrams should show all communication between actors or systems that a malicious actor could target.
Review OWASP’s threat modeling best practices for an extended overview of what this entails.
Application
OWASP Top 10
As stated above, web applications should protect against the OWASP Top 10 most critical security risks. Many of these concerns are also addressed directly in this document.
Passwords
DO NOT store passwords in plain text. There are several strategies for avoiding or mitigating this problem, but we recommend following Coda Hale's article on How To Safely Store A Password.
Passwords used by members of the team must be stored in a Password Manager, whereas passwords used by applications should be stored in a Secrets Manager such as AWS Secrets Manager or HashiCorp's Vault.
User passwords in a database must be encrypted with a one-way encryption mechanism like PBKDF2 and storing the result of ciphering the password entered by the user. Keep in mind that user-entered passwords should be converted into hashes as soon as viable to avoid or reduce the need to transfer a password over the wire.
Management of sensitive data
There are many pieces of data that shouldn't be stored in the database in plain text. In these cases a solution like the one described for passwords won't work because information needs to be decrypted before it can be used. This can include personally identifiable information (PII) or any non-public data. In these cases, the team should develop a security strategy for sensitive data and, beyond encrypting the data in transit and at rest, also encrypt the columns at the database level.
Credit Card Information
The team must not store credit card numbers. Even in encrypted form. There are laws and standards to follow when dealing with credit card information, so unless it is mission critical, the team should rely on a client-approved and fully compliant third-party service.
When references to credit cards must be stored (such as in an ecommerce scenario), it is permissible to store the cardholder's name, expiration date, card types and partial card numbers. For partial numbers, the first six and last four digits are the maximum number of digits that may be [stored or] displayed.
External input and XSS
The team must consider all user input and external data as potentially malicious. XSS exploits occur when such data is allowed to be directly stored in a database or when rich content (HTML, CSS, XML, etc) is directly rendered to an HTML document.
We recommend that:
- User-generated data sent to an API endpoint should be sanitized.
- Data received from an API endpoint that renders any type of rich content should be sanitized.
- Data fetched from an URL location, query string parameters, hash string or similar should be sanitized.
- In a Service Oriented Architecture, each service should not rely or make assumptions on the security steps (such as data sanitation) and compliance of the other services. This is also related to the “Isolation” and “Defense in Depth” secure architecture principles.
External References:
Cross-Site Request Forgery (CSRF)
Cross-Site Request Forgery is a type of attack where an attacker tries to trick or force the user into making a web request that the user didn’t intend to make. CSRF specifically targets state-changing requests, e.g. injecting unwanted data, deleting data, changing passwords, etc., since the attacker has no way to see the response of the forged request.
A trivial example would be a vulnerable web site, where the action of changing a password is executed via a GET request, like this: https://vulnerable.com/user/changepassword?pass=newpassword
All it takes for an attacker is to trick a user into visiting a web page where, say, an IMG tag executes the aforementioned web request, in the context of the currently authenticated user session.
Since the web request comes from the user's browser, this type of attack can effectively be used to target web applications deemed secure behind corporate firewalls, or otherwise not accessible to the outside world.
Defending against CSRF
- Make sure that all requests are coming from your application (verify the origin of the request).
- Use and check CSRF tokens where appropriate.
External resources:
Dependency Management
Dependencies (libraries, containers, etc) should first be retrieved from a public repository or source, security-scanned and then made available via a private repository. It is preferable that all dependencies are combined into one application package that is then promoted through environments, however, if this is not possible, explicitly pinning versions of dependencies is an acceptable alternative.
Beyond this article
Security is a vast topic and we can't even begin to cover everything in the scope of a single article. There should really be nothing new in this article for most Software Engineers, but it is always useful nonetheless to reinforce the fundamentals, and we heartily recommend that any Software Engineer reading this continues their journey from this point forward to dig deeper and stay up to date with modern security best practices.
If you would like to chat with us about security and other Software Development topics, whether you are a Lemma client, Software Engineer, Product Manager or any other role, we invite you to reach out to us directly with any questions or suggestions, as well as joining us in one of our monthly Virtual Meetups or similar events.
