An expert's guide to CRUD APIs: designing a robust one

APIs have become so prevalent, you might think that developing a robust one is quite straightforward. But after a few years in the thick of it, we do think keeping in mind a few best practices to designing a good one is worth it. CRUD APIs being one of our core know-how, we’ll be focusing on them for this article.

A CRUD API (Application Programming Interface) is an interface that allows clients to perform CRUD operations (Create Read Update Delete)  on resources managed by a server. It is extremely useful, allowing both external but also internal clients - that can be very demanding -  to access and edit information in your IT system.

When we provide clients with an efficient CRUD API, we allow their applications to interact with our server to perform basic data management operations. Having a thoughtful CRUD API approach when developing one will allow :

• Your clients to more easily access your data through a consistent and standardized way of handling it across different applications and systems.
• Your team to better withstand scaling and to avoid much troubleshooting

In short, happy clients, and less hassle for you.

So what exactly needs to be done to design a robust CRUD API? Hint: it involves planning ahead and some standardization. In this article we will cover:

I) What is a CRUD API: what elements and definitions to be aware of if you aren’t already

II) How to design one and implement it: what steps to cover to build your API

III) Best practices: what to keep in mind while building your API

I. Understanding what is a CRUD API

1) What CRUD operations are

Create, Read, Update and Delete are the four operations central to data-centric IT system. These terms were first used, beginning of the 80’s in relation to databases before expanding to Application Programming Interfaces.

CRUD operations relate to the four main needs one can have while managing a database: place a data point in the storage, be able to find it, and edit it or remove it : CREATE, READ, UPDATE, DELETE. To each CRUD operation corresponds a specific statement for each database. Below are the ones for SQL:

Please refer to our article All the basics (& more) you should know about CRUD operations if you want to specifically learn more more on CRUD operations. We will now look at how CRUD operations apply to APIs.

2) How CRUD APIs work

So what is a CRUD API ? It is an interface that allows client users to do create/update/delete/read operations on your own database resources. Your clients need to create a data point in your storage, be able to access it and read it, update it and delete it, very similarly to what you need to do on your own data storage. They just do so through the interface you provide, and thus CRUD operations in API linguo become requests and calls from your client server to your own, rather than statements. It is a translation of the CRUD operations to HTTP methods and status codes:

In this article we will focus on the RESTful APIs. As a reminder, REST (REpresentational State Transfer) is an architectural style for designing networked applications, which was introduced by Roy Fielding in 2000. RESTful APIs are thus a type of web API that adhere to the principles of REST architecture.

Nowadays RESTful APIs represent the majority of APIs worldwide due to their simplicity, scalability, and compatibility with the HTTP protocol. Thus RESTful APIs use standard HTTP methods (e.g., GET, POST, PUT, DELETE) to perform CRUD operations on resources. As RESTful APIs use the HTTP protocol, it is the widely known  HTTP status code (the three-digit numbers returned by the server to indicate the outcome of the request) we will encounter in case of a CRUD operations gone wrong.

3) Requests and Responses formatting

RESTful APIs focus on architectural principles such as statelessness, resource-based interactions, and a uniform interface, but do not dictate a specific data format. But that does not mean you should not pay extra attention to selecting a data format for representing data in your CRUD API for requests, responses and serialization and deserialization.

You should actually pay extra attention to the data format you select  as yourself  - and your clients -  will probably need to deal with this data format for a long time to come. Here are three points to consider when choosing a data format, whether it be for API requests API responses, data serialization and deserialization:

• Criteria specific to your business nature
• Developers experience
• Performance

a. API requests

In API requests, information can be sent or requested using various methods, including via URL parameters, HTTP status codes or methods, headers, and request bodies.

• URL Parameters: URL parameters are key-value pairs appended to the end of a URL, typically separated by "&" symbols. They are commonly used for passing simple data values in GET requests.

https://api.example.com/resource?key1=jane&key2=doee

• HTTP Methods: HTTP methods, also known as HTTP verbs, indicate the action to be performed on the resource identified by the URL. The most common HTTP methods used in API requests are the one cited in the section above GET POST PUT PATCH DELETE.
• Headers: HTTP headers are key-value pairs that provide additional information about the request or the client making the request. They are included in the HTTP request headers section and can convey various types of metadata, such as authentication credentials, content type, language preference, and caching directives. Headers are useful for passing information that does not belong in the URL or request body.
• Request Body: The request body contains the data being sent to the server in the request. It is commonly used for submitting complex data structures, such as JSON or XML payloads, in POST, PUT, and PATCH requests. The request body is included after the headers in the HTTP request message, separated by a blank line. A request body for a Create User action POST/user could be:

{
    "firstname": "John",
    "lastname": "Doe"
}

In addition to these methods, there are other ways to request information in API requests, such as using cookies, query strings, or path parameters. Cookies are small pieces of data stored on the client-side that are sent with each request to the server. Query strings and URL parameters but are typically used for filtering, sorting, or paging data in GET requests. Path parameters are placeholders in the URL path that are replaced with actual values when making requests.

b. API responses

In API responses, information can be sent using various methods, including via HTTP status codes, headers, and response bodies:

• HTTP  Status Codes: we covered them in the How CRUD APIs works section
• Headers: HTTP headers in API responses provide additional information about the response or the server sending the response. They can convey metadata such as content type, content length, caching directives, and authentication information. Headers are included in the HTTP response headers section and are useful for providing context or controlling the behavior of the client.
• Response Body: The response body contains the data sent by the server in response to the client's request. It is commonly used for returning structured data, such as JSON or XML payloads, to the client. The response body follows the headers in the HTTP response message and is separated by a blank line.

{
    "key1": "Jane",
    "key2": "Doe"
}

In addition to these methods, there are other ways to send information in API responses, such as using cookies or content negotiation. Cookies are small pieces of data stored on the client-side that are sent with each request to the server. Content negotiation allows the client and server to agree on the content type and format of the response data based on their capabilities and preferences.

c. Data serialization and deserialization

Data serialization and deserialization are processes used in APIs to convert complex data structures such as an object or a collection of objects into a linear format that can be easily transmitted over a network and then reconstructed on the receiving end. So serialization converts data into a format suitable for transmission, while deserialization reconstructs the data into its original form for processing and manipulation.

Once again, be considerate in choosing your serialization and deserialization format with your dev team preferences, while also taking into account your clients or potential clients needs and existing systems.

The most common formats used for data serialization and deserialization are the following:

• JSON: lightweight and human-readable, it is also compatible with JavaScript, which is widely used in web dev. JSON is also easy to parse and manipulate in various programming languages
• XML: may be preferred in certain scenarios, such as when interoperability with legacy systems or compliance with industry standards is required. XML provides a hierarchical structure and supports more complex data types and schemas, making it suitable for representing structured data in a standardized format.

And there are a couple of more advanced formats that can be used:

• Protocol Buffers: with efficient serialization while being language and platform agnostic and allowing for versioning, Protocol Buffers commonly used in large-scale distributed systems, microservices architectures, and communication between components written in different programming languages. They are particularly popular in scenarios where performance, interoperability, and backward compatibility are important considerations.
• JSONAPI (the one we use): providing a standardized format, including support for rich meta data and HATEOAS, JSONAPI is commonly used in web-based APIs, particularly those following RESTful architectural principles. It is well-suited for scenarios where standardization, interoperability, and self-descriptive APIs are important considerations, such as in web and mobile applications, content management systems, and e-commerce platforms.
  

II. Building a CRUD API

1) Crafting a CRUD API

a. Naming conventions

As a CRUD API is all about doing CRUD operations on data storage via an interface, building an efficient one first requires having clean and well organized data storage. We cannot emphasize this point enough. But right after that, a clean, organized naming convention is a must to properly use this data storage.

As you craft your CRUD API it would thus be auspicious to think of what would be a good naming convention for your resources, to document it and share the information for proper implementation. Consistent naming of your data storage resources is the foundation to ease CRUD operations on both sides of the API.  Each naming discrepancy is what will make usability for you and your clients cumbersome.

Here are a couple of the naming conventions to decide on that often create unnecessary hassle in our experience:

• Singular or plural? (e.g., /user or /users). Really much more of a personal preference.
• Versioned resources (e.g., /v1/users or /v2/users). Useful to manage backward compatibility when introducing breaking changes in newer versions.
• Introduction to nested resources (e.g., /users/123/posts)? Useful to organize API endpoints, to represent relationships between entities more intuitively.
  

b. Versioning strategies

Generally, you version APIs to handle breaking changes. A basic example would be if you have an API endpoint on POST /v1/user that takes firstname and lastname as a body. Calling this endpoint correctly creates a new user in the database. At some point in the development of the API, you realize that you need to also save in the database a password regarding those users. So you code something that allows you to also retrieve the password in the body of the request. And you could create a new url POST /v2/user, that takes firstname, lastname & password in the body.

In this scenario, a customer can now call  POST /v1/user &  POST /v2/user, so he can create users with or without password. In this scenario, you can easily locate which customers still call POST /v1/user.

The choice of versioning approach depends on factors such as API complexity, client requirements, and development team preferences. Of course each approach has its advantages and considerations, and it's essential to carefully evaluate trade-offs and choose the most suitable approach for your specific use case. Here's a further explanation of each approach:

• URL-based versioning e.g. :

/v1/users

Pros: easily visible and accessible, allows for efficient caching

Cons: can lead to cluttered and potentially lengthy URLs. plus is immutable

Example:  A payment gateway provider enhances security measures and adds support for new payment methods. By maintaining API versioning in URLs, the provider can introduce changes to payment processing flows while ensuring seamless integration for existing merchants. Example Endpoint: /v1/payments for processing payments and /v2/payments for processing payments with additional security checks.

• Header-based versioning e.g. :

Accept: application/vnd.myapp.v1+json

Pros: clean urls and flexibility

Cons: increased complexity, and clients must remember to include the versioning header in every request.

Example: A financial services company updates its API to introduce new account management features and improve security protocols. They implement header-based versioning to maintain backward compatibility while rolling out changes. Clients can thus continue accessing account information and making transactions using the existing API integration while gradually transitioning to the latest version to leverage new features and security enhancements. Example Header: Accept: application/vnd.financial.v1+json for clients requesting version 1 of the API and Accept: application/vnd.financial.v2+json for clients requesting version 2.

• Query parameter-based versioning e.g. :

/users?version=v1

Pros: easy to test different APIs versions

Cons: cache management to be checked

Example: A location-based services provider enhances its API to deliver more accurate geolocation data, routing algorithms, and location-based recommendations. In this case query parameter-based versioning allows clients to specify the desired version of the API. Navigation software vendors and location-based marketing platforms can thus continue leveraging the location-based services provider's API to deliver location-aware experiences while gradually adopting newer versions to access advanced geospatial data processing capabilities. Example Endpoint: /locations?version=v1 for clients utilizing version 1 of the API and /locations?version=v2 for clients utilizing version 2.

As for naming convention, developing a consistent documentation and communication with API consumers is crucial to ensure smooth adoption and transition between API versions.

c. Error handling and exceptions

One key point to keep in mind for better efficiency (for yourself later on while troubleshooting and trying to understand what is the source of an issue, and for your clients for the same reason, plus for good user experience) is error handling. Handling errors properly ie with precision, and handling errors gracefully needs to be set up.

Make sure to set up specific error handling rather than very generic ones to allow for faster troubleshooting:

• 4xx: Client errors with all their specific numbered versions (e.g., 400 Bad Request, 404 Not Found…)
• 5xx: Server errors with all their specific numbered versions (e.g., 500 Internal Server Error…)
  

d. Authentication and authorization mechanisms

Authentication and authorization are crucial aspects of API security, ensuring that only authorized users or systems can access protected resources. Requirements can even be legal and a prerequisite to your business or the one of your clients. Be careful to implement it  from the beginning to not find yourself in a quagmire later on. Implementing Authentication & Authorization usually have a big impact on a code base, especially when not started at the beginning.

Authentication & Authorization also play a critical role in securing APIs and protecting sensitive data. So it's essential to implement adequately robust authentication and authorization measures to safeguard API endpoints and prevent unauthorized access to resources.

• Basic Authentication: involves sending a username and password encoded in the HTTP request header. As credentials are sent in plaintext with each request, it makes them vulnerable to interception. The server verifies the credentials against its database and grants access if they are valid. It does not provide fine-grained access control. Once authenticated, the user typically has access to all resources for which they have been granted permissions. Pros: easy to implement. Cons: not very secure. No fine-grained access control. Use case: no real life use case. Too easy to bypass.
• Token-based Authentication: involves issuing a unique token to users upon successful authentication. These tokens can also contain information about the user's permissions or roles. This token is then included in subsequent requests as an authorization header or query parameter. The server validates the token and grants access if it is valid and has not expired. Pros: stateless and scalable, making it suitable for distributed systems and APIs accessed by multiple clients + fine-grained access control. Cons: tokens must be securely generated, stored, and transmitted to prevent unauthorized access. Use case: An e-commerce marketplace employs token-based authentication to secure its API endpoints, enabling merchants to integrate their inventory management systems or order processing software with the marketplace's platform. Thus a merchant's backend system obtains an access token by authenticating with the marketplace's OAuth 2.0 server. This token is then used to authorize API requests for uploading product listings, managing orders, or retrieving sales data from the marketplace's platform.
• OAuth: is an authorization framework that allows third-party applications to access a user's resources on a server without exposing the user's credentials. It involves a multi-step process where the client obtains an access token from the authorization server. OAuth provides a standardized mechanism for delegated authorization. The access token is then included in API requests to authenticate the client and authorize access to the user's resources. Pros: enables secure access to resources across different services and platforms. Cons: can be complex to implement and understand + secure token management. Use case: a healthcare application adopts OAuth to enable patients to grant consent to healthcare providers and applications for accessing their medical records and health data securely. Thus, if a patient uses a telemedicine app to consult with healthcare providers and share medical records, the app authenticates with the healthcare provider's OAuth server on behalf of the patient, obtaining access tokens to access their electronic health records (EHR) and transmit data securely between systems while maintaining patient privacy and confidentiality.
• JSON Web Tokens (JWT): is a compact, self-contained token format that can represent claims such as user identity and permissions. It is digitally signed to ensure its integrity and authenticity. JWTs are typically issued by an authentication server upon successful authentication and are included in subsequent requests as a bearer token. The server validates the JWT's signature and extracts the claims to authorize access to protected resources. Pros: stateless and scalable, easily verified without the need for server-side storage or database lookups. Cons: debugging complexity. Vulnerable to attacks if not properly implemented and secured. Use case: They are widely used in modern web applications and microservices architectures for secure authentication and authorization. More specifically, a digital banking platform can utilize JSON Web Tokens to authenticate customers and third-party applications, enabling them to securely access banking services and perform transactions via the platform's APIs. Customers authenticate with the banking platform's authentication server to obtain a JWT, which grants them access to APIs for checking account balances, transferring funds, or paying bills. Additionally, third-party financial management apps or budgeting tools authenticate with the platform's APIs using JWT-based authentication to access transaction data and account information with user consent.

In addition to these somewhat standard authentication and authorization protocols, you could need to set up more advanced ones such as the following:

• OpenID Connect (OIDC): OpenID Connect is an identity layer built on top of OAuth 2.0, providing authentication and single sign-on (SSO) capabilities. It enables clients to verify the identity of users based on the authentication performed by an authorization server. OIDC introduces additional endpoints and flows for authentication, token issuance, and user information retrieval.
• OAuth 2.0 Extensions: OAuth 2.0 supports various extensions and profiles that enhance its capabilities for specific use cases. For example, the OAuth 2.0 Device Authorization Grant (Device Flow) allows devices with limited input capabilities, such as smart TVs or IoT devices, to obtain user consent for accessing protected resources. Other extensions include Token Exchange, Dynamic Client Registration, and Client Credentials Grant.

e. Rate limiting and throttling

And, as usual, do not forget to prevent abuse and DoS attacks. Depending on your business size, and its growth stage, it might come up at a later stage. It might stay on the back burner for a while, but do not forget it.

2) Implementing a CRUD API

a. Framework or library choice

Choosing a framework and library for building your CRUD API is your first step to get going and be able to handle the data and all http requests and status codes we’ve been discussing. Making such a choice is usually linked - as usual  - to your project business requirements and your team expertise.

So a first filter to selection is the language you wish to use depending on your team expertise. Here are some of the common frameworks used:

• Node.js: Express.js
• Python: Django REST Framework , Flask-RESTful
• Ruby: Ruby on Rails API
• PHP: Laravel Passport
• Java: Spring Data REST

Then you’ll have to look into more detailed attributes for each framework. As an example, the following Node.js frameworks are usually known for :

• Node.js: Express.js, Koa, Fastify ( for performance), Nest ( for developer experience)

At this point, you need to spend the time investigating each framework or library and its adequation to your needs. Other than that, selecting your framework should be  quite straightforward.

b. Setting up routes for CRUD operations

To set up routes for CRUD operations in your API, you typically use a routing system provided by the web framework you selected ( cf paragraph above). Your routes determine how incoming requests are matched to specific controller functions for creating, reading, updating, and deleting resources. For this you need to focus on the following two elements:

• Mapping HTTP methods to CRUD operations: how do I get information from http and convert them into useable code?

First, define the routes for each CRUD operation. These routes map specific URLs to controller methods or functions that handle the corresponding CRUD operation, then implement Controller Actions, map routes to controller actions and test.

• Handling URL parameters and query strings: using named parameters really improves readability and maintainability. Provide clear documentation for your API endpoints

c. Validating input and enforcing data constraints

Handling requests and processing data in CRUD APIs involves implementing logic to create, read, update, and delete resources based on incoming HTTP requests.  You need to parse request bodies and query strings: receive the data, extract the data, to take the correct action, return the success response (or error).

Validating request data to ensure it meets expected formats and constraints will save everyone’s time and resources. You can use validation libraries or custom validation logic to check data types, required fields, and other constraints. Reject requests with invalid data and do not forget to return appropriate error responses. Keep it simple, and document the expected format and structure of request bodies and query strings in your API documentation

This is usually at this point that you'll start thinking about business logic coherence. Getting back to that POST /user example, "Validating input & enforcing data constraints" will look like considering the following questions:

• Are you ok with firstname or lastname containing special characters, spaces?
• Is there a maximum length for those field
• Are you ok with one not being sent?

Also do not forget to pay attention to security limit payload size, secure against injection attacks.

d. Integrating with databases (e.g., SQL, NoSQL)

To integrate your CRUD API to your databases you can follow three strategies:

• Directly write SQL request in your code. Pros: fine-grained control  (inc. debugging) + flexibility+quick start (developers are used to writing SQL. Cons: boilerplate code + manually handle data mapping and conversion between database rows and application objects + security concerns.
• Use an ORM -  tool or library that facilitates the translation of data between the relational database model (tables, rows, columns) and the object-oriented programming model (objects, classes, methods). Some popular ones are Entity Framework (for .NET), Hibernate (for Java), SQLAlchemy (for Python), Django ORM (for Python Django framework) Active Record (for Ruby on Rails). Pros: working on different database systems with the same code + increased developer productivity. Cons: performance overhead + learning curve.
• Use a query planner - a component of a database management system (DBMS) responsible for generating an optimal execution plan for a given query. Pros: ensuring efficient query execution and optimal resource utilization. Cons: overhead + complexity and potential instability.

Executing SQL queries or using an ORM is a trendy discussion, which answer is in my opinion related to your business needs and team constraints plus personal preferences.

III. Designing an outstanding CRUD API

1) Best practices to Designing a RESTful API for scalability and maintainability

There are two use cases for building your CRUD API:

• To provide access to your data to external or internal clients to access your data and use it, in which case you need to expose your API
• To help code a CRUD app that will enable direct CRUD API interaction? This is the case for most businesses. Most web applications use CRUD APIs.

In both use cases, as users are expected to grow sometimes to dozens of thousands, addressing the challenges for scalability and maintainability should be paid attention to.

Below are what we’ve found over time to be the best practices in designing a RESTful API for scalability and maintainability.

a.  Using standard HTTP methods for each CRUD operation

Use standard http methods to enhance the clarity and predictability of your API, making it easier to understand and use.

You might be thinking that http is just a way to convey information and that you can write yout own code on top of it. But really, http has been around for decades and using its standard greatly improves the clarity and predictability of your code.

b. Use appropriate  HTTP status codes

It could be a subset of the paragraph above or part of the following one about consistency and predictability, but we have decided to dedicate a whole section to it in order to draw your attention to this specific use case. Please use appropriate and detailed http status codes to help troubleshooting down the lane. Nobody likes having to second guess what’s going on, or dedicating time to investigating it.

Just taking the example of error handling, if a developer received a 404 code, they know the resource does not exist. It’s a common language. If you were to send a 200 status code saying ‘resource does not exist’, then your response would not be standard, so how are they supposed to handle it? Your client developer cannot use your status codes to trigger their own code.

c. Keeping APIs consistent and predictable

Use clear and consistent naming conventions for endpoints, query parameters, request/response fields, and HTTP headers.

Really, nothing is worse when you’re starting using an API for the first time, getting to do complex operations, and realizing that sometimes parameters end with an s and sometimes they don’t. Sometimes the parameter you’re handling needs to be input in a data object, but sometimes it doesn't.

As much as possible, select and follow common standards: go for restful, go with https. Use clear and consistent naming. The more predictable and consistent your API is, the better. Documentation is then almost a bonus as your API functions become intuitive.

For ex. being used to typing /api/ user, one would automatically type /api/group to access groups without needing to use the documentation. Or if one already have an idea for a group, they might try get group/id123 to try and fetch group id123.

d. Documenting APIs for developers (e.g., OpenAPI, Swagger)

Provide comprehensive and up-to-date documentation for your API. Or more exactly, if you’re building a CRUD API to expose it, then provide proper documentation; if you’re building a CRUD app, then you’ll be documenting the app for the end user. And if part of your business is selling the CRUD API you’re building, then you’ll definitely need to provide proper documentation.

We know it’s not the most exciting part of the journey, but it really needs to be done properly and if you followed the above best practices, you’ll be able to use:

• OpenAPI: the specification for defining and documenting RESTful APIs
• Swagger: a set of tools and libraries for working with OpenAPI specifications

Both will make it easier for you to document your API, and easier for your clients to consume.

e. Design your API around resource

Design your API around resources, representing entities or objects in your system, rather than actions. Resources are usually database tables. By using nouns in the URL paths to represent resources (e.g., /users, /products), your API endpoints become self-descriptive and intuitive, making it easier for developers to understand and navigate the API.

Use nouns in the URL paths to represent resources (e.g., /users, /products). This helps maintain a consistent and intuitive API structure. Plus Resource Oriented Design aligns with the principles of Representational State Transfer (REST).

f. Version your API to ensure backward compatibility

Version your API from the onset to ensure backward compatibility and facilitate potential future changes. The more your business develops, the more your user base grows, the more use case changes might come to your API. Maybe user name was not mandatory in the first version of your API, but later becomes mandatory for some business reason. How do you manage client access ? What happens for your existing customers that are still calling this API without user name? Do you expect the API to throw an error? Or do you expect the API to send a Header stating that this API will soon be removed? Or maybe it is OK for your business to keep both versions for now? Versioning your API would allow former clients to keep using their former address while new ones would use a newer address version. So including versioning information in the URL (e.g., /v1/users) or as a request header allows clients to choose the API version they want to use and prevents breaking changes from affecting existing clients.

2) Advanced expertise

In addition to the best practices mentioned above, we do consider both security and performance key areas for developing a robust CRUD API, so much so that we created two dedicated blogposts on the topic:

• Addressing security risks in CRUD interfaces
• Performance considerations in building CRUD interfaces