Inai: REST in the Small

Nov 23, 2020   #REST  #Javascript 

(originally posted on the Imaginea Labs blog - https://labs.imaginea.com/inai-rest-in-the-small/)

The Inai project (forked here) began as a light hearted attempt to explore the consequences of bringing REST principles to organizing components within a single server node. It started off with the following question -

Given we know that REST principles are useful to organize the development, deployment and evolution of large collections of services, would those principles be useful in the small - i.e. within a single process?

While I started working on it out of hacker-like curiosity, it has now reached a point where I’ve been enjoying a continuous deployment process with Inai and I think this appoach may have value in production systems - at least in early stage product development. If this post reads like an unorganized collection of thoughts and ideas, that’s because it is that.

While the history of https://github.com/srikumarks/inai is relatively short, the concepts and the goals behind it are about 7 years in the making with several incarnations, before settling on the current one. I will document the evolution of ideas in another post perhaps. This one is focused on the current incarnation.

Why would I bother with Inai?

While everyone has their favourite tools, I’m quite enjoying the following attributes of Inai and will likely use it in several projects going forward for the following reasons -

  1. The ability to do continuous development and deployment from day one. The process is practically “deploy first, develop on the go”.

  2. The underlying organizing principle is simple and broadly applicable to both server side code and (this came as a surprise to me) browser side code.

  3. You get the same “save file and refresh browser” development experience that front-end developers enjoy, with server side code too … live … at scale even. I’ve debugged even the core auth service in such a live mode.

  4. The programming model doesn’t differentiate between calling services within a single process and calling services across processes and networks. This is one of the stellar qualities of Erlang that I’ve wanted in other ecosystems.

  5. Automatic logging of application component activity for monitoring.

  6. Out-of-the-box fine grained RBAC - managed by the “policy” service.

  7. You can reuse other components and frameworks within Inai too since each “service” is self contained.

  8. I can easily get into “flow” developing with Inai since only a few concepts need to be kept in my small head.

  9. I can now exert fine grained control over server side code as well any corresponding browser-side visual components - i.e. with Inai, a team can own (i.e. independently develop and deploy) a piece of the front end and the backing server-side code without any build dependencies with other running modules.

  10. I can talk to and test individual parts within a server using Postman.

When would I not bother with Inai?

I really don’t know. Perhaps when some other specific tech stack or doctrine is mandated?

I’ve been thinking about introducing Capability Based Security (CBS) principles into Inai and I think it is possible with some modifications, but for now I’m sticking with a more open system. So if CBS is required as an application security principle, Inai currently won’t be a fit .. but drop me a note.

The ideas in Inai are still evolving even if a set of core ideas and interfaces are unlikely to change.

What does “Inai” mean?

The name is a Tamil word that means “connect”, “link”, “integrate”, “cooperate”.

Show me

I’ve been developing a prototype of a “return to work” service as a guiding prototype for Imaginea’s COVID19 employee protection framework . The prototype is available at https://labs.imaginea.com/novid .. though it is intended only for employees of Pramati Technologies and its subsidiaries. As a teaser, here is me posting a fake notification on “my status” page (has test data) by posting to the client side notifier service.

Posting to notifier

You can spot the markup on the page quite easily -

<div id="notifier" inai="notifications" data-context="personal" inai-id="e7"></div>

The id of the div tag is the name of the service. The Inai code base from which the service is booted is identified in the inai attribute. The data-context gives the context id for this instance of notifications. The inai-id is an automatically assigned unique identifier (within the page). The presence of the inai attribute is what identifies a service that’s attached to an element. Once you see that and you know the service protocol formats, you can directly interact with the service from the console like I did.

I.network(<servicename>, 'get|post|put|..', <resid>, <queryobj>, <headersobj>, <bodyobj>)

The I.network call is asynchronous and therefore directly calling it produces a Promise.

That’s silly.

Agreed. But here’s what’s going on with even this silly bit.

  1. The page loads a lightweight runtime (< 10k) named “inai_web.js” that manages all the goings on regarding services.

  2. When the page loads and it sees such a tag with an inai attribute, the attribute value is used to fetch the code for the notifications service. For known cases, this can be server pushed eventually.

  3. It compiles the service code and “boots” an instance of it, naming it notifier.

  4. At boot time, the service picks up the data-context and sets itself up to manage all the notifications for that context. It also notes down the DOM element under which it must insert the notifications.

  5. The above steps mean that during the dev cycle, you have a page that only downloads the code for the components that it needs and nothing more. Most such client side “services” require some configuration information. That is sent by the server via HTTP headers in response to the codebase .. or picked up from the document if specified.

  6. In production, the components required for a page to show up quickly can be marked for preload (which the Novid application does), leaving the others to be loaded dynamically. A component/service developer doesn’t bother whether it is preloaded or dynamically loaded.

  7. If a dev wants to fix a bug in just the notification service, it can be deployed without a “build everything into a single js file” step. The dev doesn’t need access to the entire application’s source code or repository - she only needs permission to deploy the particular service’s code. This is independent of whether we’re talking about server-side or client-side stuff.

  8. A simple page reload will get the new code. Server side components auto-update when they detect new code in the database (yes deployed code is stored in a database).

  9. The process of updating this client side component is the same as the process of updating a server side component .. they’re both database updates. The only difference is that the server will pick up the updated (or new) service automatically.

Novid: A test bed

I’d like to describe the service architecture of the Novid prototype application as an illustration. Some of the app-specific services are listed below to give you an idea.

Service Server/Browser Description
novid server Top level service which exposes other services to clients.
nvcert server Validates a given covid certificate and displays a page to that effect. A standalone service independent of other functionality and auth so it can be used by third parties.
nvcheckin server For employees to check into various marked locations.
nvcheckinui browser The front end component that presents check-in options to signed-in employees.
nvcounterstats browser A list component to render statistics for various “counters”, which serve as fact collection end points.
nvdashboard server Page component that collects together stats into a dashboard.
nvemergency server End point for collecting emergency counters like “covid positive” and “covid negative”.
nvepass server A page component that displays an cryptographically validatable “epass” based on status derived for signed in employee. The nvcert service can be used by a third party to validate the e-pass.
nvfacts server Provides data about various facts that can be “counted” by the application for inclusion in the dashboard.
nvlogin server For logging into application.
nvmystatus server A page component for presenting the logged-in user’s status.
nvmystatuslist browser A list component for status entries used within the nvmystatus page.
nvnav browser A common navigation component used across all pages.
nvplaces server Provides data about places into which employees can “check-in”.
nvquestions browser Browser-side component for taking a questionnaire structure and running through them according to a script which is capable of handling dependencies between questions.
nvrecommendations server Computes and fetches recommendations by the system based on questionnaires answered by an employee.
nvstore server Data store component that fronts the Dgraph based graph data store for the novid application.
nvsurvey server A page component for presenting a selected questionnaire a.k.a. survey.
nvsurveycatalog browser A component that lists available questionnaires for an employee to take.
nvuserdb server A front for user-related information stored in the graph DB.
notifications browser A component using which context-sensitive notifications can be presented to the user.
loc browser For getting location information. Handles permission getting as well.
email server For sending email notifications to management staff and employees.
vega browser Front for the vega visualization toolkit, used in the dashboard.

As mentioned earlier, each service can be developed, debugged and deployed independently without knowledge of other services and their status. Of course, if your service depends on another service existing, then you better deploy it in an environment that provides the dependencies. There is some basic support for dependency management at this point, but automatic runtime dependency management would be useful to add.

All the above services can be accessed (in their respective environments) via the same interface - I.network(name, verb, resource, query, headers, body). The server also provides gateway services using which clients can make calls to specific explicitly exposed server-side services. A service that needs to be exposed for such a purpose just needs to declare itself as “public”.

Much of the “content” in the system – including locations for “checking in”, descriptions of facts collected by questionnaires, etc. – is pulled from a headless Ghost instance and cached within the server. The services nvfacts and nvplaces front the CMS.

Why are you telling me all that?

That is just to convince you that a real world application of this nature can be built within this framework pretty naturally .. with the frontend and backend teams handling their own pieces of code without having to headbutt with the build system. This is made possible because all code is deployed to a redis instance that serves it up to the server instances.

Feverishly Anticipated Questions

Why do you say this is “REST”?

REST expands to “REpresentational State Transfer”. Here, we place a constraint on the services that they cannot pass a direct reference to an internal entity across the service boundary. They can only pass representations - which you can think of as JSON.stringify-able objects. Note that it is ok for a representation to present a serializable identifier that refers to an internal object … indeed we’ll always need this to refer to a “resource” … but a direct handle cannot be passed across the boundary.

A second principle in action here is that services cannot and do not close on each other because they do not have access to each other’s object references. They can only interact with each other through their identifiers/names. This is what makes it possible to update a service

Why does I.network(...) look like a HTTP request?

One of the inspirations for this work and all of its precedents has been Erlang, where a guiding principle has been that when a module accesses a process, it shouldn’t need to worry about whether it is a local process or a remote process. To make that possible, and since many remote processes expose their services today via “REST APIs” implemented on HTTP, I chose to stick to the same essential structure.

This gets us many benefits -

  1. A “proxy” service can mediate transactions with a remote service and internal services wouldn’t be able to tell the difference, thereby meeting the Erlang guiding principle.

  2. It is easy to pass through HTTP request details down to any internal service. This is maximally useful for special credentials checking deep down .. where the headers become important.

  3. A resource access can be serialized in a way very similar to a URL.

  4. The HTTP status codes collection is pretty comprehensive and we don’t need to reinvent error codes for services. Inai doesn’t use exceptions for this.

  5. Deviating from HTTP structure would mean new things to teach to people and special code to write to convert from HTTP requests to the new thing and back. This is pointless … provided it is efficient enough. For internal calls, Inai requires no serialization of the arguments (it only requires that they be serializable).

  6. Methods for doing RBAC on HTTP service end points can be applied to such internal services too without much ado.

  7. We can use the same principles of interface versioning for these services too. I prefer version through headers but you may want to use version numbers in resource ids. Whatever works, we can keep doing that.

  8. To handle sequencing of requests to a particular service, Inai provides an I.atomic function that can execute an async function without new requests interfering with it.

Why do you use a database to deploy code?

This has been one constant design factor in all the incarnations of Inai. Seriously, source code in files is .. quaint. For the purpose of deploying code, a file system is essentially a poor database. So I just use a database and benefit from additional features databases can provide. I particularly like Redis for this due to the rapid change notifications and master-slave syncing. Redis suffices for this primarily because code deployment, from a server’s perspective, requires only read-only access.

Despite the quaint Smalltalk link, Inai takes a medium stance on this. The code does reside in files at dev time, but the server cares less about files and picks up entire structured bundles (code + resources) from Redis directly.

How is this different from normal OOP?

This is kind of closer to OOP than most of what we’d call OOP. The original idea behind objects was to have them interact with the world via a rigid walls without letting any of their insides show through. Inai’s “services” do that rather well … much better than classes and objects in most OOP programming languages. This is in the same sense as that Erlang is, ironically, the best OOP language around.

A REST API for the DOM? Seriously?

Early on, I tried to challenge myself by turning most things I’d encounter into Inai services. The DOM was a challenge. While I don’t stick to it strictly as a matter of principle (not really needed), it turned out to be quite elegant to encapsulate some DOM functionality via REST. A dom client-side service now mediates access to the DOM (example). It provides a JSON representation for a subset of HTML that can be passed around, parsed, checked easily, with assurance of having no security risks because it doesn’t support script. The representation is a simple “defunctionalized” version of a composeable function form of manipulating DOM entities defined in dom.js.

The ‘spec’ used with the dom service is a JSON-serializable object from which a DOM representation can be constructed. Here is a sample -

{div: [{classes: 'container'}, {attrs: ['id', 'd123']},
       {ul: [{styles: ['font-family', 'sans-serif']}, 
             {li: "Item one"},
             {li: "Item two"}
             ]}]}

That maps to -

<div class="container" id="d123">
    <ul style="font-family: sans-serif;">
       <li>Item one</li>
       <li>Item two</li>
    </ul>
</div>

The bootstrap code inai_web.js can auto-wire buttons and some controls to send their events to components named via attributes. For example (from [here][here]) -

[here]: https://github.com/Imaginea/inai/blob/dc0e3221b9f9e6d4cb58d9087cf9ab4e8fa8255f/services/app/template.html#L23)

<button id="showdoc" class="level-left button is-info" inai-target="/doc/g1">Show doc</button>

The inai-target attribute indicates the service and resource to which a post of the click event must be sent.