Introducing Yoctoservices

Published Aug 1, 2017. 11 minutes to read.
Tagged with Meh.

Let me tell you a story.

Everybody loves microservices oriented domain driven distributed web 2.0 scalable cloud functional services…? Right?

I know a guy, let’s call him Peter, who knows a guy. Let’s call this last guy Jim. Their connection was mostly professional, as Peter was an engineer working for the same company as Jim.

Jim, unlike yours truly, is not a software practitioner. Jim, in fact, is “Principal software architect” and Jim’s first and foremost responsibility is, as you might have already guessed, software architectural design and development. Which involves almost incomprehensible amount of theoretical knowledge, which Jim amassed in full by having filled his Drive almost exclusively with endless PDF’s and E-Books on the topic. Jim’s collection might or might not make some librarians jealous.

No, no, no, don’t you make that face just yet, software architecture is a good thing, we are getting there. I thought Jenkins pipelines have already taught you some patience, no?

Apart from tasks and duties of your usual God emperor of the known uni… I mean, Principal software architect, after an impressive list of other responsibilities, duties and literally everything came Jim’s duties as the de-facto head of The Team, which compromised of Jim himself, project manager Billy, and two developers, Sam and Tina.

Jim has spent couple of years working in American Midwest for “Your Usual Consulting and Kickups Inc”, commonly abbreviated as YUCK. YUCK was by no means a small company, having engaged in pretty much every industry it could reach with it’s apocalyptic tentacles, ekhm, also know as sales and marketing department.

Came the fateful Tuesday morning and deliverance by Billy, the PM.

— “Hey Jim, listen, you remember that lease tracking software we did for our generic car rental customer last year?” Billy said prompting Jim to, as usual, not pay any attention whatsoever. “They have expanded their business since and they now have more shops, so they would like to have the particular piece of software available online in all of their locations, interconnected as to see availability of stock and such” Billy continued “so, what do you think? Not a pressing issue but… “

— “Microservices”.

— “What?” Billy uttered, mostly in disbelief for actually getting some life signs out of usually dormant Jim.

— “I said micro… services. We will build a microservices based web platform for them. I remember the requirements, should take no time at all!” Jim explained, and to Billy, Jim sounded almost happy. Excited even. Billy stared in disbelief, as if stricken by most electric kind of lightning right then and there, and started to contemplate about what has just happened. It usually took about infinite amount of weeks to get Jim past planning and design stages, but this? No design diagrams? No meta programming? Just “should take no time”?

— “Are you sure, Jim? This is not something we can build from scratch, existing data needs to be preserved, plus we need to merge it from all of their local installations” Billy inquired, unsure of what to expect next. Ragnarok?…

— “Yeah, I know, I know what they need, don’t worry. This is an entirely new concept I have read about and I really want to try it out on a production product. I already have the solution and they are going to absolutely love it. I will have it by the end of the month” Jim waved “I really have to finish this, do you mind…?”

— “No, of course, carry on then, I will email you the details” Billy shrugged and started her way back to The Cubicle.

Days passed, weeks went on, Jim designed and worked on The Architecture, while Sam and Tina all in all where mostly left wandering around countless source code repositories, “lambda” SDK’s and other interesting concepts that seemed to flow like never ending river, backed by almost exclusive and religious quarter-hourly SCM commits by Jim.

— “So, what’s the latest?” Billy inquired to Jim.

— “We are almost there, just a couple more services. We had a couple of bumps along the way, but we are about to be there” Jim retorted, business as usual.

— “How are we on storing their current data in this new platform?” Billy inquired again, hoping for the same answer.

— “What do you mean?” Jim asked, face on as if Billy had just spit in his morning coffee.

— “Well, their production data, remember? We have to load all the customer info, all of it, I sent you the samples weeks ago”

— “This is not possible. It’s microservices, we can not. They will have to type it by hand”

— “Can we just write some script to transform the structure and import it in the new database?” Billy raised an eyebrow

— “No”

— “What do you mean no, we need to, I mean, customer needs it and…”

— “Billy, we can explain to the customer this is a new kind of technology, we can not, it just does not work that way. They will need to type it in by hand”

— “This will take months, this is not what we agreed upon, it needs to happen, and that’s that”, Billy’s frustration grew and patience decreased with each sentence Jim produced.

— “No”

— “Tina, Sam?” Billy addressed the rest of the team in hope of salvage.

— “It’s just technically… Not possible. Sorry Bill’. We…”

— “So what you are saying is, I need to call the customer, inform them about how we spent all this time developing this… thing… and now they need to spend as much on data entry?”

— “Yes”

— “No”. Exit stage Billy.

Days went by, and as you, my dear reader, might imagine neither Billy, nor the customer representative was very happy about the result. Names where called, blame assigned, directors of both sides involved, monetary losses calculated. This is how it would end, unless the mastermind of Peter’s manager to allow their team to try and unravel the mysterious microservices platform Jim had spent all this time architecture-ing away. This task fell on Peter’s shoulders as the most senior developer of the team.

Disassembling the future.

— “What the f**k?”

— “Why would you?”

— “What?!”

This is Peter. Peter is a software developer. Peter, as you might imagine, is currently a little bit frustrated. Just a tiny bit.

— “Hey Tom, come here. Look at this”

— “What’s up?”

— “I can not believe this piece of s**t, I mean, I have seen things, but this. Just wait, look, tell me Tom, what do you see here?”

— “A repository listing, why?” Tom shrugged.

— “How many pages of repository listings do you see, Tom?” Peter inquired half enraged, and half laughing about the absurdity of the situation. But mostly enraged.

— “17. Why? Wait, what project is this?”

— “10 repos per page, 17 pages, all in all, about 170 repositories.” Peter replied, as if struck a gold mine. “That’s the car rental project. One of Jim’s”.

— “And what’s in there, I mean what could possibly prompt him to…”

— “Check the language tags, look…” Peter interrupted, continuing the kill-streak.

— “Ruby, Python, Java, C#, PHP, some JavaScript, wait, do you mean to say…”

— “Yes” Peter exclaimed excited like a small child, opening his hands wide open.

— “Nooooo…”

— “Yes, look at this s**t” Peter expanded couple of the repositories “Each one of these contains some kind of service if you dare to call it that way, each repo contains an app that basically runs an embedded version of web server or whatever that platform provides. “ Peter continued his rampage hitchhikers guide to what could only described as mayhem “These apps seems to talk one with another in some kind of obscure HTTP based RPC protocol, but not always, there seems to be a couple of TCP servers running, for a purpose I can only guess.

— “Then there is the databases. Each of these apps has their own database, and I don’t mean different logical databases, it’s different instances AND different database engines altogether. Like this Express app seems to interact mostly with Mongo, while this one,“ Peter pointed his finger at the source repo like a naughty child “this one here uses Postgres. And this one uses Elastic for some reason.”

— “Now look, I had to local install this to actually see IF and how it even works together, and of course, I had to install all of the dependencies, most of which where undocumented by hand, because why the f**k shouldn’t I”

— “Now this is a great example on how this works, the customer rep system. It consists of some 30 of these apps bound together, and the actual web is built by some 5 of them as far as I can tell. The login system is especially interesting design. You enter your credentials, those credentials get sent to another one of these, the *login_auth *app, here, and that app creates something called “AuthenticationTicketRequest”.”

— “That get’s you an auto generated ID in response, which then get’s sent back to the login app, where it is passed to UI and then the user get’s redirected to another app, called login_verify, this ticket request thing ID in the URL. That service then compares your credentials with another service, called user_store, this is in back end, via RPC request. “

— ”If your credentials are confirmed, you get a session ID which then get’s attached to the URL as you get redirected back to the app you were initially going to. Then, whenever you ask for any resource on the server, and I mean any, including what appears to be most static assets, which, by the way, have services for retrieval and compilation, and what not, of their own, you must send this token along with the request via GET or one of many headers defined for this purpose.”

— “Then, for each request, whichever app you access to, it invokes RPC function in the background to the user_store to validate the token. If the token is identified as invalid, you get kicked back to the login page.”

— “On top of that, there is a roles management system as well. All would be nice and well, but there is a separate service for that. Whenever you need to access some section or whatever kind of isolated feature, you, well you guessed, you need to make a RPC call to the remote service to validate access. Each section makes tens if not hundreds of these, every single time you access it. No caches, in fact, it was explicitly documented to not cache anything because… someone could change the access rules”.

Peter took a pause to evaluate expression on Tom’s face after this epic monologue. As you expect, Tom’s facial expression was an equal mix of “Wut”, disbelief, disgust and serious reconsideration about his career choices, all in one.

— “He actually described and named this whole thing in the application docs somewhere. Yoctoservices. Apparently it is advanced kind of microservices.”

Bottom line

On a bit more serious note, stop. Stop *microservices *literally everything. Microservice architecture is good for some very specific applications, as is pretty much everything SOA. No, your corporate with average reader base of 5 per eon does not necessarily need microservices to sustain it.

If you are not able to explain why you should use microservices as base for your arch without using “scalability”, “decomposition”, “isolation”, “performance” at all, you do not need microservices.

If you can not explain why you need microservices in less than one paragraph, you do not need microservices.

Chances are excelled your microservices will be even bigger pile of crap than your aging monolith, mostly because you underestimated problems a networked infrastructure might and will have, and you greatly overestimated your own ability to design, develop and maintain large scale distributed systems. It’s like drunk driving. Everything is unicorns till you hit a tree doing 120mph.

Disclaimer

Let me just end this rather nicely assembled article by saying, I am a software practitioner. I always have been, and always will be.

I have, in my entire career, read exactly zero books about software design. I have, in my entire career, written exactly zero books about software design. I have no intention to do so either.

In fact, my personal library consists almost exclusively of project management, operations and strategic management textbooks, and related literature, most of which is sourced from NYU/Stern, MIT and anywhere in between.

When I say I am a software practitioner, I want you to comprehend fully what exactly do I mean with that, and that is, I give less than negative infinity of, you know the f-word, about what your typical highly paid consultant slash speaker slash author thinks about how I, or anyone else for that matter, should write software. There are very limited number of exceptions to this rule, but, as usual, you should assume the worst. And no Martin, you are not one of them ‘xceptions.

On the other hand, I have met, and even better, had the opportunity to work alongside with some incredible people, who like me, are practical people and does not seem to care much about theoretical issues, people who know what they are doing, how, why, when and when not, and why when not. If you ever read this, you know who you are. I thank you. I have learned from you so, so much more than I could have ever imagined.

Also, as you might have noticed (at least I hope so), I care rather very little about political correctness. If any of the contents offended you in any way, deal with it.

© Matiss Treinis 2019, all rights, some wrongs and most of the lefts reserved.
Unless explicitly stated otherwise, this article is licensed under a Creative Commons Attribution 4.0 International License.
All software code samples available in this page as part of the article content (code snippets and similar) are licensed under the terms and conditions of Apache License, version 2.0.